In terms of our process, so we'll just walk you sort of through this at a high level in terms of what we do So we have a three-phase digital design strategy approach So the first phase of that approach is an identification phase So this is where we're often doing our research and a lot of our initial future-state visioning when we're looking at a product For us because we kind of have a technical background You know this will often move from user research into more robust business analysis when needed But we also are very motivated to look at the broader customer journey when we're working with clients So we do a lot of journey and experience mapping as part of this phase So from there we move on to a conceptualization phase so above and beyond just wire framing and mock-ups This is where we actually want to establish a blueprint for what we're building So this might mean technical documentation for what we're building information architecture user interface designs prototypes The works everything we need to make sure that we have a vision that's established for our product And from there Validation is really critical to everything we do. So we believe that when we're delivering a product We need to test it both with our users our end users Also with anyone involved in using it. So you're all familiar with working with Drupal. You know the content management side is often neglected user acceptance testing with our Product donors is really critical to our process and then in the long term we also think that qualitative research is a really critical component to Assessing what you've done when you build something So on the technology side, so we predominantly work with technologies of the web So our teams are agile We our teams are scalable. We have a combination of front-end back-end project management QA And we think this approach is really strong because we think iteration is really important to delivering really good customer experiences On the technology side, we're at a Drupal conference You know, we believe in open-source technology as much as possible But we also believe in integration and we think that the competitive benefit of open-source technology is that it can integrate with Other open-source technologies, but also with proprietary systems So integration is the core of what we do content management is also core to what we do And you know as we're looking, you know, if you kind of follow Drees You know this idea of build elsewhere. It's definitely mandate for us So we also, you know, wherever possible lean in technologies like JavaScript to build the right experiences And then just another part of our core offering. So we also have a long-term customer success program So we call this there for care So, you know to put the simple as opposed to long-support, but in the long term what we're moving towards more is UX research and strategy is a part of that. So this idea of continuous product improvement over time So just introduce us So our team so we have UX designers technical architects you can we have another A Talker talker speaker tomorrow Justine who's in the crowd here So she'll be doing a talk on what exactly technical architect means to our team at there for if you're interested And then we also have, you know, our specialized developers front-end back-end full-stack And our team is always growing. So if you're in the market for career change look us up So my name is Sean Rio. So I'm the director of strategy at there for so I mean in terms of what I do A lot of stuff at the operational level, but I also work with our clients on UX research strategy and business analysis I've been with there for for six years Like also spoke at many different Drupal events such as this and worked with Drupal for both basically my entire career Hi, I'm Clemens I'm a technical architect at there for and I'm mostly doing the back-end here and it's been five years I guess All right, so I get to the meat and potatoes here So what we wanted to walk you guys through today was case study for one of our recent projects so the Sort of the some of the background here. So we were engaged to work with Imagine Canada, which is a nonprofit here in Canada To redevelop a core product of theirs grant connect. So just to give you the deeper background here So imagine Canada they have a number of mission-critical programs that serve Canadian charities You know part of these are you know, basically we have a model where a social enterprise Supports a broader sectoral mandate For growth and advocacy in the sector So one of those social enterprises is grant connect So in the sort of history of grant connect that started off as a print guide 50 years old Which would list all of the different? Organizations in Canada that support charities through grant programs But that's evolved over time into different iterations of a digital product So they've had these online versions online database versions throughout that history And you know as a tool to charities It's instrumental to finding funding and operating in Canada But more than that it also allows Imagine Canada to support that mandate Because it's effectively a product with a fairly expensive subscription service So in terms of the project so grant connects gone through many iteration cycles through its history So it's been redeveloped over and over again. Obviously.  They wanted to move to a more iterative model to To developing and improving their product some of the reasons behind this so their new competitors in the marketplace for them There's also American incumbents that are you know potentially moving into the Canadian market So they kind of need to establish themselves. They do have a fairly unique value proposition in their in their their research and background But you know the risk of a new competition is the same in any industry where a company can come in with UX and You know totally disrupt the marketplace So that was a big you know Impetus to them to move to a more iterative model and be able to start actioning the customer feedback they've been getting So like I'm saying here to really be able to move this forward and continue to support that Social enterprise mandate they needed to change their approach So just to kind of dig deeper into some of the technical challenges or even project constraints You know we basically what we're trying to do here is take this product that has a long lasting history and future-proof it So if we look at some of the specifics so we have a massive amount of legacy data So we have over a hundred thousand different funder records with organizations from across Canada We have a gift history actually sourced from the CRA So we actually have Decades of data and and millions of individual records basically this represents every single disclosed funding for a nonprofits in Canada, so And a lot of organizations are giving away, you know hundreds of these per year. So it adds up We also have complex content structures So you know each of these funders represents dozens of individual data points and a complex taxonomy structure To organize it all And then we were also looking at all of this data being stored in an old MSQL database that nobody knew Why or how or any of these things so we needed to figure that out as well So in terms of some of the project constraints So coming in imagine Canada had a budget for what they wanted to do with the project and it was you know a fairly firm Project budget which meant we needed to kind of work back from this You know sort of sprint allocation And so six agile sprints was our goal We had a long list of legacy features. You know the application had kind of gotten bloated over the years We needed to deal with that, you know Change management on that side and then the you know as grant connects competitive advantage really is in its robust Data and also this legacy customer data going back years They really needed to make sure that that got carried into a new system even if the system was going to change And sorry so in terms of the customer needs so very challenging in this case as well So, you know different than I'm just your you know run of the middle website This is actually a tool that people are using to To perform their work in entirely technical work. So it was you know, we had to understand that grants seeking process We also have a lot of different needs that it serves Canadian charities can be really diverse and support a lot of different causes and populations You know in a lot of that, you know, there's some sensitivity around specific population groups in Canada as well and Then the yeah that that process itself the mechanics of it of course are very critical to How the whole application comes together So in terms of our approach So, you know, it's effectively what one of the pretenses of the prop this project was to move to a more integrative model For what we're going to do and we're also working within this budget constraint So to be able to you know provide a scalable solution. It was our recommendation to start with an MVP approach so basically what we the goal here was to ensure that minimum viable product to sort of set the baseline and to be able to build on that over time in an iterative way by keeping things simple keeping user experience simple and be able to action customer feedback through metric analysis and And user research so that the project can improve over time So again, just to give you some key points here on how this project works. So we did about a month of intensive UX design We worked with react on the front end and both on the design and development side We worked inside of the parameters of material design We worked with the content to Drupal 8 decoupled CMS, which we'll get into later and The project was basically three months Although we've added additional sprints since because we're already starting to action user feedback So to kick things off we engage our strategy process So the first part of this process as I mentioned before this identification process So in this project, obviously we're dealing with subject matter experts and we also had some, you know, very involved product owners So these stakeholders really were coming to the table with that desire to iterate the platform folder forward and build in their internal capacity to do that kind of research and improvement So while you know most projects even if people like say they just want to rebuild something there's always new features new things They want to do but for them the user experience was really the primary goal to improve that And in fact a measurable goal that they wanted to attend We also had a really good buy-in from our product owners young team at a legacy organization. So they Came with a lot of buy-in to this having a novel approach, you know That's obviously what you can't suggest an MVP in every scenario, but they bought in And then the team that we're working with was also and they come from the industry They're subject matter experts and without them we wouldn't have been able to dissect all the vernacular and terminology Because going into this I knew nothing about fundraising So speaking to that the initial process of this was kind of a crash course in fundraising for us I mean if anyone who's familiar with sales process or marketing process gets the model of the funnel So there's very much a specific fundraising funnel and this model informed a lot of what would become the user experience for this application So they we're lucky enough to have access to a lot of additional Or initial research that they had done in the background. So they had done a series of both most flows and Also, obviously Google analytics So reviewing that I mean while we were looking at that from the through the lens of trying to validate You know what we would build it was also very useful for us in terms of prioritizing our MVP Because we basically had data to back up what users just weren't touching and what users really needed So this data was crucial to us in you know negotiating the specific scope of the MVP But also in prioritizing user experience around actual core activities that people do So her stick evaluation, I'm not sure if everyone's familiar with the term But essentially what we do here is we take a set of established usability Horistics or basically models and we use that to assess an application based on you know kind of these established things So you can think of like accessibility is you know being kind of a broad usability heuristic So for the for us this kind of serves two purposes It gives us the ability to look at the current experience to empathize with the existing user base And to identify some of the pain points they might be facing at the moment some of the complexities that we're going to face from user interface standpoint But it also gives us you know this opportunity to just look at every friggin tiny little detail of the site An inventory it to really understand what it is We then did some landscape analysis, so this is sort of competitive similar to competitive analysis But because we were looking at an application that in a lot of ways shared features with things like sales Like CRM tools and other you know things like faceted search tools We wanted to look not just at the competitors in the landscape, but also contemporary analogous experiences So you know looking at things like a pipe drive, which we use internally as our sales tool Looking at things like Kijiji, which just so happened to have kind of a similar facet-driven model And then again looking at competitors in the space So another key component for us in terms of getting that buy-in and establishing the scope of the project was Value proposition thinking again because we're kind of re-approaching this old product with a new team and an MVP approach We could kind of lean on some of these startup methodologies like Value proposition canvas to really get to the bottom of what the core features should be in comparison to the tasks or jobs that a Customer is responsible for so if you haven't done this before I highly recommend it for any product oriented project You know effectively what we try to do is establish on the left-hand side the features of the product On the right-hand side we establish the jobs that users are responsible for and then we try to find an alignment between the two So the way we try to find that alignment is by identifying You know possible gains that they might experience in using the product and identifying how those features respond to those gains Similarly, we want to use that product to relieve existing pain points in their workflow And we can identify things within the product that actually do relieve pain so it's a great mental model exercise and you don't think it's effective and powerful for prioritization because if a customer says What about that feature that's supposed to be super awesome? We can say well, it's just not part of their core tasks So from that phase the initial identification phase obviously there was other things involved there But I think some of those were some of the most valuable things in terms of the MVP Moving on to our conceptualization design phase So it's a little washed out. I apologize for that but you know because we're dealing with a really really robust data there was a Strong need to ensure that information was prioritized So working with our subject matter experts We started to build a visual system around groupings and information Basically starting to set some buckets And then we worked with our those subject matter experts to really to drop in all these specific data points in groupings So that we could start to build that UI up around prioritization of information so This is a kind of exercise that can be done really nicely with sticky notes But we were actually workshopping this directly in Lucid chart with the client So mentioned before material design So I mean with any project where you have an MVP or prototype approach You don't always have the opportunity to get really deep into visual design I mean we certainly I think we put like a decent amount of polish on the designs for this project But we didn't want that to be a bunch of additional technical debt for the developers to make up so we tried to stick is strongly to a material design Principle as we could and use the material design spec to establish to ensure that on any interaction that we introduced in the application Was sort of built around a standard? So the additional benefit for this of this for us was that moving into development the developers were able to adopt material UI, which is a React based framework and Immediately start actioning some of the things that we mapped out in the designs because it was all just that a material design Now of course just us you know we want to make this thing look good It's a priority, so we certainly put a lot of effort into that But I think from one of the key things here was to just achieve that balance between material design Something that's highly functional and also the fact that this needed to be a product So, you know grant connect fits into this ecosystem as a sub brand And you know it does have its own identity and this is an identity that over the course of our engagements with Imagine Canada we're gonna be responsible for helping them to market So we did wanted the application to have its own look and feel and so certain attention was provided for that as well So yeah, so I'm gonna now move into just walking you through the experience that we came out from After looking at all this I'll try to walk you through a bit of the thinking and how we arrived at it But basically once we just filled down all of our research We basically established a three screen based application. So three different, you know possible interfaces With a search-based navigation and I think if there's any sort of like a broad takeaway for this is look at the you know It's a there's a lot of information on the screen But it's three experiences and I mean it's very rare that we get to design something or build something Especially in the Drupal space, but we only really have three templates So so I think it's a it's a powerful model and it's there's looking at things to that lens of simplicity I think worked really well here So to start this off so this the experience has been very much built around a Global search So this global search has you know three separate facets or areas that people can go in from The existing application had an autocomplete search that a lot of users would use as their primary way to direct themselves to an Individual profile so we didn't want to take that away from them. So we maintain that functionality But we also created some cross-pollination in here So we're able to pull out specific profiles which people are able to action directly But they can also Highlight individual search terms because we have such a complex and diverse hierarchy of charities in Canada This allows people to find those cause level or population level things really quickly And surface keywords, you know, right off the bat So this is probably our primary way people can get in but we're also giving people vectors for causes and their specific community The community they belong to You know as part of those search vectors So once if a person doesn't find a specific profile they're looking for we then send them to a faceted search screen so In terms of some of the information sort of the system that we established we were looking at this Each of the on the left-hand side. We have our facets very, you know sort of standardized way of organizing facets Above the search we have secondary filters, which are used to kind of narrow down things once we've established our search parameters We also have the ability to change the the sorting and also The and the sorting itself actually there's a few different algorithms in place here based on, you know different Ranking scores that are going on in the background So we wanted to be able to provide two experiences for this list Both an experience a card based experience where we have rich actionable or information On the card displayed, but then we also wanted to create a compressed view allowing for people to look at things from more of a high Level because we can do things like sorting by average gift size, which you know from an analysis standpoint is super beneficial To begin just digging to the card detail. This is where that subject matter expertise came in really handy. So Basically the system we created here was the left-hand side of the card was reserved for informational information Or information specific to the organization its mandate and its location Whereas the right-hand side is tangible actionable information and actions the user can take on the card So you'll note that we have the median gift You know whether the organization is open or closed for requests and information about their revenue for organizations that we have that and then the user is actually also able to assign a stage or in the The fundraising workflow or pipeline right in this card So we also have that in the compressed view So we're just in that compressed view. We're only losing a few key things But we still have lots of rich information and the ability to action it So yeah, so from there once the users found a funder that they're potentially interested in They want to be able to dig deep into information about that funder So keeping that card metaphor moving that forward. We have The on the left-hand side some of the more robust information about the organization and their mandate and their program And then from the right-hand side over we have our actionable information Users are able to add notes to the card They're able to add a request size and again They're able to assign a specific stage in the fundraising process to the funder Identifying it similar to what you do in a CRM or sales tool From there, I'm sorry just on the right-hand side mobile experience again. Sorry. It's so washed out So digging deeper into some of the information that we're displaying here So when you're doing fundraising a big part of the process is looking at comparable Organizations to see what kind of funding they've gotten in the past So if I'm working at a specific organization with a specific grant-seeking mandate I need to find the right match in a funder and the best way for me to do that is to look at other Organizations that they've funded in the past. I can also look at the gift size so I can optimize my specific request So you can think of this as like if you were doing a sales process And you knew how much money the company that you're you're going after spent on their previous website We very valuable information and when you're doing gift analysis, you have this so so this is CRS source data that they're Managing in a spreadsheet, but we're also providing some you know visualizations here I'll just clarify that not everything in here made it into the MVP But this does represent that vision of being able to do this analysis So to just move on to that third screen So once we've identified a funder we want to take a look at That funder over a long period of time. We have to prepare a solicitation much like in the sales process preparing your proposal So we need a pipeline So looking at tools like pipe drive or you know, obviously we're developers, so we are familiar with stuff like Jira You know, we wanted to use the same kind of model of a drag and drop manageable funnel so Effectively users are able to move these different opportunities through the pipeline from left to right from that identification process right through to that stewardship and The you know the goal here to be able to provide them with the visualization of everything they have You know that they're actively managing from a Solicitation standpoint And similarly to the search we're providing that compressed view So this also exposes some additional functionality like being able to look at lost deals one deals archive deals You know just a and again that that you know Longer list or more in-depth compressed list here as well So So I'll be handing off to you'll get a break from my voice shortly I'll hand off to Clem on to talk about this But I just wanted to talk a bit about our using agile for this process for this project so in terms of You know some of the reasons that I think agile it was a great model for this project and I mean you know often I think a lot of startups who are approaching a new MVP would lean on something like agile lean on stuff like lean to be able to you know Be able to accomplish the often constructive goals of a startup So I mean I think agile was really strong for us here But one of the key things I think was that engagement with the product owner So our product owner actually joined our daily stand-ups, which was amazing we had you know because of that we had a Clarification from them on anything that needed clarification and they were being someone who's actually done the job They were able to provide that you know, I'm really immediate feedback on a daily basis I think also Once the Stakeholder starts to take on the agile vernacular it because they're there every day that can be a really compelling thing Actually after going through this process with us imagine Canada started to do stand-ups internally And you know, I'd like to think that was just us, but I think I think you know It's a it represents a broader experience that people may be having around the efficacy of those shorter feedback loop cycles And yes So I think also another key point of having those product owners involved all throughout being able to do really tight uat cycles Was also building those internal champions People knew how it worked right off the bat because they watched it grow They watched it come together. So it helped with them onboarding other people both internally and externally I think another sort of key thing with agile here is starting to build up that vernacular around being able to move into phased future releases Approaching this with this MVP perspective It was very easy for us to start to take features or things that the customer wanted and say let's move this to the backlog Let's move this to the icebox. Let's you know, let's prioritize this for a future sprint as a way to Both create you know future opportunity to work with them to improve the product but also to manage the scope of that MVP and So in terms of some of the product benefits because obviously a lot of these benefits more influence our team But I think the product benefits here, you know The product being simple, you know keeping things there were a whole bunch of other features that could have made their way into this But simplifying it simplifies the onboarding process for new users It also allows us to get to that point of iteration in a little bit more of a managed way because as we start getting user feedback for the existing features We can refine them and then we can be very careful to add additional features because you know if a product is simple It's harder to blow and then yeah, I think that's really the key thing It's just making sure that you're having that simple scope of the application allows us to much easier Do much easier analysis as to how users are using the product and thus make tangible improvements to it over time So I will hand it off to come on here to talk a bit about You know how we implemented some open-source solutions for this project All right, so Come in me to this project as a technical architect. We got like other strategy already like smart Did So we knew some stuff. We knew that we it was a MVP with a short deadline. So we had to be fast We need we knew that there's gonna be iteration on it and improvements So like the MVP needed to work, but we can refine it later and we knew that there was a lot of data structure and that's Was like a big issue and we also knew that the experience needed to be really great so we choose for the back end Drupal For the content management because like it's always to like do a really great work with Data structure and taxonomy and everything and it also like shipped with a restful web services and authentication and also data migration which was needed on this project And especially we use contenta you guys know about contenta. Yes. Okay, cool So it's API first Drupal distribution So it comes with like end point API endpoint directly and also documentation about the end point So it's really useful because like you can work on the back and and give the documentation directly to your front-end user And they're gonna use that And yeah, it's also come with user authentication with OAuth, which is really useful for us on this case So that was really cool And for the front-end we use a react for multiple reason actually because first one is like it was it's pretty easy to learn for Web developer if you know JavaScript and HTML it comes like pretty naturally There was a big ecosystem. There was a material UI Library which was we need did and there's a large community too So we know that every like small stuff we need it already been done by someone else and Another way It's like also because we can use it in parallel So the front-end can work on the react app when we are like migrating all the data in the back end And so we are like to we are completely parallel on it. That's what's really cool. Okay Let me get that technical here Okay, so cool we can go into Deeper all right. So the big challenge is on this one There was 1.5 million raw data of gift And those gifts are referenced to entities into the Drupal So that was a lot of data and it was coming from a Microsoft SQL application And the user needed to be able to edit add and find those that are easily on the content side like Yeah, and there was permission on those data on every one of it this permission and We know that the clients use that spreadsheet inside internally. So We were like thinking about Doing a custom stuff with Google sheets. So what we do is like we create on the Drupal side customer entities for gift and and that and we export all the data from a Microsoft SQL into Google into Google sheets. There was multiple Google sheets and With migrate API we just import from Google sheet directly so the client still using spreadsheet on Google sheets and they're able to edit it and we import them when they need it and There is this directly feedback directly on Google sheets. So here's an example So you can see that all is green is means that the entity reference has been resolved in Drupal So we know that it's good and when it's read there's only one here But it means that Drupal didn't find the reference So it allows the client to work directly into spreadsheet as they're doing internally and still have feedback from Drupal And in a whole side, we don't have to do the UI. We don't have to do the administration UI for dealing with that. We just like import with Migrate API Another big topic was the search It's in react And there's filter this sort this facet a lot of facet this user customization So a user can be can ask to not show a funder for example if you don't want to see it anymore So we need like to take that into consideration and there's also permission So we couldn't like link directly the react into the solar. It was not possible. We have to go through Drupal So for that we use solar because it's the best And search API to customize the search as you will do on the basic Drupal website and After that you use facet API and view rest Display to deliver into G Sun directly into the react component. So for example in the facet There's only one component for all the facet. It just read the facet API rest G Sun and like build all the facet all together. So that's pretty awesome There was other challenges as a client one was on on the call every day and When you're building an MVP and you have not a lot of time And they ask for changes or they just realize how much powerful We stuff we can do and they say oh, that would be cool to do something like this You have to have a project manager will say no just copy that and we won't like move from that and can be a bit like harsh sometimes but it's how it is Another big challenges is more with react. You guys know Redux a bit yeah, all right, it's a store manager for react and We didn't like did it properly at the beginning and like we ending Ending adding more and more like a store into the Redux and that was an issue We had to reread it at one point and it's still right now. It's not perfect. So It says some stuff to do here pipeline functionality so it's a drag-and-drop interface and You are you need like instantly Feedback from Drupal to say if it save it or not. Otherwise, it's weird for the for the client to move it to another Column and waiting until it's done. So that was a bit hard and there was a Salesforce service authentication with oath To manage and the biggest show is that is Salesforce is using account with multiple people on it, but you don't know which people is Authenticated so it's really hard because it's customization on it That was really hard. But yeah, so what we learn as a technical guys First thing first you have to design your redux store from the beginning of the project I guess you already know that no, we know it so just Go with your Drupal architect and your react developer Just sit down and like design the redux store from the beginning because you cannot know what's you're gonna be And if you do it from the beginning you won't mess with it and One other thing is even if you do a MVP or waiting for performance first cause like It's something like as a MVP you say what I need to do is finish the product but you Like screw yourself because at the end you have a product which is not really cool And if you do it from the beginning it's way more easier to like put it every time on every like component and Especially with server-sized rendering on the react side if you can do it from the beginning it's way better Because implementing it after it's Sometimes tricky. Okay, so the benefits of the approach so being totally decoupled Allows us to have like the best in class experience So the react library library and community is moving really fast into like really cool new project And every library that we can use directly and we don't have to wait until a dual module is there Or we don't have to implement it ourselves. We just like can use our create as it is rain right now It's future proof like if you want to do an app We already have the back end and the API and everything and we can use some component from react to move it into react native For example, I don't know if we can do that, but if one day we need we can do that and also it's really parallel so You your developer your front-end developer like you give him the documentation about What content is giving you and is using only that sometimes you need a new endpoint because Content I can be like what just an API can be a bit verbose and you won't like something faster But most of the time it isn't like you don't have to communicate that much as As long as you have a good documentation and a good content structure I think it's your time Hey, thanks so With this project because we knew that we were going to move pretty quickly through MVP. We didn't do a lot of user testing on our prototype our initial like mock-ups and you guys we did do obviously we had our subject matter Matter experts internally, but we didn't really get to do that prototype testing But part of that was you know Understanding that we're gonna have that opportunity to get a product in front of users that they'd actually able to interact with Which I think is a lot more powerful So the reason that we want to do this and why we think it's important the Really, I think to get to that point of continuous iteration on something research needs to be a part of the entire process It doesn't end with Design it doesn't end like during development. It's research and and being you know Really verbose about research is really that advantage to any product So in terms of where the project currently is so we actually just finished an additional Two sprints with them leading into Their beta release it's currently in an alpha release. So they do have some users You know testing it and dealing with things as we break them occasionally but What's really great is how we've already started to be able to get feedback from users about what features they really feel is Are missing or what refinements might be useful moving into the into the future So another big thing that we wanted to do that we prioritized towards the end was ensuring that we had a Way to actually start to pull in Metrics from the application that were useful to user experience So often what happens when you build a website you turn on Google Analytics and then you go away and that's it And and then even the people who work as marketing managers or product managers or wherever They are never really look at the metrics to that I of UX So this was my request that I put on the team But I think it'll be really useful in the long term So we implemented this react Google Analytics library, which allows us to trigger custom events from basically any imaginable user interaction Which allows for really cool data to come through so Pardon the verbosity of this, but we basically have Every single time a user is actioning an individual profile. So they're moving it through different phases We get data on it And a specific kind of declarative data. So we have I mean, it's a code here But we know this for the user I should say but we know exactly what Profile or whatever funder they've identified and we know exactly what they've done So we're really tracking how they're using that pipeline on a granular level And the goal will be for us to be able to like over time to actually develop some aggregate data around this There's a bunch of other data points. We're looking at but really our goal is to get to that point where we're able to look at The users flow through the application In a really really granular way pull out trends and do all kinds of rich analysis On how people are using things. So, you know, I think I'm of the mindset that like, you know Quantitative research is like the most important thing that you can do but often we get stuck Doing a lot of qualitative research because we have bad data not because it's better You know, I think it's really important to obviously get in front of people and test them and get their reactions But that's like this emotional journey thing which is very easy to Obstocate and it's also very easy to get wrong because your data pool is going to be tiny You ask 20 users or 40 users and then you test This will allow me to get a hundred percent visibility on every single user that uses the application Look at behaviors at this high level and really dig deep into how people are going to use things And I think from our standpoint in terms of how we're working with a grant connects And in the team to really be able to provide them with guidance and feedback and recommendations And to be frank from a business standpoint identify new opportunities for us to make money by showing them exactly What's going wrong? What's going right and how we can improve things? So this kind of leads into our next thing. So, you know as mentioned before we have our there for care program Which you know in the learn we were sort of pivoting towards this right now But the goal is really to work with the organization on those larger strategic goals So working with Imagine Canada. This was actually one project with them We actually started working with them in 2017 on a broader organizational strategy So this project Basically factored in as the first step in that roadmap with them So in terms of what's next with them, you know, obviously the product itself isn't where that customer journey begins So we're actually in the midst of a project right now to redesign their marketing properties for this project for this product and two other Products that they offer they're technically programs, but it's all products to me And then so we're going to be helping them deploy a marketing platform for this also some customer journey orchestration around how users interact with different platforms and in general right we because we're We're this kind of interaction that we're having with Imagine Canada We're looking at different properties and looking to grow something like grant connect in terms of its user base And so by looking at their marketing strategy looking at all these things together and having access to that It's really powerful because it allows us to provide them with recommendations From that holistic level and it's I mean we're kind of blessed to have that opportunity So in terms of our you know ongoing Preview obviously what we want to continue to do is Research and improve grant connect which means we have to figure out ways to make more money for Imagine Canada So that they can pay us more to keep to improve the product So yeah, so we really think that that's a valuable way to work obviously we don't get that opportunity all the time But yeah, I mean whenever possible I think being able to have that deeper level of research and engagement with customers can Put together some pretty fruitful stuff So yeah, that's pretty much it So, you know for anyone in the room that's not a developer or working for an agency You know certainly reach out to us if you have a unique challenge similar to this one But obviously if you're a developer, you know, and you think that this kind of work might be exciting to you Definitely reach out to us. This is our marketing campaign email Don't take it personally, but we're just trying to work on our own lead generation So if you want to reach out to us, please reach out to us at this email and we'll definitely make sure that the right person gets back to you. I Don't know if you've been by our booth, but we are giving away these creepy Amazon echoes that watch everything you do Seems to be a concern amongst a lot of people. So I apologize next time. We'll try to give away something more innocuous But yeah, so feel free to come by the booth for that or just to chat We intentionally didn't include a demo of the actual product Because live demos are terrifying, but also because we want to do to come by our booth. So if you're at The camp tomorrow definitely come by and we'll give you a video preview of the app So you can see what all of these features got drilled down to when we actually talk about an MVP and You know finally we actually have a copy of this case study up on Drupal org slightly different version if you're interested in reading and the Yeah, so if you're interested in reading that go ahead and visit that slightly long link from bit.ly And that's pretty much it So thank you. And so I guess Just address any questions if anyone has any and today so everybody does everybody have their gift cards to the rec room Everybody like cashed in on this no Okay, well if you need gift cards to go to the rec room the after-party we have some panty on how some will be happy to Get you guys $20 so you can go enjoy VR experiences, etc. Thanks


Today, you know, the Scotty works five weeks now on training and the last four weeks I'm concentrating teaching him for the concentration and also right side of awareness. The kind of testing day for the last four weeks but he learns so as I observed he started showing initiative to make a decision for the right side. It's time to stop introducing the stopping curve so I start you know introducing you know giving him command stop while approaching the curve. The last couple he start already start stopping the curve by himself that's great. Self Scotty is really clever boy, he's very sensitive, you can see through the film but he's very fast learner so far so that's great.


Hi. Excuse me. Hi. My name is Ian Ramanick. I work in the team at Intel that does the open source drivers for all of our GPUs. And I've been working on Mesa and other drivers since 2001. And today I'm going to talk about some recent work that we did to reduce peak memory usage in a couple of the compiler pads. For the compiler pads for the shading language compiler. So this is basically a talk about optimization. So I'll bring up a couple of the rules of optimization and I think everyone knows the first rule of optimization is don't. Or maybe don't unless you really, really have to because at some point things are going to happen in your project. People are going to start using it in different ways, trying to do more things with it. And you'll have to. But any work that you would do on optimization before you reach that point is going to be speculative. And it will probably be wrong. You'll do the wrong optimizations. And so in the best case you'll either waste your time and in the worst case you'll be optimizing for things and making the stuff that's actually important worse. So we ran into a case where we had to do some optimization recently. Some of our new GPUs that are coming out for Gen 11, they have removed on some of the lower power parts. They've removed support for double precision math in the execution units. But FP64 is still required for OpenGL 4.0 and all the later versions. So we're not just going to dump from OpenGL 4.6 or 4.5 all the way back to 3.2 because there's no double precision. So we're going to implement it in software, just like you used to do in the olden days before CPUs had FPUs. And the performance isn't going to be great, but so far we've literally encountered zero applications that use this feature of GLSL. See first rule of optimization. Nobody cares about this feature, so it's okay if it's slow. And that's a big part of the reason why it's not a required feature for Vulkan, because just nobody actually wants it. So late in 2018, work on soft FP64 was getting pretty close to done. Pretty much all the test cases were passing, but the guy that was working on it noticed, huh, there's a handful of these tests that I'll start them running, they're going to run for a while, and I'll go make a sandwich or something and come back, and OomKiller has just wrecked my system, like what the hell. And we tracked it down to a couple of test cases that seemed pretty innocuous. So we mocked those tests up and ran them. We have this big, like 80 core server that we have for doing big compiles and some shared DB runs and some other things on. It has 80 cores and I think it's, I put in the slides 128 gigs of RAM, which I think is right. We actually got the test case to run the completion on that and it peaked at 80% memory usage. So it was, I think, 85 gigs of RAM, which seems really bad. So we kind of dug around and tried to figure out what was going on, because we looked at the shader that was in this program. And in this particular test case, it has a bunch of uniforms of every possible double precision type, so the scaler, all the vector types, and all of the square and rectangular matrix types. But then it has to actually use all of those or the compiler is supposed to dead code eliminate anything that isn't used.  The shader would fit on a single slide, but the compiler right now inlines every single function.  The thing that turns out to be the real disaster of it is this tiny shader that starts off with no flow control ends up with a little over 16,000 basic blocks in it, because each of those functions for doing FP64 operations, they all have to check for things like should it generate NAND, did you end up with a D norm that needs to be flushed? There's all these sort of exceptional cases that it has to check for, so all these functions have hidden flow control in them, so that all just explodes out. So, you know, most shaders that come from real applications aren't usually that big, and even the ones that are big, people have tried to optimize them to not have flow control, because generally lots of flow control performs poorly on GPUs. So we have this extraordinarily huge shader that doesn't look like other huge shaders that applications would normally give us. I guess no one should have been surprised that there was going to be some kind of problems with it. So the second rule of optimization is, you know, optimize the right thing, and I sort of condensed it down to a little bit simpler test case that I ended up submitting to Piglet for inclusion in Piglet that is able to hit the problematic paths without having to rely on the soft FP64 code because that hadn't landed yet, and also that I could run on my laptop without, you know, needing 85 gigs of RAM. It's one thing to have a passable test case, and it's one thing to just wreck everyone's systems, and also an entirely different thing if I completely wreck our continuous integration system because the guy who maintains that will come find me. You added tests that wouldn't kill everything, why? So for collecting data, I turned to Valgrind's massive tool, which is a, like, I'm going to expect that most people are at least a little bit familiar with Valgrind. You run it to run your app and it kind of inserts itself in pads in your applications so that it can collect data, and so what Massif does is it collects data through time about every memory allocation so that then at the end it can sort of show you this timeline of here's how much memory you were using and when, and I'm actually going to show some of that. Let's see here. All right, so here's the, so you run Massif and it collects some data, and then you have another tool called MS print that you used to actually display the data in a human, human readable might be an overstatement, but human puzzle outable or something. So the important bit here is the first thing that you get is this timeline showing how much memory usage you had through the lifetime of the program, and you can see right at the beginning there was this huge spike up to around five gigs that then dropped off and then there was a couple of smaller spikes a little bit later on, but the big peak is the important one. We can go down and we can look at the output where it will sort of show where memory got allocated so you can see who the big consumers are, and so right here 98% of the memory that's allocated was allocated, I guess no, 93% was allocated out of this same function, nerfee builder add value, for some reason that was hard to say. So that seems like a smoking gun of maybe look here. Okay, so fee builder is, ner is the kind of the mid-level IR used in Mesa's shader compiler. It's SSA based, and so fee builder is one of the, is the data flow analysis pass that inserts the fee nodes in the SSA form of the program. So we've got a couple of things that we could spend some time optimizing. We know where all the memory is going, but then that doesn't necessarily tell us where to go start writing code. And when deciding what to do, it's not as easy as just, you know, apply Amdahl's law and go work on whatever thing that points at, because in a real project there's other concerns, risk, amount of effort involved, schedule issues, all those things matter. So we kind of had three places to look. We could optimize the input shader, which in this case is the mangled input shader with all of our soft FP64 stuff inserted into it. And we looked at that and it was really unclear if adjusting the soft FP64 code would have that much effect on the final memory usage. We could probably adjust a few things and bring it down a bit, but that code's actually fairly complicated and it has a lot of twitchy edge cases that it has to handle. So it seemed like even if we could get a good amount of benefit from that, that by adjusting that code, because we're not floating point experts, that we might suddenly break things. So we didn't want to do that. And strictly speaking, the reason that so much memory is used in the first place is because every single function gets inlined always.  But once we get down to a certain point in the compiler stack, functions have never existed beyond that point. So through the whole backend compiler and all the instruction generation, there's no support for functions at all. So there's a huge amount of code that we would have to go right to support that and we needed to ship something. So we decided we'll take the obvious approach and we'll go work on optimizing the memory usage of the fee builder. So I'm not really going to talk very much about the process of putting a program in the SSA form and adding fee nodes because it's complicated and I don't understand it even that great. But at sort of the high level, what the process of inserting fee nodes involves is you're going to analyze each variable in the program and you're going to look at every basic block where that variable might get modified. And then at the points where through the flow control graph where multiple paths of those modifications could come together, you're going to insert a fee node because eventually what's going to happen is those won't be rights to the same variable, they're going to be rights to new variables and then when you get to that join in the control flow in which route you actually took, the compiler is going to get modified where that variable gets modified. Now, you can assign an ordering to basic blocks in a program and NUR does this and it sort of assigns an index to each basic block. So the fee builder just says, well I'll use a simple data structure that I can index with a unique value which just has an array at the bottom of the basic blocks, essentially of the basic blocks, it's actually the rights in the basic blocks and just indexes it by basic block. So this works great when you have normal programs, when you've got programs with 16,000 basic blocks, now you've got an array of 16,000 pointers for each variable in the program and almost 15,990 of those pointers are going to be null. So replaced that simple array with a hash table and it slashed the memory usage. We went from just a little over like around 5.4 gigabytes to about 1.3. I was going to show the massive output after the change but I'm running a little bit low on time. I might come to that. But the cool thing that shows up in the after massive output is that first peak is basically gone and that part of the program is no longer the long pole. So there's a couple other peaks later on but fee builder is not the critical memory usage path anymore. So at this point we had cut the memory usage enough that we probably could have stopped but I continued on to look for some more low hanging fruit and found that, so we have a fairly complex system in Mesa throughout the compiler stack where we have sort of a self implemented mark and sweep garbage collector. It's not that exactly but it's roughly analogous so we won't ever actually leak memory but we can have transient leaks. So along with a program, NER will track a bunch of metadata that gets used by optimization passes. So one of the things that it tracks is live ranges for a value. During what range of the program does this value need to exist because it might be read. And various passes will change the shape of the program and will invalidate that data. So if you have live range data and you delete a read of a variable because you can optimize out an instruction, well now the live range changes so the live range data becomes invalid.  And by looking at, by really digging down pretty far into the massive data I kept noticing that there was some of the metadata that showed up at points in compilation where we were never going to need that metadata again. And it turned out that it still existed there because it got marked as invalid and then no one released the memory. So changing literally added four lines of code and that cut another third of a gigabyte out of the peak memory usage from the worst case shader. And then just to continue on looking at low hanging fruit I did a couple of micro-optimizations using PA-hole or PAH-hole or however you want to pronounce it. It's kind of a cool program. You run it on your object files and it will analyze your structures and tell you exactly how the compiler actually laid that structure out. And so it will tell you where there's holes in your data structure. So for example here, after the tight field of the structure you get proper alignment of the pointer that follows that the compiler inserted four bytes of padding. So it's just dead space.  And it's definitely a micro-optimization. In this case it ended up being pretty useful. I didn't have any data about it which I should have collected because the NUR instruction is the base of every single instruction in the IR. So if you've got a thousand instructions in your shader you've got a thousand of these. So having that four bytes just wasted everywhere adds to death by a thousand really tiny cuts. So in this case what I did is I marked the enum for NUR inster type as packed so that instead of taking up a full int it would only be a byte. And I shuffled some things around. I think I moved the block pointer up to right after the exec node. And then they basically sorted all of the fields by the size of the underlying type and that eliminated all the padding. And I think it cut, what did it do? It cut eight bytes off the size of the structure? I want to say? Four or eight because of padding. And it adds up. So possible future work, there's still a bunch of places where we use way too much memory. We could implement real functions and I suspect we're going to have to but that's going to be a huge bunch of work. There's also another data flow analysis pass way, way down in the back end that operates at the machine code level that does a very textbook implementation of the data flow analysis algorithm and so it has these huge bit vectors with one bit per variable in the program and then you have a copy of each bit vector for each basic block in the program. There's other algorithms that you can use for that that don't need to have these massive bit vectors. I had looked a little bit at because most variables won't be live or accessed in most basic blocks. A lot of the bits in the vectors are either all zeros or all ones. So I tried using a sparse data structure to track that more compactly and it shaved off about 7% of the memory usage but since it was a more complex data structure instead of just a simple big flat array, the 7% memory savings came with a 7% run time cost. It wasn't the trade off that I wanted but if we basically chuck that algorithm and replace it with something that's not just from Compilers 101, we wouldn't take the performance penalty and it would use a lot less memory. How many minutes do I have left? Two? Okay, all right. So here's my pointer. So then here's the after graph. So the first spike is completely gone and there's just a couple of spikes later on. The peak spike is actually during register allocation. There's a shared component in Mesa for a graph coloring register allocator and it has some big data structures in it too, especially when you've got lots and lots and lots of basic blocks. I haven't analyzed that code to see if that can be helped very much because it's really complicated code and I'd rather not go in there if I don't have to. But okay, yeah. So that was what I had for that. Okay, any questions? I'll start here. In retrospect, would you consider saying to the hardware guys again to re-enable FB64? No, because I'd rather have that chip space used for stuff to actually make programs that people care about go faster.  They're mostly targeting open GLES that doesn't have FB64. So I mean we... it's frustrating that we had to do all this work and then had to do a bunch of other stuff to make the work actually work. But I think it was the right choice. And then do I have time for the one more question? I have the same question. Oh, okay, all right. Then you get the same answer.