Go ahead and start. So hello. Thank you for coming to Translating Success in Drupal 8, the Manhattan Associates story. I am James Rutherford. I'm director of client services at Media Current. And I've been working with Drupal for about eight years. Spent the last almost five years at Media Current as a developer and then a lead developer and architect and then finally as director of client services now. So my role at Media Current at this point is to engage with our clients, really understand what their business needs are and make sure that the projects that we're working with them are delivered on time and on budget. You guys can follow me at James Rutherford on Twitter and I tweet a lot about Drupal and about client services issues and I also tweet a lot about Boston Sports. So hopefully you're interested in that. So a little bit about Media Current. Media Current is a full service digital agency that specializes in Drupal. We're based in Atlanta and have about 50 plus employees ranging from a full development team to full design and theming team as well as a digital strategy team. So Media Current is going to strive to work with our clients and be that full service partner that allows them to excel in any area of a project whether it's planning and understanding how to execute a successful business strategy to actually doing the back end development and making sure that you're building Drupal correctly. They're also a lot of fun to work for and we have a booth down in 219 if you guys want to come by and see us. We have the bags tossing thing. So let's talk about our agenda. As the title alludes to, this is the story of Manhattan Associates and Drupal and them implementing Drupal 8 which is a project that we're in the middle of with them right now and that should be launching at the end of this week on Friday. So we're gonna talk a little bit about the backstory of how Manhattan Associates chose Drupal, how they went from working with Drupal previously to deciding to move to a Drupal 8 site for their current platform and then what were the lessons learned and the challenges of actually implementing Drupal 8 at this point in the beta life cycle. And then we're also gonna take a deep dive into the translation component of Drupal 8. That's really one of the killer features in core and it's one of the main reasons why Manhattan Associates decided to move to Drupal 8 which we'll get into in a little bit. When we go to the translation component, I'm gonna be handing off to Calvin from Lingotech. They are a partner of media currents and really the experts in translation in the Drupal space. So, and then at the end we should have time for QA. So let's go back in time to 2011. Manhattan Associates had just launched a brand new Drupal 6 site, fully translated in six languages. Featured multi-domain publishing capability so they could have content that was translated in several languages and publish it out to like a sub-site or a brochure site. Really a great marketing tool for their team. And we also featured CRM integration on all of their assets so that they could do custom eloquent campaigns for different assets and publishers in different languages. So, really, really powerful Drupal 6 site built sort of right at the end of the life cycle of Drupal 6. Drupal 7 had really just launched a few months before we launched this site. We had a great project, great successful launch with them and so Manhattan Associates and Media Current really enter into a long-term partnership at this point. We moved into a support phase with them and did a couple of small projects and then once they were ready for their internal team to support Drupal, we parted as friends. So, that's the history and let's move forward to 2014. Drupal in 2014 is an incredible platform. You guys have all hopefully watched the innovation in Drupal 7 and watched it really take over the enterprise space for open source. And an unfortunate side effect of Drupal 7 taking off through the stratosphere is that Drupal 6, really innovation in Drupal 6 kind of stopped to slow down and by 2014 it's really hard to do new things and solve new problems in Drupal 6 without paying a really high development cost. So, just to recap, Drupal 6 was actually launched back in 2008 so by 2014 we're almost eight years in the life cycle of Drupal 6 and at the time that we were launching their Drupal 6 site, the Drupal 7 community wasn't really robust enough or mature enough for us to make that decision. So Manhattan really got a great website out of it but it was unfortunate that they got in at the very end of the life cycle of Drupal 6. So from the Manhattan's team, their web team perspective they recognize that they need to upgrade. They have a good tool set that served them for a while but they wanna go fully responsive, they wanna serve to all different devices, they wanna improve translation tools and they're very involved in the Drupal community, they see all these great innovations in Drupal 7 and they wanna use them but they're hamstrung by being stuck on 6. So Manhattan is convinced that they're ready to upgrade, they know that they have to but how do they choose between Drupal 7 or Drupal 8? I think it's a lot of question that maybe people in this room are asking that might be why you're in this session or something that is being talked about at the con all the time and that's really what we're gonna try to highlight here is what are the decisions that Manhattan associates made and how did media current assist them working through that due diligence in order to choose a and feel comfortable with that decision. So a little bit more background. We're at DrupalCon Austin last year and Manhattan usually sends their team out to DrupalCon and to conferences and so we see them and catch up with them all the time and they came out to our after-party and really engaged with us and said, hey guys, our internal team has decided that Drupal 8 is what we want. We've done some local builds, we love the infrastructure, it's amazing. You guys are the Drupal experts, tell us can we go to Drupal 8? What are the problems that we're gonna have going to Drupal 8 and what do we need to do to prepare for a project in Drupal 8? And then finally, the really big question is when is Drupal 8 gonna be released? This is a full year ago at Austin and so here we are today and you guys are asking when is Drupal 8 gonna be released? Everybody is kind of interested in how do we answer that question? The more important aspect of this question is for Manhattan and for media current to be able to move forward is do we have to know when Drupal 8 is gonna be released in order to do a successful project? We already know that Drupal 6 has reached the end of its life cycle and its usability for that team so do they even have the ability to wait even if they knew a concrete deadline, this is when 8 is coming out, can they wait that long? So we needed a plan that would allow us to move forward looking towards Drupal 8, recognizing that we wanted to build for Drupal 8 but also recognizing reality that Drupal 8 might not be finished. The community has committed to building Drupal 8 as a platform that will be done when it's done and not setting an arbitrary deadline. So as we worked with Manhattan, our plan was to make sure that Drupal 8 core could do what we needed to do for Manhattan, make sure that we had a good fit there because we knew from prior experience that even if the Drupal 8 was released the likelihood of the contrived module community being caught up to that point and actually being usable was extremely low. So we can plan for Drupal 8, we can move forward but we can't be betting on a lot of contributed modules to be there for us. So we took a look at core and we recognized that right out of the box we get internationalization, which is the killer feature and it's baked into the initiative is pretty much complete and it's extremely powerful. We get the ability for them to do responsive theming with their own internal resources using twig and the ability to grow and change layouts on the site in a way that is sustainable and solves their goal of making sure that they're responsive all the time. And then we saw that problem that they've had since they adopted Drupal 6 is that they wanna be on the forefront of CMS innovation. So Drupal 8 looks like a good fit from that perspective. Now we still have to move forward, we have to do a discovery, we have to look at their architecture and actually come up with a proposal but we don't know when Drupal 8's coming out. So the way that we saw that problem was to make sure that the architecture that we were working on with them, the way that we designed the IA and the solutions that we were providing could all match up in Drupal 7. So if we get to the end of this process, we have a great design, we have a great new digital strategy, we have a great new information architecture and Drupal 8 is still eight months away, nine months away. We have the ability to say, okay, it would have been nice to go to eight, there was a lot of pluses of going to eight but it's okay if we go to seven. Seven is still gonna be a robust platform and we can move forward. And they felt comfortable with that plan as comfortable as we did. So we went ahead and started a design and discovery project back in May 2014. It was about two to three months and we worked really closely with Manhattan to figure out what their Drupal 8 website is gonna look like and build an architecture that could actually launch, assuming that eight was ready at the time that we finished discovery. It was key for us to make sure that we restrained ourselves with scope. I mean, anytime you get a new toy, the idea is to solve every single problem that you've ever wanted to solve and we can do this now, we can do that and we really work together with their stakeholders to say, let's rein that in, let's make sure that we're solving your core business needs so that we don't add a lot of unnecessary risks to eight and we actually have a timeline that's achievable. And then finally, as part of the discovery process, we were able to really dive in and understand, what is the level of effort to get Drupal 8 ready for Manhattan? Which control modules are the ones that are out there? Do we actually need and how much work will it take for us to get them fully ready and working with this system? And can we do a real estimate that will put a real price on this and understand the risk based on our timeline and on the estimate? During this process, which stretched into November of 2014, we got some great news. Drupal 8 Beta was announced in October. This is something that our team was looking forward to happening. With Drupal 8 Beta being announced, we knew what the core feature set was gonna be, at least from a standpoint that APIs had pretty much gelled. It really wasn't going to be too many more core structural changes. And we really had a good idea of what Drupal 8 would look like at launch as far as what we imagined or hoped it would look like and what we were actually going to be launching with. And so, from our point of view, the site that we needed to launch for Manhattan Associates, we needed views, we needed internationalization and we needed CMI to be rock solid. And at the time that Beta was announced, those systems were far enough along for us to feel comfortable with moving forward with a proposal. So, just to recap, we spent the last year planning and trying to find a way to make Drupal 8 work from Manhattan Associates. Trying to do it a way that is actually not a pie in the sky endeavor in a way that has low risk and allows them to be early adopters without concern that we launch and the website falls over. So, we complete planning and deliver a proposal to them in early January of this year, which is right around the timeline that we had wanted. Or they have a mid-May launch. Unfortunately, it wasn't good enough. So, we had been working very closely with their web team and their marketing team and the people that we had these long-term relationship with. And we gave them what we thought was a great proposal, but when they turned around and presented that to the leadership team, the leadership team had major concerns. We had not done enough to let them understand, why is it important to spend this much money to go to Drupal 8 right now and how are you going to mitigate the risk? Manhattan Associates is an enterprise-level software company and the idea that their new CMS was going to be based on a beta release was something that a lot of the leadership team was rightfully so sort of concerned about. So, we had to work immediately in order to alleviate this roadblock and sort of help justify our decision and then also do so in a way that allowed us to actually make the proposals projected launch date. We couldn't sit there forever and try to figure out how to mitigate this issue. So, the first thing we did was do a cost comparison with Drupal 7. We went through that whole exercise that we had talked about previously and gave them an estimate based on, hey, if you guys do 7 right now, this is what you're going to see and give them the ability to understand what the cost difference is between 7 and 8 at this point. That took a little bit of work on both sides, obviously, to understand what the 7 aspect of it was. And then we still wanted to reduce the budget further for Drupal 8 in order to say if budget is really an issue right now, if you're not willing to spend as much to get to 8, what can we do to make that happen? So, we worked with Manhattan, cut some scope, and then we also reorganized our project around an MVP that reduced risk. So, the MVP really was, what exactly do we need from core Drupal 8 and what's the minimum amount of contrib modules that we can use to get there. With the buy-in from Manhattan's team, we were able to reduce the budget to within what the leadership group felt was worthwhile on spending on Drupal 8 and move forward. And then at the same time, we had to show, what is the ROI of moving to Drupal 8 right now over staying on 7? One of the major factors that we talked about was, do you want to pay for going through the move to Drupal 8 three or four years from now? So, we adopt 7 now and 8 releases become stable and then becomes the center of innovation. If you're going to keep up with that innovation, do you want to switch CMSs again? If that cost, which is hopefully reduced, because Drupal as a community is trying to learn from the pain of further migrations, but there's also the hidden costs of your organizational costs, if you in three years have to move from 7 to 8, then that's the time that your CMS team and your leadership team is spending planning that instead of doing what they should be doing, which is executing on their digital strategy. So, by demonstrating a cost savings there and going early and continuing to educate them on the Drupal model of how the contrib community works and where innovation happens in contrib, we're able to demonstrate the ROI to their leadership. And then, I think the hardest task that we had to do was demonstrate that we had an expert level understanding of risk, right? That's the big unknown with Drupal 8. There is so much unknowns. Even in beta right now, we could see architectural changes that will cause your data to change or key features that you expected to be able to work with to work differently. So, how do we understand that risk and provide a clear roadmap that they can feel comfortable moving forward with? Well, the very first thing we did was do a real analysis of where Drupal 8 was. We dove into the issue queue and looked at how many critical issues do we have right now for this beta? And what's the velocity of critical issue over time, right? If we look back in December of last year, there was 110 critical issues in the issue queue. We moved forward to the time we're having this conversation in January. We're down to 72 critical issues. So, moving forward two weeks later, we expect that to continue. We talked about how with the API's jelling that development momentum would continue on Drupal as well. And then finally, we took a dive into those actual issues and said, what do these issues mean to your project? How do they actually affect us? So, of those, I think we said 72 issues at the time that we're having this conversation, we were able to determine that 53 of them were affecting the actual core components that we wanted to use in Drupal 8. And of those 53, only 22 of them were actual bugs. So, as part of our proposal to them and as part of our justification, we went through that list of 22 and showed them how those were, you know, we could mitigate those as part of the risk analysis that we had done. A great example is a bug in the core menu system that would mean that updating menu items post launch if, which is a giant if, that hadn't been addressed by the time that we had to launch, you know, how would they maintain the menu? And we're able to demonstrate that their team had the programmatic expertise and the partner and media current that would allow them to do that through, you know, fairly easy code deployments. So, addressing each of those issues, showing that we had planned for them and then showing them where in the budget we had estimated for those tasks instead of just saying, you know, here's a pile of money and we hope that we don't run out of it. I think really was what helped to sway the leadership team at Manhattan. So, what does that all add up to? January 30th, I got an email from one of the key stakeholders at Manhattan saying, hey, you know, we've got the green light. We're moving forward. And I took a screenshot of it because it was such a great email to get, but this is us finding out that we're doing our first enterprise level at Drupal 8th site. So, next we're gonna talk about how to build an entire enterprise Drupal 8th site in three months, which is really could be like three or four sessions. And it's hopefully some of the stuff that you guys have been getting in the other sessions. So, what I'm actually gonna do here is stay pretty high level and we can dive into some of that in the QA, but we'll talk a little bit more about some of the leadership challenges with doing enterprise Drupal 8th. So, we had an extremely tight deadline. If you remember, we actually lost the whole first month of January working with Manhattan to help them, you know, make that internal cultural decision to bet on Drupal 8th. So, now we're looking at February to middle of May to execute on the project. And the way we did that was by, you know, continuing to leverage our partnership with Manhattan. We've been working with them for over four years. We know their team members. We know what they're capable of. They know what we're capable of. So, we divided the project up into three different teams. We had one team working independently on the contrib modules that needed to be upgraded, allowing them to work in a vacuum and really just work against what the latest beta release of Drupal 8th was. And then we had our other media current team working on the actual site build, building out the theme, building out the more complicated aspects of the configuration. And then we worked with Manhattan to have them do some of the more basic Drupal configuration. They found that Drupal 8th UI was intuitive and they'd already been using 6th for a long time. So, they were able to build views and build content types and move the data entry aspect of it forward. So, we have all three of these teams working in parallel. And we also, before we even got to the point where we kicked off, had done preparation. So, Manhattan's team has been working with Drupal 8th to vet it for over a year, testing building views, testing building content types, et cetera. And media current committed our entire development team to Drupal 8th and Symphony 2 training, which actually came in handy as we moved forward into the project. And then we talked about this throughout the whole project, but we budgeted for risk in order to say, we understand as a company that there are unknowns in Drupal 8th. And by budgeting for risk and forecasting for risk, making sure that we had team members that would be available if a certain aspect of the project hit trouble and we needed to scale up team members, we were prepared to do so. I wanna highlight that in budgeting for risk, you don't just wanna create a big bucket of money that you're hoping will cover whatever deficiencies in the planning you have are at all, that's the recipe for disaster. What you wanna do is that do diligence and make sure that you're prepared to take the time and spend the money to do the due diligence to understand how am I utilizing Drupal 8th? How does it match up with my actual business use cases and where are the deficiencies in Drupal 8th right now that we need to mitigate? So because we were able to do that work, we were able to have an appropriate budget for risk and move forward as things got complicated. It wasn't all easy as I'm painting the picture here. Hopefully there was a few hard lessons learned. So I'm gonna stay pretty high level, but one of the toughest ones for us was that at this point in the life cycle, and I think it's probably still true right now, contributing modules are not as far along as we expected. We plan to use core as heavily as possible, but we still needed to target the correct contrib modules and not bake in a totally custom solution that as contrib matured Manhattan would have to pay then to move on to these contrib solutions. So we did our due diligence, but really when we went to implement a lot of these contrib modules, we found that they either weren't as far along as we expected them to be, or that if a Drupal 7 version of that module behaves in a way that you expected it to behave for a long time, don't bet that the Drupal 8 version will work exactly like that. We also learned that twig is not a magic bullet. For those of you that don't know, twig is the new templating layer in Drupal 8, and it's also used in several other projects. It's extremely powerful. It's according to the front end developers, and it's great to work in. So it's not really twig, but twig's integration with Drupal at this point lends itself to problems. Your expectation is that all of your data with minimal effort will be there in the theming layer in order to use, and the reality is that you'll have to be writing bridges at this point in order to do things like to get sub menus to show up in your template. So that was something that we expected to be a little bit easier, and had to mitigate as we moved through the project. And then finally, even with all the training that we did, and even with an entire staff of expert Drupal developers, the ramp-up speed is a factor for new Drupal 8 team members, or for really anybody at all that's coming into it. It's a new platform, and the fact that it's a rewrite means that you're relearning lots of convention and lots of little sort of veteran things that we already know about Drupal 7 and Drupal 6 that ultimately end up slowing down that time and making it so that even with training, it's very difficult to say, hey, we've got two developers over here and we need 10 more hours in this project. Let's just throw them at it because they need that time to get up to speed. Okay, so what we have is a new Drupal 8 website launching for Manhattan on May 15th with an asterisk because it hasn't actually launched yet, but we're expecting it to launch. It's not my call whether or not it launches, it'll be Manhattan's, but look for that on Friday. I think the big takeaway for anybody that is endeavoring to move down to Drupal 8 right now, you can't wait to get off the block, is make sure that you can afford to do all of that preliminary research and to match up all of your feature set with what's actually in Drupal, not from a high level bullet point, but get into the issue queue and get your hands dirty and make sure that your expectations are correct. And then the other bullet point that I wanted is that even though this has really been high level, one thing that our team has shared with me is that it's Drupal 8's benefits, even with the difficulties we're having now are pretty clear. Once the developers are up to speed, they're able to develop efficiently and for the most part are enjoying working in the new framework. The same with the theme layer. The translation functionality is, you know, another leap year ahead of Drupal 7, which already has incredibly robust translations, so we're gonna talk about that here in just a minute. And the improved UI and editorial experience is something that Manhattan has been very happy with, so. Okay, thanks. So, Calvin, do you want to transition? How's everybody doing today? Good. How many people here have translation needs or do they do translation on their websites? Everybody, oh, good. And everybody's in seven, or six, six, seven? Seven. And you guys are looking to move towards eight at some point. Great. Well, we're excited. Let me just give you kind of a high level. I'm from Lingotack, hold on. Lingotack, we call ourselves the translation network. I'm gonna go through these slides really quickly, but I'm gonna show you the power to translate is now inside of Drupal. Drupal actually, Drupal 7 and Drupal 8 are have great multilingual functionality baked into the core and contrived modules. I'll start out by giving some problems and solutions that we help solve and also show what the differences are between D7 and D8 are. We appreciate James and his team, Media Current, great partners of ours, appreciate them allowing to take some time here to describe some of this stuff today. So I'm gonna just talk about today's challenges with multilingual websites. I'm gonna show you a flexible, agile solution that's inside of Drupal. We'll take a high level tour of how it works currently in seven and then an installation setup and just some quick workflows. And then we're gonna take a closer look at D7 and D8 and what the differences are and why we're excited to see D8 come out and part of the reason, or big part of the reason these guys decided to go with D8. We'll do some Q&A. So the problem that we have is that today's websites are highly dynamic and it has a continuous life cycle of change so you're always updating and there's different types of content and whatnot. And the legacy process for translating, sorry, that's really kind of hard to see, is laxed up in speed and agility so it's really hard to translate stuff so currently a lot of people are copying and pasting stuff out and having to send it over to a translator they're hooking into Google Translate to do some translation and it's just kind of challenging. And so you have this legacy translation process and so why is it, what are we trying to solve here? So if you look at a typical project and this is just the translation piece of it, you have some project manager stuff and some web administrator stuff but you have to identify content, new pages, edit pages, download the pages, export them as either PO files or X-LIF or something like that and most translation companies that you talk to only do one box of this and so you have all of these steps in a translation agency or what's called a language service provider is only going to do the translation part of this and so what our software does is it starts to take parts of the system and well actually when we go back into this when you type this by how many languages, number of pages, words per page, how involved the process is, time zones, steps in the process, it starts to get kind of unwieldy and so even ourselves we have a 500 page website we're in 10 languages that makes 5,000 pages so it's a lot of content right because you have to keep track of and position so it's inefficient man hours available to keep up with the pace of the change and the volume that you're looking at so there's too many steps if it's manual just to even keep track of I mean we have a lot of clients you see they're taking care of the stuff in spreadsheets and Google Docs and that kind of stuff it's just kind of crazy there's not enough automation in the process and so what we do is we provide a flexible solution it allows you to automate for managing all these workflows and saves you time, money, mistakes and makes the process much quicker so how does it work in our particular case we have your content in your website in Drupal we have a module, Lingotech module that you can download on Drupal.org that identifies or helps identify the content that you want to translate that stuff is pushed up into our translation, our cloud-based translation network you can run through different workflows here in just a minute and then when those are done it's published back into Drupal we're not replacing the core functionality of Drupal Drupal and all of the contrib modules in seven and eight are really the plumbing to make your site multi-lingual what we do is help to translate that content like how do you actually get the content out and translate it and replace it or get that back into the system so not all content is the same so some people say I have to professionally translate or I don't we actually have what we call a content value index where you can actually do different levels of translation so some translation or some content may only be machine translated it's super inexpensive to do that it's like 0.03 cents a word basically you can do community or crowdsource translation so if you have in-country marketing managers or folks that can help you translate you can actually have them log into the system and do the translation some cases you need professional translation and review so just consider these all workflows and you can actually do a different combination of these some people will actually do machine translation and they'll do a post-edit by a professional or a community member our system does a flexible enough that you can do it based on languages so one language might go through one workflow another language might go through another workflow which is really handy especially when you have like 10 languages and a lot of times you have 10 different locations that are each have their own offices and they kind of want to control content or at least have a say in that so that's helpful for that as well so all of these pieces that you saw in this earlier thing are handled actually by the module so we automate the process of mailing the files or you know FTP or emailing the files around and whatnot so all of these steps actually are handled inside of the Drupal site inside of the module and then the translation process is handled by the translation management system and again those different workflows and then the rest of these pieces are handled by the translation management system itself and so what we're eliminating is not the actual translation itself word by word but we're actually eliminating all of this project management or webmaster stuff around translation that makes it very complicated so as you can see if you can start to eliminate a lot of these steps and all of them are handled in the system then it makes translation a lot easier so that's what we're trying to do so I'm gonna do a quick tour of the product itself and this is just the power of translation inside of Drupal it's super easy to install we recommend if you're going to install it to use a Drush command there are dependencies that make in particular this is D7 make it multilingual so the Drush command will actually install all those dependencies if you've ever tried to install or create a multilingual website in D7 it can be challenging because there's about five or six contrib modules that you have to turn on in a certain order and turn on certain content types and that kind of stuff we automate all of that in the install process so it makes it super easy to do so when you have the modules in you can go in and just enable the Lingotech module you create an account which is the account that sends the content up to our translation management system just basic information you can do your source language which is in this case we're showing in English this is actually a commerce kickstart distribution that we're inside of once you get inside you have this translation dashboard that shows your content how much it's been translated if you can see these there's different languages you can add languages you can sync translations it gives you a really robust way of seeing what is and is not in sync in this kind of dashboard of you it's actually very easy to tell if content is translated and you make a change on a page and you retranslate that page you only have to translate the changes because we keep track of those in translation memories anyway it makes it super easy sync the translations in this case we're going to just translate the entire website into machine translation this is an example on this this is really handy just to get an idea of what a your site looks like in a different language like I said we don't necessarily recommend doing all machine translation but you can do the machine translation and it certainly can show you what your menus get translated and all of that and where you need to work on different pieces of the product once the translation happens of course you have this kind of drop down of different languages and you can do a language switcher wherever you want but you now have this English site is now been translated into a machine translated site so it's really that simple you can you know a 5, 10 page site or a 50 page site can be machine translated translated in like 5 or 10 minutes it's actually pretty fun to watch because it seems like it's almost like magic because the whole thing is translating menus and everything they've done a great job now if you want to go into into editing you can you know send different translation engines we hook into different machine machine translation engines if you want to use those so we can do like Google and Microsoft and there's a variety of paid machine translation engines as part of that now you have all these settings I'm showing you this really quickly but there's you know all these settings where you can synchronize stuff you can include source text automatically send stuff up for translation you can do it manually or automatic you can pick different different engines you can decide to translate your URLs if you're using switcher on that but anyway so you know switch the page you can also do what we call community or crowdsource translation or this might be someone who's an in-country marketing manager and this is someone obviously who was logged in and they do have permissions to see this and do the editing but we do a box here that says you know this page has been machine translated and someone can click on the link and it actually brings up the Wingotech workbench that allows them to do a translation so this is a post edit of machine translation and you can use that with you know your crowd or folks or on your team once they do the translation and they click save and continue that page is automatically API back down and the translations are now live now all the translations are stored inside of Drupal and we do that for indexing and for search purposes and you obviously for SEO and that you wanna make sure that those are there so you know the it's not like a a proxy based server where stuff is stored and then on someone else's server and it's not indexable so think of Drupal sites are you know highly dynamic pages come up with different content based on who's logged in and that's all served up from Drupal and we don't replace that functionality at all now if you wanna do professional translation this is actually where we start to make money our module's free to download and free to use we get free machine translation just so you can try it out so I recommend you just check it out to see how it works for you we also do professional translation it's based on the content type and words and languages so you can click on professional translation we give you an estimate based on good, better, best quality of translation and give you an estimate you can pay for it and have that professionally translated so that's actually how we make the money once the translation's done then the site again APIs back and you have that translation so I'm gonna jump into what the difference are between D7 and D8 are the Drupal 7 language Drupal 7 landscape had a lot of there's a lot of trouble spots and most of it's been kind of patched and worked around there were many things that were not translatable in core and English was not the only true source language so in D8 you actually pick your core language when you start so in D7 it assumes everybody's English and it starts in English and you can change it but when you first start you have to start in English so that gets fixed in there but when you look at the actual landscape you have this huge graph of all the different dependencies to have D7 work out green ones are in the core and then there's contributed modules I know these are too small to look at and this is not by any means a representation of everything but there's just a lot of stuff going on there's a lot of relationships in that now with that being said D7 is actually still a really good translation and multilingual CMS it's we deal with a lot of different ones as a company and it actually does handle as messy as this may look it actually handles it better than most content management systems so D7 taxonomy translation we have the I18N versus entity translation both are young both are active and who wins so you have to pick which direction you're gonna do in your translation some have pluses and some have minuses and you can have those in the Q&A and I can have my developer give a little more explanation of which is which but if you pick one and then you want to change it is really difficult to change in mid-process and so you really have to kind of know what you're going to do many translations is a source of a lot of headaches it's kind of hard to set up and you know you have this language neutral when do you use it, when do you not use it a lot of times I just ignore it in certain circumstances and whatnot so you really have to start to understand and do a lot of research to figure out which of these languages you want to use so the field collections is always a favorite of folks to do it's easy to, you know, fun to set up it's easy to work with but it's not very friendly with translation and it's very fragile and so we have to do actually quite a bit of work around in our module to work through those issues and field collections is not maintained very robustly I mean I think it's there's a lot of people use it but not a lot of time is spent fixing bugs and stuff in there so we have a lot of work arounds for that so the easiest way to get and make your Drupal 7 multilingual ready is to use our translation we'll take translation like I say use the following commands it's at this project even if you don't use LingOtac I would suggest using us to set it up because it'll set up all the dependencies automatically through command line it'll save hours and a lot of headache to figure out what things you have to turn on and off there's books written about it that are 150 pages on how to get this set up and we can solve all that you can have it all set up in like five minutes so use that just to make your site multilingual ready and of course you can certainly try us out and use the free machine translation so let's talk a little bit about the Drupal 8 landscape really quickly so D8 is a huge multilingual effort it's one of the was it six main core pieces in it? there's four, right, four and language interface, the content and the configure, the ones that we're looking at as the main parts of that and they've spent a lot of time and effort obviously as Dries is international I think it's kind of core his heart it was interesting to see this morning I think internationalization was in 2.0 or 3.0 and it was one of the very first things that they've done so it's been a core what they've looked at and obviously eight they want to take it even more deeply so what's interesting is when you open up Drupal you actually pick a base language and if you guys have anyone's downloaded D8 and started to play with it you'll start to see some of this stuff but there's no longer the sense that it's in English and everything's going to be translated you pick the core and then once the core has started it's now in that language now that's a fundamental shift to say now everything is in this particular language and we're going to start from this point it also makes some interesting connections where you're translating stuff because in the United States we have a tendency to say everything's in English and we're going to translate into another language and actually there's a lot of other languages that now want to translate into English so I might have a French side that now I actually want to translate into English or Japanese that I want to translate into English this makes that easier for that to happen the translation interface has the ability to set up count settings, site information titles and descriptions on the buttons and that you can go into translations of Chimps and Chumps but it makes it which is based on Entity Translation in D7 configuration all of these configurations have been standardized so remember in D7 you could translate text on many terms in two different ways and now it's all done one way so that's all been combined the I-18N Entity Translation more clearly defined as long as it's been drawn so the difference between Drupal 8 I-18N Entity Translation those are both contrib modules I-18N Entity Translation they're now part of core so you actually have people that are maintaining it at the core level as opposed to a site project for folks so what is the winner? the winner is all of us so we've gone from D7 which has all of these different dependencies and this kind of structure to a very clean and easy interface now again Lingotek sits on front of this the D8 is the plumbing again for multilingual translation and Lingotek is the means of translating that content so people come up to us and say are you going to go out of business and D8 comes out and actually no we're not it actually expands it makes it more prevalent in the community and it's a bigger thing and more internet so just from our point in Lingotek we actually do have a D6, 7 and 8 version we are in dev version we support I think up to beta 6 or 7 10 we got to 10 and we're going to have when D8 comes out we'll be 100% compatible and feature parity to D7 so we do have that as a company that's committed towards that as well and here's my here's my contact information if you want to know more about Lingotek Lingotek.com slash Drupal you can go to Drupal.org project slash project slash Drupal and download 6, 7 or 8 on there but we're very encouraged by the response we've had at this conference and people coming by I know both of our booths and asking about D8 and translation as well got some time for some questions from either side here yeah there's a microphone if you can go because they're recording all of this if you have a question you can go up and ask that so were there any pain points in content migration from D6 to D8 from Manhattan I'm in the position of trying to convince my boss who makes all the decisions to go with Drupal rather than some of the other content management systems out there and I was wondering if you guys particularly with Manhattan had explored any of the other CMSs or were you pretty much they were sold on Drupal you were sold on Drupal and it was just 7 or 8 and if even if not like is there any takeaway that I can bring back and say this is why we really need to go with Drupal rather than WordPress or something yeah yeah so fortunately for us we didn't have to make an argument against another content management system even though we had to do a lot of work to justify our proposal to the leadership at Manhattan which makes sense that's the same thing you're going through now we had the buy-in from the core team that Drupal and from the leadership team that Drupal was the right way to go regardless of whether or not it was 7 or 8 it was really just justifying that making the jump to 8 early because of their deadline they had to go to a new platform was the right way to move so it was fortunate for us that we didn't have to really deal with that question but if I'm in your shoes I mean I think that the absolute killer is that Drupal 8's core integration as opposed to it just being a module is what differentiates it from the rest of the open source community by the fact that every single API and every single aspect of core takes the idea of translation as part of its fundamental step I think makes it an entire class above other open source CMSs yeah certainly other open source CMSs have a little bit of the D7 problem where you have to have extra modules or plugins to make the plumbing work so to speak in WordPress for example WordPress out of the box is not multi-lingual ready and so in that case you have to decide whether you're going to do multi-site or you're going to get a third party plugin to do the plumbing piece on that and then there's a whole security issue that comes along with WordPress there's a lot more security holes and issues along that if multi-lingual is the is the kind of the gating factor we deal with lots and lots of different content management systems and Drupal even Drupal 7 handles it better than most if not all on the market right now so I would be happy to help you make that case with your boss if you want to do I think you either said this or implied that there was some sort of system that in which so if I have a website and it's in five different languages and everything's all translated and suddenly I decide I need to change the content on a given page is there a subsequent like for every other page or node with which is there like a revision that's automatically created with the notation for what needs to be translated subject to you know translation and then publication of that node I'm just wondering how that if there's like a you know say a waterfall effect where I've got my site in place I mean I've you know developed multi-lingual sites and it's like once you've published a page everything is completely independent if you know if I change the content on one page I just have to know that I have to go to the other page and change the content which is there's some sort of system that lets you know that it has to be done for the other so Calvin do you want to take that from Michael? Yeah so in our platform so our translation management system and I mentioned this and I didn't go into too much detail because of time but the benefit of using a translation management system is what we call translation memories and translation memories are previously translated content that has been approved by a human so it could be machine translated and then someone post-edited and they said this is the correct you know sentence so to speak when that is stored in a memory that becomes a kind of the corpus for that particular content on a page you say you have five paragraphs and you've written the content you've sent it off and it's been translated and a week later you change one paragraph or one sentence you can resend that content back up to Lingotech for translation and we match it against the translation memory and we only have the differences translated now we call that content re-use and recycle so you can re-use that content over and over again what that means is you don't have to pay for that content all of the content to be re-translated only the new piece you don't have someone do the post-editing on it if you have content that's similar on page to page that actually gets picked up too because it's not just the page you can match against any page in your site and so most marketing sites support sites knowledge bases have similar content in them same writers are writing them or they're copying and pasting one to the other if the translation memory has been set you can run any page against that translation memory and the differences are kept now in the old days before we were using a translation management system you would have to keep track of the change so you would say here's the page I've made a change and then put that in some sort of a spreadsheet and then send it to the translator and then when it came back you would have to either replace the content of the entire page into the page or you'd have to remember which sentence it is and maybe it's a different language you don't know how or where to place it all of that is eliminated so when you see all those steps in the boxes and you say how am I eliminating that's what the system does is it programmatically keeps track of all of that stuff and all those changes that's really where you save the time and money so we have clients that you will save 25% in translation costs because of memories and using the system but they actually can translate 50% faster because of these changes and so if you're going to market the product and you can go 50% earlier than you were planning to you can see the kind of the benefits of that does that answer your question? that's a good question and that's a big benefit of what we do with the web connecting people from all over the world and people being able to come to sites and discovery and all that stuff is multilingual something that media current pushes to your clients, prospective clients not just from a localization perspective but also from an accessibility point of view yeah thanks that's a great question I don't like the term push that's something that we try not to push anything we try not to push anything on to any clients it's really speaking strictly from a media current perspective we have processes that we like to work through with clients so that essentially what we're doing is becoming a partner with you so what that means is we're trying to understand what your business problems are why you're trying to achieve your goals so that any solution that we're recommending or working with you on really shouldn't feel like a push it should feel like a natural fit if we have a client that we feel would benefit from internationalization because their goals their digital strategy, their potential market then we're going to try to demonstrate them why this would be a great move for you as a company and here's how we can do it fairly painlessly for you to move forward if we feel like it doesn't fit not because they wouldn't have gained some benefits from translation but because it doesn't match the budget they have currently or they have bigger issues the budget they have would probably be better off suited than in that case we'll kind of mention it as this would be an advantage for you but probably something that should go in the backlog or be a deep prioritized Billigence and your risk management were you able to look outside the company if you needed the additional resources if you hit a particular problem with Drupal 8 were you able to find resources that could help you or was that putting more internal resources into getting your own people up to speed even faster how did you manage that risk and getting additional resources that's great so we didn't plan on working with a partner certainly that would have been something that had the timeline called for that and had our own internal forecasting made us realize we know that we need to have these personnel and we know that we're not going to be able to have it if we get to that point where risk becomes reality and we need to ramp up that's something that we probably would have considered or at least discussed with Manhattan but from a forecasting perspective we felt like we kept the resources that we need and that we prepared the resources that we need in order to be able to flex so I think it's kind of a case by case basis it's always hard to see where you're going to be at any point in time so I think we have about time for one more question if anybody has one more you got a question all right well if there are no more questions I appreciate your guys time today I appreciate James and media current and I appreciate everyone coming to the session oh yeah media current and lingo tech we're doing an after party tonight at city tavern so it's a partnership between media current and lingo tech so come out and hang out with us and we'll talk translation all night if you guys have more questions that's it it's from 8 to 11 it's at city tavern it's about four blocks from here thanks guys


Hey folks, last of the day. So I'm happy to be here again after I think three, four years when we talk a little bit about the work that we were doing with Haskell long time ago when we first started using it. I'm going to talk a little bit about what we do at Haskell and kind of our experience using Haskell at Haskell and I hope that kind of this talk gives you a nice sense of the features of Haskell that are useful to us. Really simple ideas and simple topics but maybe hopefully should be an interesting case study for you. So before I get started I'll just like to quickly sample the audience so how many of you write Haskell regularly? Okay, how many of you have dabbled with Haskell a little bit but don't really regularly write it? Okay, cool. And how many of you have heard of GraphQL? How many of you use GraphQL? Alright, love it. Perfect. So I'm Tanmay. I'm one of the founders at Haskell. We have an open source project that we call the GraphQL engine and I'll kind of talk about what that does. I'll start with a demo to kind of motivate the idea behind what we do and then get into how we use Haskell. I'm going to cover a little bit of a little introduction to what Haskell does maybe a brief intro to what GraphQL itself is for those of you who are not familiar with it and then the architecture that we have and why we've built Haskell the way we built it. I'm going to talk a little bit about the compiler pipeline touch upon two key ideas. The ideas of how ADT makes it very easy to kind of represent syntax of a different language and how ASD transformations are particularly easy for us because of the compilation or transpilation step that we do and then I'm going to go into the real-time component that we have and how that was also spectacularly easy to build because of exploiting STM constructs in Haskell and then I'm going to do a quick review of the stuff that we use our tech stack, our libraries. I'll talk a little bit about the problems and the challenges that we have now going forward and then kind of wrap the whole thing up together. We'll do questions towards the end. I also have Bamshi here with me who is the lead architect. He honestly knows ten times more than me about everything that's happening here. Cool. Okay. So what we do at Hasura is that we want to give users, teams and application developers a GraphQL API instantly on top of an existing or a new Postgres database. And so the idea is that if you're building a mobile application or a web application and you want to use APIs from your database you don't have to write cruddy real-time logic. This entity kind of just sits in the middle in that little architecture diagram and that kind of replaces a chunk of the business logic that you would have written, especially if it's just cruddy or real-time. We launched this as an open-source project about four months ago. We started writing this four years ago but then we just open-sourced about four months ago and since then it's picked up really well. Our users include Fortune 500 healthcare companies, government transport authority, banks, AI startups, DevTools startups and recently crypto startups as well. So it's been fairly, the adoption's been very great for us and we've been able to move very quickly as the community wanted us to move. A lot of that is due to Haskell. So I'm going to start off quickly with a demo just to show you what this is. I'm going to deploy this on Heroku. How many of you are familiar with Heroku? All right, cool. So Heroku is a provider on AWS. It kind of helps you deploy Docker containers and sets up a database for you so that you don't have to set it up. So I'm going to create a domain called fncon and you can see that this is provisioning Postgres for me as well and I'm deploying this. So what's happening in the background right now is that I have a Postgres database and we're deploying a Docker container and the Docker container is essentially the Haskell GraphQL engine which gets deployed as a demo and I can afford to do this live because the Docker image is just 15 MB in size which is also one of the benefits of having a language that you can compile down and then have a really tiny Docker image that makes it faster shipped. If this was on the JVM, I wouldn't have been able to do a live demo. It also wouldn't run on Heroku's free tier. Not hitting on anything. All right, so this is kind of the GraphQL endpoint. If you can see, I'm just going to zoom that up a little bit. That's the GraphQL endpoint that your application will hit. You'll make a post query to that and you'll make GraphQL queries to this endpoint to fetch your data. So what I also have set up is I also have a sample database and I'm just going to import that sample database right now to show you what the GraphQL API on a sample database looks like. So this is the database that I'm going to use. This is the Chinook database. So that's the database that I'm going to import and it's just a standard database. There's Album, there's Artist. Every album has a bunch of tracks. Just a standard relational database. I'm going to connect directly to the database on Heroku and then I'm going to import an SQL file. So that's me connecting to the database very slowly. There's nothing in the database right now and so what I have is I have an SQL file and I'm just going to import that here. So you can see that that's creating your sequences and then eventually it'll start creating the tables. Let's give it a few seconds. So what happens is that on the UI, Hasura tells you that we introspect the database and so we connect to Postgres and we say, hey, what are the tables that you have? And then we show you those tables and we say, do you want to expose these tables over GraphQL? And you can expose them selectively or because we're in God mode, I'll say expose all of them. And then we say, okay, these are the foreign keys that we've seen in your schema. Do you want to convert these into GraphQL relationships? And so we'd say yes, convert all of them. And then as soon as I do that, what I can start doing from my application is I can start querying it with... I can do an auto-complete and I can say albums. Every album has an ID and a title and then there's an artist for every album and every artist has a name. And I can make that query. So you're seeing an empty response there which is because the data import is happening at its own leisurely pace. And so as that data import happens, we'll start seeing some data here. But that's the idea. To motivate the idea behind GraphQL a little bit, I hope you're worried about security. There is permission and access control layer so that you're not actually querying your database raw. But let's assume that that's set up so that I don't have to show that to you. But the idea behind GraphQL essentially is that instead of having REST endpoints to give you an album where you fetch the album or to give you the artist of a particular album or having to iterate with your backend developer and say, hey, can you give me a precise API that fetches exactly this data? Instead of doing that, I as a mobile application developer I can make the exact query that I need. Let's say, for example, in this query now, I also need the ID attribute of every artist. And so all I need to do is add that column here, make that query, and there will be some data that comes up. So that's kind of the crux of what Hasura does and hopefully that gives you a sense of what GraphQL looks like as well. So GraphQL, if you look at it, is a structured language. It kind of looks like the shape that you base on that you want. And we don't have any data yet inside our... Let me see if we have any artists here. ID and... All right, so we have some artists imported. So if you look at what the GraphQL query is, the GraphQL query is essentially the shape of the data that you want. It's like a language that is essentially removing the values and just having the keys and then removing the quotes. That has a particular grammar. And so queries to fetch data are called queries. If you want to write data, that's called a mutation. So mutations are equivalents of like post requests, patch request, put request, delete request. All of that gets called a mutation. So you can use a mutation to insert an album or to update an album and stuff like that.  And so what I can actually do is maybe show that to you. I have a number for albums to see how many albums have imported so far. I have 126 albums and that number is gradually increasing because I have the import running live here. But what I can do is I can convert this query to a subscription and you'll see that this is changing real-time. So you're seeing that like you have that count that is changing real-time for you as the data is changing in the database. So that's the kind of API that we generate. Socket API that allows you to take like 5,000 clients or 10,000 clients and run that on one database. So that's the real-time component of GraphQL and the idea is that this is called a GraphQL subscription. It's like having a watch for your rest. Imagine if rest had get, post, put and also watch. It's kind of like a watch on rest. So that's what Hasura does and that's what GraphQL looks like. And now let's kind of get into how we built this. So if you were traditionally building your own GraphQL back-end, has anybody built a GraphQL server before? I know Harindra has. He's on the edge. So the way you built a GraphQL back-end is that instead of having the way you would build a traditional rest back-end is that you would have URLs and every URL would map to a function. Every path or route would map to a function. That function would do something and so every time you hit that URL with a particular method the function does something and returns some data to you. With GraphQL it's a little different. You have a GraphQL query. There's some component of your code that uses the GraphQL query and calls out functions. It traverses the GraphQL query instead of looking at a route and then calls the right set of functions and then all of those functions run and give you a response that you send back to the client. So let's say for example you had a really simple query that was fetching author and the id and name of an author. So when you send this query to an endpoint which is a web server, the web server has a controller. The controller starts traversing the query. There's a node called author and the node called author is mapped to a function which is called a resolver function which is the author function. The author function knows how to talk to the database and fetches data from the database. Simple. And so the author function decides okay, you've asked for id and name. Let me give you id and name. If you'd only asked for id, author will only fetch id from the database. If you'd ask for id, name and age it will fetch id, name and age from the database. Let's say you had author which would be a controller. The controller would then kind of call the author function and the article's function. The author function will independently fetch data from the database. The article's function will also independently fetch data from the database. You will merge that data together and then you will respond back to the client. Imagine that every article had tags. So you would now call three functions. Each of those three functions would call the database individually and then you'd return the response. You can see where I'm going with this and the idea is that this is a problem in ad hoc queries which are very large. You'll essentially be hitting the database several times even if you make just one single GraphQL query. This is also like the N plus one query problem. I remember the first time that I was writing a web server with Django.  I was like, what is happening? Everything was working fine and I have 10 users. That's because in the ORM when I was trying to fetch the article and the author for each article I had a 4A in author fetch articles. And that's iterating in the loop and that's hitting the database 10 times. So if I have N authors I'm hitting the database N times to fetch N articles. Now imagine if each of those N articles has N tags or each of those N articles has tags. So now you'll hit the database again N times for every article and N times for every author.  So that's kind of one problem that you have with GraphQL with a typical approach to writing GraphQL. So we take a very different approach. We say that if you look at this GraphQL query what we really wanted was that SQL query. We don't want to resolve we don't want to traverse this query we don't want to make that SQL query. And that's what we want as the final output. Obviously this restricts what you can do with your GraphQL API but the performance benefit that you get is slightly serious. So this is what we want to do and obviously what this looks like is a transpiler where you're saying hey let's take that GraphQL query and let's compile that to that single SQL query and let's add the right access control in the middle let's add the right type information in the middle and let's create a SQL query. So that's what Hasura is doing in the background. This is super useful because for every GraphQL query now you have just one SQL query so it's very easy to debug or to see if there's a performance problem. Thank you. But let's say for example you make this query. What I can do now is look at the actual SQL query that is being generated and then I can give this SQL query or I can look at the cost of this GraphQL query by DBA and the DBA can figure out what are the right indexes to add and stuff like that. The performance is also super, super high. On nested queries we can serve about 2,000 requests per second on a nested GraphQL query that is hitting a join in 50 megs of RAM.  This is insane for a lot of people. You can fetch about 1,500 records in just 4 milliseconds because it's just one great SQL query that Postgres knows how to optimize. This has been running for no good reason. GraphQL queries that fetch 400 MB of data joining several tables that have hundreds of millions of rows and that works within a 1-5% performance difference from if you just made a raw query to Postgres directly and not had an HTTP layer at all. So that is the advantage that you get with this approach. Let's talk a little bit about what that approach looks like and why Haskell was a good choice in hindsight to have done this. It would end up being useful. So what you're doing is you have the GraphQL query. We're going to parse that into a GraphQL syntax tree which is an internal representation of the GraphQL query. We'll then convert that to an internal data structure that we have so that we can add the right information to it. Like for example with author and ID and name we want to know that ID is an integer. Name is a string so that later on if you're adding any operators to name like you're saying I want author where name starts with a T. If you want to do that we want to validate that it is generated the right SQL. So for doing that we need the right type information for every column. So that internal AST is then converted into an internal AST that has extra information. Then we finally render that into a SQL AST we convert that to a SQL AST and then we render the SQL and we hit the database with it. There's a more optimization that we do after we've generated the SQL as well but that's okay. The first bit is obviously that you need to represent the GraphQL grammar in your language and Haskell makes this spectacularly beautiful to write. What you see on the left is a screenshot of the GraphQL spec. What you see on right is a screenshot of the code that we have which basically means that you can literally look at the GraphQL spec and map that one to one into an ADT that will represent your GraphQL AST internally. This is useful. This is trivial but this is useful because as the GraphQL spec itself undergoes changes it's very easy for us to kind of keep up the date. Of course you're representing everything in a type safe kind of language which is also a fun thing to do for those who have not done it before. What you now need to do is let's say you now know what the final structure you want to get to is right.  If you want to do this if you want to write the lexer and parser by hand A you shouldn't do it but you can kind of imagine the work that goes into it. You kind of read character by character see that there is a here's a word, here's a space, here's a curly bracket after this I expect a node name the node name should be author and that's fairly tedious work usually with most languages that you use like if you're reading JSON you need to parse JSON as well but you don't do that because boss road is on but you don't but we don't have that with GraphQL we did have that with GraphQL but we needed to work at this layer because there was a lot of other stuff that we needed to do at the GraphQL layer as well so we kind of ended up writing our own parser to do this and writing the parser is also spectacularly easy this is for example the code that it takes to figure out whether you're seeing a query a mutation or a subscription and you don't really need to grok the syntax but kind of just look at the at a high level you're dealing with the concept of what the GraphQL query is you're saying I want to parse the operation type because the operation can be a query or a mutation or a subscription and so you're saying I want to parse the operation type and so I expect this to be a token called query or a mutation or a subscription this is really the code that you need to write again helping you kind of keep iterating at this layer fairly easily and the rest of the kind of heavy lifting is taken care of with the combinator libraries and with the tokenizing libraries underneath so that kind of makes it easy again I'm not going into too much detail but it's just one example of where writing that parser yourself has a lot of benefits so this part is going to be little involved but and don't worry too much about the syntax but let's look at a particular part of the syntax tree and let's look at a possible transformation that we have to do right so I'm going to go back to the GraphQL interface and I'm going to make a query to fetch artists where the ID is greater than 4 and less than 2 is a bad query let me change that to 50 right and let's say that's the GraphQL query that I want to make ID and name right and so if that's the GraphQL query that I'm making I unfortunately don't have any data that has that sorry sorry oh thank you let me change that to less than 4 alright cool and now let me make that greater than 2 cool so I have just one which is 3 and so that's the data that I'm catching right and so if you look at let's just look at this part that we ultimately need to compile down to SQL right and so what we have is a Boolean expression here this is a part of our overall tree and we need to convert that part to SQL right and let's zoom in on that particular task so what we have is the JSON syntax there which is basically you have a column and the column will take a list of operations and values right and if you want to do that the natural way to kind of model that with the radity is to say I'm going to have a column expression type and the column expression type will have a name and a list of operations right and the Boolean expression that I give to the where is going to essentially be this column expression which is great if this is all it was but actually what we also want to do is we want to have a list of operations and or a not so if I do and would be a bunch of columns I want column one to have these conditions and column two to have these conditions and column three to have these conditions or an or or a not right like a typical Boolean expression and so what you end up doing is you have a tree and you say that I have a Boolean expression which could be an and and a list of expressions or an a list of expressions not of an expression and ultimately that node that has to be there has to be a column expression right so so internally when you read that portion of your GraphQL query this is what is going to internally look like right you're going to have a tree which is going to be and this is one example of what a query might look like you'll have and of three expressions two of them are column expressions one is not of an expression right so now what we need to do if you if you remember one part of the compilation step we needed to add extra information about every column right so I have this tree but in this column expression I need to now take the information that I have from postgres and I need to add that extra information in every column expression along with a column name saying that this is the type or this is the actual postgres column name not the column name that is being exposed right so we need to add that extra information so if you think about it this is the transformation that we need to do we have a column expression that just has name and operation and we now want to convert that into something that also has the information that is the rest of the expression object right so that green column expression is going into a yellowish annotated column expression right that's what I want to do as just one tiny little part of this whole process right and so what I really want to do at a higher level is that once I know how to do that I want to transform my entire tree instead of having green nodes on the bottom I want to have yellow nodes on the bottom right and the simplest way of thinking about this oh cool let me just have one AST for this for the red green tree which you can't see so I'm going to call it the left tree and let me have another ADT for the right tree right and so on there we have the left tree which is just a simple Boolean expression and on the right you have the annotated version of the tree right and so what you want to do is you will write this function called add annotation which will traverse the first tree you know go through everything and then convert that into the annotated version of the tree this is this is yuck work right because I don't want to I don't want to write code that traverses through the entire tree I know I just want to do a tiny thing I want the green to go to a yellow that's the code I want to deal with I don't want to deal with the code that is dealing with this whole tree finding out the right like nodes and converting those nodes into the final type and so what you do is you can parameterize parameterize the ADT that you have right and so instead of having two different trees I would represent that as one tree but where the where the leaf node is a parameter right the type is parameterized and then what I do is you can derive and there's a typo that should be deriving and then I'll derive a traversable on that which for free gives me a traverse function and the traverse function will do is it will kind of go through the entire tree for me right if I give it a function it will apply that function through the tree right and this is useful because now what I can do is the mental model that I have is I know how to go from green to yellow and I have a function that I write that takes a column expression to an annotated column expression and then what I do is to apply that through the tree I get this ability to traverse the tree for free right and this is just one like super tiny example of a super tiny transformation that we need to do right but overall the transformation that we have are fairly significant you need to do these one to one transformations you need to reduce your tree into a different kind of tree right because this Boolean expression is not the Boolean expression that you have in SQL right so all of those transformations all of those modifications all of those reductions kind of become easy because of this ability to manipulate your trees without worrying about the implementation of the actual transformation at node level right so that's the second thing that kind of makes Haskell super easy it makes it a good choice one of the other things also at this point that I should point out is that back when we started writing this four years ago we didn't really have there was no GraphQL there was GraphQL but we were looking at GraphQL and saying right so that's literally the sound that we made and we regret making that sound and so what we used to have is we used to have a JSON AST so what we had back in the day like two years ago is we had everything after the internal AST right and then and we had our own JSON language and then GraphQL came along and we had to add support for GraphQL and adding that support was actually really easy because we looked at the internal AST we made that internal AST a little more generic and then we just added the front end which was accepting a different language and reducing it back to the internal AST right so whatever work we'd done on the back of that pipeline kind of continued working for us and we just changed the front end and it works really well because now we're adding support for something called relay and that's going to follow the same idea right and that makes life convenient alright so the last portion that I'm going to touch on is the real time portion so if you look at GraphQL subscriptions that's kind of what's happening underneath and the idea behind this diagram is to show you that it's a pain but I'll kind of walk you through it so let's say you have a client or you have a bunch of clients who want to get back to data so everybody wants to see what the latest score is so you say subscription score whatever team name team name and some value right and you're subscribing to this so what does the web server do the web server for every single connection creates a connection object and has a delivery thread that will take whatever data comes into the queue and deliver it to on that particular web socket but at the same time by doing Postgres Listen Notify or listening to the change stream or reading from the database whatever you do you don't want every single application to start its own event sourcing thread in the database because then you'll have 5,000 clients and all 5,000 of them are hitting the database simultaneously to fetch the live score so when you know that you have you have thousands of people requesting for the same piece of data you want the event sourcing thread to be the same so you separate that out and you say I'm going to have a bunch of event sourcing threads independent portions of data that will then pass that data along to the relevant connection objects and then send that to the application GraphQL also allows you to have over one GraphQL subscription subscribed to multiple things and that makes the connection object a little harder to deal with so one connection object can accept data from many different event sources right when you're creating a connection object you need to make sure that the connection object is created and is validated against the GraphQL schema that you have so I'm querying or subscribing to a live score the live score model needs to exist if something happened and your GraphQL schema went through an update any connection object that is using the score object should get invalidated or if you're creating a new connection object live score should not exist so there's a lot of shared state in completely functionally independent portion of the code something that is managing your GraphQL schema that is doing event sourcing and that is managing your connections right they're all concurrent within themselves and now there's shared state where these kind of multiple systems need to kind of talk to each other so it's a pain in the ass pardon my French and sorry if anybody speaks French for real but concurrent programming is a pain it's a gigantic pain this GIF is very dark but it is painful it is very very painful it's an example of how you would try to solve this problem and why solving that problem becomes painful very quickly so let's say I have that event sourcing thread and I have the connection thread and what I want to do is I want to pop from the event queue and I want to push into the connection objects queue pop is a safe operation inside the event sourcing threads push is a safe operation inside the connection thread and I want to compose them I want to say hey pop and then push I want that operation together to also be to also happen atomically this is a slightly contrived example you don't actually need to do this in our case but let's just say you need to so I need to pop and I need to push pop is safe, push is safe but pop plus push should also be atomic so what you would do naively is you would say oh cool let me create like a lock for the thread event sourcing threads a lock for the connection threads acquire lock 1 and lock 2 and the push and then release the locks and you might think this is a great idea but this is not because somebody else is using lock 2 and lock 1 so the ordering of lock starts mattering let's say one portion of your code is doing lock 1 and lock 2 and doing something and the other portion of your code is doing lock 2 and lock 1 and then doing something and then you will deadlock very quickly and all of those nice users who loved you will now hate you what you then do is then you would have like a global sequence of your locks and then this will also be painful because you then need to find out what order your lock has right this is not fun what is fun is doing this I want to write code and I will write code by doing atomically pop and push right this is nice I don't have to deal with any of what I showed you before and this is what STM gives you and this is why STM makes it easy STM is the idea of having database transaction working on persistent data but doing that to in-memory objects across threads right that's it and that's all you need to do and so you can imagine that all of the shared state that we have here you know where the scheme object is there that getting updated the connection getting created the connection getting validated connection subscribing to a same thread if a connection needs an event source thread that does not exist creating that new event source thread or telling an existing event thread that hey I'm a new connection I also want your data all of that synchronization becomes super easy right so STM is a huge boost and allows us to handle a vast number of real-time connections fairly easily alright so this is kind of a quick overview of what our stack looks like we build we build with stack and with Docker we ship with Docker we run our tests on a CI CD on CircleCI libraries that we use are fairly standard for the web stuff there is one web software library we use that we do not use a Postgres library to connect to Postgres directly we have our own hand-rolled library that uses liptiq we do this because most of the existing libraries the idea was that you will define your SQL query statically but in our case the SQL query is dynamic right the GraphQL query that you make will create a new SQL query and so we kind of needed to do this ourselves especially to leverage like prepared statements and stuff from the database for events and MTL for for monotransformers we have our tests written in Python we have a few in Haskell but mostly in Python because we have a part of our team that validates whether the GraphQL query that you're generating is valid whether queries work as they should it's easy for them to write those tests if they're just written in Python and then we use HPC for code coverage and try to max that we do performance analysis with Profiter, ThreadScope and a really fun tool in Lua called REC WRK2 for benchmarking HPP performance and we're going to release a tool for benchmarking subscriptions and WebSocket performance as well because I think the one tool called Sung in Erlang is there but it's not super easy to use or customize so our big challenges that we're facing at the moment are in some cases we need to start fighting laziness and this is because there was a point of time where we wanted to see how much time each step in our compiler takes and measuring that is hard because if you look at the various stages in the compiler let's say for each stage I want to see how much time everything takes the problem is that because the evaluation of the whole thing is actually done when you render the SQL and not at each stage it's been a little painful for us to when you add instrumentation for timing it ends up telling you how much time it took to build the thunk not really evaluate the thunk so that's some work that we need to do kind of refactor the code to take care of that but that has a few associated problems with it and that problem is around and that problem is that for example we wanted to evaluate moving from MTL to FRIAR for composing effects better but the idea but one of the things is that it's apparently the compiler can optimize MTL code better but we don't know whether we need to bother about this performance or not and performance is very important for the use case that we solve for our users and so without having an accurate way to measure performance it's very hard for us to now decide whether to kind of move to more modern ways of writing code another problem is now around kind of deciding to move to generic lens and this is not really a problem but this is more of a generic lens is a new thing but evaluating whether or not we should switch to that what the cost of actually moving to that is going to be is a little hard for us because it requires a certain amount of maturity with all of these various tools to be able to say which tool we should use and if something is working should you really change it? No, we shouldn't change it so we're not changing it but at a quick glance the high level business benefits that we've had because of using Haskell over the last few years one is that as a really tiny team our feature velocity is extremely high and that's because of like the three simple examples that I showed you we're doing a bunch of hard things that you need to iterate on very frequently are made really easy with Haskell especially in our context of creating a compiler like thing and something that has a high amount of concurrency that becomes easy. The other thing has been that we've been able to reduce debt-debt fairly quickly and that's because you kind of start using REC and then you're like hey I don't like this HTTP client I need to do some fancier stuff I need to have a better API internally it turns out it's just a few hundred lines of code you can just write it yourself, roll your own library you don't have to stick with old code you can refactor faster and that has another added benefit which is that there's this thing where people say that it's hard to get people who are new to Haskell to work with Haskell code base but actually that's not been a problem because if somebody is learning Haskell even if their code is quote-unquote immature it doesn't matter because as long as it works it's fairly safe because the compiler will ensure that the code works, that the code is safe and what you can do is you can refactor that later when everybody is more mature and that works because refactoring is fast anyway and all of this kind of adds up together in a lot of community love for us there has to have seen a lot of love from the GraphQL community that has nothing to do with Haskell because we're building things we're fixing things and we're shipping them super super fast and that's created a lot of love for us in the community and that's been a huge that's been super important for us what I'm going to conclude this and open this up for questions but a few ways that you can get involved with us are one, we're hiring Haskellers and FB programmers please head to haskell.org slash careers remote roles and roles in Bangalore we're also, please do try Haskell out join our discord, give us feedback use it in a side project, tell us if you like it tell us what you don't like, be nice though and then of course Stars on GitHub makes us feel like we're in kindergarten at all, like stars of course those contributions are welcome especially if you're doing Haskell stuff, we're very beginner friendly we have a lot of issues tagged for beginners so if you're doing React JavaScript, Go, or Haskell head to the repo try to build some things and help us fix a few things questions and I'd like to invite Bamshi who's here who's the lead architect who's been dealing with this over the last few years if you guys have any questions for us we'd love to take your time we have a CLI in Go because the CLI needs to work on our platforms cross compiling CLI not Node.js and not Go will not work anywhere it is open source but it's very specific to how we connect to Postgres it's open source you can try it out, it's optimized for performance but again without specific use case got it so this is not really a, it's more of a Haskell question it's not really a GraphQL question because so whenever you think if something is possible with GraphQL or not the right way to ask that question is is this possible with REST if it is possible with REST it's possible with GraphQL because REST, GraphQL, SOAP are all at the same layer but with Haskell that's a good question to ask and that's something that I didn't show you but what we allow you to do is create this access control layer which is kind of like Postgres low level security that we've added at the application layer so what you can do is you can say organization, right? So every organization has a manager and this organization manager can select data if the org ID let's assume that I had a org ID column if the org ID is equal to the org ID that comes in from their SSO and so that's the access control rule that we allow you to specify and so now whenever somebody queries you know a table and they make a GraphQL query in our compilation step when we create a GraphQL AST what we'll do is we'll look at who is this user what is their token? Let's convert their token into the role so this guy is a user manager so let's take user manager, let's take this access control rule, let's inject that into the AST so instead of doing select star from table it will become select star from table where org ID equal to session org ID does that make sense? I mean so you specify permissions individually on each table, right? each table would have its own access control rule so if you do each table you get to specify relationships and you can use relationships in permissions so let's say you have a team I get your point, so you're basically saying I'm allowed to access this if I belong to the team is that what you're saying? Absolutely you can, correct but the permissions on table are coming from table A implicitly that's what you're saying, right? so what you can do is you can create a permission here for example I can create a permission saying that for this particular album album.artist.name you get what I mean? the access control rule that you create can itself traverse your foreign key constraints but that's a Hasura specific thing it's not a GraphQL thing if you were building your own GraphQL API and you didn't build it as a compiler you would write your own business logic that would do whatever you wanted it to do in our case because we're implementing it as a compiler we give you these rules that you can add and these rules can traverse relationships so JWT token so what we do is when you make this query to Hasura what you do is your token will come in from R0 so you'll say ignore the spelling bearer and some token here so this token will come to Hasura you'll configure Hasura with environment variables that has the Jot signing key or the Jot public key Hasura will take this token will try to extract user ID and role information from that so we don't do authentication we do authorize it that authentication information comes in from R0 your own R system, password whatever you want to do that makes sense you had another question I was implementing one of the implementation from your end and one other thing which I wanted to know was how do you do multi-tenant kind of stuff like let's assume that I'm working on a source platform where each same table is multiple tennis we can take this offline it's a Hasura specific thing we can take this offline we can handle those intermediate and that's great but it becomes a problem when we need to find out how much time it takes let's say you're instrumenting an query and you want to know where the compiler is spending most of the time you want to know how long it took to parse validate how long it took to convert to SQL and then how long it took to actually execute it but now if you measure these things you just see that the final execution phase takes the largest chunk but it's not true because all of that is being lazily evaluated in the end so we are the fastest grab in the world, yes but do we know where it's faster, where it's slow, no you have shown the connectivity from GraphQL to PostgreSQL so do you have connectivity to other databases eventually we'll get the money from the sky to follow us and then we'll have all the databases okay but one of the things that helps out is if you think of it as a compiler it's like the SQL target that we render that language has to change so it's like instead of rendering the Postgres dialect of SQL we now have to render mySQL dialect of SQL which is the most god-awful SQL dialect or Oracle SQL dialect which is better but things like that or Google Spanner SQL dialect or whatever so that's what we need to do why don't you take some time so how are you handling the subscription piece in a sense because I saw those being inserted to tables and you know so I mean are you using the Postgres notification or things like that so so it's kind of a what we do is we end up using something like that but we also debounce so it's kind of like a glorified polling so what we end up doing is like if you use something like Postgres Listen Notify just by itself right the problem is that what if a million events happen will you send a million events to a mobile client no right suppose you're subscribing to something that's super active you don't want every single change on a mobile client you might want every single change for an event sourcing like if you're syncing that data to a different database you might want every single event but on a mobile client you want the latest score every single score change that has happened right and so we debounce to a second and so it ends up being like a polling thing so what we do is we take like thousands of subscription connections we convert that to one query on the database and then we watch for changes on that query if something changes on that query for any of those relevant clients then we send data down but within a maximum interval of one second which you can configure so you keep polling the query to the database correct correct correct and so we we we did start off by using a listen notify we were actually starting off with a um by reading the postgres write a head log but that that becomes impractical very very fast like it becomes useless very quickly it sounds fancy but you can't use it for example postgres does have you heard of postgres's materialize views so there's a reason postgres materialize views are not auto refresh right because it's complicated it's almost hard an impossible problem to solve so the way we expose subscriptions is that you can query across views and views connected to other views so it kind of becomes hard for us to just use the write a head log and compute what has changed underlying so this works out like across all possible use cases of our subscription system but if you're a write a head log kind of guy um we have a really nice library which streams postgres write log changes on web sockets uh it's a fun library it's useless but it's fun if you feel like I want to do write a head log stuff it won't work for real world use cases but it's nice and do you support stitching? yes that's that's how you add like custom business logic so if you see remote schemas here um I can just add I can just add a GraphQL schema like pokemon I don't know what the pokemon url is pokemon.now.sh and then what will happen is that your GraphQL query it's not a valid url but the GraphQL schema that you get will be merged the Hasura GraphQL schema and your GraphQL schema will merge together because obviously you can't do everything with the database right like payment APIs you can't handle that or a very custom transaction you can't handle that so those are your code however you write thank you I'm afraid that's all the time we had maybe Tanmay and Babshi can answer this later alright thank you thank you folks for having us


Lavrov, lack of progress between Azerbaijan and Armenia, caused by Yerevan's position. The lack of progress in normalizing relations between Azerbaijan and Armenia is caused by the position of Yerevan, which receives advice from the West, Russian Foreign Minister Sergey Lavrov said at the traditional annual press conference. I have, in general, every reason to say that the West doesn't want to allow the agreement that were concluded between us, Azerbaijan, Russia and Armenia, in subsequent decisions Lavrov noted. I don't know who is advising Yerevan, but we saw how the European Union, France, Germany and the United States, as soon as they realized that the Russian-Armenian-Azerbaijani process was producing results in unblocking routes, deliminating the border and preparing peace treaty immediately began to infiltrate these processes without invitation, the Russian Foreign Minister said. Lavrov told that Russia has never initiated the cooling of relations with Armenia. We remember that while many of the Armenian officials were still in the opposition during various political processes and election campaigns, they called to leave the CSTO and the Eurasian Economic Union. When the Prime Minister of Armenia Nikol Pashinyan came to power, there was not the slightest impulse in our relations to move away from Yerevan in this way. Everything has developed as it was during the time of the predecessor. The Russian Foreign Minister stressed.