To get started, this talk is about theft of service attacks and subscription service vulnerabilities. This talk is going to be a little bit different about than other talks, which normally focus on system exploits. These exploits that I'm talking about today are going to be more of an annoyance kind of an attack. And also, I'm going to, the attacks try not to cross the line of actual legal fraud. Some of them do, but most of them try not to. Most of them just use the holes that are in the applications that are hosted on websites. A lot of them are developed custom in-house, and they don't go through a lot of testing. And there's a lot of simple things that just seem to work with a lot of different websites that are out there. A little bit about myself, my name is Rob Sheehy. I'm with Zentec Consulting. An update of all the slides is available at zentec.com, defcon11, slash TOS. Now, what are theft of service attacks? They're an application level attack. The attacker gains increased access to some limited resource. Usually, it's something that you would pay for. It's an opportunistic attack. It's really hard to kind of scan for sites that are vulnerable to these types of things. It might be just something you almost stumble across and just find out a site is vulnerable to. And the attacks typically do not result in administrator access. It just gets you a little bit of elevated privileges or bypasses some restriction, but you don't actually own the system. Now, some example targets for theft of service attacks are, would be software registrations and downloads, adult web hosting sites, web hosting accounts, proxy and anonymity services, dial-up internet services, email and use net services, shell accounts, financial news services, and even domain name registrations. All different kinds of companies are vulnerable to the attacks I'm gonna be talking about. Some of more examples are hacker conventions and hotels that host hacker conventions. And so we made fake badges. And you can actually get the image for it at FTP to zentech.com slash pub. Everyone who sees this should have a legitimate badge so I don't think there's any harm in letting you guys know about it. We went, spent about 20 bucks at Kinko's in about an hour and got that done. This was actually, we did this last year too and it was actually harder last year because if you try and scan in pressed aluminum, it's hard to print something out that's shiny. So it didn't work, but we were able to do it with the speaker badges because they were green and we were able to walk around with those on top of our real badges and no one ever questioned us. So we also hacked and got a free registration for DEF CON from someone who didn't have to pay. He avoided the common route. He did not go through user registration. He social engineered the speaker registration. He had no notice, nobody knew. They just thought it was me for the talk and he just went in and said, hey I'm here for the theft of service attack talk and they just gave him a badge and no one ever even asked me if he had anything to do with it. Actually I walked into the speaker area with the fake badge on and when I said I was going up to get a speaker badge a goon said, you're going up to get a speaker badge? Yeah, let me see the badge you've got on and I showed it to him and he said it's a fake badge because I'm in the theft of service talk. And he said, go ahead. So and besides hacking the convention you can also hack the hotel that hosts the convention. If you have a hotel here you can actually get cheaper internet access if your room actually has internet. And when you go out the way it works is you send out any internet request through Port 80 it gets redirected to this IP address and the user is presented with two options where you can get either a public or private IP address and you pay either 10 bucks or 11 bucks. Well if you go and view the source of that form all it has is a couple of options and value equals one for the first one and value equals four for the other. And so I started playing around with the values to just hey let's see what happens if I submit other values and I only got a little bit through it because as it starts to charge you I mean you can't really go through and try and find the other ones and know if it really works but this is what I found out two and three can be homework for you guys. But if you actually go through option six you only pay 295 the access is slower but it's an option that's available but they just don't present it to you. So all you have to do is actually view the source, change the option of whatever you're going to select and then save the HTML file on your desktop, make sure all your browsers are closed and then just open up that file that you saved with the edited option and just say I wanna connect and if it's option six it'll do 295. I have a sneaking suspicion three is gonna be a private IP address and might just be 195. And the speed is just fine if you wanna check your email and not have to pay 10 bucks every day. Now who are the attackers that perform these types of attacks? It could be a technically savvy customer of some service, a competitor might try and do something to mess up somebody else's system. Email spammers will try and get free accounts so that they can spread out their email. Anyone looking for a better deal say you're searching around for services and someone just happens to be vulnerable to something to where you pay the normal price but get something better than what everyone else does, you're probably gonna pay for that service. A legitimate customer's friend, this would be for account sharing type of attacks, script could use worldwide and of course my friend Bobcat, they're the attackers too. What's stolen? This is kind of similar to what kind of services, web hosting, DNS registration fees and of course software downloads. Anything that's not physical, the security seems to be a lot more lax. Now subscription based systems seem to be especially vulnerable. Because they're left up unattended, nobody's really watching them, they set them up so that it just runs, people sign up, they do what they're gonna do and they just get their money at the end of the day. Usually they're set up kind of quick, they want to get it up and running because the subscription services, the company's only form of income so they want to get it up and running as quickly as possible so they may not spend the attention to security detail as they really should. A lot of these attacks are really, really simple, I mean script kiddie-ish but it's amazing how many sites out there are vulnerable to them. Okay and some of the problems maybe that the service can't fix the problem because they're using a third-party shopping cart or the application was developed by a consultant who's no longer there and all the code they wrote is uncommented and no one can make heads or tails out of it. So they just have to live with it. Now here's a list of common security holes used for theft of service attacks, I'm gonna go through all of these. Instant account creation vulnerabilities, there's a lot of those. If you see like instant access, instant activation, chances are likely you can instantly nail them and get whatever it is for free. Subscription data in HTML forms, another big mistake a lot of places make. Authentication data that's stored in cookies, PayPal payments, then there's also application server or operating system specific vulnerabilities. Those could be used for theft of service attacks but that's for some other talk. And there's also business process exploits where you just deal, you take advantage of how they do business, how their customer service works. Sometimes you may even know someone on the inside. There's also copy protection circumvention, getting software for free or getting a serial number to register something using someone else's serial number. And abuses of a legitimate account. So you have an account you've really paid for and you just do something that you know or think and assume that the owners of the system would not want you to do but you paid your money, you have your access, it's legitimate and you're able to do some fun stuff because of it. There's also ways of bypassing the billing systems and you can also sometimes do user defined changes to the subscription terms. Now sometimes the way some of these texts will be obscured is by putting orders in, signing up for things during the holiday weekend. If there is any human review, they might miss it because they just came back from a long weekend, they're not happy to be at work. They just want to get all these things done. They're not gonna want to spend a lot of time on it and that you can use that to your advantage. Another thing that'll happen is people will use a valid price for something else. So say there's a basic subscription that's five bucks and a premium that's 20, well they signed up for the premium but it's at the basic price. If someone's just looking in their accounting records they'll see the basic price and it just looks, it blends in with all the other valid transactions. They really have to look at it closely to see it and thus the attack is obscured. Now for copy protection circumvention, that's really easy. You can go to those websites listed above, there's search engines for serial numbers and cracks. You can also go on to UseNet and things are posted regularly and there's a few of the more popular groups to get things. I'm not gonna linger on that. Gonna move on to abuses of a legitimate account. Now when you sign up for a new account you'd be amazed how many places will let you choose your own login and password and you can choose login and password as your password. That makes it real easy to share your account too because you can say, oh hey, I've got this account on the system, it could be a valid account. You're really paying for it and the login is login and the password is password, go have fun. I don't care. And of course then it also allows for anonymous sharing because anyone who comes into a system they just type in login and password if they wanna try it and it'll work. I've signed up for accounts myself where my login is login and my password is password and I've actually found other systems where oh, login was already taken and they chose password. Another way to abuse a legitimate account again is having multiple people share it although if the provider chooses it's really easy to detect. If they try, I mean there's gonna be multiple connections from multiple IP addresses. The more people that are using an account the more easy it's gonna be to tell. Account sharing is less likely to occur though if the system exposes like your credit card data address, things like that. If I see a system exposes my address who I am and whatever I'm really not gonna want login and password as my login and password because I really don't want that information that open. But if the account doesn't expose anything to me if the account is simply a you have access or don't have access, fine. Login and password will work just fine. And also you can share it with anybody else it doesn't matter. Then there's also bypassing the billing system. You can use cookie poisoning to alter your cookie if they're using cookies to identify the user and you can actually assume the identity of a subscriber. You can take a look at what cookies you have and edit them with a program called cookie editor that's pretty good. And you can edit cookies in memory using a program called WinHack. So of course this is all under Windows. Lynx has some support for that but I'm not going to go into that too much. I've got a little bit about Lynx later on. Another way to bypass the billing system is taking advantage of free trial accounts that are available. They're open to usually repeated abuse because you sign up with a credit card that give you some type of limited access and then you can just cancel before it expires. If you have another credit card you can sign up again for another trial account. It's real easy to cancel your credit card and just get a new one. The banks are really happy to do it and usually the two or three dollar fee of doing that is cheaper than the monthly subscription to an account. Though also try to limit it with email addresses so that they want a unique email address. Well getting new email addresses is not hard. Using credit cards and email addresses to have unique users and qualify a new user as unique and never been here is not very good. It's just too easy to get new credit card numbers and new email addresses. Anything free can be abused. If they're giving you free access that's another good indicator that you can do something with it. Then there's also application specific attacks. This is something you would find more on a system that you actually use.  Maybe do some kind of obscure things when you're in the sign up process and bypass the billing or alter the billing so you get a better price. The front end when you sign up for an account initially that might be very well tested and may actually be secure but then you can cancel your account or upgrade your account and the scripts that do the upgrade and so on and cancellation may not be as well tested and there may be some vulnerabilities in there that they don't know about with some of the methods I'm gonna talk a little later on. There's subscription specific attacks. These types of things work where you are subscribing for some amount of term. Again, I talked about signing up with a premium account at a basic price, a yearly account at a monthly price and a combination of the two can be really bad. You get a bit premium account for one month at five bucks but it's really a year account with a premium account that they might be charging 200 bucks for. Spammers would love that. Porn sites would love that. They can throw something up. I mean, it's something people really could make money off of if they really decided to. Again, there's attacking the resubscription process. Another thing I found is sometimes the user data in their database will get corrupted in certain sites and some of them have actually user verification pages and the user verification pages actually recreate the account for you, sometimes just using something like a transaction ID or some kind of account ID that you used when you signed up. One example I can think of, although I'm not going to name names, is a site where they had a daily download limit and they had a verification page. The verification page linked a user ID and password to a valid subscription. While they had a daily download limit, you could get around that by going to the verification page and just choosing a brand new username and password when you verified your subscription ID and that new ID had a brand new download limit. Then there's also HTML form alteration attacks. This is similar, actually this is exactly what you would do for the Alexis Park to get cheaper internet access. For get posted forms, it's really easy because all the parameters are in the URL. You can see it right there.  It's kind of hard not to see these things when they do it. The post forms though, you actually have to view the source and take a look at things. Now they'll have parameters that are input HTML tags that are of type hidden and the quote character is optional and down at the bottom is an example of something that would look like. In type input type hidden, it'll have a name and then a value. Nothing changes, nothing stops you from since you have the source changing those values, saving them, opening the form in the browser and then submitting the page. The one thing that you might run into is for the forms there's an action equals and that's where the form data is gonna be submitted to and that's some form, some script on their site. Sometimes the address, they don't have the full URL. They only have a local path and if you're loading it off your hard drive it's gonna try and submit it to your hard drive so you would also have to edit the action to include the URL if it's not there. But that's very trivial. Sometimes the way that they'll validate that the form is legitimate is they will use the HTTP referer to verify that this form was submitted and retrieved from whatever the domain was actmeink.com whatever. So the form is submitted, the script that gets it just checks the referer to make sure that the referer is an accepted domain, one of the domains that they own. This is very easy to defeat. It's very trivial. Depending upon how the setup is very, you don't even need to do very much. Sometimes you don't even need to do the much editing to defeat the HTTP referer. And also they'll usually will not automatically disqualify a order because the HTTP referer is wrong. Usually what happens is it flags whatever order it is for more human review because just because the referer is wrong does not automatically mean the order is fraudulent and companies are not gonna wanna throw away valid orders, legitimate things. They're gonna wanna take a look at it. Now here's a couple methods for faking HTTP referer. You would place the edited HTML form onto a web server. This could be a server you have out on the internet, some hosts somewhere, some free website. Something you have control of is, a full control of is actually the easiest. If you have your own local server, that's fine. Apache is free. You can use Microsoft web's personal web server.  You change your host file so that the domain name, www.acmeink.com, goes to that website or if it's your local host, it goes to <IP_ADDRESS> so that when you actually type into your browser, that web address, it goes and hits that other page. You load that URL, then you remove the host file entry and you either wait for the DNS cache to expire or you can flush it manually because that entry is no longer in there when you're submitting the form to the same site. It's going, although the site now has a new IP address, that's not reflected in HTTP refer. It only has the domain name and it thinks the domain name is valid so that's very easy to do. The hardest part is flushing the DNS cache which is just a command line option. The location of the host file is different depending upon what platform you're using. I'm not sure what is for Macintosh or OSX. I would assume OSX is ETC hosts but in Unix, it's ETC hosts, Windows 95, 98, whatever, it's in the Windows directory and in the Windows NT series, 2000 XP, it's in Windows system 32 drivers, ETC. And of course Windows would be replaced with whatever your system directory is.  This would be for local host and it would be fakerefer.com. So if you typed in fakerefer.com, your browser is going to send a web request to your local machine. The second method of faking is similar to the first one but it uses the proxy support. You edit the host files same as you do in method one and then you load the altered page but then instead of flushing the DNS cache you can just go enter in information for a proxy, maybe even a secure proxy and then submit the form. When you submit the form with the proxy, the DNS resolution is actually done by the proxy. You say you want ecneink.com, the form is going to be submitted to them, that information is sent to the proxy and it resolves it and actually sends it. You actually don't have to remove the entry from the host file because that's no longer being used when you turn the proxy on. Now the other way of doing that is sometimes they'll use a third party cart for their order processing though and that actually makes it easier because you can edit the host file to reflect that your local machine is that site. You then can go submit the form and it's going to a third party site. The fact that the entry for the original merchant is still in the host file doesn't matter because you're going somewhere else. If that third party shopping cart is checking HTTP refer to make sure that the customer ID in domain actually match and are a valid pair, it will look like a valid pair. They won't be able to tell the difference and it just, again, you don't have to remove the entry from the host file to complete the attack. Now there's also another method of tacking third party shopping carts. You can actually place an order for an inexpensive item. You then go and start checkout and go and pay for that item. It's a valid transaction. You give them all your information and so on and then while you're in that window filling out all your information, that cart information is still on the merchant server while you're on the credit card processing vendor server filling out the information. You can actually open up another window, go back to the vendor's site and start adding items to your cart. The bill processing system is not going to know that the total price has changed. It's going to go through and give you the verification and just charge your credit card that small amount and when the transfer was successful because you used a valid credit card, everything's legit, it sends an ID, a message back to the merchant saying, oh, by the way, cart so-and-so was paid for and that's all it says. The fact that there's new items in the cart, the system doesn't know. So it thinks you paid for a lot more than you actually did. A lot of times it'll even say cart ID is paid for, the system will assume that the valid amount was paid. Doesn't do any checks to see how much was actually paid. It just trusts that, oh yeah, I know the system, it told me it was paid. It was paid. Even if they have checked the authenticity of the message, they sign it with a public key, that's going to be valid because the message really is valid. There's nothing fraudulent about the actual transaction. You really are giving them money. They're just going to send you more stuff or give you more things because of it. And doing a combination of these attacks can really result in some really neat access. Again, changing a subscription period from monthly to yearly or even beyond and then changing the price of it. The price is a premium account but it looks like it's a basic price. Again, the transactions look normal just by casual inspection. Most people aren't going to be looking for these types of things. If they were looking for these types of things and knew these holes were out there, they'd probably fix them so that you couldn't do it. And these types of things, again, it comes down to obscuring the attack. If the attack, you just put the order in for a web hosting for a penny, that's going to stick out as unusual even just on a casual glance over. So using the same prices that they have for other service helps obscure the attack. Now, sometimes you want to submit forms and if you want to do that with the get, again, it's all in the URL so you can change the variables there. But if you want to submit it using a command line, you can use links and echo to do this. Links and echo are both ported to Windows and Unix so you could run this anywhere. You just, there's the format for all the variables, the form variables that you would pass. And again, we're going to be passing an email, a username, password, password, confirm, and then a submit. You put that in there, pipe that to links with the post data parameter and the URL that it's supposed to go to and that site will process it. So the example I was talking about before with the download limit problem, you could write a script that would take advantage of that vulnerability by changing your user account every so often, say via a cron job and then just using this command line to submit the data every time you wanted it changed. Now, there's a site that used to be around called IT Knowledge. They were a bookstore. They had an online library of books. They offered a 14-day trial and they only wanted to give you, all you needed was a credit card to sign up for the free 14-day trial. Some were to Safari, or Riley Service Safari. They had a really unsuccessful attempt to stop offline archiving via cookies. WGet wouldn't work because you had to log in, it gave you a cookie. It was a little difficult to get WGet to do that, but that was actually easily defeated with another program I'm gonna talk about called Offline Explorer, which is included on the DEF CON CD for this presentation. The subscribers to the service received complete access to the library. So during the 14-day trial, you could just archive everything. You have your own copy of it. Now, I signed up for one of those 14-day trials and thought that was pretty neat, but I didn't actually keep the account. Then, a while later, I got another email saying that, hey, we want you to have another 14-day trial.  And they sent an email offering another sign-up for another trial, and the way they set it up was really bad. There was no credit card required for the second sign-up. And the second 14-day trial never expired. I don't think that was their intent. And IT Knowledge went out of business, and I can probably see why. They were giving away everything for free. IT Knowledge was owned by EarthWeb, who I believe is still around. Users had access to the entire library for free, and normal access costs 295. They sent me the email message for the second free account. I said, sure, I'll take it. And it just never expired. It was good until they went out of business. They never asked me for money. I never gave them any money. And this is IT Knowledge now. Their important notice to their customers that they discontinued service. Now, what can we learn from this? That resubscription and customer retention programs are also vulnerable, except after service attacks. They created their own vulnerability. They were opening up the doors to everybody. Accounts should be regularly audited to detect irregularities. If they had looked for accounts that no one was paying for, or even tried to track down the accounts that people had signed up for using this new 14-day trial, they would have found this, but they never bothered. And that's probably part of the reason why they went out of business, especially since they sent all this information to the customers who most likely had signed up before and might actually be willing to pay that expense or have their employers pay that expense. They eliminated the need for them to go to their employers and ask for them to pay it, or even for them to pay it themselves, because they just gave out free access. Now, O'Reilly kind of has the same problem, and I'm gonna tell you how to do a theft of service attack against O'Reilly's Safari Bookshelf. They try to enforce a no-offline archiving policy, but they do a lot better job than IT knowledge did. I actually had to try some things to figure out how to actually take an offline copy of their books. Now, the reason why I actually wanted to do this is because I give O'Reilly a lot of money. I have a bookshelf full of O'Reilly books, but it's sometimes useful to have a digital copy that you can just search through digitally, even if it's just to locate which of several books might be the best source for the so-and-so information. And I've used that just to pick out which book I wanna bring with me when I go to visit a client's site, because it seems like I might need it. Now, they do, they try and detect if you're trying to download the books that you're viewing. And after three strikes, they lock out your account. They limit you to one login session at a time. If you try and log in, even if you have two browsers on the system and you open up like you have Netscape and you use Explorer and open up another Explorer window, when you log in a second time, you'll get in there pretty much on both sessions saying that they both expire and you'll have to then log in again. They restrict the access on the book library instead of giving you access to everything. They only give you access to however many books you pay for. In the free trial, it's 10 books or 10 slots. If you do, get your account locked out. This is the message you get. Your Safari account has been locked as a result of excessive activity. And they say if you wanna have your account unlocked, just send them an email and they'll do it. When I got this message, I actually asked them to do it. They did unlock the account, but they did it just the, and this was, I had a valid account that I was paying for. This was not on the 14 day trial or anything. But my account got locked and I asked them to unlock it. I had just been billed for it not too long before and it took them about three weeks to respond. They unlocked the account, but just so that they could bill me the next day, it seemed. And I didn't think that was very nice. Now, when you sign up for their service and you pick a book, because you only have a certain number that you can view, you have to keep the book on the bookshelf for 30 days. And that is actually a limitation because the day you sign up, you may not know which books you wanna view. You can, so it actually makes sense to just have the account for a month and then cancel it and then at the end of the month, sign up for a new account and it has a brand new set of brand new slots that are all open and available for use immediately. If you're reusing a credit card, you won't get the free trial, but again, there's all the bookshelves are open. And if you do use a new card with a new number, even if it's on the same credit card account, but just you got a new card, you get a new 14 day trial. So using that, you can do free trial account abuse, get a new email address because they use the email address as your login, sign up for a 14 day trial, pick the books you want. You've got 14 days, although for archiving, if you wanna archive 10 books, it's a slow process. And then just make an offline copy of the books you want and then cancel the credit card. You'll have a new number. You'll be able to sign up for a new credit card account and you have an offline copy of the books that you already have hard copies of, of course. So offline Explorer is created by Metaproducts. You can get it from metaproducts.com and it is also included on the CD, I included version 2.8.1220. Demo version is usable. I tried doing this with WGIT, didn't have that much success. So I kind of gave up because I knew offline Explorer is more suited for this kind of task. It is a Windows only program, so you need to use Windows for this. When I set it up to archive the book, I set up just using one connection. It can go up to I think 10 connections going out grabbing different files real fast. If you really wanna archive a website, it will do a good job really fast. But I set it up just one connection, 45 second delay between retrieves because O'Reilly doesn't like you getting a lot of stuff real fast. That's gonna set off the alarms real quick. And if you want to filter it so that you only get a single book, you can actually use the ISBN number as a filename filter because all the parts of the book are going to have the ISBN number in it. So if you just wanna get a single book in the bookshelf, you just add that ISBN number and you'll just get that book when you go through the download. And you also wanna add the images filter so that you get all the graphics and the book covers and so on. Now you have it scheduled to run every hour and a half, two hours, whatever, and it runs for nine minutes. There's options in offline Explorer that lets you do that. So it's scheduled to run. There's an option that say stop downloading if you received so many numbers of files or if you've run so many minutes or if it's grown to so much size. You use the option for how many minutes, set it to nine, and then you schedule the job to run every two hours. So it'll run for nine minutes really slow grabbing stuff, stop, and then start up again.  If you're doing a whole bookshelf this way, it's gonna take a while, but it's not gonna set off the alarms. And if it does set off the alarms and it does lock you out, there's a way to clear out only the bad pages and take advantage of that. You also wanna make sure that you set your identifier so that you're using a valid browser. If you tell O'Reilly that you're sputtering and trying to grab it, they instantly lock you out. It'll definitely lock you out. That URL is there is actually the front page of the bookshelf. Now you can choose how many levels deep you're gonna go to that you wanna grab. So if you only wanna grab a little bit, you can just go, if you just grab level zero, it's just gonna be that bookshelf page of the list of all the books. And if you go to level one, you're gonna get the table of contents for all the books. And then if you go to, and at a level limit one, you're probably gonna get about maybe 20 to 30 pages, depending upon what books you have selected on a 10 bookshelf limit. And at a level limit of two, you get all the pages. And then at a level limit of three, you get all the index pages. And when you get to a level limit of three, that's when you really start to have to download a lot and you would really need to have this program run for a while to successfully get everything at a level three run. But you could start it out and just do say a level one run to get everything initially. And then from there on maybe add a filter to just get one book at a time, whatever you specifically want. And then do a level two of that one to fill in the holes. Okay, I got a little conflicting information here. Stop down in seven minutes. Another thing, do not download existing files. That way it starts up again when it reschedules and will parse the HTML files. It's already downloaded to grab the new links that it needs to add to its queue. And you can also put in your login and password. There's a configuration option where you can put that in into offline explorer. You have to log in manually first before you start it though. There is a window, a browser window in offline explorer so you can hit the O'Reilly site and just log in.  And then from then on, it's only gonna be grabbing preview pages instead of the full pages of the book. Here's some excluded keywords and included keywords I used. These are just part of the URL. It helps filter things out so you don't grab a lot of excess stuff that you don't need. And if you want to delete the pages that have errors, you just simply search for all the files that have this is only a preview or session disabled and delete them. You then start up offline explorer again. It's going to parse the HTML files and if you're logged in correctly, it's then going to get those pages and get the full version and it's not gonna have to re-download the successfully downloaded pages. O'Reilly could actually get around this pretty easily just by putting those strings in a comment in every single page. I have a feeling they'll probably do that but there's still other ways to get rid of the preview pages. You just use a bigger search string to get around that. When I did this, I downloaded, one time I downloaded 600 files and just under three hours and I was like maybe two hours and 56 minutes and I got a warning message. So you do have to go pretty slow and you don't want to get a lot of these error messages. It seems to be that after you start getting these messages, the threshold for the next one is lower. So you really don't even want to get that first one. Otherwise it just makes offline archiving that more difficult. Now, basically this is because the idea of digital rights management is really unenforceable. I mean, they can't let you see it and not let you keep a copy of it. You can do it automated and that's pretty much what we're doing. Digital rights management conflicts with ease of use too because as you're using this system, if you try and log in through two browsers or you're at your home machine and work machine, things start to get logged out and it can create a lot of problems for you. So when they're putting in all these limits of what you can do, it actually makes the ease of use harder. Now, another theft of service attack, this is kind of lame, but eBay seller avoidance, basically after you sell an item, you have to pay eBay a percentage. You can fill out a non-paying bidder form and it seems to be an automated process where it just sends the bidder a message saying, oh, was this really a non-paying thing? Did you guys agree not to go through with the deal? And if the user doesn't respond, then it's all good. The seller can get the user to agree to it or the buyer to agree to it and not turn them in just by maybe splitting the cost of it. I don't know if anyone does this, but this is again, another theft of service type attack. Now PayPal was a lot of fun. A lot of places have started taking PayPal payments and a lot of them seem to do it wrong in how they implement their implementation of PayPal. Now for software, there's a return value after you fill in the form and that return page sometimes has what you actually are going to be paying for. Here is a actual URL that adds an item to your shopping cart and this is actually for some VPN software and in blue you can see that the return URL goes to chileweld.com slash success.html. And this is what that page looks like and it just says, oh, thank you for your purchase. I mean, it's the URL that they showed you and you can just download the software straight from there. They have absolutely no security guarding their software and there's a lot of software companies that do things like this.  You can place a legitimate order for the software but alter some terms of it. It's a legitimate order.  There's nothing fraudulent. I don't think legally fraudulent about it but the system acknowledges payment but doesn't verify that you paid the correct amount for whatever you're ordering. And again, here there's the amounts. Now here they have the amount actually encoded with URL encoding where they're using the percentage characters to represent the actual values. They're trying to obscure what you're doing. Now I changed this myself. The other one, oh, wrong way. Now here it has the actual price which is 90 bucks and shipping is 12 bucks but some places will obscure it, the amount, so that people don't know what to edit that they can actually do that by just doing that encoding which is really no security at all. It's very recognizable. And there's a URL for a URL encoded character reference table. Again, the method's not infected. It just tries to hide the problem rather than fix it. Now another thing is PayPal will give you an option to get insurance for whatever you purchase and that just signing up paying for the insurance confuses the vendors, especially if it's a non-physical item. It just freaks them out, their system has not been usually tested for this type of thing and they just don't quite understand it and even if they're reviewing it by human it kinda seems legitimate. I know someone that actually got hit by this and they actually thought someone was signing up with the insurance or the PayPal guarantee and when they got a small payment they thought that was some kind of insurance money that they were getting from PayPal and they were just curious why they were never getting the full payment. They just had no idea until I explained it to them. Now here's what the HTML looks like in a form for a subscription payment and it actually shows how much it's gonna cost and then there's a setup fee, how long it's valid for and in this example it's a two month free setup valid for two months free and then after two months it's gonna bill you 295 and then you're good for a year and every year from that you get billed again 295. You can just mess with those values and you don't even have to do any HTTP referer stuff. You just submit that to PayPal and it goes with it. Now for preventing form alteration I kinda gotta go quickly because I'm running out of time. You can do CRC variable checksum so that you know that the data hasn't been altered with but that can be reverse engineered but that just kind of obscures it a little bit. Again, you can check the HTTP referer but that can be faked as demonstrated. The best way to do it is just not use form variables. They're just not the way to do it. It takes them extra programming to do it a different way and unfortunately doing it a different way doesn't guarantee it's gonna be any more secure. You really have to take security as a primary concern especially since the business lives on this. They're subscription content. Okay, credit card payment attacks, credit card fraud, price alteration, avoiding payment and subscription terms extension. Again, in the previous example here you could change the instead of being free for two months it's free for 20 months.  Okay, how to spot theft of service attacks, audit your orders, do not trust what your applications are telling you. They may be lying. Look for increases in system utilization. If all of a sudden there's a lot more usage but not that much more income something's probably going on. Make sure that whoever processes your orders there is human verification on everything. Regularly audit account activity. Do not overlook accounting and consistencies. If your system is automated, definitely don't trust it. I mean that's the easiest thing. When I see instant activation they're full of holes automatically. There's not many places that do instant activation securely. Attack response, you can't really do much because the attack isn't always fraudulent. Especially if you're using something like login and password is your password. They let you choose it and there's not much they can do. If they find it really all they're gonna do is ask you to change your login and password or maybe assign you something. And sometimes tracking down the user can be hard because it's easy to fake email addresses and law enforcement will not necessarily care. They may even think it's funny. It's your fault that you did this. Let me jump around. Not much you can do for recovering losses most likely write it off. Last thing I wanna get to is the Direct TV theft of service. Direct TV sent out a lot of letters suing people for who were supposedly pirating cable just because of orders that they placed and not all of them were necessarily doing it. The problem is really that the Direct TV service was not secure. Not that the customers were doing anything. And I just personally think it's wrong that Direct TV has legal protection to sue to recover losses when they just either didn't want to or couldn't make their system secure and now they're gonna sue the customers. If it's insecure, again, it's the provider's fault, not the customers. Suing your customers should not be a profit center. They're making money off this. People are sending in money because they're afraid and they don't wanna get sued so they're gonna pay the 3,500 that they're initially asking. And they were able to send out these letters without giving any proof to any government organization. They're just able to send out and threaten people. And the interesting thing about it is to do these hacks, the people had to actually be legitimate customers and have basic service. So they weren't getting money, they just weren't getting as much as they want. There's some more information there. Kevin Paulson wrote a pretty good article. It's available at the register and I believe it's security focused. And again, Legal Rights has some more information about this. For finding vulnerabilities, I included this on the CD, is a program called SelfSeq. It's your own web spider, it'll go out and crawl out and then you can do your own searches against your own database of websites and search if they're doing PayPal and have amount equals or price equals and search for something that you know is a common vulnerability. SelfSeq is pretty good. Again, just wrapping up, very simple attacks. These are more of an annoyance than anything else. Victims rarely get any sympathy. It's kind of like your own fault that you got hit with this.  And just because there was a theft of service attack does not mean that there was even loss of revenue for the company. They may have gotten more money because more people signed up and so because their service, because they were able to expand their access was better than any competitor. And I think that is about it. So, thank you very much. Since my time is up, I don't have time for questions and answers but I will be outside if anyone wants to ask me any questions. So, thanks a lot.


So, this talk is about Microsoft Visual Studio's journey to continuous delivery. So this talk is really not about what Visual Studio does and what kind of capabilities it has and what kind of awesome tooling it provides to help you be agile and really be good at your ALM life cycle. It's also not a remedy for all the problems. It's really a talk about the development team, which actually builds Microsoft Visual Studio and the team foundation server. It's about their experiences of how they went from a traditional waterfall style of development processes and how they're adopting the modern practices today to really provide a world class first class kind of engineering system. And build out great products. I personally have been part of this journey to some extent. I was part of the Windows Phone platform team. Then I built tools for Windows Phone application development tools. I spent some time in Redmond. I spent some time in India. And in my current role, I'm the GPM for the Visual Studio testing tools. So it's super important for me as a product owner to kind of understand how the internal team within Microsoft itself are actually progressing and adopting modern practices. So that we can really learn from there and help provide great tools to my customers. So that's my personal interest in the whole Microsoft Visual Studio's journey here. So how many of you know about what Visual Studio does and how it's doing? All right, pretty much everyone. Okay, great. For some of you who don't, the historical context for Visual Studio is that it's a very large product. It's that coupled with Team Foundation Server really helps you build applications for various kind of platforms. Be it Windows, phone, Azure, websites. It provides bulk of developer and testing capabilities, source control, build systems, name it. And it's like the full end to end tools for developers, testers, and stakeholders. So it's really a large and a very complex product. And since it's a large and complex product, it's also built by an organization which is fairly large. And traditionally what used to happen was we used to have two to three year product cycles, right? So we had Visual Studio 2005, then 2008, 2010, 12. And now we have 2013. So from two to three year cycles to, we've come down to one year cycle as part of shipping an RTM product. But of course we go through various kind of CTPs, betas, etc, etc in the middle and we'll talk about that. It's a large product, it's a large organization. But I would say the organization is pretty adaptive. At every year, every stage, we've been making a fairly disruptive changes to our processes, to our themes that we are operating on, what our focus areas are. And with any kind of change, there is always resistance. But I would classify it as a fairly adaptive organization, trying to cope up with the changes that are coming. The teams itself are kind of distributed across geographies. In Redmond, India, in Hyderabad, Bangalore, in China, Israel, and it's kind of fairly spread out. When you're working on a large product and you have multiple components of things that are getting developed across various teams, the common challenge is how do you get really a holistic picture of where you are really in terms of your development cycle? There's one team here that's building debugging for Windows phone apps. There's another team building profiling. There's someone who's building project system for the whole thing. But what does an end to end scenario kind of looks like and where, how do I really figure out what state am I really in? And with geographic distribution, there comes its own challenges with collaboration, efficiency, how do you make sure that the quality of the product is good because there are several dependencies. There's one team always breaking the other team. And there is always finger pointing going on. I was on track, but the other team kind of broke me and therefore I fell flat over the floor, right? So those kind of things happen. And with all this going on, the overall goal is still to kind of provide customer value to your customers. And how do you make sure that there is a continuous flow of that customer value? And in such kind of situations, there are a lot of excuses that people look for, right? One of the common things that back then, which I personally myself also as product owner did was, people ask me, where are you with your features? And I would say, yeah, we are doing fairly well. And then somebody would ask, hey, but you seem to have these many large number of bugs here. Well, on a relative scale, it seems fine, like that other team look at that. And that has like 400 bugs, whereas mine is only 100. So we are playing with who is more inefficient at that time, right? There's that kind of schedule chicken that's going on between the teams. And since there isn't any common criteria of what really done means or what really is the truth about the quality of the product, everyone is trying to play with their numbers. And that used to happen quite a bit in world in days. And just to give you kind of an idea of how big the Visual Studio product team is, this is data back from, I think, July 2012. Just gives you a flavor of its large number of developers, this large number of source files that are getting touched. And there is just terabytes of data that's getting generated, right? And pre-2005, right, our cycles used to be, our traditional planning cycle was there used to be an M0, where we would do the planning for the two or three year release. There would be milestones and you will develop code in it. Once milestone M3 is over, that's when you start with like mini milestones, 3.1, 3.2, because you really realize that you haven't hit anything right now and everything is kind of chaotic right now. And there is a lot of milestones, mini milestones that are getting created to really just clean up the mess that we have created there. And when we are actually doing beta, rather than using the beta to really get customer feedback on the experiences itself, you're really using beta for people to tell you what the bugs are, right? I mean, you're not really iterating on whether the product that you've built is right product, but it's about, okay, I found this bug versus I found that bug. So there's a lot of quality issues that we really used to discover during those beta timelines. And then there used to be a release candidate, you do the RTM, and any product that we ship has a 10 year kind of a servicing SLA that we have. Then you end up doing a lot of QFE's, hot fixes, or what we used to call service packs, where fine, I've shipped it. Now, there is a bunch of another 100 bugs, here's a service pack for that, and you install that. So there's a constant kind of, once you're done, you're kind of fixing all the issues kind of a phase. And I would think most of the companies, most of the organizations, most of the products have been used to it. They've gone through these cycles. But then when all this was happening and the industry was kind of changing, we really wanted to look at how do we adopt little more modern practices and set some engineering principles really behind some of these things. And when we started the journey of setting these engineering principles, we really didn't tell anyone to use agile practices or something else. It's just that there's some fundamental changes that we said need to be brought in. There was a realization that you can't really have buck tails and long buck tails to kind of fix product issues. So we wanted to move the quality upstream. There was a lot of emphasis that was put on developers to do unit testing and write testable code and write a lot of automated tests themselves actually and not really rely on the testers to kind of do the white box testing for them. So that was an effort to kind of drive up the quality upstream. We really brought the developers and testers together as part of feature crews so that testers know what is being developed, stakeholders know. There's more collaboration between the teams and there is a concept of feature crews that kind of evolved from there. And for the main branch, we said hey, there needs to be certain amount of sanctity to that main branch. If main branch has fallen over or it's not building or some basic things are not working there, you need to really feel bad about it. It's not that you can't fix it, but you really need to feel bad about why is such kind of a thing happening, right? So that kind of culture we created where main line is something that there is a customer ready tomorrow, I want to ship it to him. I can't ship it to that customer because you broke this, right? Because your change actually caused this. So there was a culture that was being created that it's not about shipping at the end of two years, your main line bill should be ready such that I can give it to the customer at any point, right? And to that extent, people created feature branches where they would actually do their feature development. Get it to a certain level of stability and then kind of add it to the main branches. We did try to define a common definition of what done really means. When you say, hey, this feature is done, kind of define those criteria for that, certain teams followed it, certain teams didn't follow it. And then kind of there was a lot more focus on automated testing. And this also came with a lot of resistance because automated testing means you need to invest upfront in certain infrastructure for automation. And there is that initial period where people don't see the value of it. So certain teams did a lot more level of automation, certain teams did less. But at least what everyone did was for whatever they were trying to build, they tried to automate that, right? I mean, there were issues related to what the legacy system is and how much of the legacy can be automated. But at least the new features that were getting built. There was a lot more focus on automated testing. And so we laid out these engineering principles and different teams adopted these engineering principles and different methodologies and different process methodologies, right? Some teams who were, for example, the teams who were building now, Agile was kind of gaining more momentum. And there were teams in team foundation server who were actually building these tools for external customers. They certainly adopted Agile practices and started doing Scrum and all. There were certain other teams who were more mindful of these things and kind of brought them into their processes, but did not really follow the practice to that extent. But largely they use these engineering principles to kind of start evolving the mindset of what a tester needs to do in this new kind of area. What is the responsibility of product owner, right? So there was a, with the model of feature crew coming into the picture, there was a notion of shared responsibilities that people started in buy-me-ing. So the road to Visual Studio 2013, really, if I were to kind of break it down into certain phases. A phased first wave was really about kind of addressing a technical debt, right? We just had lots of bugs in the system. We needed to get that down to a controllable state so that we can actually operate in a model which is more predictable. The second wave was once we actually did that, it was making sure that the learnings that we have from that first wave actually stay there because it's very easy for people to kind of go back. Yeah, sure, now the bugs have come down from 2000 to 200 or whatever. Let's go back to that previous mode where, yeah, fine, 50 more bugs is not gonna make much of a difference. But it's about maintaining that hygiene and discipline for another cycle where you kind of institutionalize what you kind of learned from your first wave. And that really resulted in kind of amplified flow of value to the end customers. And today our focus is really on kind of increasing the cycle time, right? You have an idea, you want to implement it, you want to get it to the customers as fast as possible. And coupled with that, now we are not just building an on-prem product. We have a service called Visual Studio Online Service. Or we have the load service, build service. There are several services that we are building in the cloud where we are doing continuous delivery on a very sprintly basis, like three-week cadence. So that's another set of, there's common teams who are building products now for on-prem as well as services. So kind of making sure that there is some kind of a uniform process for them to actually go deliver that. So if you look at our 2005 and 2008 debt, like this charts kind of give you a feel for what those first set of principles that were laid out really resulted in terms of what our 2005 bug backlogs look like, and what the 2008 things look like. It was almost like a 15x reduction in the debt. There was a lot of schedule improvement here. We could get out the bits to the customers faster. And in general, I think there was a huge amount of customer satisfaction raise that when we did our customer sat surveys. We found that there's a lot of improvement in that. There's a lot of improvement in performance, reliability, less number of crashes, and those kind of things. Earlier what used to happen was people would not focus on bugs. And everyone used to take a lot of pride in building new and new features, but kind of shipping them not very finished features, right? So that was definitely a marked improvement from 2005 to 2008. Since then, now as I said that our focus is really on kind of enabling continuous delivery, enabling reducing really the cycle time to get a product out. We're really following this modern application life cycle here, where during the planning phase, you have some hypothesis. You do a code and test during the construction phase, and then release it to the customers and then do a measure and incorporate that feedback. So we are following that typical build, measure, learn cycle here in this phase. And since then what we have done is for our team foundation service, which is a service oriented thing, which we actually ship every three weeks on the cloud. So every three weeks we are having delivery of that product that goes out. And then for the Visual Studio itself, which is a non-prem product, we have enabled an update cycle where every quarter we typically do an update. And then there are some CTPs that we go out that go out. So one of the fundamental kind of learnings of this particular thing was that if you really want to practice agile, if you really want to get continuous feedback, if you really want to do a continuous delivery, you need to make sure that your product really has a ship vehicle, right? If you're going to still ship the product at the end of one year, two years, you're really not enabling to get feedback from the customers. So what we did was for our on-prem product, for example, the whole process of actually installing Visual Studio and then installing updates on top of it, itself was not so smooth, right? So if you want to enable your customers to kind of give you feedback, you need to make sure that the process for the customers to give feedback is easy, right? You remove those hindrances in the way such that people can actually acquire your product easily, you have to kind of invest in a smooth acquisition. Easy for them to kind of provide feedback on top of it so that you can get it easily, and so you have to really invest a lot in your loop of getting customer feedback from the, so that's another thing that we kind of invested heavily on for us to be able to successfully use the agile practices, okay? Any questions so far? Does this kind of sound, yeah, there was no extra budget for that. We just kind of instilled that this is the focus, right? You don't necessarily have to develop new features, right? What you need to do is you need to use your existing bandwidth and actually get down, knock down those issues, right? So we didn't create new budgets for it. But there was a debt, and we were paying that debt anyways, right? Like if you saw the previous one, right? We finished the product, we were doing QFE's, and we were doing service packs. So there is large number of bugs that used to get fixed, and we used to carry on that debt. And frankly speaking, for a lot of that debt, we actually took hard calls and punted all those bugs. Like I don't want them in the system. Give it to the customers, if the customers come back, like Tester has this notion, yeah, this is a bug. We said it's P2, there's a lot of P2 and P3 bugs that's lying in the system. The customers may never hit it, right? And in that case, it's not necessary to fix them at all. So there was this notion of make hard calls if you don't think these bugs are required, because if it's sitting for three years and if you haven't fixed it, what's the point? That means your customers are not really finding those issues and they are not important, right? So that's the way we dealt with it. The team foundation service gets deployed every three weeks, right? So for the team foundation service, which is used for source control and build, right, and work and backlog management, etc. That is something that the teams who are building Visual Studio, they would consume it, you're talking about internal consumption, right? They're consuming it on a daily basis, right? So every three weeks, new bits are coming, we are consuming it, right? We are providing feedback on it. Our product that we are building may get released a couple of months later, but as development teams, we are consuming the bills, service and everything, and we are providing constant feedback on that, right? That's true, so the people who are using the service only, right? For them, they can give feedback, but the people who are using on-prem product, for them, it will go every quarter and we'll get the feedback only once or quarter from them. But remember, since we have deploying the same bits on service and as well as for on-prem, the fact that you have a customer base, which is actually consuming the service, to some extent, you have dog-fooded that quite a bit. So the chances of additional feedback coming, which is going to derail your plans from on-prem product are relatively less. And further on, I'll also show you that I think though our on-prem product, we are saying we ship it like every quarter, which is kind of go live release. But for that also, we are doing some customer technology previews, which are kind of aligned to these printly cycles. I don't know if I have a slide to that. So for metrics, what we did was we really define, I think I'll touch upon that. Maybe once I get to that slide, I'm going to talk to you about how we do our sprint reviews and what kind of metrics we use there, okay? So let's move forward. So once we started doing this, right, we started really with Scrum. So there is a lot of teams when we actually laid out that engineering principles adopted Scrum, right? And they are the ones who started then kind of training the other teams. Said, hey, you're trying to do something similar. Let's try and unify your processes if possible. Though we never said, we never mandated those processes. Because there's always a resistance when you actually mandate something. You really want to show them that, hey, every sprint I'm doing this, I'm getting this feedback. People started kind of imbibing on their own, right? And I think it became a viral effect. And now most of the teams, I think, I would say still not 100%. But 80, 90% of the team are actually following this. The things to kind of consider was what should be the sprint length. It should be two weeks, three weeks. What's the ramp up time that it requires to actually take these teams into kind of following these cycles? What should be the team organization when you're trying to do that? And one of the things for team organization that was very relevant was to kind of set them cross functional. But have a very shared responsibilities and shared goals, right? So there are three or four teams that are working together on something. At a top level, we want to make sure that the goals and the priorities for those teams are kind of aligned well. And that's how we enable them to kind of work together. Because if we said, hey, all the four teams agreed that these are the common priorities, right? Then everyone just works towards that goal, right? And if there is something else that they're working on, which could have dependency on some other team, so long as they know, right? That the other team's priorities are this and therefore it may not deliver this, right? Then it's easier for them to understand the delays from other teams and things like that. And dependency management used to become a little more easier. It's not straightforward, but still it became easier when you actually had like five teams talk to each other and say, hey, these are my priorities, these are my priorities. These are where we are aligned.  People understood, right? So that's how we kind of started with Scrum. This was very important to set the cadence of how often you're iterating on your plans. So often, people when they went through reading books of agile or they read some papers or went through some training materials, there's always chances of misinterpreting things. And they said, hey, since I'm working on agile practices, I can pretty much, don't ask me what's gonna happen a year later, right? I'm gonna learn and then kind of evolve. But when you're trying to build such a large product, when you have such complex organizational structure and things like that, you need to really have a common vision that you're working on, right? So we set a 12 to 18 month vision. And the vision was something that it would stick around for some time, there will be, you know, iterative changes in the vision as well, right, as you learn more. But it's like, you know, at least the destination that you want to reach. But the route that you may take for that, reaching that destination may differ. So we set out a 12 to 18 month vision there. We really had like, okay, over the next six months, we'll draw a line in the sand based on the velocity of the team and whatever we've learned that, you know. At the end of the six months, you know, if you can light up these end to end scenarios for our customers, then that'll be great. And then we used to have a little bit more kind of better understanding of experiences that we will deliver in, you know, three sprints. And every sprint is like a three week, you know, sprint there. So that's how we actually use even the kind of world breakdown to say, hey, there's end to end scenarios. There's a vision and there's scenarios, there's experiences. The user stories, and roughly try to align them to, you know, how often do you think you will need feedback on some of those things, right? Effectively, it's that, like, why are we trying to do these three weeks sprint execution? Because we thought that the kind of pace at which we will be able to deliver and get the customer feedback, that seems like the right, you know, level of thing there. If I had a mechanism where every day my product was getting used by a large number of customers and every check-in is something that I would need feedback on, probably I can evolve to that system, right? That system also has its own benefits. So that's how we kind of traded off these cadence, if you will, for stories, experiences, and scenarios. And this kind of gave people a fair bit of picture on, yeah, okay, I know what my, you know, what Brian had to use, my distinguished engineer of application life cycle tools wants me to do, you know, 12 to 18 months, and here are the reasons, here's the customer data that he has given me to actually go and build that vision.  So then I kind of build the scenarios that are required for testing tools to be able to do that. And then, you know, the team within will actually say, okay, now here are the experiences, and here are the user stories that help me get there. While we're doing that, there was really two metrics that we were using, right? One is the idea to a working software. You know that this is what you want to build, and here's the working software for that. And then, you know, once you have the working software, what's the mean time to repair? Once you put it in production, how fast can you actually fix issues if you found any issues there? Those were the two metrics that we were kind of, you know, wanting to build here. Now, in terms of how we kind of involved really the stakeholders and different teams, right? There was an envisioning that actually the leaders did and, you know, presented a vision. Then we did experience reviews, right? So every once in two months, three months, we did, you know, experience reviews with our different stakeholders, different teams. And that's where you would say, okay, yeah, you're building this experience. There is this team that's building a similar experience. Why are you not kind of incorporating that? And that used to be a good discussion to really see how the cross team, you know, things are kind of aligning towards building that uniform experience. At the end of every three weeks, we will do a feature team chat, right? We would actually say, hey, you know what, this is what we have built. Here's what we are trying to build in next two, three weeks, right? So that kind of give them an idea of what our overall backlog looks like. So this is mostly around how we kind of take care of, you know, there is independent teams who are building their features. We trust them, but how we are kind of involving the management and the stakeholder in the overall process. Because, you know, finally, that's also very important as to, you know, for them to see how this all is kind of aligning towards the broad vision. And then at the end of the sprint, we actually send a sprint mail. We do a recording of a, do a sprint video recording and actually share it with everyone. And that is like a, you know, a super way of actually getting, letting everyone know what, you know, what you did really and how does the product really look like. And people can then go play with it. Whether it be cross team, organization, we just, you know, broadcast it everywhere. And sometimes we actually send this sprint videos also, use them to kind of talk to our early adopters and customers and kind of, you know, take some feedback from them. All right, so this is about how we are kind of doing the planning part of it. And how we are kind of executing the things. In the cloud delivery, right, when we are actually shipping it, it was very important to see what is your shipping cycle. So like I taught, we used three week cycles. At the end of the third week, we'll have a feature team chat. The fourth week is when we are trying to use deployment, and you know, that's the cycle we use. Now this is where we are. Certainly we would love to be in a shape where at the end of week three, you've done enough that you can just deploy with confidence. We are not there today, right? We kind of end, use up some time to actually do the deployment. Now from one week, it's reducing to three weeks and we are kind of constantly improving what we can do there to really make sure that there are smoother deployments at the end of three weeks.  Yeah, so I think all the exploratory testing, etc, gets done in week three. The deployment is predominantly about taking it to pre-production environment. Running certain kind of tests that you might not have done. For example, today, we are not doing load testing in a continuous integration way, right? You're not doing that on a weekly basis. That's some of the activities that we are doing in that cycle.  We've found issues with feature flags, right? Because once the feature flags are turned on with five different feature flags coming through, we've seen things falling off the cliff. So those are the kind of things that we are doing in deployment. And those are the things all we think is something that can be optimized for sure, right? But this is where we are and kind of trying to improve on. So if you look at the overall cycle, I would say everyone today is kind of storyboarding the ideas. I think it's a uniform practice now, right? This is kind of institutionalized. You have an idea, go storyboard it, right? Validate with your customers. Once you've built the product, you've given it to the customers, enable stakeholders to give you feedback on that. And from a product backlog item, you can link that request and people are able to see it. So we've set certain processes in place where all this becomes kind of smoother and easier. You kind of capture the results in the backlog. If there is a feedback response, let's say it says, Brian Harry, he gave some feedback on certain things. You capture it in the backlog and do prioritization in the backlog. You use these iteration backlog on a task board and go through the cycle of actually going from one place to another. And then once you have your backlog, what are the coding and testing that you're doing? So you kind of, even from a developer perspective, right? We've kind of had developers doing kind of continuous coding and a lot of teams have started using TDD, right? And more teams are getting cost to do test driven development. So as they are, they're writing tests first and then coding. And that's a different paradigm shift at all, completely. To do certain things in that fashion. The people who are doing it, who are doing it successfully just love it, right? The people who are not doing it, they're just allergic to it, right? So there are two extremes of people that we are seeing. But overall again, from an engineering principle, what we're saying is you don't need to do TDD. What you need to do is you need to do upfront quality, right? And that's what we're kind of going to make you accountable for. The way you want to do it, you can do it this way. If they see that the other teams are producing features with a higher velocity, they'll get motivated to adopt some other practices, right? For testing, yeah, so now once you do your automated testing and all kinds of conflict testing, really we are focusing a lot on exploratory testing. So the testers are actually doing a lot more what we call customer experience validations. They're sitting in the shoes of customers now and actually trying to see if the customer were going to use it like this, right? Though the user story experience may not have been defined like that, but if he's going to use it like this, right? They're going to explore different areas and create issues and create more, generate more backlog items, right? And once these things are, performance under load is again, this is something that I said traditionally these used to be like an activity which was done like at the end of six months, right? Now with our service and other things, we are at least doing it at a sprintly cadence. And we want to see how some of these things can be done earlier. Because what we've seen is when we find performance issues, they're hard to debug, right? And even when you look at your bug counts, you can easily hide them behind your bug counts because but the fact is like 10 UX bugs will take probably two days for people to fix it, but one perf bug can take a week or two, right? And it can cause potential changes to your design. So this is something that we still want to explore as to how I can get some amount of performance testing. Can I do some, should I be doing some micro benchmark kind of testing more upfront in the cycle such that there is lesser chances of getting hit by these performance issues? So there are things that we are kind of exploring in that area. But we recognize that this is an area where we sometimes get into problems. This is there, yeah. Yeah, and once we get into the production, it's about life side, right? This, I'm gonna talk a little bit more about, we have a life side culture where everyone's kind of monitoring the life sides and what kind of processes that we put in place to actually make sure that we can recover from those life side issues faster. We can learn from the experiences of what other people are actually experiencing on the life side. This is something that we've been doing. I was completely new to it. So I think I at least see that we are really evolving as an org there and really learning from the experiences of each other. So life, yeah, so this is not for the on-prem product. It's more for the service that we're building. Yeah, let me load, go a little bit faster on this one. Yeah, and in the end what it says is at least for the service, we're doing all this, but what really matters is how the real users are using it, yeah. No, no, so there are certain class of issues which ops can fix, but I think when we are talking about user experience issues, they will get translated into, they really go back to generating a product backlog item for the team. And we are investing heavily in things like analytics and usage analytics and things like that where things can be more automated to achieve some of those things. But what it's really showing is that we have like dashboard culture. So there are these dashboards if you have, for example, my team owns the load service and we have dashboards all over our offices, right? And there are TV screens and monitors where you can actually see, right? What's going on in your site? Of course, there are ops people and there are DRIs who are actually constantly monitoring it and are responsible for it. But it's just kind of creating that awareness that you need to be seeing how your customers are kind of using it and being kind of constantly kind of watching it, right? So there are such kind of small tools that we have enabled in the system for you to kind of do those things. From an engineering system, I would say that roughly these are all the kind of areas of the engineering system, right? Where it's about making sure your deployment processes are automated. You're using tools for deployment. I'll talk a little bit more in detail about managing quality and code flow. Exposure control is important. You want to do AB testing, you want to do incremental deployments such that you're deploying it to a smaller set of people, such that you can incorporate feedback from there. Dependency management and engineering backlogs, something we talked about earlier. For managing quality, really there is a whole set of things that we are doing. So one is gated check-ins, right? Where the developers itself are building some unit tests and actually doing the check-ins. Then there are some larger integration tests that run as part of nightly builds, then you have the sprint sign-offs. And this is where we actually kind of do the customer experience validations. And really define the sprint criteria, what is a done criteria? So for example, today we say, hey, the done criteria for a user story is that there should be no P1 bugs, right? There should not be any customer experience kind of impacting issues. Of all the things that we have around performance, reliability, acquisition, etc., etc. We've defined certain criteria which should be met, right? And that's when you actually close out the user story or the experience, right? And then for the P2 bugs or some other bugs that are there in the system, ideally there should be zero, right? But what we've done is we've kind of defined some kind of criteria or number that there shouldn't be more than these many or they should be fixed within this should be the age of those bugs and those kind of criteria we've defined to kind of keep them under control. But sprint sign-off criteria is predominantly like, yeah, it is ready for the customers to actually go consume this build, is predominantly the sign-off criteria. And what we do is we have our leaders, our managers, our stakeholders. Actually, we set up a system for them and we say, hey, go play with this product, right? So that's what we call experience walkthroughs, right?  We'll give them a kind of a five minute overview of what the feature looks like and we'll ask them to go play with it, right? And they will provide feedback. So this is just a way to kind of make sure that your stakeholders, your management, everyone's involved. You are able to ship your products to the customers, get the feedback. And then you have tons of feedback, you can then go analyze it based on certain criteria and prioritize it. So we talked about the exposure control. The life-side focus also we kind of briefly covered. It's like our operations team and the developer team are sitting side by side, right, and we are just working with them together all the time. They come and participate in the life-side reviews. If there was a life-side incident, they will explain what was the mean time to detect it, what did the ops people do there, how was it given to the developers. So we've kind of established those practices and we kind of do those things on a weekly basis to kind of ensure that life-side issues are taken under control and we are constantly learning from there. For example, we've had issues where your life-side availability shows 100%, but there are large number of customers who are actually complaining about the service. What does that really mean? Are we really measuring our availability correctly, right? So we had certain way of looking at availability and we said, okay, the way we should look at availability is really how the customer experience really is there, right? If you have some kind of a failure today, are you actually looking at your system in entirety and seeing that you won't have that kind of system again? So there's a lot of scrutiny and a lot of discussions that we have around that. And then there's a weekly life-side and then there is a monthly service review that we do at again our VP level, who actually goes and looks at what were the issues. Okay, this is an issue that the Azure team needs to fix. Where is it on your backlog? So there is a whole culture of making sure that the life-side is up and running all the time. When we talked about box product, I think it's much harder for box product. Like some of us talked about earlier, right? Because you don't get constant feedback. But what we've done is, we've decided to align our CTPs, I say every sprint. I don't think it happens every sprint. But even though we are shipping at four months intervals, but we're kind of giving some early bits earlier to the customers there as well. And we had to go through a lot of NDA and licensing and this and that and lots of crap to actually deal with it was not easy to do something like that, right? There's a lot of disclosure issues, especially Windows and Windows phone, like our big platform. They don't want to disclose what's coming in the next release. Whenever you give the tools out, there is a chance of those things getting leaked there. So it's about maintaining some of those privacy, making sure that the features that you really want to protect for a year or so, those are hidden in such a way that people can't jailbreak and get to know all those things. So you need to put that kind of system in place to be able to do that. So it's much harder and that's why when we talk to our teams today, everyone wants to build a service because they think that, yeah, if I build a service, I can get the feedback early. I really am in control of my customers. I really know what's getting used. But of course, that's not the way to prioritize whether to do service or on-prem, but it's harder there. So I think I'm kind of towards the end of the talk. I think if I were to summarize what the key takeaways are. From a culture perspective, I think every change will meet resistance. It's about managing that change, believing in that, showing the value to the people and then kind of incrementally getting there, right? As you see, for example, we are not at a point where we can say every check-in directly goes to production. We would like to get there, right? But we are kind of moving slowly towards that. Like teams like Bing and all are doing that already, but probably not Visual Studio. You really want to automate your recurring processes, be it deployment, be it your testing, be it your dashboards. We really want to kind of invest when we say life-side issues. You can actually see those metrics automatically. Or for on-prem product, we used to have something called Squim, right? Where people have to enable some flag and that's when you get to know which feature people are using. You want to kind of make sure that some of those things are easier and you automate some of that, even the analysis of some of those things. Eliminate the friction of collaboration. To know when you're doing sprint, you want to do sprint emails. You want to do sprint reviews. You want to get a larger number of people into the room to actually dog food your product, that's one way to get rid of collaborations. Make sure that the management level, that the leads level, you have relatively shared goals and priorities, right? That's what you want to do. Make operations really part of the process. Ops is not a separate team. Make them part of the process. Think about what the operations will have to do up front. And really use the cloud cadence to rapidly respond to the customer feedback. I mean, that's critical, right? If you're giving bits out early, customers are providing you feedback. You also need to be empathetic to the needs of the customer and actually respond to the customers with the feedback in time. And something that I think we, I think do a fairly good job of striking balance of what I had planned earlier versus I got this customer feedback. And therefore I need to change this. And something that all the management leaders, everyone kind of pays attention to. Even though you might have said that you will build this experience in the coming sprint, you got a lot of feedback the way you've built it. Go change it, and the management is going to be completely fine with it. So the management needs to be supportive of the changes that come with a feedback driven approach. Yeah, I guess that's pretty much that I wanted to share in this talk. If you guys have any questions, I can take some questions now.


The results of this study suggest that global efforts to increase births at healthcare facilities may not be effective in reducing maternal and neonatal mortality unless quality of care is improved. This study found that the quality of basic maternal care functions were significantly lower in primary care facilities compared to secondary care facilities, even when adjusting for other factors such as volume of deliveries and presence of skilled staff. Additionally, facilities with caesarean section capacity, especially those with high birth volumes, had better quality of care. These findings highlight the need for increased investment in improving the quality of care in health facilities in order to reduce maternal and neonatal mortality. This article was authored by Dr. Marguerite Cruck, M.D., Hannah H. Leslie, Ph.D., Staphan Bergett, Ph.D., and others.


Welcome back to Tennis Talk, my name's Cam Williams, and we have the Madrid Open happening in a couple of days time. It's an ATP and WTA 1000 event. It's a combined event where both men and women play at the same time. And we've had some big names pull out of the tournament, so let's have a look at the withdrawal list. We'll start with the withdrawals from the men's side of things, and we've got Matteo Barattini. He's out until at least the French open, maybe beyond, so we don't know when he's gonna return. Daniel Mevedev's still out with that hernia, and Stan Vavrinca has also pulled out of Madrid this year. On the women's side, a much bigger list. We've got Sophia Cannon. She's pulled out, Kerber has also pulled out, and Neck Kondivade's pulled out, which is really strange because last week she played in Stuttgart and played pretty well. So interested to see that she's not playing in Madrid. Kredgikova's also pulled out of Madrid. We haven't seen her for a long, long time, and she is the French Open champion of last year, so we'll see if she can defend her title based on no matches. Osorio's out, Svetlina's out, and Vundrusova is also out of Madrid next week. So there you have it. Big tournament in Madrid next week. The fourth Masters 1000 tournament for both men and the women is happening, and we've got some big names pulling out now.  So the women's side of things, the draw comes out in about 12 hours or so. So we've got a lot of the women who have confirmed they're not playing. We could have some more withdrawals from the men in the next 24 hours or 48 hours, so let me know down in the comments below. Who are you gonna miss the most that is not playing in Madrid? I guess the other question is, are you excited because it looks like Rafa's gonna be back? He hasn't pulled out of the tournament yet. I don't know, a lot of people out there are waiting for Rafa's return. Looks like he is gonna play, but I'm sad, Medvedev and Baratini aren't gonna play. Baratini made the final here in Madrid last year, so that kind of sucks. And also Kredjikova, where's she gonna be when we get to the French Open? She's got a lot of points to defend in the next couple of months.