All right, good morning. I say good morning because I know by this point in DEF CON, we're better off using the same time zone as Tokyo, which makes this about 7 AM. So thank you for getting out of bed. The good news is we have an interesting topic ahead. We'll get to spend the next hour talking about boobies. Or more specifically, strategies designed to protect an execution environment for manipulation with a specific eye towards ensuring cryptographic keys can't be exfiltrated via physical access. Try three times fast. Hey, good morning. Oh, man. There we go. Yeah, I tried to change the slide over here. And Google was asking me for a question. Your presenters today are myself, Ladar Levison. And I'm Hanimbo. Hanimbo Tales. What we're going to be talking about are essential skills for life in the world of 1984, where the maids can't be trusted and your big brother is out to get you. For my part, I'm the operator of LavaBit and encrypted email service. I'm a technological warlock, a corporate kingpin, and some would say a team linchpin. My assistant, Hanimbo, is the proprietor of hacking and coffee. And he enjoys robbing banks under the cover of darkness. Assistant, come on, man. I'm your web host at this point. A little more than that. Let's talk about security. As an industry, I think we've come a long way. Speaking for myself, it's been a few years since I've seen anybody try to log into their box via telnet. One of the key factors pushing that security is encryption and its proliferation. Encryption is important because it provides a mathematical guarantee that the data can't be accessed without the corresponding key. Of course, the problem is that as we've evolved, so have the attacks. Unfortunately, as good as we are at architecting and building secure systems, we still need a reliable and friendly environment to run our bug-free, ultra-secure code. With encryption specifically, we've seen a new breed of attacks where all the security measures we put in place get simply via physical access. Now, there's a picture of an altair because that was probably the last computer you could look at and kind of understand how it worked on the inside. Before we go any further, we should talk about our assumptions. For our purposes, we're focused on targeted physical attacks, which means we're making the assumption there is no universal backdoor in x86 hardware. As always, when it comes to assumptions, your mileage may vary. Unfortunately, we don't have time to cover it today, but it makes sense that if your threat model requires you to install booby traps in your hardware, you probably want to do to your IME as well. For those of you who've spent some time in this building, what we're talking about will look familiar. Essentially, we're talking about taking commodity hardware and modifying it to afford you some of the security guarantees provided by FIPS Level 4. So we can run our common criteria EAL4 plus software in a secure environment. Unlike the actual FIPS Level 4, our recommendations don't come with certificates from the government though. If you aren't familiar with those standards, don't fret. It only means you haven't spent much time working for the man. For those folks, I'll summarize by saying FIPS Level 4 is what the State Department would use for a classified information terminal. They have located in the basement of the American Embassy in Taylor. Some place interesting like that, if a machine can stay secure in an environment like that, is that the same computer would be worthy of being placed in a co-located facility and be safe from legal, extra-legal, and illegal access, probably. There are plenty of commercial solutions available. And they'll work great if you don't mind paying too much for five-year-old technology. Personally, I have a problem trusting black box crypto modules, particularly when the vendor's largest customer happens to be the government that you may be trying to protect yourself from. There's some notable open projects, the most notable being Orwell as in 1984. And for those of you looking for a secure solution to place at home just to protect your Monero private key, that might be a good way to go. Unfortunately, running a service like LavaBit takes a few more MIPS than a sixth-gen mobile processor can give you. Unfortunately, even if that was enough, unfortunately, their rack solution leaves something to be desired. So I decided to go with commodity hardware. And while I have my concerns about Dell, it's easy to find warehouses that let you pay cash and carry away the equipment. It's always good to purchase your gear anonymously. As a tip, if you have trouble finding a local vendor, try searching eBay for sellers selling what you want in your local area.  More than happy to sell it to you. Whoa, am I? If you're a high-value target, I'd certainly recommend against ordering equipment online and having it shipped. You never know what can happen to it in route. In case you didn't catch the subtext in this note, Apple even has problems with their servers being intercepted and modified during shipping. If the largest company in the world can't do it, what chance do we have? Interesting factoid. If you look at the SEC filings for Intel, you'll find out the three of their biggest customers for custom x86 chips are the NSA, Google, and Facebook.  If there are any FOIA fans in the audience, they may want to go after those details. Be sure to pick a commodity hardware vendor with a reasonably robust and competent security team. So you could probably meet a big portion of their security team here, and you can make your own judgments about whether or not they're competent. You also want to look for one that's going to use signed firmware updates and will continue to publish firmware updates for some time after you purchase the hardware. Unfortunately, when it comes to commodity hardware, most servers do ship with tamper detection circuits, but they tend to be rudimentary and thus easy to bypass. It doesn't help that their location is known in advance. That's one of the benefits to the do-it-yourself approach. You can randomize the location, even add a reverse switch. To demonstrate just how easy it is to bypass these tamper detection circuits, Honimbo is going to demonstrate on this R710. So we need some volunteers in the audience who can see the front LED and tell us if he manages to trip the circuit. The trick in this case is to take the shim and go through the regulatory label hole in the top of the case, find the tamper circuit, depress it. They said it would be fast. You may have had a malfunction earlier. You know that always helps. Who's got something? Hey, you try getting up in front of 1,000 people at 7 in the morning and doing this. Once you have the tamper circuit depressed, you can lift the top cover and remove the case, remove the lid, giving you access to all sorts of different ways of attacking the system. Most notably, being those exposed JTAG ports. Moving on, if you think about security as an absolute where something is either secure or it isn't, then you're going about it all wrong. I like to think about security in the same way physical safe manufacturers do. The first and most important job of a safe is to defend against spoofed access. If you can break into the safe without leading a trail, like by using this particular dialing spot that was demonstrated last year, the safe has failed. So our first mission in modifying our servers was to ensure that nobody could access them without us knowing. The second consideration is the degree of difficulty that an attacker would need in order to brute force their way past the locks. Presumably, with a physical safe, it's a blowtorch. With our approach, it's the same thing. And the goal is that the amount of force and heat required to remove our modifications are sufficient to destroy the machine, thus accomplishing our goal. To continue, I'm going to turn the mic over to Honimbo who's going to talk about how we made those modifications. So the first thing to know about all of these Dell systems is there's a very large debug headers, expansion modules, and available on these boards. Some of these are for legitimate things for businesses. You've got your iDRAC, you've got various OpenManage, you've got these that sometimes have additional heat. It's whatever hardware you end up with and you got to do a little bit of engineering in. In our case, we took the little bit messy approach for some of this. Some people may recognize this as a PC7 epoxy that we went through a lot of the board. So anything that can't be physically removed from the board, we basically are unable to remove it, so that they're not going to be able to practically do it while the sensors are still in place. So all over the other things that we also do the connectors. So these whole sides of the chassis, all for a very easy spots to pick probes through. And with those probes, there are various things like the front header of the USB bus on the underneath the front bezel. And part of this is not just the debug interfaces, but the ones that you're actually using to conduct operation you have. Because there's only so much that you can actually turn off and still have a working system. PCI risers, we just went ahead and go for that. And just wear some gloves to take skin off. I know this the hard way. One thing some people may notice, I have some very cruddy solder joints on here. There's a reason for this is if someone were to start physical damage, we would have the server fail off. So you may notice that we have a few different types of contacts here. For one, we have just a sample of a simple lever switch or the roller next to the original Dell chassis sensor. So we have a few different types. Again, solder joints, and this is really interesting. So we take advantage of the actual disassembly for these servers. In this case, assembly module inside of one of these Dell R710s. The fan module comes out as a big tray and we can pull out any of the fans. Oh, luckily for us, Dell has lots of alerts when it comes to fan failure. What we started doing is in the modules, we added a lot of our circuitry and sensors for some of the units. And part of this is that in order to get to the initial components, not only do you have to try and bypass our sensors, but at the same time, you've got the existing diagnostic sensors that weren't even tamper intrusion working for you. So all of these extra fans, the speed sensors, there are tons of diagnostic information that lets you know something's wrong with your environment. It's inside of a co-location facility, any decent quality data center, you know what the environment is and you take that data whether or not the airflow is starting to lessen that something's in the room with you, whether or not the temperature is really going up in a way that's not related to hardware. And it comes down to a lot of monitoring. Yeah, I'd just like to add that bearing the switch in the assembly here, you'll notice the lip, which protects an attacker against an attacker accessing it unless they go directly, which is because they can't get the shim past this little lip here. It's important to take advantage of the natural character of your particular server when deciding where to place it. So we take advantage of that from the top, but also we actually will modify the frame as well. One thing to drill to the top and risk that power supply are causing some kind of faults, which is nice enough to do this with various types of equipment from a drill on the server laser on the government side. But going through the actual motherboard on the bottom, you're getting into the SATA buses that run underneath the particular area that we chose and you're getting into a lot of other interfaces where if you started drilling through it's gonna be very, very difficult to get to those sensors. Take advantage of everything from your traces to your layout, just how tiny you can get these little microcontrollers in place. So we use the internal headers for connecting up our equipment rather than running it back out, obviously. And we just slathered everything on there and you may notice we explicitly did not use pin headers. Again, just very, very cruddy solder joints in this epoxy as a lot of people like my challenge here know is that it's very easy to make a mistake and cause something to slip. And we want to take advantage of that where a minor slip that normally wouldn't trigger will cause damage. As again, failing off is exactly what we want. And when you get to the code that we have available right now, it's currently in its older state. It's just a simple Arduino script that's currently up. We're uploading our new version of this talk. And we've since added tilt sensors. We've added other types of contact sensors besides shown. One of the ones that we're actually working on right now is a challenge response NFC tag inside of some of the components. And with that, you'll actually be able to authenticate not just a little magnetic read sensor, but the particular tag inside of the chassis being used. We're doing great on time, so keep talking. So when we get into these challenge response systems in the box village just next door, you will notice a lot of people just bring magnets and bypass the sensors directly. One of the unique things that you can get away with when you get into the DIY approach is you can have sensors that aren't documented outside of your own design. So we actually can have unique tags per contact point inside of this chassis. On top of the more traditional methods because in a certain environment, you may be able to get the NFC signal out reliably. For practical purposes, these servers might as well be Faraday cages, as well as they're well-grounded. If someone were to try and tamper with the power supply to try to damage that, you have another Dell sensor that has a potential to kick in. And with this environment, you also have some other interesting approaches you can take. So we actually also control the server cabinets as well. And with control of the server cabinets, you know whether or not someone's supposed to be doing maintenance or whether or not someone's opening your doors to try and get access to the fiber line that's running into it. And it actually, this is where you get to a little bit of the balance act that you have to do is that it is very easy to shoot yourself in the foot with these sensors.  Yes, test your switches before you install them. The hard drives that were originally going to come here succumb to that. The other thing to test is not just whether or not your own, the individual server sensors work, but whether or not what you're sensing for the environment as a whole is not going to just come back to bite you. So overall, any operation, any home lab, any rack in a data center, it's gonna need maintenance. So if I was servicing this box right now, I could easily just shut off the sensors from the servicing purpose, lock the data, do whatever is necessary. So what about the server above and below it? Well, contact sensors won't be an issue, but let's say you start adding in things like various motion, tilt, and one of the more interesting ones to play with is light sensors. You have to maintain good airflow through your server still. So such sensors become very effective at detecting anything in your environment that's changed, but at the same time, you have to make sure the servers are still functional and what you can actually block off. The tilt sensors are actually one of the ones that we found the most problematic. Those server racks in particular, if you're not fortunate enough to have the ones anchored to the ground, will vibrate quite a bit when unracking the server beneath it and promptly lock or erase everything. Yeah, I mean, I can take over. Yeah, the great thing about those ball bearing tilt sensors is that if somebody tries to pull the server out on its rails, the tilt sensor will get switched from the movement, the inertia. So try drilling a hole in the top of a case when you've got two other computers on top of it that are equally protected. Like I said, it's about taking advantage of your environment. I wanted to talk about briefly how you go about building these particular modifications. It's a pretty simple process, takes a little bit of practice, but it involves purchasing switches like this along with some ribbon cables. You solder it to the two pins, mount it in your chassis in various parts, and then wire everything back to the port on this little Arduino, which we protected with electrical tape to keep it from grounding. And then, of course, epoxy did in. You then connect that to your internal USB port and you can use the Python script, the rudimentary version of it, which is in this repo, to monitor that Arduino and detect when any of your switches or detection modules get tripped and then choose to take the appropriate action. Presumably, if you have a fully encrypted system, it would be to shut down. Once you get all your modifications in place, one of the things we like to do, oh, I didn't bring it, is use thread locker to ensure you can't even unscrew it. We didn't have the screw in this particular lid, but most of these servers have at least one optional screw that you can use. Another great tip is to cover your joints with something like varnish or my personal favorite, nail polish. The more glitter the better. Sprinkle that on, take a photo of the glitter pattern, and if anybody tries to remove the case, they won't be able to replicate the exact dispersion of the glitter. Poor man's holographic seals. Now, I'd like to talk a little bit about operational considerations when you're working in a high-thread environment and you have one of your sensors actually get tripped. One of the recommendations that I would make is to take advantage of the fact that Lux supports multiple key slots. I don't have a photo of it here, but you can see that it supports up to eight different passwords. All you need to do is write a little script that when somebody enters the password in one of those high-numbered key slots, it wipes the slot. That way, if you need to tell somebody the boot password, they can type it in, and if that password were intercepted, they wouldn't be able to use it again. We didn't talk about it earlier, but one of the reasons that we used PC7 epoxy is because it has the right thermal and electrical properties.  We also chose PC7 because it's incredibly difficult to remove, as he mentioned. Don't get it on your hands. There are also other advantages or things that you can take advantage of if you're not using enterprise-grade equipment. One of the things you wanna look for, for example, is ECC registered memory. The reason being, it's incredibly difficult to remove ECC memory and still read the information off the chip after the system is shut down because ECC memory, by default, cycles itself every time it powers on. You also wanna add things like this project, USB kill, which will detect if anybody inserts a USB drive into any of your ports and can take appropriate action in that situation as well. Anything else we should cover? Let me pull up my notes. I feel like I'm forgetting something important. So actually, that's a really good question. So there's a couple of options. Repeat the question. So he asked, how are we killing our hard drives? So the first method is simple. If you have LUX enabled, you're going to zero out your memory and shut down. The second is a little more aggressive and the reason that I had to drive out here. The hard drives that we use for some of the equipment actually has a two ounce thermite charge embedded inside of it, next to the platters. Yeah. I thought we weren't gonna mention that. Well, good luck dealing with it. Yeah, don't, nobody tell our colo. It's okay, just a little smoke. It won't throw much fire. So the big trick to using a thermite drive, however, is choosing the right composition. The most common alloy used is FE-304 based, I'm sorry, FE-203. And FE-304 is the variant that you want to use. It has a higher instantaneous heat output. And depending on how aggressive you really wanna get, there's various other types of compounds that can be added in with it. Using a hobby rocketry igniter with a potassium permainate mixture gives a reliable ignition at a low voltage that can be delivered by the computer's power supplies. In terms of actually building the drives, the big difficulty is doing it in a clean environment. So with the hard drives, they're so dense now that a single speck of dust can kill a lot of sectors. So either A, if you can finagle your local university to let you use your clean room for a little bit in exchange for some beer, which is what I did. Or you can instead do the improvised clean room method in which you actually raise the humidity very high inside of an enclosed space. But the problem is that when you do this, you have to have some very good method of drying out the drives air after the fact. I take some fresh desiccant and I make sure it's in every drive when I do this. So when the air has a high moisture content, the dust will actually sit to the ground and you have a safe working environment to modify your drives. Don't assume a single drive will have good sectors afterwards. They won't, but over a large array of drives like a sand, you'll end up with enough reliability that you can safely do it. Yeah, one of the things to consider is the fact that the old fashioned spinning magnetic disks are easy to destroy. All you need is to breach the case. And like he said, get a little dust in there and it'll be impossible to read the data. The problem with SSDs is they're a little bit more robust. And that's where the thermite patch or something of that nature might be required to physically destroy.  In lieu of that, if you're using something like Opal to do your encryption, you can do an instant key erasure on the flash ROM and then nobody will ever be able to recover that data. The downside to Opal is that it's a black box module. It's implemented by the drive manufacturer. You're never gonna get a chance at looking any code for it. There's a free PBA available from the Drive Trust Alliance and the self-encrypting drive utilities. However, that just talks to the drive. The actual encryption is still done on the drive itself which if you choose to trust Samsung or Micron or one of the other manufacturers, it's a very fast and high performance encryption system that actually erases it instantaneously. The downside is it's not testing a black box again. Did we talk about the NFC switches? I guess we have time for questions. I wasn't expecting that. Any questions? How do we test it? So, I will tell you right now, do not test in prod. So, what we actually do is I'll have usually a couple friends of mine that are various involvements and levels of tamper evidence or bypassing and I will not tell them what I did to a particular system and I will let them have at it. Beer goes a long way to bribe hackers, pay your testers. Yeah, we intended to actually bring a system out here for the tamper evident village but travel disruptions got in the way. Yeah, but basically in terms of testing you have to think about the entirety of your state machine, what possible attacks that you're considering, what your switchers are designed to do and what exactly you're working on in the event of an intrusion. So, you may choose to say I have different ranks of intrusion that I'm going to switch to just simply raising an alert or actually shutting down or even erasing the data. So, for example, the internal contact sensors, when we test those, we'll have people using various types of shims, various types of cutting tools and various types of probes to try and bypass them. We actually use a randomized mixture per deployment on whether or not it's an openly contacted switch or normally closed switched. So, that way you actually don't know and except per device for the person who built it whether or not that switch is going to open or close if someone tried to short or bypass it. So, when testing, we'll load up some data and the goal is for whoever's looking at it can they get this data off whether they do it through a cold boot attack via USB, whether they do it through actually pulling out a drive and trying to keep power to it if you're using an OPAL method or whether or not they can simply get, if they can get the chassis off without triggering it that's considered a failed unit. You can just get the lid off regardless of whether or not they do any memory-based attacks yet. Yeah, one of the photos that we showed, Chromebooks having trouble keeping up with the, there it is, the 700 megabyte PDF is we actually epoxied the power cable from the PSU to the motherboard to ensure that somebody couldn't swap it out and remove the board that way while maintaining power to the system. And this is a photo that we took after just doing the connector. We actually slathered parts of the cables as well that are exposed. One of the attacks that actually is done for forensic purposes is they will actually slice the sheath of a power cable and apply power from a secondary source to try and keep the system running. Hot jacking. Yeah, hot jacking. And with that as a concern, we took kind of precautions in terms of epoxying over the existing power so that basically any attack that will remove the sheathing with it and cause a short. So we will accept the hard fail and data corruption as a path in that environment. Be sure to epoxy all of the places that an attacker could plug into and get direct memory access. As you can see here, we covered up the ports on the riser card so that nobody could insert anything into them and add something to the system. I didn't show it, but you want to look for the BIOS chip and we don't have a picture of it here, but the pins on the side of some of these integrated ports, you also want to cover those up so that nobody can add a lead to it and directly manipulate the integrated circuit. You also want to cover up the battery. So nobody can kill power to your BIOS, particularly if you're using hardware level protection methods like Opal. So most of the modern UEFI BIOS is for Opal. Well, actually, if it detects a warm reboot, it will send the lock commands to the drives. However, if you don't have the ability for the BIOS to actually process and update to the current system status that enables a possible hot jacking attack, this is again specific to Opal when you have the encryption on the drive itself rather than using a method such as lux where you're offloading it to memory. Also, be sure you don't overlook your integrated lights out management board. One of the problems we found with Dell equipment in particular is that it's possible to disable IPv4 access to the DRAC controller. What we ended up doing was, what, setting the IP address to... A slash 32. Yep. Make your subnet all 255s so that it can only talk to itself. And then you can sit there and wonder what it's saying. A non-existent broadcast domain is something that, in some of the Dell firmwares, you can actually set. So take that to your advantage since they don't allow you to actually turn the features off. You also want to cover up some of these unused ports in the back. You can swap out these default covers which have convenient little holes in them with ones that don't... You don't really have to worry about airflow because the rear air can still eject out the back of the power supply. And of course, it's a lot easier, or sorry, a lot harder to get into the machine by sticking a fiber optic camera through an active power supply. We tried this and it ended in a lot of fireworks. We were curious. We had an old system. Any more questions? Yeah, we do it at the rack level. One of the things you want to look for is a motion. Oh, so he is asking if we install webcams inside of the hardware so we can watch anyone trying to manipulate it. So we install the webcam at the rack level and then if you have a good one that can trigger on motion, you can see when anybody approaches the environment. And then presumably email the photo to you. Make sure that the photo actually gets transmitted. You don't want to have a situation where the only copy of the image happens to be in the device in the environment that the person is accessing. We actually also recommend that you have a secondary link going out besides your primary routers, especially in a Colo facility that provides every bit of internet access otherwise. We actually experimented with Cytomodules. Kalex actually has the sprint devices. And you can also, of course, for the various embedded hardware, various Arduino add-ons, etc., add GSM chipsets as well for Excel trading. Don't overlook hardening your routers as well. We use PF Sense Machines, which run on commodity hardware, so we can do the same modifications to that as we do to the machines that run the encrypted services. So the question was is that it seems this is geared around denying access in an undetected fashion to our data and whether or not the switches should just kill power to the system. So the reason that it doesn't just immediately kill power in every case is that first, some of the sensors are particularly sensitive. Regular maintenance in the racks around it will trigger them. So you first want an alert for that because you do have to balance what your actual operational goal is and what your security goal is. The contact sensors are designed to just kill things immediately because the contact sensor is not something that will go off via traditional rack maintenance like the tilt sensor might. Yeah, you'll find out through experimentation which sensors are more reliable than others. That's where the testing comes into play. And if you look at that Python script I posted a link to, all it does is monitor the sensors. The area of the script where you actually have to decide what happens is left blank. But presumably, as I mentioned earlier, our recommendation isn't physical destruction. It's to rely on the mathematical guarantee provided by the encryption when you cut power. That at least affords you the option of restoring. You do have to be careful about using the same hardware that may have actually been tripped. So if you have a confirmed breach situation, you may just want to remove the hard drives and place them in a new chassis. I actually have a funny story. At Hope a couple of years back, the guys from Rise Up had the server that the FBI had confiscated and held on to for a year. And they brought it back to the hacking conference. They lifted the top of the cover off and the challenge was to see if you could spot the modification. Any more questions? We're almost out of time. Not yet. We've had a few false alarms when a prior facility of ours would actually do maintenance in our racks without notice. And they generated zero audit logs when they did this. They didn't tell us, they told no one. They wouldn't tell us what they were doing at one point. And I had to pride that information out of them. And while they didn't open up any of the chassis, they were moving the racks around and at one point re-cabling some of the, we found out later with the PDUs when we looked at the camera. So while that wasn't a particular intrusion attempt from the hard drives perspective, the sensors did catch it and that's already a very suspicious thing just to be inside of someone's rack. Then re-cabling above the racks is very typical. Re-cabling the PDU inside of the rack is a very rare thing. The question is, what encryption do we recommend to guarantee the security of the drives? I use AES-256 for my symmetric cipher. I've traditionally, for all my pub key stuff, have been switching to elliptical curve. Although I heard yesterday at one of the talks that it's possible the National Security Agency might have a quantum computer in the very near future capable of breaking ECC and is now recommending that we use very large RSA keys. I don't know if I believe it yet because I'm not looking forward to switching back to 8K RSA keys. When it comes to the AES, you also have to choose what mode you want to use, whether you're using something as XTS or the CVC modes. Stick with an authenticated protocol if you can. If you have it and it supports it, you want to be operating in GCM block mode. Unfortunately, that won't work with a full-disk encryption because it's a chain cipher. In order to decrypt block, you have to know the previous block. It's stream cipher, sorry. You have to use a block mode like XTS. I do have pretty high hopes. I haven't implemented it yet in prod, but Chacha and Poly 1305 look like very promising replacements for AES. But when it comes to crypto, what I like to say is that speed kills. You move too fast and you're guaranteed to make a mistake and do something wrong, particularly when it comes to picking a new cipher or a new implementation. It's always good to go with the tried and true. The nice thing about symmetric ciphers is that they're much harder to break because of the mathematical principles behind them. They're essentially random number generators that are doing a one-time pat. So if nobody can break the random number generator, either by brute-forcing the key or by taking the stream and determining what comes next, the algorithm stays secure. It's the public key algorithms that you have to worry about. And on a related note, cryptographic hash algorithms are an interesting discussion because traditionally SHA-256 and 512 have been very secure alternatives. But what we're seeing is that with the emergence of Bitcoin and all of the investment in A6 is the speed at which they run reduces the security of those older algorithms. One more question if we have it. Well, if you think of something later, just come find me at the pool. I'll be at the 303 party tonight.


Hello, everyone, my name is Flavia and I'm currently Director of Product at Spotify. Right now, I'm based in sunny Barcelona, but before landing here, I worked in Portugal, Ireland, Silicon Valley, and spent most of my career between product growth and product management. I worked in startups, scale-ups, and in larger orgs, which I think helps with this conversation, both as a full-time employee as well as an advisor. At Spotify, I'm part of the experience mission, which is responsible for the whole mobile and other platforms' experience and where most innovation happens. And today, I want to share some thoughts with you around the unknowns in product management. I think you'd be surprised with the number of people I come across, product people, that want or expect predictability. And this always scares the heck out of me because, in my opinion, product management is everything but predictable. The environments, the variables that we play with are rarely controllable, and there's always an almost infinite amount of known and unknown unknowns that we have to deal with. So being a product manager, or a good one for that matter, means accepting that uncertainty comes with a role and also learning how to navigate it. We only have 20 to 30 minutes today, so it's impossible to explore every dimension of this topic. So instead, I'll share the recommendations I find most useful that I think are less talked about and that can be applied in various different settings. So let me start by defining uncertainty and the tools we as product people can leverage to chart our way into building impactful products. I define dealing with uncertainty as being in a situation that requires a decision for which you have considerably less information than you need for the decision to be obvious to everybody involved. No results are guaranteed for sure with any product decision because users and markets have way too many moving PCs for anybody to be able to kind of express the logic of all the dynamics at play. And unfortunately, there's no one plan that fits all the challenges or that can guarantee good outcomes. But I do believe, though, that there are some strategies that can improve the odds of building great tech products. So let's start from the beginning. You don't have your product yet or you're fundamentally reimagining it. These situations and especially the prioritization process are like charting your path in a heavy fog, right? How many times have you found yourselves in a situation where you were asked to make a decision with limited to no information? I think the majority of us have been there. And there's a great deal of uncertainty that we have to deal with. So let me talk about some tips overlooked most of the time on how to navigate this. The first one is frequently talked about, but I'm putting a spin on it. One, move quickly to get a first rough iteration in front of users to start getting high quality data back. This is the old question, should we explore to build or build to explore? Well, first and foremost, you need to know what your problem is. What you're trying to solve and what goal you want to achieve. This is where research, qual and quant is essential. So start by using data to frame the problem, define your goals, create the hypothesis to test, but then pedal to the metal. Don't spend your time doing very complex analysis, planning all the details and working out a perfect plan just built to explore. And the reason why I say this is because it is key to move quickly and get that first iteration in front of your users so that you get a flow of high quality inputs that will then help you refine, redesign your strategy and or your roadmap. And now you ask, but aren't we all doing that? Isn't that the like the most important thing in product management? Yes, a lot of people talk about this. And I think we've come a long way implementing this advice when it comes to delivery. We no longer or a lot of companies no longer think and work six months on something and then put it in front of users. But that's primarily thought for the delivery process, the delivery track, not so much beforehand. And I still see in a lot of companies, teams spending an awful lot of time planning and re-planning and analyzing everything before actually committing to do something. Whereas I think that it's much easier to build something and then explore. What did you get from that experiment or whatever you put in front of users? What are users telling you and how can you refine work? How can you set the next goal? Obviously, by saying this, I'm not saying don't plan. Don't think ahead of time. We have to think ahead of time. But our thinking should go into creating the right hypothesis, should be into creating the goal, making it abundantly clear to everybody around us. Let me tell you a quick story. A few years ago, I was asked in an interview process to define a long-term roadmap in what I think was an awful lot of detail. And to be very transparent, I knew very early on that there was no cultural fit whatsoever, but I decided to use that process as an opportunity to see how set companies were on this way of thinking. And also if I could change them or change their minds, I couldn't. And so I challenged them and explained how doing back of the envelope calculations supported by either an existing data or way too many shaky assumptions would be useless. Think about it. If you come up with a plan that relies on 20 different assumptions and the 20th relies on the first, 5th, 10th, and then you start your process and the first one is not true, then what happens to the 20th? You kind of have to revisit your plan. So you might as well put all your effort into creating that first assumption or hypothesis or first couple of hypothesis, having very clear what the goal is and then take it step by step. At the end of the day, we're not supposed to be futurologists, right? We're product managers. So to sum up by moving quickly and creating a flow of high quality inputs will help you pave the path forward. It doesn't, however, make the path 100% clear. It just helps you kind of navigate and create the path step by step. So the best advice that I can give you is to accept from day one the uncertainty and the long odds. Is it easy? It's not humans have been using categorization and classification as ways of understanding something unfamiliar and also making decisions for centuries. So you can see that it's very easy to fall into the temptation of organizing, planning, classifying, to feel safe and in control. However, endless hours of that planning will unlikely change the odds. But what it will do is give you probably less time to iterate. So going back to the role of research and bear in mind that when I talk about research here, I'm referring to both qual and quant. It should never be used to soothe our fears, right? Or internal fears, but to advance the discovery to help us generate realistic hypotheses and a plan to test them. So make sure that you intentionally recognize and accept that most of the time the path isn't clear and the path to a successful product comes from standing on a pile of failed tests and iterations that hopefully will help you move forward. So if you are desperate because you have ran 10 different tests and neither of them yielded the results you were expecting, don't be discouraged because you can learn so much from those and it will help you if you look at them the right way to actually move forward. So you shouldn't fear failure, but instead embrace it, accept that the path isn't clear that you will only be able to see one feet ahead of the time. So be humble, nimble, learn as you go, iterate based on your learnings and take one step at the time. Lastly, I know that the unknown is very scary, but and that the majority of the time we don't feel like we can be cheerleaders, but it is part of our job to keep everybody motivated and with their way their eyes on the future. It's certainly daunting to look at the odds, especially if you work in a startup. You know what I'm talking about. And it's also daunting to look at the decisions that need to be made in the midst of so much uncertainty. But the truth is, and I hope that this helps you, most great tech products started from a similar position. They all have their own unique, incredible stories. But I think what unites them all was having one, a capable team, two, being very relentless in their pursuit and third, iterating just long enough to have to get an incredible product out the door. And as product people, junior or senior, we play an outsized role in keeping everybody motivated, aligned with the goals, knowing exactly what their role is to get to where we set ourselves to go so I can't stress enough how important it is to keep the team motivated through these difficult times of uncertainty. And I'll tell you a story. A while back, I was working with a team that was fundamentally broken. They were not broken, the whole setup was. And I'm not going to lie, I had my doubts. I could turn things around, but never for a second, I let others see it. After we went through that transformation, a lot of people, I had this conversation with a lot of people, but I specifically remember an engineer that came to me and said that oftentimes he was 100 percent sure, convinced that there was no solution to the problem that I was giving them. But he saw me so confident, he saw that I was looking 100 percent convinced that there was a solution that he'd never gave up. He was trying, going to try and find it out. Was I convinced? No, absolutely not. Most of the time, I didn't even know if it was possible, but I didn't let them know that because I wanted to instill confidence so that they would believe they could find a solution and to that difficult problem. So long story short, a happy and a motivated team can move mountains, but a paralyzed team could be the end of your product or your company. I've seen it.  OK, so we talked about the importance of moving fast, accepting uncertainty and also motivating those around us. What else? Let's talk about data, which is something I think we usually turn to when we don't know what the future holds. I'm sure you've heard before that good product managers have to be data driven. Well, as somebody highly analytical who loves your data and uses it to make decisions on a daily basis, if I'm honest, I'm not going to tell you otherwise, it's incredibly important. Now, the art is one, knowing when and how to use it and two, to strike the right balance between the two extremes of 100 percent intuition driven decision making and the other polar opposite of fear induced paralysis, because for whatever reason, there's no solid data to go to work with. So I'll share now some pitfalls. I often see PMs falling into first one, proving assumptions. When we start recognizing familiar patterns and we can tell what seems like a reasonable story, it's easy to get complacent. And the reason why I say this is because sometimes something feels familiar. We write a narrative that sounds plausible. And before we know it, we are unintentionally planning our tests to make sure that it holds true. So while most product managers work to prove their hypothesis, I recommend that you focus your efforts on disproving them instead. After all, after all, if even what is a stable user behavior or a trend today might not hold true in a couple of quarters, COVID being an extreme example of that, shouldn't we challenge our deeply held beliefs, our assumptions, whenever possible to make sure that we're still chasing the right thing? I think so. So when we're prioritizing writing our hypothesis or planning our testing phases or anything related to the whole planning process, our goal should always be to prove that our assumptions are wrong. I know it might sound weird, but think about it for a second. If after trying to disprove your hypothesis, it still holds true. Shouldn't you have a better chance of being in the right path than if you had invested everything you had to prove the plan? I think you do. So in my daily practice, not only do I ask all the squads to plan experiments that can prove their assumptions wrong, but I also ask everybody to poke holes in my strategies or plans. These days, I don't spend as much time planning experiments or those sorts of things, but I do spend a lot of time thinking about the next big thing or putting together a strategy. And the one thing I do is ask everybody to poke holes in my thought process. And the more I believe in it, the more I want others to find potential problems because I know my bias and my will to make it work can make me blind. So I want them to go read the document and then based on what they know or even instinct tell me I don't think this is going to hold true or I don't think this is going to work out because I know this data point and I know that happened in that experiment or I know that in this specific step of the funnel, this is happening. And yes, is it tough in the early days to do this? It is. I remember feeling it's thing a little bit when I started doing this and asking people to criticize the work that I fundamentally believed in. But then I realized a few things. One, I might as well fail now than later on. The result is the same. Somebody saying that something doesn't end up now or me finding out made me go through the delivery process or even after we put it in front of users. The result is the same. I just I just learned earlier, which allows me to kind of plan differently. And second, our job is not to be right all the time. I know that it feels like we should be, but we're not supposed to be right all the time. So if somebody can poke holes, it doesn't say anything about your quality as a product manager. Actually, if the way you react to that says a lot about who you are and as a professional and how far you're going to come. So stay humble and question everything and learning and giving up on strongly held beliefs takes humility. I know that, but it will make you a much better product manager. And your journey will be a lot easier because you can anticipate problems. You can start the delivery knowing with the better chances of actually succeeding. OK, so now imagine your experiments disproved your hypothesis or that the experiment just didn't give you any meaningful information. And now you don't know what to do next. Please do not stretch the numbers to tell a beautiful story and don't jump to conclusions who here has looked at the data of a fail experiment and felt tempted to slice and dice the data a hundred different ways and come up with a convoluted narrative of what users were telling us through the data. I sure have the majority of us have, but data snooping while it can be a useful tool to iterate and define next steps. If it's it's flawed, right? If the data points are not coherent or even sometimes when they are, you might just be looking at noise. There's no there's no narrative. There's nobody's telling you anything. So the solution to deal with uncertainty is to do more discovery, to dig deeper, to chart a path to the learnings that will drive the strategic decision ahead, not find patterns where they don't exist or jump to conclusions just because it makes our lives easier. So it's key to have a good sense for when you need to repeat the experiment or when to continue to experiment to make decisions that ultimately stand the test of time. Finally, Phil Carthagoras saying that you don't have enough answer to answer. You don't have enough data to answer the question. While it might not make you popular, been there done that, I'll tell you a story in a second. I assure you that you will read the benefits down the line. So let me tell you this story. In my previous company, there was a time where when we started seeing our KPIs declining quite aggressively, PENIC was installed, my peers all got together. And in a matter of hours, they came up with a list of actions to fix the problem, to solve everything that was happening. And I got everybody nervous because every time they asked me, I would say, I don't have enough information about the problem to design a plan of action. So I had colleagues reaching out to me privately, showing concern about my ability to do my job and asking if I wanted help, if I wanted them to do it for me, because somehow I wasn't able to do it. And I thankfully declined and explained that I just needed a few hours to dig through the data to really understand what was going on. If you ask me, everybody looked really smart, except for me.  I kept hearing comments and people talking behind my back about whether or not I was going to keep I was I was going to be capable of doing that. And why wasn't I capable of doing that? If I was running the whole show in a specific area, so I should be able to know what levers to pull in order to achieve what we wanted to achieve. So it was it was it was hard. It was I didn't like it, but truth is it was the best decision that I made because there was a plan for each person in the leadership team. There were squads being asked left and right to do certain things. Whereas I had nothing. But the answer gave me a couple of days to really understand what's going on here. And that was tough. That was a tough period. But by the end of that exercise, not only those initial plans that my colleagues put together were not used, not because they don't know how to do their jobs, but because they didn't really understand the problem. And therefore those solutions were were not usable for that specific situation. But I had to sit with them and explain what was going on for them to then go their way and figure out, OK, I know what the problem is. This is my area of action. This is how I how I can contribute to fixing this particular situation. So my message to you is the loss of trust by stating something that is demonstrably false is really hard to learn back. So try answering the queries you don't feel comfortable answering with. I will look into it. I'll come back to you with with an answer or a plan on how I'm going to get that answer, or I don't think we have data for that yet. Here's why and how I'm going to go about fixing it. And when you feel pressured, because there will be a situation where you're pressured to express your opinion, be careful and state it as a hypothesis and how you would test it. So instead of saying, this is what's happening, say, hey, based on what I know, which is kind of limited right now because of A, B and C, this is my hypothesis. And this is how I'm going to go about proving or disproving it. So intuition has its value. You have to develop product intuition. You have to use your intuition throughout your career. But it's usually a lot less valuable than most people give it credit for. Bear with me, we're almost there. Last section, dancing to the tune. So product management is a dance between paradigm breaking changes and optimization or expansion of existing offerings. What we do and which side we invest more or less on depends on the end goal. What we're trying to achieve, what is it that we need to do for our business? So let's talk a little bit about strategy, optimization, how to identify the need for a paradigm shift and how to innovate without building a Frankenstein product in the process. So there are many tactics that I use to navigate the unknown. But if I had to choose the one thing that makes the process easier other than first accepting that there is uncertainty, there are long odds, it's tough, is that is having a North Star and a strategy. No matter how unclear the path is, if you have one North Star or a good strategy guiding you, you'll probably make better decisions or more good decisions than bad ones. And because, you know, you know at all times that regardless of the situation or how you want to what you want to achieve, what is it that you're striving for? And by knowing that, you can see how the several options that you have. So you have A, B and C, you know what the North Star is. You can better gauge what impact that the several hypotheses that you have will have on your North Star objective you have. So if there's one thing that I recommend that you do is having that guiding goal that you can always fall back on in case of doubt. Now, assuming you have that, you know what you're trying to achieve, you know what the business is for, you know what the business is trying to do. Generally speaking, then you're probably faced with probably one of these two scenarios, either you're in a zero mode, so conversion rate optimization, you're looking for an uplift or in innovation mode. So let's start with the first. You're chasing an uplift and you're not sure where to invest because it's unclear, it's unclear where you can actually optimize. The most likely place to find the next positive outcome is in an area where you just succeeded and certain needs usually lower. The potential is a lot higher where past iterations generated a significant uplift. So if you find if you found an underutilized area where there are multiple opportunities to explore and you have already some learnings that can help you make better decisions, double down. This will help you with prioritization, even if and it will make you partially blind to new opportunities because it steers your attention only to small marginal improvements. So don't be afraid to invest further in an area that you know is under optimized. Just make sure that you keep your eyes on the overall goal and that you don't go on, you don't go down the rabbit hole of going overboard in the attempt of squeezing that last point zero one percent in that little step of the funnel that you've been working on for four months. So if it's very unclear, your objective is to optimize. If you know that there's an under optimized area, that's probably going to be your best investment because you already have data. You already have some information to guide you. Just make sure that you don't be that you're not completely blind and just obsess about one single thing that you need to improve to the max. But at times it's not about optimization, right? Sometimes it's about innovation or it's a mix of both. So one thing to pay attention to are the signs that a paradigm shift or change in focus is needed. So I collected the most three, the three most common signs I've seen in my practice. One, your positive impacts become less and less frequent and or margin marginal increments reduce over time. Second one is the majority of experiments that you run actually return significant negative impacts. This normally means that you have arrived at a close to optimal stage in the current paradigm and three, it becomes very difficult to do discovery and come up with hypotheses that do not require a large rewrite of the user experience. This usually means that your current paradigm is kind of locking you in and you need to look elsewhere. And finally, if you're shifting your focus or planning a more fundamental change to your product, you also have to be careful not to build a Frankenstein in the agile world we live in. It's super easy to think that any hypothesis that needs more than two sprints to be put in an experiment should either be deprioritized or broken in chunks. However, this does not apply to larger changes in EWECS as a Frankenstein mix of the current state and the proposed state are not usable. I'm sure you've either been in a situation where you were you had to make this decision or you use the product that was in this process of completely reimagining how the entire functionality and you had a mix of the old and the new and nothing adds up together. So that's what you need to be careful of. And then obviously, small iterations are always preferable, but ultimately, the new proposed state needs to be super functional. So we need to be realistic that changes in paradigm carry uncertainty and risk, not only because you disrupt the value that you are delivering right now, but also because you need to invest more than you would for a couple of small tweaks to the UI, so you have the experimentation cost and the opportunity cost together and you and your teams need to go in cognizant of the challenges and determined to be dead. So just be aware of this and then you're set for success. So to summarize what we discussed today, first and foremost, accept the uncertainty can't stress this enough. You won't change it so you might as well embrace it. Also, build to explore and not the other way around because you'll learn more and faster, make sure everybody is motivated and might feel difficult at times, but it's crucial for success. And then data is very important, will always be very important so long as you know when and how to use it. Don't overestimate what you know and always challenge your strongly held beliefs and don't stretch the numbers to tell a story. Don't jump to conclusions. Don't try to find patterns that don't exist. Might feel easier, but long term, it won't do you any good. So feel comfortable saying that you don't know yet that you're going to have to explore because earning trust once lost is really hard work. Have a North Star that can guide you through the toughest decisions you'll have to make. And if you're in an optimization mode, double down your investment in areas that you know are under optimized and where you already have some knowledge. Watch out for signs that a paradigm shift is needed. So get the product in front of users as soon as possible, but avoid building a Frankenstein product. That's the last one. And I know that being a PM in the more modern days isn't easy, isn't an easy job, but it's incredibly helpful feeling when you know how to navigate these challenges. So have fun more than anything else. You know all of this if you're starting your career, if you're in the middle of your career or you're very, very senior. I think the best advice of all is just have fun. Building products is amazing. I think it comes down to how we approach the problem. I've been in situations where it was hard. The setup was not in my favor. And I found a way of seeing light at the end of the tunnel. So have a lot of fun. Make sure that you bring people along with you in this journey and just build crazy things because that's amazing. That's an amazing opportunity that all of us have as product managers. I hope this was useful and that you can implement some of these tips in your daily practices around. So if you have any questions, feel free to drop them in the in the chat or reach out by email, you have it in the in the presentation. So reach out if you have any questions. And thank you so much for listening. It was a pleasure being here and I hope to see you soon in the chat. Bye.


So we all know that volunteers are really the lifeblood of any community, especially those that serve public safety in volunteer search and rescue and firefighting roles. And it's because of their selflessness. Each of us rests a little better knowing that BC has made stronger by the support of their safety net. So it's my pleasure today that as a province and as a result of our strong and growing economy and as a part of our 2017 balance budget, we are providing a new tax credit to these volunteers who serve their communities in search and rescue teams and in volunteer fire departments. We are pleased to recognize their leadership and volunteerism through a $1 million yearly commitment from the province to in a small way give thanks for the large contributions. I'd like to thank the minister as well as our local MLAs for their support. Search and Rescue volunteers contribute an incredible amount of time away from their families and workplaces to save the lives of lost or injured people. They also contribute financially buying personal protection equipment, clothing and ready packs to operate safely in a very invariable terrain and weather. This tax credit will relieve some of the financial burden carried by these volunteers and shows the commitment the province of BC has to volunteer search and rescue program and its members. So this tax credit matches the federal tax credit for those with at least 200 hours of volunteer service which means that between our two levels of government we're putting up to $600 back into their search and rescue volunteer team members wallets each year as well as volunteer firefighters. To have this recognition from the province for the work they're doing for the communities truly is a great gesture for us. So on behalf of our associations we thank the province for really stepping up and recognizing what we do for them. So again thank you very much.


You are somebody for people that don't know that is all about sex all about femininity and teaches within the realms and expressions of that so When that comes down to it What what is femininity to you? I'm sure you could man. That's Jesus Christ.  I just answer this in a probably wanky way for anyone hasn't meditated the greatest sex You'll ever have is very similar to meditation You're in this moment with a person. There should be a lot of eye contact. You might be saying I love you You might be in a powerful position and you know, I'm not as Haven't done as much as James, but I've done a lot of meditation. It was Wasn't important to my life.


Welcome to my channel Lam Arpata Karwa and in this video I am going to talk about some really really important things that I would like to tell you before you go and hit the exam hall. I know that net exam is just a week ahead and all of you are all set to rock this net exam.  So, the first important thing that I would like to bring to your notice is that guys please keep your confidence boosted, okay. It is very important that during the exam preparation and also on the day of your exam you feel good about yourself. You should always remember that you will do very very big in life. It doesn't matter whether you performed according to your expectation in the past years or you have performed well in other academic field or non-academic field in the past year, what matters is that please have a belief in yourself that you will surpass your own expectations this time. You will definitely reach the goal, you will be able to do something really big in life. See confidence doesn't come from the fact that I will do good in this exam. Confidence comes when you think about your future life, you think about the life you want to lead and you are able to imagine that life and you can see yourself going in that direction that is what gives you confidence. So just close your eyes for five minutes and think about your dream, think about the position that you would like to attain in this life and just see yourself moving in that direction and this net exam is just one milestone in this entire journey. So if you look at the bigger picture you will definitely feel proud about yourself. The next advice that I would like to give you is that you need to take care of your thinking pattern. Guys there are people who tell you that thinking positive or positive thinking helps you a lot and if you look at this thing scientifically you will feel that positive thinking actually charge up your body cells and they are able to perform in the best possible ability. So you must remember this thing that always you have two options, either you can perform out of the desire for gain or you can operate out of the fear of loss. There are people who keep on questioning themselves that what if I forget everything? What if I am not able to complete the paper on time? Whatever I have studied does not come in the exam paper. All these questions show the mentality of people who are operating out of fear of loss but there is another set of people who succeed way more than these people and these people operate out of the desire for gain. They will keep themselves positive, they will boost their confidence by telling themselves that they will be able to complete the paper on time. They will have questions in the paper which they know the answers for. They will not forget anything when they sit down to write the exam and such positive affirmations build their confidence and help them to maximize their examination result. So always remember this thing that in the last days of your exam preparation keep this thing in mind that you keep yourself positive. Also you can practice self affirmations. I am giving a link of a video which I used to listen to every day before I sat for my net exam so that I can rejuvenate myself, I can feel confident about my abilities. It is a very short video which will help you imagine the entire examination setting and will help you to think the best that you want to see yourself in on the day of your exam. So if you too believe in self affirmation, you believe in positive thinking, you want to charge yourself, you want that your mind should perform the best of its ability on the day of your exam then start practicing this self affirmation every day before you hit the bed.  The next important tip is that after finishing paper one, please don't start calculating your score because your performance in paper one will affect you mentally and will also affect the performance of paper two. Please don't demoralize yourself after finishing paper one that there you made a mistake and this is the wrong part and this you should have done more carefully. All these thoughts will affect you emotionally and will definitely show the effect in the subsequent paper which is paper two. So please don't let the performance of one paper affect the performance of second paper. Draw a line to your past what has been done has been done you cannot change. Once you do your paper one just keep that paper aside don't even look at it. Focus all your energy and time on the next challenge which is paper two. I have seen this thing in many students that once they do paper one and in the meanwhile when paper two is coming they'll keep on looking at paper one and they'll keep on calculating their marks. This will have a very demoralizing effect in your brain. So it's better to keep paper one aside focus all your attention all your energy on paper two and just do your best what is there in front of you. So before I end this video I would like to give you a very short tip which is very dear to my heart and I keep on telling my students also that there is a force beyond you and me and before you begin answering the question paper it is important that you devote one minute in remembering that force in evoking the presence of that force and asking that force to carry you through the exam paper and it is also important that you devote at least one minute to pray and meditate before you start writing your paper so that the energy that flows in your body while you pray is able to help you to remain calm during your exam time. So I would always advise you to devote at least one minute to pray those who are a part of my online course they are aware of this thing that there's a section called prayer in my website in which I have uploaded a prayer by Mahathriya Ra which is a one-minute infinite prayer okay it's just a one-minute prayer and I make sure that all my students listen to that prayer first and then they start their studies for the day so the same way I'm also advising you that before you hit the books before you start studying before you sit down to give the exam just take one minute and pray to whichever god you believe in so that he is able to take you through the exam preparation and take you through the those two hours of your exam so with that note I end my video I wish you all the very best I really wish that you are able to give your hundred percent and you are able to achieve your goals and your targets and you're able to clear net with flying colors so with that note I take your leave thank you so much happy learning keep loving literature and stay tuned to ArpitaKarwa.com and don't forget to subscribe to my channel because I post videos every Saturday and every Sunday for UGC net aspirants apart from this you can also go to my social media platforms and you can like my social media pages so that you are notified every time I posted go net quiz


I deeply appreciate being included in this important conversation as Carl has noted my interest in electronic access to primary legal materials stretches back more than two decades to a time when that meant dial-up access to bulletin boards and disk distribution. And it was that interest that led 18 years ago to the establishment of Cornell's Legal Information Institute with my colleague Tom Bruce. The first dedicated law site on the internet still going strong at www.law.cornell.edu. From our early outpost on the net we watched through the 1990s and into this century as court sites gradually appeared providing full text versions of appellate opinions on day of release. Well at first these developments seemed to give us much to celebrate. My principal observation in the year 2010 is how far short of the full potential of this new environment the U.S. courts have fallen. And my focus in these brief remarks will be on courts rather than the full panoply of lawmaking bodies that Carl has sketched. Particulars in a moment but first two other general observations grounded on our institute's experience. First a decade and a half of processing the text files released by the U.S. Supreme Court and New York's highest court for full featured web access has provided countless painful examples of how powerfully small, invisible to the eye technical details in the files that courts release affect the cost usefulness and quality of case law collections built from them. Those are barriers of a third kind alongside those that Carl mentioned. Second our institute's users have taught us time and again of the meaning and value of public access. From the start we've been stunned by the depth of the interest in and need for primary legal materials among educators and health care workers those employed in the financial services industry and high tech endeavors those running small businesses as well as the public sector folks from police officers to agency officials. We have found that in truth there is no sharp boundary between the information needs of lawyers and of the public. Fifty states countless variations it would be all too easy to get tangled in the details of Alaska, Georgia, Illinois and Texas. Time is limited so I will for the most part describe dominant practice and significant variants identifying only a few illustrative states where that's useful. By now the judicial branch in most states maintains a website that not only serves diverse public information and educational functions but also provides access to the jurisdictions appellate decisions. As soon as opinions are released by the court they are posted and in a majority of states subsequently stored in an open archive. As generally practiced however these measures fail seriously as a means of improving access to the component of our legal system that professionals refer to as case law. Let me illustrate using three state court decisions of last week all posted promptly on the respective website. A decision of the Florida Supreme Court last Thursday setting out the responsibility of real estate lawyers for overseeing access to escrow accounts. Second, a Kansas Supreme Court opinion released last Friday laying out the standard of proof confronting insurers seeking to rescind the policy on grounds of fraud. And third also from Friday a ruling of the California Supreme Court resolving a circuit split on an issue of bail forfeiture. For anyone closely following a particular case or the work of these courts in general or monitoring one of those issues immediate web access to these decisions is an incredible boon and redistributors publishers no matter their type of purpose are free to harvest this data add value and republish it. However, because of the heavy hand of print sector practices and entrenched interests all three decisions were posted in a form that is seriously incomplete and destined to remain so. Two years ago in an address to the association of reporters of judicial decision I argued that taking electronic distribution of case law seriously required judicial bodies to do four things. And the first and most critical is that they must furnish the general public and publishers with access not just to slip opinions but to opinions in their final official citable fully edited site checked quote checked form. The judicial sites of Kansas, Florida and California are typical in failing to meet this threshold test. Institutional factors vary but in all three instances the day of release versions of the opinions I've just mentioned will not later be replaced by their final citable text. During the latter half of the 20th century well over half the states surrendered the dissemination of case law in the form of print law reports to the commercial sector and in nearly all cases that meant ceding the field to what was then the West publishing company. Florida was one of them. As a result the editorial work on the published decisions of its appellate courts the case summaries and head notes and the compilation are all the work of the editors of the national reporter system and copyrighted by Thompson formerly West. The Florida Supreme Court's website warns users that the opinions contained in its archive which now reaches back over a decade are quote subject to revision before publication in the southern reporter third series and since the court's publication process is in commercial hands and the court hasn't altered its decision preparation and post release revision practices so that it even possesses electronic copies of the decisions in their final official form Florida fails my threshold test and that failure widespread through though not universal among the states that during the 20th century abdicated responsibility for law report publication bestows a huge competitive advantage on a single commercial publisher now at first blush it's odd that one finds a similar approach in kansas one of the 14 states in which a public official a reporter of judicial decisions is still responsible for the publication of law reports the case law archive at the kansas judicial branch website reaches back to 1996 from top to bottom the decisions are in slip form with the site warning quote slip opinions are subject to modification orders and editorial corrections prior to publication in the official reports consult the bound volumes of the kansas reports and the kansas court of appeals reports for the final official texts here statutes and deeply resistant bureaucratic practices continue to assert the primacy of print despite the dwindling demand for official reports in print form change here would not so much threaten commercial advantage but public jobs turf and status and my third example california is also among the few states still employing a public reporter to oversee publication of law reports but the size of that state's legal information market generates incentives to cling to the nominal primacy of print that smaller states like kansas need not contend with the california reports like the new york reports and those of washington state georgia ohio in illinois are published under contract by one of the major commercial publishers tomson west holds most of those contracts but lexus nexus is at present the publisher of the of california's official reports a contract for the next five to seven years will be let later this month like the current contract for the rfp for the next one seeks to exploit the competitive advantage it will bestow on the successful bidder to secure a host a host of tangible benefits for the california judiciary performance of much of the post release editorial work on decisions free access for the reporter and staff to the publishers online legal research service and complementary copies of the print reports for california judges where the california or new new york appellate courts to release final edited and revised versions of their opinions with full citation information such contractual exchanges of privilege and access for free services would collapse for the final and official text of opinions california directs users to a lexus nexus website at that site internal page numbers necessary for precise citation have been stripped out and license terms for bid commercial use direct linking and all forms of redistribution of the contents this removal of internal pagination draws attention to the citability component of my threshold standard in the current environment citation norm norms that still rely on volume and page numbers in a subsequent print edition impose large and unnecessary costs on all publishers other than the one originally assigning them beginning with louisiana in 1993 a quarter or so of the states have moved to court applied public domain citations appellate courts in these states simply embed all necessary citation information in opinions at the time of release the approach places all publishers on an equal basis first it removes any proprietary copyright claim to the citation information tomson has still not relinquished its long-standing claim to copyright protection and the pagination of the national reporter system in addition court applied citations remove the logistical burden of extracting citation data from a later source or sources for insertion in opinions that were released without them the models are there and to date those models as embraced by the association american law libraries and the a b a have been adopted in varying degrees of completeness by fourteen states for impressive examples of state judiciary sites meeting my final official citable standard i point you to those maintained by north dakota and oklahoma my second criterion involves the attachment or inclusion of what geeks call metadata most state court websites awful offer digital files principally designed to replicate the paper documents for which they substitute therefore might reflect serious concern with how an opinions file will print but next to know attention to how easily and accurately computer software can extract from it such basic information as which circuit or district decided the case what judge wrote the opinion how it's organized when it was filed taking electronic dissemination seriously requires encoding the structure and key data elements of decisions not merely their appearance third it's far more important today than it was during the print era for researchers to have access to a single authoritative version of a decision against which they can compare critical language from a text they've downloaded from any one of the many unofficial sources available publicly produced print reports once served that purpose the contemporary analog is a digital archive of authenticated copies and of course by authenticated i mean more that the file that the files have emerged from a court computer system specifically that each decision carries a digital signature applied by the appropriate public official so that any question raised by discrepant versions can be put to rest for examples of this level of public assurance take a look at any recent public law available in digital format from the gpo or decisions available at the site maintained by ohio's reporter of decisions fourth and last some judicial opinions cast a very long shadow and for that reason the value of a digital archive of decisions grows over time a few jurisdictions have sought to accelerate that value accrual process by recreating or acquiring retrospective case law data from the print era whether or not that's done since case law is cumulative courts taking digital release seriously must not only provide initial access to courts in final citable authenticated form but they must commit to maintenance of a permanent archive of that data dominant commercial interests have encouraged us courts to believe that adapting their case law dissemination methods to a digital age is beyond their capacity and unnecessary fortunately within the experimental space created by our 50 state federal structure vision leadership and capacity have aligned in a number of jurisdictions to provide solid examples of what's possible on which other states and the federal courts can draw i've mentioned north dakota and oklahoma there are others but all too few my hope is that this law dot gov initiative will catalyze many more by providing visibility to the jurisdictions that have led and support to those setting off down the path of change arkansas for example last year arkansas ceased publication ceased publication of its print law reports pursuant to a rule of the arkansas supreme court the electronic version of appellate decisions posted on the judiciary website are the official report of those decisions the same rule charged the state's reporter with applying public domain citations to those decisions and with maintaining a secure database of authenticated copies recognizing that the previous distinction in weight between published and unpublished decisions made no sense under this new scheme the court's rule also provides that with the switch from print reports to official electronic publication all appellate decisions will be regarded as precedent to be relied upon and cited needless to say this past year has been a period of furious activity as the document management system capable of delivering on this ambitious charge has been designed and assembled arkansas reporter of decisions susan williams told me last week that she expects the system to be open to the public via the net by the end of this month the volume of my applause will be affected by what i see then and in the months and years to follow but i fully expect arkansas to meet and surpass the four criteria i've outlined and properly realized law.gov could assure that this initiative by one state is more than another isolated case against the national background of an airship. Thank you.


So, everyone, welcome to the first episode of the Spokes Podcast at the Urban Cycling Institute. Today here I have with me from Lithuania. So yeah, I'm Anton Nikitin, I work for the City of Wellness Lithuania and I am the Sustainable Urban Mobility guy. And how is cycling in Lithuania? I imagine it's coming up to winter so things are getting cold, do you have any snow there? Yeah, we have a lot of snow in winter, well, it depends actually. Last couple of winters are not that bad, but yeah, the cycling is getting better, we're building networks, we've built some 50 kilometers of cycling infrastructure in the last four years and we are about to build another 70 kilometers in the next couple of years. So yeah, it's getting better. Cool, and you're here with me on the Cycloalk project, which is part of the European Union funded inter-reg stream, and what have we been doing so far? Well, the thing is that the project is about to end, basically its first phase is about to end by the end of this year, so it's year 2019 for the record, right? So the project is about cycling and walking and how do we improve our cities. And basically it's about learning about this, what can we do? We had our first meeting in Amsterdam and had this really great experience of how everything could be and then we visited all other partners, so we went to Austria, we went to Italy and Slovenia, we went to Sardinia, we went to Romania, guys came to Vilnius and we were learning from each other, what do we do, what good things we have, what bad things we have and yeah, that's probably it. Cool, and you were here in Amsterdam, well in the Netherlands, but in Amsterdam a few years ago or months ago, how did that compare to the Netherlands? I'm in love with the Netherlands. You're in love with it. I'm in love, I'm actually, well the thing is that yeah, so first time we were here, it was two years ago in April and then I happened to visit Netherlands every now every half year. Even on your own money? No. Well, yeah, I visit warmer countries on my own money, but no, but the thing is that I happened to visit on different kind of occasions and I really love it, it's, well you know, I'm always liking the weather, so I don't know. Wait, the weather is good here? Yes. People in Spain who come in Spain and Italy, this is pure suffering for them. I know, but well, I'm so lucky. I don't know, people say that it rains a lot here and there's winds and everything and I'm so lucky that it rained, I don't know, totally during my all four or five visits here, it rained totally two hours maybe. So yeah, so what are the odds, right? Uh oh, look ahead, we're having a traffic jam here. Oh yeah, okay, it's traffic. Oh, yellow light, that's white. Uh oh. Wow, good work. You did well. No, no, it's okay. We're late, we're late, we're late. It's great that. So yeah, what was the question? The question is, how do you feel about the weather given that you're from a much colder place? Yeah, so the weather is great for me, I don't know, oh we can go. So the weather is great, but the thing I enjoy most is this experience, of course, of the cycling and before visiting Amsterdam, I just, you know, you kind of, well, I love cycling from as far as I remember myself, but until you visit Amsterdam, until you sit on the bike and until you just really try what nicely designed network looks like and until you really see the rush hour and all the people going everywhere, you're like, oh, this is how it all can be.  So that's what I basically do every time I visit. I take a bike, I rent a bike, I just ride around, I take pictures of all the strangest things I can take. It's a curb, it's an intersection, it's a marking, it's the asphalt, it's the piles or whatever. So, and then I bring those pictures back to Lithuania and, well, try to do not something similar but adapted to our local conditions. So Anton, how'd you get started in this whole cycling planning business and how'd you get involved in this EU project? It happened to be that I'm working as the cycling and not only the cycling person, but I started my career in transportation as a cycling project coordinator, like a cycling officer or you name it, then it really quickly expanded to cycling and pedestrian issues or topics and then it again quickly expanded as we had our SCMP approved. It really quickly expanded to all the SCMP management because, well, I think you can't have good cycling network or pedestrian comfort without this holistic approach or how you name it. You have to look at all the infrastructure and try to make safer streets, you have to make enough space and comfort for everybody. So that's, I guess, how it should be. Cool. We're going through a tunnel here. It's amazing. Oh, let's just enjoy this for a bit. So I think it's cool here because they try to fill the tunnels. And you can see I'm making pictures of people. Yeah, yeah, at the same time. That's illegal, by the way. So I'm turning this footage over for the police. I hope I'm not going to get a fine. So they shaped the tunnels at an outward angle so it makes it feel more spacious. And I was involved in this simulator project where they actually tested in virtual reality the different light spacing. So here you would have a bridge and then you would have some open air and then you have another bridge. And they were trying to figure out exactly how much open space do you need for it not to feel too much like a tunnel. Oh, wow. So they actually deliberately space out the bridges so there's light. And I'm curious what is the incline because it should be in the chrome manual. This is like very mild, right? What do you think? Well, if it doesn't look mild, but it's easy to cycle, so that's the, I don't know, everything is so easy here. It's too easy. Oh, it's too easy. Yeah, yeah. Give us some adrenaline. Yeah. They're legally allowed, but they clearly don't have speed limiters. Yeah. They're supposed to go maximum 45, but. Or is it 25? No. It should be 40 on the bike path. Wow, 40. 40 is a lot. Yeah, it is. So at the end of this EU project, what are some of the things that you're seeing to bring to your city? I think I already brought some. So we can help. So crow is one of the things. Another actually, it's not about this project, but basically inspired by this project is the Sustainable Safety, which is, well, kind of my dream that we, not completely, but really have it implemented in our city. But basically, I guess this project and these experiences, they do change the mindset of me. How it can be made, what can be done. And that's, you know, a lot of small and big things that are important. So it might be infrastructure, but as well, you know, not less important is the soft measures, all the things educational and then so on. So I think that's what I'm learning. And then, you know, design-wise, or how do we change perception. I'm also bringing it to illness from this project, yeah. I'm looking, you mentioned crow, that's interesting. I'm looking into doing a YouTube series on the book. Oh, cool. And I think that, like, what do you think about that value added for practitioners? Well, it would be great, actually. Well, you know, reading all crow, it gives you a lot of good thoughts about how the infrastructure could be planned. But the thing is, and the limitation of crow, and it's not only because of crow, that you can tell people, you know, well, I need 3.5 meters bike path or 4 meters bike path because, well, we have our plans and we will have this capacity and so on. That's one of the examples. But then the politicians or decision makers will say, well, that's wide, that's very wide. You know, what do we need to build it? And I think along crow, we have to think about changing perception of other people. So because, you know, in Netherlands, I guess there is more or less understanding that, okay, we need to follow that and then we, and that's the thing to be done. But in other countries, it's a bit different. Just because there's a manual doesn't mean people actually follow it. Yeah. And if they follow, clearly, it's school hour. Wow. So, it's interesting, there's this kid on the bike. Look how these parents are picking up their kids. I don't know if these cars are from the residents, but these bikes on the left and that school bus bike thing, the orange, that's cool. That's how that's how that's your version of school bus. So how do you think cycling here in Delft, which is a smaller Dutch city compared to cycling in the much busier Amsterdam? Amsterdam is busier, so no, actually, it's not that different. I think during the rush hour, it's also quite busy here, because we were, this morning, we were walking from our hotel and it was really, really busy, a lot of cyclists. So, and I particularly enjoy that, I know how to put it and how to describe it, but the way people move on the cycles, it's that eye contact and, you know, and just speeding up and slowing down a little bit so that it's kind of a ballet dance with the bicycles. So yeah, that's probably the, you know, it really is, it is really fascinating. Do you watch their feet, because we're on coaster brakes, actually, I want to ask, how do you feel about coaster brakes? What's the coaster brakes? It's a back pedal brake, so we've got nothing on our hands. Yeah, oh, yeah, well, it's a funny thing that I have a similar bike at home. I have the front brake, the hand front brake. They quite rarely use it, so I'm fine, but for those who never used it, it's quite challenging, I guess. Yeah. So I'm fine, but yeah. Have you paid attention to people's feet when, when you're negotiating? If they have a coaster brake, I find it much easier to see what their intentions are.  Oh, wow. Or if they prepare to pedal backwards. That's nice. Yeah. No. Yeah, next level, that cycling. Next level. Yeah, yeah, yeah. Oh, wow. No, I'm just, you know, enjoying the surroundings and enjoying the quiet, actually, the quietness. Oh, here's more. I guess they get priority here. So these speed bumps here, they're, they're mostly to slow down the scooters that are going that, the really fast ones that came up from behind us a while ago. Yeah. So they put these wave things as a kind of a fun way to let the scooters reduce their speed a bit. They're not very comfortable for cyclists, too. I actually noticed the differences, you know, about all the infrastructure. So one of the differences is the speed bumps for cars in the, in these residential streets. So with them, sweet. So with them, they're so bike friendly, those speed bumps for cars and they're, they don't seem very friendly for cars, you know, so that they keep up the 30 speed, for instance. So that really fascinates me because our standard speed bump is not friendly for bicycle because you have this hit and then you jump a little bit if you're, you know, going on the bike. So there is, it's not that, that very friendly. It's more of a gentle wave pattern, right? I think it's, and actually the, the speed bump that they built here, the 30 ones, they're high enough that if you drive really fast, your suspension won't absorb all of it. Whereas if you put those tiny, but, but really the ones that are small, but they have a very high impact and you have a really big car and sometimes you can go really quickly and just drive through them. And then your suspension just takes up all of the bump, whereas these ones, it's, you have to really slow down. And they're so good for the cycle as well. And another thing actually, probably, well, the difference between Amsterdam and Delft, the cycling, so in Amsterdam people cycle closer to each other, I think because of the space. Is it too close? No, it's not close. It's fine. It's pretty good. It's pretty good. Yeah, basically because I guess, well, as, as I said, I didn't cycle here during the rush hour, the real rush hour, where everybody goes to school or work or back home. But it doesn't seem that, you know, people are cycling very close here. So because of the space problem, oh, those kids are close. Okay. Yeah, this is a nice four meter wide bike path here. Yeah. No, no challenge for space. Yeah. And have you got any other Dutch cities that you want to visit at some point? Actually, I visited Hague and Utrecht. I think I would like to go back to Utrecht one more time. I would like to go to Rotterdam, which seems, for me, that is the city that could be quite perfect to learn from. Is it very far center? Yeah. So as, you know, as a lot of European cities, so, and they're doing a lot as far as I know, and as far as I follow. So yeah, so I'd like to visit it just to see what could be made when you have wide streets and stuff. Cool. And with that, and with this traffic jam we have up here, we'll call that the end of our podcast. Yeah. Yeah. All right. That looks quick. So thank you, Anton, for joining us. Thank you. And we're going to have a blast here in Delft in the next couple of days. Cool. Take care, guys. We'll see you next time.