netsekure rng

random noise generator

How to approach forking Chromium

One really nice thing about Chromium is its source code is open and released under the BSD license. This allows people to reuse code, extend the browser, or fully fork the project. Each of those are probably worthy of a blog post on its own, but I will focus only on the last one.

Taking Chromium and forking it is fairly easy process, just clone the repository. Make all the changes you would like to do - add missing features, include enhancements, create a totally new UI - it is only limited by one’s imagination. Building the binary from the source code is a little bit laborious, though not too hard. It does take beefy hardware and some time. Once it is built, publishing it is deceptively easy. However, what comes next?

Software in today’s world is not static. As a colleague of mine likes to say - it is almost like a living organism and continuously evolves. There is no shipping it as it was the norm in the ‘90s. The web is in a constant release mode and its model of development has trickled to client side software - be it desktop or mobile apps. Chromium has adopted this model from its initial release and is updating on a very short cycle - currently averaging six weeks between stable releases and two weeks between intermediate stable updates. It is this constant change that makes forking it a bit more challenging. However, there are few steps that one can take to ensure a smoother ride.


With constantly changing codebase, having a continuous build system is a must for project as big as Chromium and is very useful even for much smaller projects. Setting one up from the get go will be tremendously useful if there is more than one developer working on the code. Its value is even higher if the project needs to build on more than one platform.

What is more important and I would argue it is a must - using a continuous integration system. Running tests on each commit (or thereabout) to ensure there are no breaking changes. It is a requirement for any software project that needs to be in a position to release a new version at any point in time.

The system used in the Chromium project - buildbot - is actually open source and can be adapted to most projects.

Making changes

The most important action one can take when forking Chromium is to study the design of the browser before diving in and making any changes. There are multiple components and layers involved, which interact through well defined interfaces. Understanding the architecture and the patterns used will pay off tremendously in the long run.

Chromium has two main component layers - content and chrome. The former is what implements the barebones of a browser engine - networking stack, rendering engine, browser kernel, multiprocess support, navigation and session history, etc. The chrome layer is built on top of content to implement the browser UI, extensions system, and everything else visible to the user that is not web content.

Each layer communicates with the upper ones through two main patterns - observer and delegate interfaces. Using those interfaces should be the preferred way of extending the browser and building on top of it. Whenever this is not possible, changes to the core are needed. I would strongly suggest preferring to upstream those, if possible of course. It will make maintaining the fork much easier by reduing the burden of keeping up with changes and also shares the improvements with the whole community!

Finally, do yourself a favor to keep you sane in the long run - write tests for all the features you are adding or changes made. It is the only way to ensure that long term the regressions and bug rate is manageable. It will save your sanity!

Keep it moving

The Chromium codebase changes constantly and gets around 100 commits each day. The sane way to keep up with the rate of change is to rebase (or merge) your code on tip-of-tree (ToT) daily or at most weekly. Letting more time lapse makes resolving conflicts a lot harder.

Updating the install base is key to long term success. The update cient used in Chrome on Windows, called Omaha, is also open source. The server side code is not available, though, since it depends heavily on how Google’s internal infrastructure is setup. However the protocol used to communicate between the client and the server is publicly documented.

Development for Chromium relies quite a bit on mailing lists. Subscribing to the two main ones - and - is very helpful. It is place where major changes are announced, discussion on development happens, and questions about Chromium development are answered. The security team has a dedicated list for discussions -

Keep it secure

Security is one of the core tenets of Chromium. Keeping up with security fixes can be a challenging task, which is best solved by keeping your code always rebased on tip-of-tree. If this is not possible, it is best to subscribe to the list. It is the communication mechanism the security team uses to keep external projects based on Chromium up-to-date with all the security bugfixes happening in the project.


The web is moving more and more to a world without plugins. For me, this is a very exciting time, as plugins usually tend to weaken the browser security. There are two plugins bundled with Chromium to produce Chrome - Adobe Flash Player and a PDF viewer. The latter is now an open source project of its own - PDFium. It can be built and packaged with Chromium, though the same care should be taken as with the browser itself - keep it up-to-date.

Overall, maintaining a fork of Chromium isn’t trivial, but it isn’t impossible either. There are a bunch of examples, including the successful migration of the Opera browser from their own rendering engine to building on top of the Chromium content module.

Last, but not least - feel free to reach out and ask questions or advice.

Be humble

The topic of this blog post has been long on my mind, but I did not have a good example to use. Finally I found one.

Software security is a very complex field and a asymmetric problem space. Arguments whether defense or offense is harder have been fought for a long time and they will likely never stop. I think most of us can agree on those two statements:

  • offense needs to find only a handful of problems and work hard to turn them into a compromise
  • defense needs to architect software to be resilient and work hard to ideally (though not practically) not introduce any problems

Each side requires unique skills and is extremely rare for people to be really good at both. What really irks me is that lots of people in the security industry tend to bash the other side. It is easy, one understands their problem space very very well and knows how hard it is to be an expert. Also, it feels that the other side is not that hard. I mean, how hard can it be, right? Wrong!

In this post I will pick the side of the defender, since this is where I spend most of my time. The example I will use is the recent events with the Aviator browser, because it is near and dear to my heart. One thing I want to make clear from the get go - I totally respect their efforts and applaud them for trying. Forking Chromium is not a small feat and not for the faint of heart. The goals for Aviator are admirable and we definitely need people to experiment with bold and breaking changes. It is through trial and error that we learn, even in proper engineering disciplines :). What can we use more of the security industry?


It is no surprise people on the offensive side bash software developers for “stupid” mistakes, since the grass is always greener on the other side. The problem is that many trivialize the work required to fix those mistakes. Some are indeed easy. Fixing a stack-based buffer overflow is not too hard. In other cases, it is harder due to code complexity or just fundamental architecture of the code.

What humbles me personally is having tried the attack side. It is not too bad if you want to exploit a simple example problem. Once you try to exploit a modern browser, it is a completely different game. I admire all the exploit writers for it and am constantly amazed by their work. Same goes for a lot of the offensive research going on.

I have secretly wished in the past for some of the offensive folks to try and develop and ship a product. When WhiteHat Security released the Aviator browser, I was very much intrigued how it will develop. It is not a secret that Jeremiah Grossman and Robert Hansen have given lots of talks on how the web is broken and how browser vendors do not want to fix certain classes of issues. They have never been kind in their remarks to browser vendors, but now they have become one. I watched with interest to see how they have mitigated the issues they have been discussing. Heck, I wanted to see clickjacking protection implemented in Chromium, since it is the authors of Aviator that found this attack vector and I have personally thought about that problem space in the past.

Chris Palmer and I have played around with the idea of “Paranoid Mode” in Chromium and as a proof of concept we have written Stannum (source) to see how far we can push it through the extensions APIs. It is much safer to add features to Chromium using extensions than writing C++ code in the browser itself1. So when Aviator was announced and released initially, I reached out to WhiteHat Security to discuss whether the features they have implemented in C++ could be implemented through the extensions API. My interest was primarily motivated by learning what they have done and what are the limitations of the extensions subsystem. Unfortunately, the discussion did not go far :(.

Where do I believe they could have done better? You might have guessed it right - being humble. The marketing for Aviator is very bold - “the most secure and private Web browser available”. This is a very daring claim to make, hard promise to uphold and anyone who has been in security should know better. Securing a complex piece of software, such as a browser, is a fairly hard task and requires lots of diligence. It takes quite a bit of effort just to stay on top of all the bugs being discovered and features committed, let alone develop defenses and mitigations.

Releasing the source for Aviator was a great step by WhiteHat. It gives us a great example to learn from. Looking at the changes made, it is clear that most the code was written by developers who are new to C++. When making such bold statments, I would have expected more mature code. Skilled C++ developers that understand browsers are rare, but it is a problem that can be solved. It takes a lot of time, effort and desire for someone to learn to use the language and most importantly understand the architecture of the browser. Unfortunately, I did not see any evidence that whoever wrote the Aviator specific code did any studying of the source code or attempted to understand how Chromium is written and integrate the changes well.

What really matters at the end of the day, though, is not the current state of a codebase. After all, every piece of software has bugs. I believe there is one key factor which can determine long term success or failure:


Security vulnerabilities are a fact of life in every large enough codebase. Even in the project I work on we have introduced code that allowed geohot to pull off his total ChromeOS pwnage! We owned up to it, the bug was fixed and we looked around to ensure we did not miss other similar instances.

However, what I was most disappointed by was the reaction from WhiteHat when a critical vulnerability was found in the Aviator specific code:

“Yup, Patches welcome, it’s open source.” 2

Our industry would go further if we follow a few simple steps:

  • Do not trivialize the work of the opposite side, it is more complex than it appears on the surface.
  • When working on a complex software or problem, study it first
  • Share ideas and collaborate
  • Own up to your mistakes
  • Be humble

1. Even Blink is starting to implement rendering engine features in JavaScript.
2. Nevermind that there is no explanation on how to build Aviator, so one can actually verify that the fix works.

30 days with isolated apps in Chrome





  "name": "netsekure blog",
  "version": "1",
  "app": {
    "launch": {
      "web_url": ""
    "isolation": [ "storage" ]
  "permissions": [ "experimental" ]

The way to read the file is as follows:










Pass-The-Hash vs cookie stealing

I saw a few talks at the BlueHat conference at Microsoft and the funniest of all was Joe McCray’s (@j0emccray) “You Spent All That Money And You Still Got Owned????”. At some point, he touched on Pass-The-Hash attacks and asked why those can’t be prevented. That struck me as an interesting question and an analogy popped in my head:

“pass-the-hash attacks are functionally equivalent to cookie stealing attacks”

If you think about the pass-the-hash attack, it requires administrator privileges, which means you can get LocalSystem level privileges, at which point you own the operating system. Then you extract the user’s hash out of memory or from the SAM database and you inject them into the attacker’s machine. Then you rely on single-sign on built on top of NTLM/Kerberos to authenticate to remote resources.

What if we assume the following mapping: OS -> Browser, LocalSystem code execution -> Browser code execution, User’s hash -> User’s cookie, Single Sign On -> HTTP session with cookies?

It can be easily observed that the pass-the-hash attack is equivalent to attacker having code execution in the context of the browser, stealing the user’s cookies, injecting them into the attacker’s browser, and accessing remote resources. Actually, in the web world, one doesn’t even need code execution in the browser to steal the user’s cookies, it can be done through purely web based attacks.

Is it possible to defend against attacker using your cookies? It is extremely hard, because to the remote server, your cookie is *you*. From that perspective, a Windows domain is no different than web HTTP domain, so remote resources have no way of telling apart the real you and someone having your token, be it a password hash or a cookie. I haven’t gone through the thought experiment of mapping best practices for securing against cookie stealing attacks to see if those will nicely map into best practices for defending against pass-the-hash attacks, so I’d leave that as an exercise for the reader.

How to approach fixing the TLS trust model

TLS is an exciting protocol and its wide deployment makes it even more interesting to work on. It has been said many times that the success of online commerce is due to the success of SSL/TLS and the fact that people felt safe in submitting their credit card information over the Internet. These days a lot of people have been speaking openly about how broken the TLS trust model is because of its reliance on Internet PKI and the Certificate Authorities infrastructure and there is some truth to that. We have seen two cases already where CAs trusted by all browsers have issued fraudulent certificates for high profile sites. Those incidents revealed two key problems in the existing TLS infrastructure today:

  • Any CA can issue a certificate for any web site on the Internet, which I call “certificate binding” problem
  • Revocation checking as deployed is ineffective

To counteract these deficiencies, multiple proposals have either existed or emerged. The most notable two are using DNSSEC for certificate binding and Convergence, which is based on independent system of notaries. While both have merits, I believe both have sufficient problems that will prevent them from being deployed widely, which is required for any successful change.

Using DNSSEC for storing information about the server cert is actually very appealing. The admin is in control of which cert is deployed and controls the DNS zone for the site, so it makes sense to be able to use the DNS zone information to control trust. There is even a working group inside IETF to work on such proposal. The problems with DNSSEC are multiple though, here I list just a few:
  •  It is not as widely deployed yet, but that is being fixed as we speak
  • Client stub resolvers don’t actually verify the DNSSEC signature, rather rely on the DNS server to do the verification. This opens the client to attack, if a man-in-the-middle is between the client and its DNS server.
  • There is non-zero amount of corporate networks, which do not allow DNS resolution of public Internet addresses. In such environments, the clients rely on the proxy to do the public Internet DNS resolution. This will break DNSSEC based approach, as clients don’t have access to the DNS records.
  • DNSSEC is yet another trust hierarchy, which is not much different than the current PKI on the web, just a different instance.

Moxie Marlinspike has the right idea about trust agility and his proposal, which he calls Convergence, has a very good foundation. Where I believe it falls short is the fact that many corporate networks block outgoing traffic to a big portion of the Internet. Unless all notaries are white-listed for communication, traffic to those will be blocked, which will prevent Convergence from working properly. Also, the local caching creates a problem with timely revocation - if a certificate is found to be compromised, then until the cache expires, it will still be treated as a valid one.

My take
I actually don’t want to introduce any new methods of doing certificate validation. My goal is to point out a solution pattern that can be used to make any scheme actually deployable and satisfying most (if not all) cases. There are few basic properties any scheme should have:
  1. All information needed for doing trust verification should be available if connectivity to the server is available
  2. Certificate should be bound to the site, such that there is 1-to-1 mapping between the cert and the site.
  3. A fresh proof of validity must be supplied

There is already existing and deployed, although rather rarely, part of TLS called OCSP stapling. It does something very simple - the TLS server performs the OCSP request, receives the response, and then supplies that response as part of the TLS handshake, the last part being the most crucial. The inclusion of the OCSP response as a TLS message removes all of the network problems that the currently proposed solutions face. As long as the client can get a TLS connection to the server, trust validation data can be retrieved. This brings property 1 to the table. In addition, OCSP responses are short lived, which satisfies property 3 as well. So the only missing piece is the 1-to-1 property.

So, there are two ways the problem can be approached - either bring certificate binding to OCSP somehow, or use any other method to provide certificate binding. The latter can actually be achieved rather easily with minimal changes to clients and servers. RFC 4366, Section 3.6 describes the Certificate Status Request extension, which is the underlying protocol messaging of how OCSP stapling is implemented. The definition of the request message is:

      struct {
          CertificateStatusType status_type;
          select (status_type) {
              case ocsp: OCSPStatusRequest;
          } request;
      } CertificateStatusRequest;


The structure is extensible allowing for any other type of certificate status to be requested, as long as it is defined. I can easily see this message defining DNSSEC and Convergence as values of CertificateStatusType, then define the appropriate format of the request sent by the client. Conveniently, the response from the server is also very much extensible:

      struct {
          CertificateStatusType status_type;
          select (status_type) {
              case ocsp: OCSPResponse;
          } response;
      } CertificateStatus;

      opaque OCSPResponse<1..2^24-1>;

Currently, the only value defined is for OCSP response, which is treated as opaque value as far as TLS is concerned. Nothing prevents whatever information the above proposals return from being transmitted as opaque data to the client.

Just like Moxie explored in his presentation, using the server to do the work of retrieving trust verification data preserves the privacy of the client. It does put some extra burden on the server to have proper connectivity, but that is much more manageable and totally under the control of the administrator.

It is true that there will have to be change implemented by both clients and servers which that will take time. I fully acknowledge that fact. I do believe though, that using the Certificate Status Request is the most logical piece of infrastructure to use to avoid all possible network related problems and provide an inline, fresh, binding trust verification data from the server to the client.

One thing I have not yet answered to myself is - how do we make any new model to fail safe. Having hard failure and denying the user access has been problem forever, but if we keep on failing unsafe, we will continue chasing the same problem into the future.

So in conclusion, for any solution to fixing the TLS trust model must satisfy the following:
  • Provide timely/fresh revocation information
  • Work in all network connectivity scenarios
  • Preserve the client privacy
Ideally, it will also fail safe :)

TLS Client Authentication and Trusted Issuers List

One of the common questions I’ve seen asked lately is related to TLS client authentication, which likely means more people are interested in stronger client authentication. The problem people are hitting is described in KB 933430, where the message the server sends to the client to request client authentication is being trimmed. Let’s look at why this occurs and what are the possible solutions, but first some background.

When TLS server is configured to ask for client authentication, it sends as part of the handshake the TLS CertificateRequest message. The TLS 1.2 RFC defines the message as follows:

      struct {
          ClientCertificateType certificate_types<1..2^8-1>;
          DistinguishedName certificate_authorities<0..2^16-1>;
      } CertificateRequest;

where the supported_signature_algorithms is addition in the 1.2 version of the TLS protocol. The certificate_authorities part of the message is further defined:


When the server sends this message, it optionally fills in the certificate_authorities part of the message with a list of distinguished names of acceptable CAs on the server. The main reason for this list is for the server to help the client in narrowing down the set of acceptable certificates to choose from. For example, if the server only accepts certificates issued by the company private CA, there is no need for the client to send a certificate issued by a public CA, as the server won’t trust it. Nothing in the RFC prevents the client from sending any certificate, but it is in the best interest of the client to send appropriate certificate.

On Windows, the way the TLS is implemented, the server picks all the certificates that are present in the “Local Computer” “Trusted Root Certification Authorities” store (or in short the local machine root store). With Windows Server 2008 and later, the default list of trusted authorities is very small as I’ve described in a previous post, so including those distinguished names in the message does not pose a problem. However, if the server has most of the trusted roots installed or has additional root certificates, it is possible for the combined length of the distinguished names to exceed the limit of the TLS record size, which is 214 (16384) bytes. The TLS standard supports this, as it breaks messages up into records and a single message can span multiple records - this is called record fragmentation. Windows does not implement this part of the RFC though, so it cannot send messages that are bigger than what a TLS record can hold. In most cases this works just fine, but in this particular instance it is a problem.

Now, what can be done to solve this problem. The are two solutions - either decrease the list of root certificates in the message or do not send that list at all (allowed by the RFC). The former approach is possible, but is more error prone and I wouldn’t recommend it for most people. I would argue that the latter approach is the preferred one, but I always get backlash when I propose this. If one were to think about the original purpose of this message and the way Windows has implemented this, it will be easier to understand why this is better. Remember:

  • the server wants to “help” the client do an informed decision on which certificate to send as its identity
  • Windows sends the contents of the local machine root store as the list

If we take those two factors together, the end result is that the server is not helping *at all* the client to make a good decision.  A few sources - my own research,’s SSL Survey, and the EFF SSL Observatory - point out that 10-15 root CAs issue the majority of the certificates seen on the web, therefore if we send 16k worth of these, the probability that any certificate the client has is *not* issued by someone in the list is close to zero. Therefore, in this configuration, the list the server presents to the client doesn’t effectively filter down the set of certificates on the client. It is almost equivalent to sending an empty list to the client and ask it to chose randomly.

In short, if you are hitting this problem, you are better of using Method 3 described in KB 933430 and setting SendTrustedIssuerList to 0, which disables sending of the list of CAs than any other method.

Automatic CA root certificate updates on Windows

I was recently listening to Chris Palmer talking about SSL on the PaulDotCom podcast and one thing caught my attention – the discussion on IE behavior with trusted roots certificates. It was discussed that IE is violating the “No-Write-Up” policy of the integrity level (IL) mechanism in Windows. While the end effect looks like it, the inner workings of how this is accomplished is more complicated and the behavior is not restricted to IE.

Let’s start with some preliminary background information:

  • Certificates are validated through a process of building a chain up to a trust anchor, the underlying of which is constructing a graph of nodes (certificates) and edges (issuance relationships) and then the graph is traversed to find all possible complete chains (in most cases just one). The smaller the graph is, the quicker it is to find a complete chain. Once the chain is complete, the certificate at the root of the chain is checked for trust. If the chain ends in a certificate present in the list of trusted root certificates and all other verifications pass, the certificate validation is successful.
  • Microsoft has a specific program called “Microsoft Root Certificate Program”, which is how certificate authorities (CAs) submit their root certificates for inclusion in Windows. The end result of this program is a *fixed* list of root certificates that Windows considers trusted. The entire list is available on TechNet and is updated whenever there are any changes. This list (or the equivalent of it at the time) was present in full in the root certificate store in Windows XP and earlier, but starting with Vista, this default list in the root certificate store is much smaller in order to increase performance while validating certificates. If a chain ends up in root certificate which is part of the Root Program but is not present in the list of trusted roots currently on the machine, Windows downloads the appropriate root certificate directly from Windows Update. The full process for all versions of the OS is described in a KB article. I’m not going to rehash the explanation of how it works, but the key point is that only those certificates accepted through the root program will be downloaded from Windows Update.

The IE low integrity processes are not instructing the broker to do anything, it all happens under the hood in the crypto APIs. Now, here is why the IE behavior is observed. You go to a web site, which certificate is issued by a CA not yet in your trusted root list. Because IE uses the standard cryptography API that Windows provides, when certificate validation is performed, Windows itself (not IE, nor its broker process) goes and fetches the root certificate from Windows Update *if* that certificate is part of the Root Program. The same behavior will be seen for any program that is using the same API to do certificate validation. Chrome as far as I know uses the Windows crypto APIs to do certificate validation and relies on the trusted roots list from Windows, so if you browse with Chrome, you will see the exact same thing happen (though I haven’t verified it personally).

Some people have asked me how they can prevent Windows from going to Windows Update in such cases. This can be achieved by disabling automatic root update through policy as described on TechNet. It should be noted that one should exercise caution in doing so, because disabling root update means that Windows will no longer manage certificate trust for you. You will have to manage the set of trusted root certificates on your own. To that end, import the full list of CAs part of the Root Program such that those are available in the list of trusted roots. Microsoft provides a small program – rootsupd.exe, as part of KB931125 which accomplishes this task. With this approach, you have control over trust management, but you need to keep the list updated whenever the set of roots in the root program changes. Microsoft has a wiki page which includes the information about new certificates that are part of an update. This page has RSS feed available which you can subscribe to so that you are notified of new updates, which makes keeping track of the updates an easy job.

I hope this helps explain how this all works and clarify that there is no real violation of the No-Write-Up policy, even though it might seem like it from high level.