Saturday, December 10, 2016

Mystery of the Misumena fidelis Crab Spider

Mexico and the United States are missing a crab spider. It's not the most inconspicuous of spiders, either, being one we should see on flowers. Read on to see how biological taxonomy is sometimes detective work about the names and shapes of things...


A fortuitous series of circumstances, along with some sleuthing, reveals that Misumena fidelis Banks 1898 is properly Mecaphesa fidelis (Banks). It is very likely synonymous with Misumenops volutus F.O. Pickard-Cambridge 1900. It may or may not also be synonymous with Misumena decora Banks 1898, which Gertsch 1939 might have mistakenly synonymized with Misumenops volutus. An examination of the Misumena decora type specimens should resolve that. Mecaphesa fidelis ranges along the western half of Mexico, south to Guatemala, and possibly north into southern Arizona.

Last Seen in 1901

Cotype of Misumena fidelis Banks 1898, from MCZ

There are supposed to be two species of Misumena crab spider (family thomisidae) north of Mexico. Misumena vatia ranges across most of the United States and much of Canada. It's also found in Europe and northern Asia. It's rather large for a crab spider and makes for some great photos. The other spider, Misumena fidelis, is supposed to be in southern Arizona and western Mexico.

The problem is that only one person has ever knowingly seen Misumena fidelis. That person was Nathan Banks, who described and named the species in "Arachnida from Baja California and other parts of Mexico," published in the 1898 Proceedings of the California Academy of Sciences. Banks reported "several specimens from El Taste, La Chuparosa, and San José del Cabo." The first and last of these locations is in southern Baja California. La Chuparosa appears to be in Chihuahua in northern Mexico.

Banks reported seeing the species again in "Some Spiders and Other Arachnida from Southern Arizona," published in the 1901 Proceedings of the United States National Museum. This time he saw "a female from Catalina Springs" in Arizona, USA.

And that was it. The species has never been found again. Or has it?

Type Specimen on Hand

While researching for a new key to the genera of crab spiders north of Mexico, I happened across two specimens that Banks himself had labeled Misumena fidelis. In fact, Banks designated these specimens as "types" for the species, meaning that these specimens are considered to define the group of spiders that are to be called Misumena fidelis. (Technically, it's a "syntype.") Any other spider I find that has the characteristics of these spiders should then also be Misumena fidelis.

Misumena fidelis Banks, adult female

Misumena fidelis Banks, subadult female

One of the type specimens is an adult female. The other is a penultimate female, which means that she's one molt shy of being an adult. The adult has a body length (cephalothorax + abdomen) of 7 mm. Both were in a single vial from the Harvard Museum of Comparative Zoology, collected from El Taste in Baja California by Eisen and Vaslit.

Banks' 1898 paper does not describe males, so he apparently only had females. Male spiders are typically more distinctive, so it's an added challenge that we have to solve the mystery from females. Typically only the adults of a species are distinctive, so we'll have to do this from a single female.

With this type specimen, we can now scour museum collections to find other records of the species and see what else we can learn about the spider. But there are lots of spiders in collections. Where do we begin looking? The best we can do is guess how others may have labelled spiders of this species and double-check the specimens so labelled. This all depends on what the spider looks like.

Looks like Mecaphesa or Misumenops, not Misumena

The first thing that jumps out at me is that Misumena fidelis has heavy spines (macrosetae) on the carapace and abdomen. Actually, they've all broken off the abdomen, but the spine bases are still present, so it's clear she had them while alive. This jumps out at me because one of the most distinctive characters of our well-known Misumena vatia is that they have no such spines on their body, though they do sometimes have them around the eyes.

Misumena vatia's dearth of body spines are apparent in these two beautiful photos. (Okay, I couldn't post such a long, technical blog without including some fun-to-look-at photos.)

Misumena vatia. Photo courtesy of Sean McCann

Misumena vatia. Photo courtesy of Christy Pitto.

Let's compare the carapaces (tops of their heads) more closely. Here we more clearly see that Misumena fidelis is distributed with robust carapace spines, while Misumena vatia is not.

Female Misumena fidelis, having carapace spines

Female Misumena vatia, lacking carapace spines

Several other North American genera of crab spiders are known for having spinose (spiny) bodies, so it may belong to one of those instead. Genera are human constructs intended to approximate real-world relationships, so our first step is to see how we humans have defined the various relevant groups. This looks like a misumenine spider, so we'll restrict ourselves to those genera. In North America, the misumenine spiders are mainly the flower crab spiders.

Once upon a time it was fashionable to call all New World misumenine spiders Misumena.  In 1900, F.O. Pickard-Cambridge divided Misumena into Misumena and Misumenops, mostly relegating any spider whose ALEs (anterior lateral eyes) were larger than the AMEs (anterior median eyes) to Misumenops. Misumena vatia eyes are usually the same size, so Misumena vatia remained in the genus Misumena, while many other crab spiders moved to Misumenops. Notice from the first photo in this blog that this 1900 rule would have moved Misumena fidelis to Misumenops.

Misumenops didn't get revised again until 2008, this time by Lehtinen & Marusik. Their paper divided Misumenops into several more groups. As with most arthropods, the most distinctive character of a spider is its genitalia, so it's not surprising that the paper largely divides out new genera by genitalia. The next step for us is therefore to look at the Misumena fidelis female genitalia—the epigynum.

Misumena fidelis epigynum, ventral view

Misumena fidelis epigynum, angled posterior view

One of the groups that Lehtinen & Marusik separated from Misumenops was Mecaphesa, and this epigynum looks very much like Mecaphesa to me. It also has some similarity to the Misumenops epigynum. For completeness, we also compare the epigynum with that of Misumena vatia.

Left and right: Two Mecaphesa dubia specimens, ventral epigynum

 Left: Misumenops bellulus, ventral epigynum
Right: Misumena vatia, ventral epigynum (Photo by Nicole Miller)

The Misumena fidelis epigynum looks nothing like the Misumena vatia epigynum, except for apparently having a small central hood-like structure. Instead, the overall look is suggestive of Mecaphesa, particularly the upward narrowing sweep of the central epigynum. However, the apparently small hood on Misumena fidelis seems more suggestive of the small hood on Misumenops. None of the other genera have epigyna that look so similar to Misumena fidelis, so our working assumption is that our spider is either Mecaphesa or Misumenops. We don't know which yet.

Mind you, that little hood on Misumena fidelis is not quite what it seems.

Superficially neither Mecaphesa nor Misumenops

Let's dive into the details of what it means for a spider to be Mecaphesa or Misumenops. Lehtinen & Marusik 2008 provide the latest definitions. For now, we'll stick with what we can see externally on the spider, without dissecting any spiders.

Both Misumenops and Mecaphesa have robust spines on the carapace and abdomen, as Misumena fidelis does. There are strong distinctions in the male genitalia, but we aren't looking at a male (yet!). On the female's epigynum, Misumenops has a small hood, while most Mecaphesa have a median septum. The median septum is a structure that runs for at least a portion of the front-to-back length of the epigynum, along the center of the epigynum.

Now it's time to take a closer look at the Misumena fidelis epigynum. Photos don't always convey the 3-D structure well, so here are some drawings. The first drawing shows a strictly ventral view. The second drawing is what my mind tells me how a cross-section of the epigynum would look, based on my examination of the epigynum from various angles under a microscope. The second drawing should be schematically correct if not proportionally accurate.

Misumena fidelis cotype, ventral epigynum
Misumena fidelis, imagined epigynal cross-section

In these drawings, 'CO' refers to a copulatory opening where the male inserts an embolus (a penis-analogue) to inseminate the female. I believe all spider epigyna have two copulatory openings. There is a huge cavity at the anterior of the epigynum, here labelled 'cav'. There are two hoods, one part of the other. The apparently-small central hood we noticed previously is actually part of a much larger hood. 'IH' refers to the small inner hood, and 'OH' refers to the large outer hood.

It is common in Mecaphesa for the hood to span the width of the epigynum, as we see here. We call the broad depression at the center of the epigynum the 'atrium'. The outer hood here spans and encloses the anterior atrium. This seems to rule out Misumenops. However, our spider also does not have a median septum that Lehtinen & Marusik 2008 says we can use to identify most Mecaphesa. Instead, the middle atrium is flat, gradually sloping into the anterior cavity.

Lehtinen & Marusik 2008 provides a clearer distinction for females using features of the dorsal epigynum, which can only be seen by dissecting the spider. We don't want to dissect a type specimen if we don't have to. Fortunately, we have a clear enough understanding of the epigynum at this point to go looking for other Misumena fidelis specimens. So that's what we'll do. For now, we're left still not knowing whether the spider is Mecaphesa or Misumenops.

More Misumena fidelis Specimens Found

It didn't take long for me to find additional females whose genitalia exactly match the Banks type specimen. I had sorted out Mecaphesa-like spiders that I was not familiar with. Among them was a vial from the Canada National Collection containing two adult females and one adult male, collected from Xechi milco in the Federal District of Mexico by H.E. Milloron in 1962. The structure of the epigyna of both females was identical to the Banks type specimen for Misumena fidelis:

Misumena fidelis, ventral epigyna, two females from CNC

Misumena fidelis, ventral epigynum schematic

The spiders themselves are also quite similar-looking. Recall that the Misumena fidelis once had the abdominal spines, because it still has the spine bases. One of the CNC females has a body length of 7 mm, the other a length of 6 mm.

Misumena fidelis females. Left: Banks cotype. Center and right: CNC specimens.

The vial was labelled Misumenops decorus, because the genitalia of the male in the vial exactly matches the drawing for Misumenops decorus given in Gertsch 1939. The male has a body length of 3 mm. Lehtinen & Marusik 2008 renamed Misumenops decorus to Mecaphesa decora.

Male accompanying female Misumena fidelis, frontal view

Male accompanying female Misumena fidelis, dorsal view

The females in this vial were probably identified as Mecaphesa decora because they were found with the male. That seems like a reasonable assumption, but there's a small problem: the female epigyna look little like the only drawing that exists for the Mecaphesa decora epigynum:

Misumena decora Banks 1898, ventral epigynum
(Banks 1898 Plate 16, Figure 13)
Left and right: Misumena fidelis from CNC, ventral epigynum

Banks provided the drawing of Mecaphesa decora (then Misumena decora) in the same 1898 paper that described Misumena fidelis. These are the original descriptions of the species, so they are considered definitive. Therefore, according to Banks, who named these species, these additional CNC females are Misumena fidelis. If Banks' drawings for Mecaphesa decora are accurate, we can further say that these females are not Mecaphesa decora. It's still possible that Banks was mistaken to think that Misumena fidelis and Mecaphesa decora are different species, but the strong differences in the epigyna drawings make this seem improbable, at least for the females.

So we appear to have a situation in which two female Misumena fidelis were found with a male Mecaphesa decora. But is it really a male Mecaphesa decora?

Identifying the Accompanying Male

We'll identify the male from scratch, carefully heeding the history of North American thomisid descriptions. Banks did not know what the male Misumena fidelis looked like, and no one has since (knowingly) described the male, so this investigation cannot yield a match for Misumena fidelis.

The male genitalia that are distinctive are the enlarged "bulbs" at the ends of the pedipalps. Traditionally, we just refer to these as the "palps." Arachnologists typically draw the palps of the males of new species, so for our investigation, we can scan the literature for palps that look like those on our male. There are a lot of spiders in the literature, so we'll limit the scan to North American thomisids that have ever been called Misumena, Misumenops, and Mecaphesa.

The earliest definitive match for the palp that I can find in the literature is Misumenops volutus, described by F.O. Pickard-Cambridge in a 1900 paper. We normally draw a strictly ventral view of the palp and maybe also an outside (retrolateral) view, but neither of these views provided a match. Instead, there is a great match in a view that is partially ventral, distal, and retrolateral:

Misumenops volutus palp. Left: ventral view. Right: retrolateral view

 Misumenops volutus palp, view angled to match F. O. Pickard-Cambridge 1900 Plate 10, Figure 3.

Okay, now we have a male Misumenops volutus in a vial with two Misumena fidelis that are all together identified as Mecaphesa decora. That sounds crazy, but we can explain some of this. In his 1939 paper, Gertsch decided that Banks' Misumena decora and Pickard-Cambridge's Misumenops volutus were actually the same species, renaming both to Misumenops decorus, which appears on the vial label. Lehtinen & Marusik moved Misumenops decorus to Mecaphesa decora in 2008.

Let's revisit Gertsch's decision to synonymize Misumenops volutus with Misumena decora. Gertsch 1939 reports, "Cotypes of Misumena decora Banks from Mexico, all destroyed except one male and one female in the Museum of Comparative Zoology." But Gertsch 1939 only describes the male and only lists six male records, no female records. Moreover, F. O. Pickard-Cambridge 1900 only described the male of Misumenops volutus, not the female. So it appears that Gertsch made this decision to synonymize these species entirely based on the male.

In order for Gertsch to decide that the Misumenops volutus male was the same as the Misumena decora male, he would have had to compare their palps. Because Gertsch 1939 does not describe the female and does not list any female records, it seems reasonable to conclude that he did not examine the male and female cotypes he mentioned; and if he didn't examine the cotypes, then in particular he didn't examine the male Misumena decora cotype that Banks designated. Odds are that his basis for the synonymy was a comparison of the Banks 1898 drawing of Misumena decora with obvious male specimens of Misumenops volutus that he could examine.

Left: Misumenops volutus palp, distal-retrolateral view.
Right: Misumena decora Banks 1898 palp, unknown view (Banks 1898 Plate 16, Figure 13).

Let's do this comparison ourselves. The above-right drawing is Banks' illustration of the Misumena decora palp. Notice that the embolus appears to originate from the center of the tegulum. (That is, the tapering black thing originates from the center of the large, smooth round thing). This would normally be a distinctive character, but no view of the Misumenops volutus palp presents this way. Instead, the above-left photo is the closest I could get to presenting this apparent character.

There is enough similarity to make it conceivable that Banks intended to draw a Misumenops volutus palp. Perhaps his microscope didn't give him a very crisp view. Just as no two people are physically identical, no two spiders are physically identical, so perhaps it's possible for a palp of this species to sometimes looks more like Banks' drawing. Because this remains unclear, it seems that we can't have a lot of confidence in Gertsch's synonymy of Misumenops volutus with Misumena decora.

To properly resolve this question, someone will have to examine Banks' Misumena decora type specimens; I do not have them on hand. For now, I'm not willing to say that Misumenops volutus is Misumena decora. The palp is only an iffy match, and the females found with a male Misumenops volutus strongly match Misumena fidelis but not Misumena decora. All I'm willing to say with confidence is that the vial has a male Misumenops volutus and two female Misumena fidelis.

But could Misumenops volutus really be Misumena fidelis?

The Most Probable Situation

Taxonomic understanding is imperfect and constantly improving over time. We can only state the most probable situation for the available facts. In our case, three situations seem possible. Let's state these situations without referring to the possibly-invalid species name "Mecaphesa decora":
  1. Misumenops volutus is the same species as Misumena fidelis, with Misumena decora being a different species; or
  2. Misumenops volutus is the same species as Misumena fidelis, Banks' male Misumena decora is actually a male of this species, and Banks' female Misumena decora actually belongs to some other species; or
  3. Misumenops volutus, Misumena fidelis, and Misumena decora are all the same species.
The first two possibilities seem the most probable for the following reasons:
  • The Banks 1898 drawing of the Misumena decora epigynum looks significantly different from the epigynum of one of the type specimens he designated.
  • The Banks 1898 drawing of the Misumena decora palp is not obviously identical to the Misumenops volutus palp, though it may fall within the limits of variation.
  • The male Misumena fidelis was unknown to Banks, the female Misumenops volutus was unknown to F. O. Pickard-Cambridge, and an adult male of Misumenops volutus was collected with two adult females of Misumena fidelis.
Adult male spiders typically spend their time seeking out females of their species to mate with, so the immediate proximity of the male and female specimens could be informative. Unless you're observing mating behavior, it's hard to tell which males go with which females except by their mutual proximity. However, because we only have one data point in our case—one occurrence of a male with females—we can't have a lot of confidence in the association on this basis alone.

In support of cases (2) and (3), there is evidence that Banks' Misumena decora palp might be an inaccurate drawing. Here is the epigynum of one of the very spiders that Banks drew, next to his drawing of that spider's epigynum. Banks clearly was not able to see all the detail that we can now see.

Misumena fidelis, ventral epigynum. The photographed specimen is one of those
from which Banks made the drawing on the right (Banks 1898 Plate 16, Figure 2).

It is still possible that Misumenops volutus and Misumena fidelis are different species, but this scenario creates an improbable coincidence. It would mean that we found a male of a species (Misumenops volutus) for which the female is unknown with two females of a species (Misumena fidelis) for which the male is unknown, and yet after this finding, the female and male counterparts remain unknown. Instead, it seems that the finding should change our working assumptions.

An examination of two remaining Misumena decora type specimens should select among the three possibilities. I'm reluctant to borrow the only type specimens in existence for a species, but maybe someone at the Harvard Museum of Comparative Zoology could check this out and report back.

Misumena fidelis is a Mecaphesa

We found that the external appearance of our Misumena fidelis didn't allow us to decide which genus it belonged to according to modern definitions. Recall that we had narrowed it down to either Misumenops or Mecaphesa. Now that we have three female Misumena fidelis, only one of which is a type specimen, we are freer to dissect one. I've chosen the CNC specimen whose ventral epigynum is virtually indistinguishable from that of the type specimen.

Internally, spider epigyna consist of sclerotized (hardened) structures. These internal structures are often the most distinctive character of females of a species. The dorsal view of an epigynum is the view of these structures from their internal side. In crab spiders, the most conspicuous parts of the dorsal epigynum are the spermathacae, which store sperm received from males.

Here is are the dorsal epigyna of two species of Mecaphesa and one species of Misumenops, for comparison with a Misumena fidelis epigynum. The large boxy structures are the spermathacae.

 Left: Mecaphesa celer, dorsal epigynum. Right: Mecaphesa dubia, dorsal epigynum.

Left: Misumenops pallidus, dorsal epigynum. From Lehtinen & Marusik 2008, fig. 9.
Right: Misumena fidelis from CNC, dorsal epigynum.

Lehtinen & Marusik 2008 says that the Misumenops spermathecae are "tubular U-shaped," as we see with the drawing at the lower-left. It also says that the Mecaphesa spermathecae have "large compact basal parts," as we see with the top two images. The Misumena fidelis dorsal epigynum at the lower right is not tubular-looking but has basally large spermathacae, matching it with Mecaphesa.

We can definitively say that Misumena fidelis now belongs to the genus Mecaphesa.

However, this was my suspicion before the dissection, so there may be something about the ventral epigynum that allows us to make this determination without dissection. It seems to me that the copulatory openings of Misumenops are always lateral to the hood, while the copulatory openings of Mecaphesa are always posterior or interior to the hood. In the species of Mecaphesa that appear to have multiple hoods, the relevant hood is the broadest median hood at the anterior of the epigynum. This rule seems to work for all the Misumenops and Mecaphesa I have so far seen, whether as drawings or actual specimens. However, I have not seen many species of Misumenops.

In particular, Misumena fidelis makes it clear that not all Mecaphesa epigyna need have a median septum or anything resembling a median septum. Had the copulatory openings of Misumena fidelis been closer together, their margins might have been construed as the margins of a median septum. It's hard to argue that they form a median septum when they are as far apart as they are in M. fidelis.

Lehtinen & Marusik 2008 also says that Misumenops has a "mostly small epigynal hood" that is "widest in Misumenops bellulus." The inner hood of Misumena fidelis looks small, but the outer hood containing it does not. The size of outer hood might have been enough to rule out Misumenops, but it could also have been the case that there's a Misumenops having a wider hood than Misumenops bellulus. Relative sizes probably aren't a stable diagnostic over the long term.

Misumena fidelis is Mecaphesa fidelis

Now that we know Misumena fidelis is actually a Mecaphesa, we can ask what the full species name should be for this spider. Mecaphesa what?

The name given to the first description of a species is the name that takes precedence. Our names of concern are Misumena fidelis, Misumena decora, and Misumenops volutus. Banks described both Misumena fidelis and Misumena decora in the same 1898 paper. F. O. Pickard-Cambridge described Misumenops volutus in a 1900 paper, so Banks' names take precedence.

We have three scenarios to consider, according to our prior probable situations:
  1. Misumenops volutus is Misumena fidelis, but Misumena decora is another species. In this case, the modern name of the species is Mecaphesa fidelis.
  2. Misumenops volutus is Misumena fidelis, as is Banks' male Misumena decora, while Banks' female Misumena decora is another species. In this case, we could call the species either Mecaphesa decora or Mecaphesa fidelis. However, if we call it Mecaphesa decora, Banks' female Misumena decora would no longer have a species name, and nothing would have the name Mecaphesa fidelis. To ensure that the first to name and describe a species actually got to name the species, it seems that we would give the name Mecaphesa fidelis to Misumena fidelis and restrict Misumena decora to just the female that Banks' described.
  3. Misumenops volutus, Misumena fidelis, and Misumena decora are all the same species. In this case we could name the species either Mecaphesa decora or Mecaphesa fidelis. However, because there's a chance that someone may later prove this interpretation incorrect, it may be wiser to allow for the possibility that Banks' female Misumena decora could have its own name, thus again suggesting the name Mecaphesa fidelis.
In all three scenarios, the best modern name for Misumena fidelis appears to be Mecaphesa fidelis. Moreover, the spiders that we have been calling Mecaphesa decora since Banks are also more properly Mecaphesa fidelis, and Misumena decora becomes more of a mystery.

The Range of Mecaphesa fidelis

These synonymies tell us more about the range of this spider, because the species has been collected under other names. Banks is the only one to have reported collecting a female, so we can assume that all of the male palps match Misumenops volutus and hence our presumed male Mecaphesa fidelis.

A proper range compilation would require examining specimens from Mexican collections. We'll keep things simple and report the minimal known range by combining the records of Banks 1898, F.O. Pickard-Cambridge 1900, Banks 1901, Gertsch 1939, and our three CNC specimens.

Mecaphesa fidelis ranges along the Sierra Madre Occidental mountain range in Mexico, westward to at least southern Baja California, southward to Guatemala, and possibly northward into southern Arizona. The records from Mexico are as follows: El Taste; La Chuparosa; San José del Cabo; Lake Chapala, Jalisco; La Buena Ventura, Veracruz; Pedregales; and Xechi milco, Federal District. There is also one record from Guatemala and one record from Catalina Springs, Arizona, USA.

Gertsch reports the Arizona record from a single female despite not having a clear understanding of the epigynum, so we probably shouldn't be confident in that record. Mecaphesa fidelis certainly ranges through much of Mexico but may or may not range into the southern USA.

Work to be Done

The following work remains to be done to resolve outstanding questions about Mecaphesa fidelis:
  1. Examine Banks' type specimens for Misumena decora to see whether they match the male or female Mecaphesa fidelis. These specimens are probably at the Harvard Museum of Comparative Zoology. They should select among our three probable scenarios.
  2. Locate the female specimen that Banks reported for Arizona in 1901 and identified as Misumena fidelis. If that specimen matches Mecaphesa fidelis, we can definitively include southern Arizona and the United States within its range.
  3. Search collections of misumenine thomisids from southern Arizona and maybe southern New Mexico to attempt to locate other specimens of Mecaphesa fidelis.
  4. Collect misumenine thomisids from southern Arizona and New Mexico in search of this species. At present, identifying this species requires examining the genitalia under a microscope, so they would have to be collected and preserved for examination.
  5. Locate drawings or specimens that match Banks' 1898 drawing of the Misumena decora epigynum. There may yet be a match for this spider other than Mecaphesa fidelis.


Banks, N. (1898b). Arachnida from Baja California and other parts of Mexico. Proceedings of the California Academy of Sciences (3) 1: 205-308.

Banks, N. (1901). Some Spiders and Other Arachnida from Southern Arizona. Proceedings of the United States National Museum XXIII: 581-590.

Gertsch, W. J. (1939b). A revision of the typical crab spiders (Misumeninae) of America north of Mexico. Bulletin of the American Museum of Natural History 76: 277-442.

Lehtinen, P. T. & Marusik, Y. M. (2008). A redefinition of Misumenops F. O. Pickard-Cambridge, 1900 (Araneae, Thomisidae) and review of the New World species. Bulletin of the British Arachnological Society 14: 173-198.

Pickard-Cambridge, F. O. (1900). Arachnida - Araneida and Opiliones. In: Biologia Centrali-Americana, Zoology. London 2, 89-192.

Friday, December 9, 2016


(I originally wrote this story on September 29th, 2001, but shortened it a bit over the years.)

"Dang roaches, get out of my life!" The old man coughed and coughed. From where he lay on his hospital bed, he pointed his cane at the floor and expertly crunched a cricket.

The machine beside the bed exploded in frantic beeping. "Please, you must rest," the nurse said. "That was only a cricket."

"Hmph! I hate bugs. All of 'em!"

The man's chest heaved, and he gasped a sudden heavy gasp. The machine beeped at the nurse, the nurse called the doctor, and the man passed away.

The man woke. He breathed easily, feeling well rested, as if from seventeen years of sleep. It was dark, the air was stale, and it smelled like dirt.

"Hmph. Figures," he thought to himself.

He began to scratch at the dirt overhead. Progress was slow but steady. Now and then he would pause, look down, and muse, "Yup, shoulda been a hole digger."

Finally he broke through the surface into fresh air. After resting, he ambled over blades of grass and deftly climbed the nearest tree.

Part way up the trunk he stopped. There, unbeckoned, he molted, sloughing his cuticle skin. "Hmph! Figures," he thought, and then flew away on cicada wings.

Friday, July 22, 2016

Weed Rage

(I originally wrote this on August 6, 2008. I've edited a little for posting today.)

I hunkered down in the garden and uprooted a weed. "Unh!" A root nearly two inches thick! An adjacent weed had unbelievable two-foot leaves. I tugged and tugged and up it came, revealing a dozen weeds underneath. I yanked and yanked and yanked.

Still more monstrous weeds! Heaving, heaving, they came up one by one. But now there were baby weeds and sister weeds and even grandfather weeds. I pulled and yanked and tugged. "Hmph! Bluh! Hargh!"

I didn't see the morning go, and I didn't notice the long shadows drawing. Instead I squatted and lifted and grabbed and shovelled and tugged and jumped. Spitting dirt and raining dirt, I watched arms and legs flail. I watched weeds fly. "Ahg! Unh! Grr!"

Unbelievable! This weed had a stalk eight inches thick and it was covered in bark. I whacked at it with the side of my trough. I whacked and whacked and whacked, my grunting now as frenzied as a chainsaw.

Finally the behemoth fell. Thunk. The ground shook.

I looked around, panting madly. My knees weakened at the sight. Flowers, bushes, trees and weeds alike lay leveled flat.

Saturday, June 18, 2016

The Unspoken Vulnerability of JWTs

JSON Web Tokens (JWTs) are the new thing. Blog after blog and book after book tell you how to generate them and use them to authorize access to web services. But there is one little detail that everyone is leaving out: it is much harder to secure a server that generates JWTs than a server that generates session IDs. This is because the JWT signing key must be protected, whereas there is little need to secure session IDs, and session IDs are easily secured by hashing, anyway. As a consequence, the push to use JWTs for local authentication is making sites more vulnerable.

Here you might dismiss me as a random loon for questioning the JWT love, but I do have years of experience professionally evaluating systems to assess and document their security. I've performed IV&Vs for NSA, evaluated NetWare's file system for a TNI Class C2 rating, and developed a reputation for being able to quickly identify security flaws in large software systems. Mind you, that was COMPUSEC, not INFOSEC, and that was years ago, but I like to think some skill remains.

A Convenient Assumption

It seems that many developers believe that if a server is ever breached, then the attacker is able and willing to do anything with the server. Under this assumption, there is no need to protect JWT signing keys because the attacker can add sessions, change keys, or obviate session and signature checks and directly manipulate software and data. This assumption gives developers permission to do whatever is most convenient with the keys, because all is apparently lost with a breach anyway.

I was unable to find statistics indicating what percentage of breaches result in attackers modifying software or data, so until such statistics show, let's look at this logically. First, accounts can generally read more data than they can write, so if attacks track account distribution, attackers will generally be able to read more than they can write. Second, software is fickle, so attackers who want to remain undetected are ill-advised to attempt to modify software or data, particularly when they don't have access to the source code. Attackers are better off quietly hijacking accounts that can do what they want done. Finally, if the breach is of an authentication system that is isolated from client services, the attacker has not yet breached client data. If this system does not expose credentials (e.g. passwords or signing keys), the attacker is forced to either patch the software or change credentials, one of which is risky and the other of which users or database triggers can detect.

In short, it's neither clear that breaches normally give attackers unrestrained access nor clear that attackers who wish to remain undiscovered would take full advantage of unrestrained access anyway. Besides, if after a breach the attacker finds a key that provides easy, unchallenged access to APIs as any user, why would the attacker risk directly mucking with application behavior or data? It seems that guarding against attacks that only steal data might be valuable after all.

The Vulnerable Scenario

Consider this scenario. You have deployed a server that authenticates users by username and password. Your server generates a JWT on successful authentication, signing the JWT with either a symmetric or asymmetric key. The client hands the JWT to web services to authorize access. Each web service verifies the signature, and if the JWT is valid and unexpired, allows the client to access the service as the user identified in the JWT payload. The payload may also specify the permissions granted to the user (e.g. "scopes"), but our scenario does not require that permissions be present.

Now suppose an attacker breaches the authenticating server and we don't learn about the breach right away. This risk always exists. There are many different ways to breach, but we do know that it is common for the attacker to gain access to sensitive data, such as usernames and passwords. We're not particularly worried about passwords though, because our server has hashed them.

But the server has not hashed the signing key. It cannot hash the signing key, because the original key itself must be used for signing. Without taking measures that exceed what we normally do to protect passwords, the attacker may readily acquire the signing key. Maybe we thought to protect the key and split it between a file and the database, or maybe we stored the key encrypted in the file system. In these cases, both key halves might still be as readable as the password hashes, and the key for decrypting the encrypted key might be as readable as the encrypted key. Simple measures for securing the key are likely insufficient. In our scenario, insufficient measures were taken.

Specifically, we assume that the attacker has acquired the JWT signing key.

Our scenario is taking advantage of the most touted benefit of JWTs: by trusting the payload claims of a JWT, each server is spared from having to hit the database to authorize each request. When load balancing, the servers of a cluster are also spared from having to cache user data and from having to dedicate themselves to clients for "sticky" sessions. If we were not trusting the payload claims, we would be ignoring them and treating the JWT as an opaque token or session ID.

Exploiting the Scenario

Once the attacker has a signing key, the attacker can forge unexpired JWTs forever—or at least until the breach is detected and keys are changed. Having seen the data files, the attacker probably has user IDs and permission codes and so can put together functional JWTs. The attacker may therefore be able to access any web service as anyone, and no one will know until after the damage has been done.

It's possible to detect attacks by confirming IP addresses and device names, but servers that do so will not be benefiting from the touted scaling benefit of JWTs, because these confirmations require hitting the database. To scale, servers must trust the payloads of properly signed JWTs. Besides, the IP addresses of mobile clients dynamically change as the devices switch from network to network.

The exploitation exists even when using refresh tokens. It's common for an authentication server to issue both refresh tokens and access tokens (which are both JWTs). A refresh token controls the maximum period of client inactivity, while an access token controls the maximum period of unchecked access. When the access token expires, the client requests new refresh and access tokens. At this time, the server can validate the refresh token against a database to make sure that the token is legitimate and that the user's access hasn't been revoked for some reason. However, the attacker can keep forging unexpired access tokens to prevent the refresh token from ever being checked.

If the attacker gains both read and write access to a server database or file system, the attacker could potentially do anything without having to forge JWTs. But software is extremely finicky, so attackers taking this approach are at greater risk of impairing servers and being detected before gaining an advantage. Attackers wishing to exploit a service are best off secretly employing the service as implemented. In any case, because server accounts generally have permission to read more data than they can write, attackers are more likely to only gain read access. Forging JWTs is therefore both the preferred approach and the approach that's more likely to be available.

JWTs vs Session IDs

It is tempting to dismiss this vulnerability because no system is ever completely secure, but JWTs introduce this vulnerability. Traditional session IDs need not be vulnerable this way. Moreover, this vulnerability can be seriously damaging when a breach goes undetected. Traditional session IDs here include session tokens and otherwise opaque authentication tokens.

News of big name websites being breached is all too common. Sadly, it is also common for breached sites to report that they were storing passwords in the clear, allowing the attacker to gain access as the user undetected and possibly to exploit accounts on other websites. Companies are proud when they can report that only password hashes were stolen and accounts not likely compromised, because hashes don't give attackers free access to accounts having strong passwords.

It is uncommon for companies to report whether they were storing session IDs in the clear. An attacker can use the ID of an active session to gain access as the owner of the session. However, because servers typically expire sessions after short periods, it can be hard for an attacker to successfully exploit a session ID. Sessions with short expiration periods can also be tied to IP addresses to harden them further. The savvy server will generate strongly random session IDs and only ever store their hashes, making sessions virtually impossible to exploit.

It is easy for servers to secure session IDs.

Contrast this with our JWT scenario. If a server that stores unprotected or naively-protected JWT signing keys is breached, the company may have to report that all accounts were exposed despite having hashed passwords. The company that incautiously jumped on the JWT bandwagon finds itself in a less enviable position than the company that stuck with conventional session IDs. Mind you, that word "incautiously" is important here, as it is possible to secure JWT signing keys.

No Inherent Protection

Seeing that tutorial after tutorial places JWT signing keys in the file system, we may be inclined to assume that something as-yet-unmentioned about the file system or about JWT protocol provides the protection. Let's consider some of the possibilities.

A JWT can store a session- or token-unique in the "jti" claim of the payload. To gain a security benefit from this unique, the server would have to both confirm the unique on each request and ignore the rest of the payload. This allows the JWT to be used as a session ID, but it does so at the cost of obviating most of the benefit of JWTs. There does however appear to be one benefit to signing a unique: the server can verify the signature before looking the unique up in a database. This can reduce the impact of a DoS attack by keeping the database from being accessed.

Perhaps an attacker can be kept blind to valid JWT payload values by having the server only store hashes of these values. That way, it would not be sufficient for an attacker to steal a signing key, because the attacker would also need the unhashed values that are only known by clients. Let's try this out on the user ID, given by the "sub" claim, one of the crucial pieces of the payload. The unhashed user ID would need to be returned in the JWT to the client at authentication so that the client can include the ID in the JWT with each service request. Well, because a given user can authenticate multiple times, whether on multiple devices or on the same device periodically after logging out, the server must be able to repeatedly put the unhashed user ID in the JWT. It can only do that if it is storing the user ID unhashed, leaving the server unable to blind the attacker this way.

JWTs are commonly used with the Oauth protocol. Oauth 2 provides a service for "token introspection" that allows a server to verify the validity of a token such as a JWT. This service exists to allow tokens to be revoked on events such as deletion of the user, change of password, or change user permissions. The service can report that a token is invalid even if the token is properly signed because the service does not rely on the signature. A server could use this introspection service to validate each token on each request. But doing so would obviate the benefit of having JWT payload data: there is a lookup hit for each request, and the introspection service could simply provide the data in response to an opaque token. A server could validate tokens against the service at intervals, but the JWT could be exploited for the duration of the interval, and servers would need to maintain indefinite blacklists to prevent themselves from accepting JWTs that were previously found invalid.

Many JWT tutorials that offer advice about storing the signing key suggest putting the key in an environment variable that is set in the shell profile or in a script that runs at server launch. In these cases, the server runs as the user that can read the file that contains the key. It is hard to confidently secure anything but a very simple server, and an attacker working server's entry points may find a way to get the server to return the file. If the file is accessible to a group of users, such as the site developers, the attacker has more chances to acquire the file. In any case, the site admin is now set with the task of carefully protecting file privileges, a task that's less crucial when using session IDs.

The approach of storing the key encrypted in the file system has already been mentioned. While providing a level of indirection is an impediment, this just moves the problem to protecting the decryption key, which most of us would again make somehow readable by the server's user.

The Problem of Protecting Keys

It seems that if we're going to use JWT tokens in a way that is at least as secure as traditional session IDs, then we must take extraordinary measures to protect the signing keys. In particular, we should not take the most obvious approach of putting the key in a file that is readable by the server. I am not an expert in how this should be done, but I do know a few ways it can be done:
  • Hardware security modules (HSMs) may provide the most secure way to protect a key. A special hardware device is attached to the server. Private keys resides in the device and no where else. The server delegates responsibility for signing JWTs to the device. Anyone who breaks into the server would then have to find a way to break into the device to get the keys.
  • The server that receives authentication requests can delegate responsibility for signing keys to a simple backend server. The backend server is not be on the Internet, and its network interfaces are carefully designed to ensure the security of the system. The key may be stored on the file system of this backend server, but after breaching the Internet-facing server, the attacker would have the additional task of having to breach the more-protected downstream server.
  • The signing keys are only stored in RAM and not in the file system. When the server goes down, the keys are lost. If asymmetric keys are being used, the public keys can be stored in the database or file system because they can't be used to sign JWTs. If the server goes down, the private keys are lost, but all JWTs that were signed with them can still be verified using the public keys. On a load-balanced cluster, each server can generate its own keys at launch and store the public halves in the database where other servers can read them to verify JWTs.
  • The signing keys are stored in the file system accessible only to a single user that is not used for any other purpose, and a second process runs on the server as this user to provide JWT-signing services to the web server process. The web server process communicates with the JWT-signing process via strictly memory-based IPC to get JWTs signed.
These approaches are probably listed most-secure first. This StackExchange answer lists some other possibilities as well. OWASP provides the following general guideline for storing keys:
Symantec's Internet Security Threat Report 2016 (p. 72) has this to say:
Make sure to get your digital certificates from an established, trustworthy certificate authority that demonstrates excellent security practices. Symantec recommends that organizations:
  • Use separate Test Signing and Release Signing infrastructures.
  • Secure keys in secure, tamper-proof, cryptographic hardware devices.
  • Implement physical security to protect your assets from theft.
If a key is ever used on a server to sign a JWT, the key must occur in plaintext in RAM. It is not clear that RAM is any more secure than the file system. For example, there are several ways that RAM can end up in the file system. When more software is loaded than there is RAM to hold it, the operating system can page-swap RAM out to files as needed to make room for the software that is waiting to run. Similarly, when an error occurs or a diagnostics operation is performed, the contents of RAM can be written to a file as a core dump. There may also be ways for an attacker to read RAM. This exposure could come from within the server by the attacker executing software, or it come through faulty interfaces to the server that allow an attacker to remotely extract RAM for inspection. Hence, keys are safest residing completely outside of Internet-facing servers.

Most programmers seem to find all this overkill. Most seem content just to make things tricky for attackers, such as by splitting a key and storing its pieces in multiple locations or by transforming the key after loading it from a file and using the transformed key. Security professionals call this class of approaches "security through obscurity." Security through obscurity may slow some attacks down, but it only stops novice attackers. Real security requires compartmentalizing accessibility. For example, security is gained by putting one half of a split key in the file system accessible by one user and the other half in a database that is not accessible to that same user. This requires an attacker to make two successful breaches instead of just one. A web server that has both privileges to the file and credentials for the database provides a single point of access to both halves of the key.

Unfortunately, the problem of protecting keys is becoming more pervasive as web service APIs grow in popularity. When a company such as Google or Flickr provides web services, they issue an API key to the software developer to use for accessing the services. If the developer writes software that uses these services to provide additional downstream services, the developer must store the key on a server that is somehow connected to the Internet, even if indirectly connected. Each API uses its own key, so developers may find themselves needing to protect multiple keys. Protecting keys requires careful design for security, and it's likely that most sites instead keep things simple and insecure. Read more about this growing problem at ProgrammableWeb and the Cloud Security Alliance.

Trusting Authentication Servers

While the vulnerability we're highlighting affects all clients of a service, the authenticating server is the source of the vulnerability. This is the server that receives client credentials, such as username and password, and returns a signed JWT in response to successful authentication. The risk to the client depends on the degree to which the authenticating server has secured its signing keys. As we have seen, it takes extraordinary measures to protect keys. Most servers that roll their own authentication and issue JWTs probably cannot be trusted. But what about the big name authentication services?

Google, Yahoo, Facebook, and Twitter all offer Oauth 2 single sign-on services for third party websites. The user chooses which service to use for login, and the third party website delegates authentication to that service. The authentication service provides access tokens for the client to hand to the third party website. The third party website trusts these tokens. I've read that Google uses JWTs, but I don't know about the others. These third party websites can use the Oauth introspection service to verify each token on each request, but they don't have to. If they strive for scalability by trusting the signature and the JWT payload, these third party services are potentially vulnerable. The developers of third party websites need to know whether to trust the JWT tokens.

It seems crucial that authentication services say something about how they are protecting JWT signing keys, so that third party services can decide whether to leverage JWT scalability as a function of the authentication service. If this information exists for these services, I cannot find it.

There are also a number of tiers such as Auth0 and StormPath that provide backend authentication services for websites. Websites can delegate authentication and possibly authorization to these services so that users can login with usernames and passwords on the website without the site having to implement its own authentication mechanism. That way, developers can leverage the security of a single base of code that has been hardened across many applications. However, I am again unable to find where these sites are reporting how they protect keys from theft by breach.

At the moment, we seem to be assessing trustworthiness purely by brand name.

The Missing Warning Label

The takeaway here is simple. Developers should not be implementing JWT authentication unless they take pains to protect the signing keys. Developers should instead either stick with traditional session IDs or strictly delegate authentication to a trusted brand name authentication service. Moreover, authentication services should really be describing their efforts to protect keys so that developers can make informed decisions about trusting access tokens.

This information is missing from the dozens of treatments of JWTs that I've seen. I've created a warning label that really ought accompany any discussion of developing JWT-based servers:

Unless the convenient assumption really turns out to be true, it is worth designing to protect against breaches in which data is only stolen. In that case, I offer this JWT authentication warning: Only implement authentication that signs JSON Web Tokens if you also compartmentalize access to the signing keys. Otherwise delegate authentication via Oauth or stick with session IDs.

Thursday, April 7, 2016

A Splash of Liquid Sunlight

(I wrote this poetic story on September 10, 1995, except for a few small edits.)

A gentle breeze blows across the mountain meadow. Waves ripple over the green and yellow grasses. Thistles and Queen Anne’s lace rock quietly from side to side. Small purple flower buds draw circles through the air. A yellow flower also sways slowly to the rhythm, alone among the grasses of the field.

The flower shines fiery and effulgent, glowing and radiating like a splash of liquid sunlight. Its petals are long and narrow, and they ride the wind as if floating up and down on waves of water.

A bee flits by and leaves a memory of its gentle hum.

The bee returns and swings around to a side of the flower, hovers there briefly, and then swings around to the other side. The shifting winds turn the bee's hum into melody. Hovering over the flower, the bee stretches its legs forward, and dips its abdomen to land among the golden petals.

In one swift blur, the petals retract sharply from the bee and swing the flower onto the side of its stem. The bee darts backwards and turns itself around. Once upon the stem, the flower hurries down the stalk, its petals scurrying like spider legs. The bee has meanwhile flown far away.

A lazy breeze meanders across the mountain meadow. Waves ripple over the green and yellow grasses. Flies crisscross the air and grasshoppers click their ratchedy sounds. A soft waft of pine visits the meadow and just as quickly disappears.

Saturday, March 19, 2016

A child dances in the flame

(I wrote this prose poem in October of 1992 while sitting on the grass one evening at the University of Maryland, College Park.)

A child dances in the flame

The sky is a ruddy flood. Basalt clouds pump brilliant blood. And a child dances in the flame.

The ground is a surface of darkness, an empty sheet shaping hill and plain. Fiery footprints lash upon nothing, spontaneous flares quick-to-die.

A body races, the body dances, a beating frolic, torchlight lurid. Glowing knees hurl high and down-hammer hard, torso twists, face flashes. The hair strikes out across the clouds, the head thrown back, the hands thrown high. A silhouette is seen in silent laughter.

A child dances in the sunset flame.

Monday, March 14, 2016

My Favorite Quotes

Here is a collection of my favorite quotes, in no particular order:

"Because the people who are crazy enough to think they can change the world, are the ones who do." ~Apple Computer, Inc., 1997


"Against the assault of Laughter nothing can stand." ~Mark Twain


"[W]e find a capacity for fulfillment we never knew we had when we accept love and joy as it is given, not just in the forms we desire it." ~Kara Dittmer Savvas

"Don’t bother me with facts, son. I’ve already made up my mind." ~Foghorn Leghorn

"It’s important to figure out how to be ruthlessly efficient and disciplined with your time, and do only those things that matter." ~Kristin Muhlner

"Let your identity be your religion." ~Lady Gaga


"I don't know the key to success, but the key to failure is trying to please everybody." ~Bill Cosby

"It is not what is poured into a student that counts but what is planted." ~Linda Conway

"There is no quality in this world that is not what it is merely by contrast. Nothing exists in itself." ~Herman Melville (Moby-Dick)

"If you can't handle me at my worst, then you sure as hell don't deserve me at my best." ~Marilyn Monroe

"If you want to tell people the truth, make them laugh, otherwise they'll kill you." ~Oscar Wilde

"Once you begin watching spiders, you haven't time for much else -- the world is really loaded with them." ~E.B. White

"When you change the way you look at things, the things you look at change." ~Max Planck



"Pity the poor insomniac dyslexic agnostic. He stays up all night wondering if there really is a dog." ~origin unknown


"May I have the courage today to live the life that I would love, to postpone my dream no longer but do at last what I came here for and waste my heart on fear no more." ~John O'Donohue



"We live in a world in which ninety-nine percent of all beautiful things are destroyed in the bud." ~Kurt Gödel (per Palle Yourgrau)

"I think I shall never see a poem lovely as a tree." ~Joyce Kilmer

"May our haters make us humble and our critics make us wise, may we learn from every stumble and if we fall may we always rise." ~Cory Booker, U.S. Senator from NJ
