Routing Algorithms for DH Ts : Some Open Questions

dicedknockemstiffΔίκτυα και Επικοινωνίες

13 Ιουλ 2012 (πριν από 4 χρόνια και 9 μήνες)

265 εμφανίσεις

Routing Algorithms for DHTs:Some Open Questions
Sylvia Ratnasamy
Scott Shenker
Ion Stoica
1 Introduction
Even though they were introduced only a few years
ago,peer-to-peer (P2P) ﬁlesharing systems are nowone
of the most popular Internet applications and have be-
come a major source of Internet trafﬁc.Thus,it is ex-
tremely important that these systems be scalable.Un-
fortunately,the initial designs for P2P systems have sig-
niﬁcant scaling problems;for example,Napster has a
centralized directory service,and Gnutella employs a
ﬂooding-based search mechanism that is not suitable
for large systems.
In response to these scaling problems,several re-
search groups have (independently) proposed a new
generation of scalable P2P systems that support a dis-
tributed hash table (DHT) functionality;among them
are Tapestry [15],Pastry [6],Chord [14],and Content-
Addressable Networks (CAN) [10].In these systems,
which we will call DHTs,ﬁles are associated with a
key (produced,for instance,by hashing the ﬁle name)
and each node in the system is responsible for storing
a certain range of keys.There is one basic operation in
these DHTsystems,lookup(key),which returns the
identity (e.g.,the IP address) of the node storing the ob-
ject with that key.This operation allows nodes to put
and getﬁles based on their key,thereby supporting the
hash-table-like interface.
This DHT functionality has proved to be a use-
ful substrate for large distributed systems;a number
of projects are proposing to build Internet-scale fa-
cilities layered above DHTs,including distributed ﬁle
systems [5,7,4],application-layer multicast [11,16],
event notiﬁcation services [3,1],and chat services [2].
With so many applications being developed in so short
a time,we expect the DHT functionality to become an
integral part of the future P2P landscape.
The core of these DHT systems is the routing al-
The interfaces of these systems are not all identical;some
reveal only the put and get interface while others reveal the
lookup(key) function directly.However,the above discussion
refers to the underlying functionality and not the details of the API.
gorithm.The DHT nodes form an overlay network
with each node having several other nodes as neighbors.
When a lookup(key)is issued,the lookup is routed
through the overlay network to the node responsible for
that key.The scalability of these DHTalgorithms is tied
directly to the efﬁciency of their routing algorithms.
Each of the proposed DHT systems listed above –
Tapestry,Pastry,Chord,and CAN – employ a differ-
ent routing algorithm.Usually discussion of DHT rout-
ing issues is in the context of one particular algorithm.
And,when more than one is mentioned,they are of-
ten compared in competitive terms in an effort to deter-
mine which is “best”.We think both of these trends
are wrong.The algorithms have more commonality
than differences,and each algorithm embodies some
insights about routing in overlay networks.Rather than
always working in the context of a single algorithm,
or comparing the algorithms competitively,a more ap-
propriate goal would be to combine these insights,and
seek newinsights,to produce even better algorithms.In
that spirit we describe some issues relevant to routing
algorithms and identify some open research questions.
Of course,our list of questions is not intended to be
exhaustive,merely illustrative.
As should be clear by our description,this paper is
not about ﬁnished work,but instead is about a research
agenda for future work (by us and others).We hope that
presenting such a discussion to this audience will pro-
mote synergy between research groups in this area and
help clarify some of the underlying issues.We should
note that there are many other interesting issues that
remain to be resolved in these DHT systems,such as
security and robustness to attacks,system monitoring
and maintenance,and indexing and keyword search-
ing.These issues will doubtless be discussed elsewhere
in this workshop.Our focus on routing algorithms is
not intended to imply that these other issues are of sec-
ondary importance.
We ﬁrst (very) brieﬂy review the routing algorithms
used in the various DHT systems in Section 2.We then,
in the following sections,discuss various issues rele-
vant to routing:state-efﬁciency tradeoff,resilience to
failures,routing hotspots,geography,and heterogene-
2 Review of Existing Algorithms
In this section we review some of the existing routing
algorithms.All of them take,as input,a key and,
in response,route a message to the node responsible
for that key.The keys are strings of digits of some
length.Nodes have identiﬁers,taken from the same
space as the keys (i.e.,same number of digits).Each
node maintains a routing table consisting of a small
subset of nodes in the system.When a node receives a
query for a key for which it is not responsible,the node
routes the query to the neighbor node that makes the
most “progress” towards resolving the query.The no-
tion of progress differs fromalgorithmto algorithm,but
in general is deﬁned in terms of some distance between
the identiﬁer of the current node and the identiﬁer of
the queried key.
Plaxton et al.:Plaxton et al.[9] developed perhaps
the ﬁrst routing algorithm that could be scalably used
by DHTs.While not intended for use in P2P systems,
because it assumes a relatively static node population,
it does provide very efﬁcient routing of lookups.The
routing algorithm works by “correcting” a single digit
at a time:if node number
received a lookup
query with key
,which matches the ﬁrst two dig-
its,then the routing algorithm forwards the query to
a node which matches the ﬁrst three digits (e.g.,node
).To do this,a node needs to have,as neigh-
bors,nodes that match each preﬁx of its own identi-
ﬁer but differ in the next digit.For a system of

nodes,each node has on the order of
bors.Since one digit is corrected each time the query is
forwarded,the routing path is at most
(or application-level) hops.
This algorithm has the additional property that if the
node-node latencies (or “distances” according to
some metric) are known,the routing tables can be cho-
sen to minimize the expected path latency and,more-
over,the latency of the overlay path between two nodes
is within a constant factor of the latency of the direct
underlying network path between them.
Tapestry:Tapestry [15] uses a variant of the Plaxton
et al.algorithm.The modiﬁcations are to ensure that
the design,originally intended for static environments,
can adapt to a dynamic node population.The modiﬁ-
cations are too involved to describe in this short review.
However,the algorithmmaintains the properties of hav-
neighbors and routing with path lengths of
Pastry:In Pastry [6],nodes are responsible for keys
that are the closest numerically (with the keyspace con-
sidered as a circle).The neighbors consist of a Leaf Set

which is the set of

closest nodes (half larger,half
smaller).Correct,not necessarily efﬁcient,routing can
be achieved with this leaf set.To achieve more efﬁcient
routing,Pastry has another set of neighbors spread out
in the key space (in a manner we don’t describe here).
Routing consists of forwarding the query to the neigh-
boring node that has the longest shared preﬁx with the
key (and,in the case of ties,to the node with identiﬁer
closest numerically to the key).Pastry has
neighbors and routes within
Chord:Chord [14] also uses a one-dimensional cir-
cular key space.The node responsible for the key is
the node whose identiﬁer most closely follows the key
(numerically);that node is called the key’s successor.
Chord maintains two sets of neighbors.Each node
has a successor list of

nodes that immediately fol-
lowit in the key space.Routing correctness is achieved
with these lists.Routing efﬁciency is achieved with
the ﬁnger list of
nodes spaced exponentially
around the key space.Routing consists of forwarding
to the node closest,but not past,the key;pathlengths
CAN:CAN chooses its keys from a

toroidal space.Each node is associated with a hyper-
cubal region of this key space,and its neighbors are the
nodes that “own” the contiguous hypercubes.Routing
consists of forwarding to a neighbor that is closer to the
key.CAN has a different performance proﬁle than the
other algorithms;nodes have
neighbors and path-
lengths are

hops.Note,however,that when
,CAN has
neighbors and
pathlengths like the other algorithms.
3 State-Efﬁciency Tradeoff
The most obvious measure of the efﬁciency of these
routing algorithms is the resulting pathlength.Most
of the algorithms have pathlengths of
while CAN has longer paths of
  

.The most ob-
vious measure of the overhead associated with keeping
routing tables is the number of neighbors.This isn’t
just a measure of the state required to do routing but
it is also a measure of how much state needs to be ad-
justed when nodes join or leave.Given the prevalence
of inexpensive memory and the highly transient user
populations in P2P systems,this second issue is likely
to be much more important than the ﬁrst.Most of the
algorithms require
neighbors,while CAN re-
quires only
Ideally,one would like to combine the best of these
two classes of algorithms in hybrid algorithms that
achieve short pathlengths with a ﬁxed number of neigh-
bors.Question 1 Can one achieve
pathlengths (or
better) with

One would expect that,if this were possible,that some
other aspects of routing would get worse.
Question 2 If so,are there other properties (such as
those described in the following sections) that are made
worse in these hybrid routing algorithms?
4 Resilience to Failures
The above routing results refer to a perfectly function-
ing system with all nodes operational.However,P2P
nodes are notoriously transient and the resilience of
routing to failures is a very important consideration.
There are (at least) three different aspects to resilience.
First,one needs to evaluate whether routing can con-
tinue to function (and with what efﬁciency) as nodes
fail without any time for other nodes to establish other
neighbors to compensate;that is the neighboring nodes
knowthat a node has failed,but they don’t establish any
new neighbor relations with other nodes.We will call
this static resilience and measure it in terms of the per-
centage of reachable key locations and of the resulting
average path length.
Question 3 Can one characterize the static resilience
of the various algorithms?What aspects of these algo-
rithms lead to good resilience?
Second,one can investigate the resilience when
nodes have a chance to establish some neighbors,but
not all.That is,when nodes have certain “special”
neighbors,such as the successor list or the Leaf Set,
and these are re-established after a failure,but no other
neighbors are re-established (such as the ﬁnger set).
The presence of these special neighbors allow one to
prove the correctness of routing,but the following ques-
tion remains:
Question 4 To what extent are the observed path
lengths better than the rather pessimistic bounds pro-
vided by the presence of these special neighbors?
Finally,one can ask how long it takes various algo-
rithms to fully recover their routing state,and at what
cost (measured,for example,by the number of nodes
participating in the recovery or the number of control
messages generated for recovery).
Question 5 How long does it take,on average,to re-
cover complete routing state?And what is the cost of
doing so?
A related question is:
Question 6 Can one identify design rules that lead to
shorter and/or cheaper recoveries?
For instance,is symmetry (where the node neighbor re-
lation is symmetric) important in restoring state easily?
One could also argue that in the face of node failure,
having the routing automatically send messages to the
correct alternate node (i.e.the node that takes over the
range of the identiﬁer space that was previously held by
the failed node) leads to quicker recovery.
5 Routing Hot Spots
When there is a hotspot in the query pattern,with a
certain key being requested extremely often,then the
node holding that key may become overloaded.Various
caching and replication schemes have been proposed to
overcome this query hotspot problem;the effectiveness
of these schemes may vary between algorithms based
on the fan-in at the node and other factors,but this
seems to be a manageable problem.More problematic,
however,is if a node is overloaded with too much rout-
ing trafﬁc.These routing hotspots are harder to deal
with since there is no local action the node can take to
redirect the routing load.Some of the proximity tech-
niques we describe below might be used to help here,
but otherwise this remains an open problem.
Question 7 Do routing hotspots exist and,if so,how
can one deal with them?
6 Incorporating Geography
The efﬁciency measure used above was the number of
application-level hops taken on the path.However,the
true efﬁciency measure is the end-to-end latency of the
path.Because the nodes could be geographically dis-
persed,some of these application-level hops could in-
volve transcontinental links,and others merely trips
across a LAN;routing algorithms that ignore the la-
tencies of individual hops are likely to result in high-
latency paths.While the original “vanilla” versions of
some of these routing algorithms did not take these hop
latencies into account,almost all of the “full” versions
of the algorithms make some attempt to deal with the
geographic proximity of nodes.There are (at least)
three ways of coping with geography.
Proximity Routing:Proximity routing is when the
routing choice is based not just which neighboring node
makes the “most” progress towards the key,but is also
based on which neighboring node is “closest” in the
sense of latency.Various algorithms implement prox-
imity routing differently,but they all adopt the same
basic approach of weighing progress in identiﬁer space
against cost in latency (or geography).Simulations
have shown this to be a very effective tool in reducing
the average path latency.
Question 8 Can one formally characterize the effec-
tiveness of these proximity routing approaches?
Proximity Neighbor Selection:This is a variant of
the idea above,but now the proximity criterion is ap-
plied when choosing neighbors,not just when choosing
the next hop.
Question 9 Can one show that proximity neighbor se-
lection is always better than proximity routing?Is this
difference signiﬁcant?
As mentioned earlier,if the
node-pair dis-
tances (as measured by latency) are known,the Plax-
ton/Tapestry algorithm can choose the neighbors so as
to minimize the expected overlay path latency.This is
an extremely important property,that is (so far) the ex-
clusive domain of the Plaxton/Tapestry algorithms.We
don’t whether other algorithms can adopt similar ap-
proaches.Question 10 If one had the full
 
distance matrix,
could one do optimal neighbor selection in algorithms
other than Plaxton/Tapestry?
Geographic Layout:In most of the algorithms,the
node identiﬁers are chosen randomly (e.g.hash func-
tions of the IP address,etc.) and the neighbor relations
are established based solely on these node identiﬁers.
One could instead attempt to choose node identiﬁers
in a geographically informed manner.
An initial at-
tempt to do so in the context of CAN was reported on
in [12];this approach was quite successful in reduc-
ing the latency of paths.There was little in the layout
method speciﬁc to CAN,but the high-dimensionality of
the key space may have played an important role;recent
work [8] suggests that latencies in the Internet can be
reasonably modeled by a

-dimension geometric space

 
.This raises the question of whether sys-
tems that use a one-dimensional key set can adequately
mimic the geographic layout of the nodes.
Question 11 Can one choose identiﬁers in a one-
dimensional key space that will adequately capture the
geographic layout of nodes?
However,this may not matter because the geographic
layout may not offer signiﬁcant advantages over the two
proximity methods.
Question 12 Can the two local techniques of proximity
routing and proximity neighbor selection achieve most
of the beneﬁt of global geographic layout?
Moreover,these geographically-informed layout meth-
ods may interfere with the robustness,hotspot,and
other properties mentioned in previous sections.
Question 13 Does geographic layout have an impact
on resilience,hotspots,and other aspects of perfor-
mance?7 Extreme Heterogeneity
All of the algorithms start by assuming that all nodes
have the same capacity to process messages and then,
only later,add on techniques for coping with hetero-
However,the heterogeneity observed in cur-
rent P2P populations [13] is quite extreme,with dif-
ferences of several orders of magnitude in bandwidth.
One can ask whether the routing algorithms,rather than
merely coping with heterogeneity,should instead use
Note that geographic layout differs fromthe two above proxim-
ity methods in that here there is an attempt to affect the global lay-
out of the node identiﬁers,whereas the proximity methods merely
affect the local choices of neighbors and forwarding nodes.
The authors of [13] deserve credit for bringing the issue of het-
erogeneity to our attention.
it to their advantage.At the extreme,a star topology
with all queries passing through a single hub node and
then routed to their destination would be extremely ef-
ﬁcient,but would require a very highly capable nub
node (and would have a single point of failure).But
perhaps one could use the very highly capable nodes
as mini-hubs to improve routing.In another position
paper here,some of us argue that heterogeneity can be
used to make Gnutella-like systems more scalable.The
question is whether one could similarly modify the cur-
rent DHT routing algorithms to exploit heterogeneity:
Question 14 Can one redesign these routing algo-
rithms to exploit heterogeneity?
It may be that no sophisticated modiﬁcations are
needed to leverage heterogeneity.Perhaps the sim-
plest technique to cope with heterogeneity,one that has
already been mentioned in the literature,is to clone
highly capable nodes so that they could serve as multi-
ple nodes;i.e.,a node that was
times more powerful
than other nodes could function as
virtual nodes.
When combined with proximity routing and neighbor
selection,cloning would allow nodes to route to them-
selves and thereby “jump” in key space without any for-
warding hops.
Question 15 Does cloning plus proximity routing and
neighbor selection lead to signiﬁcantly improved per-
formance when the node capabilities are extremely het-
USCHEL,P.Scribe:The design of a large-scale event no-
tiﬁcation infrastructure.In Proceedings of NGC 2001 (Nov.
[2] BASED CHAT,C.,2001.
Achieving a global event notiﬁcation service.In Proceedings
of the 8th IEEEWorkshop on Hot Topics in Operating Systems
(HotOS-VIII) (Elmau/Oberbayern,Germany,May 2001).
AND STOICA,I.Wide-area cooperative storage with CFS.
In Proceedings of the 18th ACM Symposium on Operating
Systems Principles (SOSP ’01) (To appear;Banff,Canada,
This technique has already been suggested for some of the al-
gorithms,and could easily be applied to the others.However,in
some algorithms it would require alteration in the way the node
identiﬁers were chosen so that they weren’t tied to the IP address of
the node.
[5] DRUSCHEL,P.,AND ROWSTRON,A.Past:Persistent and
anonymous storage in a peer-to-peer networking environ-
ment.In Proceedings of the 8th IEEE Workshop on Hot Top-
ics in Operating Systems (HotOS 2001) (Elmau/Oberbayern,
Germany,May 2001),pp.65–70.
[6] DRUSCHEL,P.,AND ROWSTRON,A.Pastry:Scalable,dis-
tributed object location and routing for large-scale peer-to-
peer systems.In Proceedings of the 18th IFIP/ACMInterna-
tional Conference on Distributed Systems Platforms (Middle-
ware 2001)W (Nov 2001).
ZHAO,B.OceanStore:An architecture for global-scale per-
sistent storage.In Proceeedings of the Ninth international
Conference on Architectural Support for Programming Lan-
guages and Operating Systems (ASPLOS 2000) (Boston,MA,
November 2000),pp.190–201.
[8] NG,E.,AND ZHANG,H.Towards global network position-
ing.In Proceedings of ACM SIGCOMM Internet Measure-
ment Workshop 2001 (Nov.2001).
ing nearby copies of replicated objects in a distributed envi-
ronment.In Proceedings of the ACMSPAA (Newport,Rhode
Island,June 1997),pp.311–320.
AND SHENKER,S.A scalable content-addressable network.
In Proc.ACM SIGCOMM (San Diego,CA,August 2001),
SHENKER,S.Application-level Multicast using Content-
Addressable Networks.In Proceedings of NGC 2001 (Nov.
SHENKER,S.Topologically-aware overlay construction and
server selection.In Proceedings of Infocom ’2002 (Mar.
ment study of peer-to-peer ﬁle sharing systems.In Proceed-
ings of Multimedia Conferencing and Networking (San Jose,
AND BALAKRISHNAN,H.Chord:A scalable peer-to-peer
lookup service for internet applications.In Proceedings of
the ACMSIGCOMM ’01 Conference (San Diego,California,
August 2001).
An infrastructure for fault-tolerant wide-area location and
routing.Tech.Rep.UCB/CSD-01-1141,University of Cal-
ifornia at Berkeley,Computer Science Department,2001.
KUBIATOWICZ,J.Bayeux:An architecture for wide-area,
fault-tolerant data dissemination.In Proceedings of NOSS-
DAV’01 (Port Jefferson,NY,June 2001).