---
abstract: 'Federated Learning (FL) enables learning a shared model across many clients without violating the privacy requirements. One of the key attributes in FL is the heterogeneity that exists in both resource and data due to the differences in computation and communication capacity, as well as the quantity and content of data among different clients. We conduct a case study to show that heterogeneity in resource and data has a significant impact on training time and model accuracy in conventional FL systems. To this end, we propose [<span style="font-variant:small-caps;">TiFL</span>]{}, a Tier-based Federated Learning System, which divides clients into tiers based on their training performance and selects clients from the same tier in each training round to mitigate the straggler problem caused by heterogeneity in resource and data quantity. To further tame the heterogeneity caused by non-IID (Independent and Identical Distribution) data and resources, [<span style="font-variant:small-caps;">TiFL</span>]{}employs an *adaptive* tier selection approach to update the tiering on-the-fly based on the observed training performance and accuracy over time. We prototype [<span style="font-variant:small-caps;">TiFL</span>]{}in a FL testbed following Google’s FL architecture and evaluate it using popular benchmarks and the state-of-the-art FL benchmark LEAF. Experimental evaluation shows that [<span style="font-variant:small-caps;">TiFL</span>]{}outperforms the conventional FL in various heterogeneous conditions. With the proposed adaptive tier selection policy, we demonstrate that [<span style="font-variant:small-caps;">TiFL</span>]{}achieves much faster training performance while keeping the same (and in some cases - better) test accuracy across the board.'
author:
- 'Zheng Chai\*'
- 'Ahsan Ali\*'
- 'Syed Zawad\*'
- Stacey Truex
- Ali Anwar
- Nathalie Baracaldo
- Yi Zhou
- Heiko Ludwig
- Feng Yan
- Yue Cheng
bibliography:
- 'main.bib'
title: '[[<span style="font-variant:small-caps;">TiFL</span>]{}]{}: A Tier-based Federated Learning System'
---

Conclusion {#sec:conclusion}
==========

In this paper, we investigate and quantify the heterogeneity impact on “decentralized virtual supercomputer” - FL systems. Based on the observations of our case study, we propose and prototype a Tier-based Federated Learning System called [<span style="font-variant:small-caps;">TiFL</span>]{}. Tackling the resource and data heterogeneity, [[<span style="font-variant:small-caps;">TiFL</span>]{}]{} employs a tier-based approach that groups clients in tiers by their training response latencies and selects clients from the same tier in each training round. To address the challenge that data heterogeneity information cannot be directly measured due to the privacy constraints, we further design an *adaptive* tier selection approach that enables [<span style="font-variant:small-caps;">TiFL</span>]{}be data heterogeneity aware and outperform conventional FL in various heterogeneous scenarios: *resource heterogeneity*, *data quantity heterogeneity*, *non-IID data heterogeneity*, and their combinations. Specifically, [<span style="font-variant:small-caps;">TiFL</span>]{}achieves an improvement over conventional FL by up to 3$\times$ speedup in overall training time and by 6% in accuracy.
