Distributed Namespace (DNE) in Lustre

How It Works

DNE spreads metadata across multiple MDTs, using a hierarchical namespace: root MDT0 and remote/striped directories on secondary MDTs. Phase 1 (DNE1, remote dirs) routes operations; Phase 2 (DNE2, striped dirs) balances loads via hash-based distribution. Supports up to 256 MDTs, handling billions of files without bottlenecks. Clients query MDS for routing.

History

Introduced in Lustre 2.1 (2011) as DNE1 for basic distribution; DNE2 (striped directories) in 2.8 (2015). Matured in 2.12 (2019) with directory striping defaults. By 2025 (2.16), enables 64 billion files per MDT in ZFS-backed systems, critical for exascale like Fugaku.

Examples

Enable Striped Directory

Create a striped dir on 2 MDTs:

lfs setstripe -D -c 2 /lustre/mydir
mkdir /lustre/mydir/subdir

Operations hash to MDTs for balance.

List MDTs

Query filesystem MDTs:

lfs df -t mdt

Output shows MDT usage, e.g., `mdt0 100% /lustre[0]`.

Ideal for metadata-intensive genomics workflows.