Kademlia DHT
A comprehensive Dart implementation of the libp2p Kademlia Distributed Hash Table (DHT) for building decentralized peer-to-peer applications. This library provides the core infrastructure for peer discovery, content routing, and distributed key-value storage in P2P networks.
π Featured Implementation: IpfsDHTv2
Section titled βπ Featured Implementation: IpfsDHTv2βIpfsDHTv2 is our flagship implementation featuring a modular, production-ready architecture with enhanced performance, observability, and maintainability. Itβs a drop-in replacement for the original IpfsDHT with significant improvements.
π Key Improvements in v2
Section titled βπ Key Improvements in v2β- π Modular Architecture: Clean separation of concerns with focused components
- π Built-in Observability: Comprehensive metrics and monitoring out of the box
- π‘οΈ Enhanced Error Handling: Structured exceptions with retry logic
- β‘ Performance Optimized: Parallel operations and efficient routing
- π§ͺ Testing-Friendly: Dependency injection for easy testing
- π§ Flexible Configuration: Builder pattern for complex setups
π Features
Section titled βπ FeaturesβCore DHT Capabilities
Section titled βCore DHT Capabilitiesβ- Peer Discovery: Find peers by their ID across the network using Kademlia routing
- Content Routing: Discover who has specific content using content addressing (CID)
- Distributed Storage: Store and retrieve key-value pairs across the network
- Service Discovery: Advertise and find services in the P2P network
- Provider Records: Track and announce content availability across the network
Network Modes
Section titled βNetwork Modesβ- Client Mode: Lightweight mode for mobile and resource-constrained devices
- Server Mode: Full participant mode for infrastructure and bootstrap nodes
- Auto Mode: Switches between client and server with the hostβs reachability: a server while AutoNAT (or
forceReachability) reports public reachability, a client while it is private or unknown
Advanced Features
Section titled βAdvanced Featuresβ- Bootstrap Integration: Easy connection to existing libp2p networks with configurable bootstrap peers
- Routing Table Management: Kademlia-based peer routing with configurable bucket sizes
- Query Engine: Efficient parallel query execution with configurable concurrency
- Retry Logic: Configurable retry mechanisms with exponential backoff
- Network Size Estimation: Built-in network size estimation capabilities
Production-Ready Components
Section titled βProduction-Ready Componentsβ- Cryptographic Validation: Signed records with anti-replay protection
- 4-Phase Bootstrap: Comprehensive network connectivity with health monitoring
- Provider Operations: Complete network-integrated provider management
- Datastore Operations: Full local record storage with validation
- Metrics & Monitoring: Real-time performance tracking and health checks
π Quick Start
Section titled βπ Quick StartβInstallation
Section titled βInstallationβAdd the package to your pubspec.yaml:
dependencies: dart_libp2p_kad_dht: ^1.6.0 dart_libp2p: ^4.7.0 dcid: ^1.0.0Basic Usage with IpfsDHTv2
Section titled βBasic Usage with IpfsDHTv2βimport 'package:dart_libp2p_kad_dht/src/dht/v2/dht_v2.dart';import 'package:dart_libp2p/dart_libp2p.dart';
Future<void> main() async { // Create a libp2p host final host = await createLibp2pHost();
// Create a provider store for content routing final providerStore = MemoryProviderStore();
// Create and start the DHT v2 (recommended) final dht = IpfsDHTv2( host: host, providerStore: providerStore, options: const DHTOptions( mode: DHTMode.auto, bucketSize: 20, concurrency: 10, ), );
await dht.start(); await dht.bootstrap(); // Connect to the network
// Find a peer final peerInfo = await dht.findPeer(targetPeerId);
// Store a value (signed). The key's namespace needs a validator: // '/pk/<peer ID bytes>' is a public-key record. See "Value Records" for // your own namespaces. final pkKey = '/pk/${String.fromCharCodes(host.id.toBytes())}'; await dht.putValue(pkKey, (await host.peerStore.keyBook.pubKey(host.id))!.marshal());
// Retrieve a value (validated; asks the network) final value = await dht.getValue(pkKey);
// Announce content availability await dht.provide(CID.fromString('QmExample...'), true);
// Find content providers await for (final provider in dht.findProvidersAsync(CID.fromString('QmExample...'), 10)) { print('Found provider: ${provider.id}'); }
// Check metrics final metrics = dht.metrics; print('Success rate: ${metrics.querySuccessRate * 100}%');
// Cleanup await dht.close(); await host.close();}Critical: Initialization Order
Section titled βCritical: Initialization OrderβYou must start the DHT before starting the host. When host.start() is called, AutoRelay immediately connects to relay servers and triggers an Identify exchange. If the DHT hasnβt registered its protocol handler yet, the first Identify response will be missing /ipfs/kad/1.0.0. Go-based peers that receive this will mark your node as βpeer stopped dhtβ and refuse to open DHT streams to it β breaking peer discovery permanently for that connection.
// CORRECT β DHT starts first, then hostfinal host = await createLibp2pHost(); // Do NOT call host.start() yetfinal dht = IpfsDHTv2(host: host, providerStore: store, options: options);await dht.start(); // Registers /ipfs/kad/1.0.0 protocol handlerawait host.start(); // Now AutoRelay's Identify will include DHT protocolawait dht.bootstrap();
// WRONG β host starts before DHTfinal host = await createLibp2pHost();await host.start(); // AutoRelay connects, Identify sent WITHOUT /ipfs/kad/1.0.0final dht = IpfsDHTv2(host: host, providerStore: store, options: options);await dht.start(); // Too late β Go already marked us "peer stopped dht"This applies to any protocol handler that must be advertised via Identify β always register handlers before host.start().
In DHTMode.auto (the default), IpfsDHTv2 starts as a client and registers
the /ipfs/kad/1.0.0 handler only when the host reports public reachability
(EvtLocalReachabilityChanged, from AutoNAT or the forceReachability host
option). It removes the handler again when reachability becomes private or
unknown. Identify pushes each change to connected peers, which add the node
to or remove it from their routing tables. DHTMode.autoServer starts as a
server and is a client only while reachability is private. dht.isServer
tells whether the node serves now. Start the DHT before the host, as above,
so that the DHT sees the first reachability event.
Value Records
Section titled βValue RecordsβputValue and getValue store and read signed records under a key. As in
go-libp2p-kad-dht, the keyβs namespace (/<namespace>/...) decides how a
record is checked:
- A record is accepted (by
putValue, by a server receivingPUT_VALUE, and inGET_VALUEanswers) only if the validator registered for its namespace accepts it. The default validator has thepkandipnsnamespaces. - A key whose namespace has no validator, such as
my-key, is refused by default:putValuethrows aDHTProtocolExceptionandgetValuereturns only a record that this node stored itself. - When a node holds a record and receives another one for the same key, the
validatorβs
select()decides which one it keeps. The recordβs timestamp is chosen by its writer and does not decide. - A node keeps a record for
DHTOptions.maxRecordAge(default 36 hours) after it stored it. Put a record again before then to keep it in the network; the DHT does not republish records itself. getValuereads the local record and asks the closest peers. It selects the best valid record withselect(). WithRoutingOptions()..offline = trueit reads only the local datastore.
Register a validator for your own namespace:
import 'package:dart_libp2p_kad_dht/src/record/namespace_validator.dart';import 'package:dart_libp2p_kad_dht/src/record/public_key_validator.dart';import 'package:dart_libp2p_kad_dht/src/record/ipns_validator.dart';import 'package:dart_libp2p_kad_dht/src/record/validator.dart';
class GreetingValidator implements Validator { @override Future<void> validate(String key, Uint8List value) async { // Throw if the value is not valid for this key. To stop other peers // from replacing a value, tie the key to its owner here, e.g. require // a signature in the value by the peer ID that the key contains. if (value.length > 1024) throw Exception('value too large'); }
@override Future<int> select(String key, List<Uint8List> values) async { return 0; // index of the best value }}
final dht = IpfsDHTv2( host: host, providerStore: providerStore, validator: NamespacedValidator() ..['pk'] = PublicKeyValidator() ..['ipns'] = IpnsValidator(host.peerStore) ..['greeting'] = GreetingValidator(),);await dht.putValue('/greeting/alice', utf8.encode('hello'));Passing validator: replaces the default validator, so add pk and ipns
again if you need them.
For keys without a namespace validator, DHTOptions(allowUnvalidatedRecords: true) accepts records that carry a valid signature of their author. A node
then replaces a stored record only with a newer record of the same author, so
the first author to store a key on a node keeps it there until it expires.
This is weaker than a validator; use it for trusted or test networks.
The legacy IpfsDHT applies the same rules: the PUT_VALUE handler, records
in GET_VALUE answers, putValue and getValue (which now also asks the
network when it has a local record) all use the same validation, selection
and expiry code as IpfsDHTv2. Its default validator also has only pk and
ipns; the v namespace, which accepted any value, is gone. A validator:
passed to IpfsDHT gets pk and ipns added when it has no entry for them.
Advanced Configuration with Builder Pattern
Section titled βAdvanced Configuration with Builder Patternβimport 'package:dart_libp2p_kad_dht/src/dht/v2/config/dht_config.dart';
// Use builder pattern for complex configurationfinal config = DHTConfigBuilder() .mode(DHTMode.server) .bucketSize(25) .concurrency(15) .filterLocalhost(false) .networkTimeout(Duration(seconds: 30)) .queryTimeout(Duration(seconds: 60)) .enableMetrics(true) .optimisticProvide(true) .build();
final dht = IpfsDHTv2( host: host, providerStore: providerStore, options: config.toOptions(),);π Examples
Section titled βπ ExamplesβInteractive P2P Node (v2)
Section titled βInteractive P2P Node (v2)βRun a fully interactive P2P node with all DHT operations:
dart run example/basic_p2p_node.dartAvailable Commands:
stats- Show network statistics and metricsstore <key> <value>- Store a cryptographically signed key-value pairget <key>- Retrieve a validated valueannounce <content-id>- Announce content availabilityfind-content <content-id>- Find content providersfind-peer <peer-id>- Find a specific peermetrics- Show detailed performance metricsquit- Exit the demo
Mobile Optimized Node
Section titled βMobile Optimized NodeβFor resource-constrained environments:
dart run example/mobile_p2p_node.dartServer Node
Section titled βServer NodeβHigh-performance server deployment:
dart run example/server_node.dart --port 4001π§ Configuration
Section titled βπ§ ConfigurationβDHT v2 Options
Section titled βDHT v2 Optionsβfinal dhtOptions = DHTOptions( mode: DHTMode.auto, // client, server, or auto bucketSize: 20, // K-bucket size concurrency: 10, // Concurrent operations resiliency: 3, // Query redundancy bootstrapPeers: [ // Network entry points; each address must end in /p2p/<peer ID> MultiAddr('/ip4/203.0.113.7/udp/4001/udx/p2p/12D3KooW...'), MultiAddr('/ip4/203.0.113.8/tcp/4001/p2p/12D3KooW...'), ], maxRecordAge: Duration(hours: 36), // How long a value record is kept allowUnvalidatedRecords: false, // Accept keys without a namespace validator);Advanced Configuration with Builder
Section titled βAdvanced Configuration with Builderβfinal config = DHTConfigBuilder() .mode(DHTMode.server) .bucketSize(30) .concurrency(20) .resiliency(5) .networkTimeout(Duration(seconds: 30)) .queryTimeout(Duration(seconds: 60)) .enableMetrics(true) .optimisticProvide(true) .maxRetryAttempts(5) .retryInitialBackoff(Duration(milliseconds: 250)) .build();ποΈ Architecture
Section titled βποΈ ArchitectureβDHT v2 Modular Components
Section titled βDHT v2 Modular Componentsβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ IpfsDHTv2 ββ (Main Interface) ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€β βββββββββββββββ βββββββββββββββ βββββββββββββββ ββ β Network β β Routing β β Query β ββ β Manager β β Manager β β Manager β ββ βββββββββββββββ βββββββββββββββ βββββββββββββββ ββ ββ βββββββββββββββ βββββββββββββββ ββ β Protocol β β Metrics β ββ β Manager β β Manager β ββ βββββββββββββββ βββββββββββββββ ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββComponent Responsibilities
Section titled βComponent Responsibilitiesβ- NetworkManager: Message handling, connection management, retry logic
- RoutingManager: Routing table management, peer discovery, bootstrap
- QueryManager: Query coordination, peer lookups, value operations
- ProtocolManager: Protocol message processing, request/response handling
- MetricsManager: Performance monitoring, error tracking, health checks
π Monitoring & Metrics
Section titled βπ Monitoring & MetricsβDHT v2 provides comprehensive monitoring out of the box:
final metrics = dht.metrics;
// Query metricsprint('Total queries: ${metrics.totalQueries}');print('Success rate: ${metrics.querySuccessRate * 100}%');print('Average latency: ${metrics.averageQueryLatency.inMilliseconds}ms');
// Network metricsprint('Network requests: ${metrics.totalNetworkRequests}');print('Network success rate: ${metrics.networkSuccessRate * 100}%');
// Routing table metricsprint('Routing table size: ${metrics.routingTableSize}');print('Peers added: ${metrics.peersAdded}');
// Performance metricsprint('Queries per second: ${metrics.queriesPerSecond}');print('Network requests per second: ${metrics.networkRequestsPerSecond}');π‘οΈ Error Handling
Section titled βπ‘οΈ Error HandlingβDHT v2 provides structured error handling:
try { final peer = await dht.findPeer(targetPeerId);} on DHTNetworkException catch (e) { print('Network error: ${e.message}'); // Handle network-specific errors} on DHTTimeoutException catch (e) { print('Timeout error: ${e.message}'); // Handle timeout-specific errors} on DHTException catch (e) { print('DHT error: ${e.message}'); // Handle general DHT errors}π§ͺ Testing
Section titled βπ§ͺ TestingβRun the comprehensive test suite:
dart testThe test suite includes:
- Unit tests for all components
- Integration tests with real network scenarios
- Performance benchmarks
- Mobile device simulation tests
A fresh clone builds against the published packages. To develop against local
checkouts of dart_libp2p or
dart-udx, create a
pubspec_overrides.yaml (git-ignored) next to pubspec.yaml:
dependency_overrides: dart_libp2p: path: ../dart-libp2p dart_udx: path: ../dart-udxThe Go interop tests in test/interop build the go-libp2p peer from a
dart_libp2p checkout (GO_PEER_DIR, or ../dart-libp2p/interop/go-peer by
default) and need Go.
π Documentation
Section titled βπ Documentationβ- Developer Guide: Comprehensive guide for P2P application developers
- Examples: Detailed examples and use cases
- DHT v2 Documentation: Complete v2 architecture guide
- Integration Tests: Real-world usage patterns
π Migration from Original DHT
Section titled βπ Migration from Original DHTβSimple Migration
Section titled βSimple Migrationβ// Oldimport 'package:dart_libp2p_kad_dht/src/dht/dht.dart';final dht = IpfsDHT(host: host, providerStore: providerStore, options: options);
// New (recommended)import 'package:dart_libp2p_kad_dht/src/dht/v2/dht_v2.dart';final dht = IpfsDHTv2(host: host, providerStore: providerStore, options: options);Benefits of Migration
Section titled βBenefits of Migrationβ- Better error handling: Structured exceptions and retry logic
- Improved observability: Built-in metrics and monitoring
- Better testability: Modular design and dependency injection
- Enhanced performance: Optimized query patterns and caching
- Future-proof: Easier to extend and maintain
π€ Contributing
Section titled βπ€ ContributingβWe welcome contributions! Please see our contributing guidelines:
- Fork the repository
- Create a feature branch
- Make your changes
- Add tests for new functionality
- Ensure all tests pass
- Submit a pull request
Development Setup
Section titled βDevelopment Setupβ# Clone the repositorygit clone https://github.com/stephanfeb/dart_libp2p_kad_dht.gitcd dart_libp2p_kad_dht
# Install dependenciesdart pub get
# Run testsdart test
# Run examplesdart run example/basic_p2p_node.dartπ License
Section titled βπ LicenseβThis project is licensed under the MIT License - see the LICENSE file for details.
π Acknowledgments
Section titled βπ Acknowledgmentsβ- Based on the go-libp2p-kad-dht implementation
- Implements the Kademlia DHT algorithm
- Built for the libp2p networking stack
π Related Projects
Section titled βπ Related Projectsβ- dart_libp2p: Core libp2p networking library
- dcid: Content addressing utilities
- dart_udx: UDP-based transport layer
π Support
Section titled βπ Supportβ- Issues: GitHub Issues
- Discussions: GitHub Discussions
- Documentation: Developer Guide
Built with β€οΈ for the decentralized web