Recipe 36: Stateful Key-Value API With Fabric-Backed Storage

The Idea

Most distributed services treat compute and storage as separate problems: the program runs on one machine, the data lives on another, and a network protocol bridges the gap. If the process dies, the data might survive — but only because a separate replication system copied it somewhere else.

On grafOS, the storage is the fabric. A tasklet writes key-value data into leased fabric memory (fast, volatile) and leased block storage (persistent, durable). When the tasklet dies, the cold tier’s block lease can outlive the crash — a replacement tasklet promotes surviving cold entries on first access.

This is the recipe that answers: why would I put my program on a fabric instead of just running it on a server?

Kill the compute, keep all the data. When a tasklet crashes, a replacement can reconnect to surviving leases via MemBuilder::attach() and BlockBuilder::attach(), then rebuild the KV store with KvBuilder::from_leases() (both tiers survived) or KvBuilder::from_cold() (hot tier expired, cold survived). This recipe shows both paths.

Cross-Domain Boundary

This recipe is about lease-backed service recovery: the data survives compute failure when the backing memory or block leases survive. It is not a shared replicated KV by itself. Independent service replicas each own their own FabricKvStore; they do not automatically share writes, quorum, or conflict resolution.

Use replicated resources when the service needs one logical KV view across availability zones, regions, or providers:

ReplicatedMap for mutable key/value state and CAS;
ReplicatedFabricLog for ordered mutations or audit records;
ReplicatedObjectStore for large immutable values;
ReplicatedCheckpoint for named latest snapshots;
PlacementPolicy and ReplicaPolicy for cross-domain availability.

What You Need

A fabric with at least two nodes (sim mode works fine)
grafos-std for fabric memory and block storage APIs
grafos-kv for the tiered key-value store
grafos-scheduler for placement and failover

Step 1: Build the Tiered KV Store

KvBuilder allocates its own leased storage internally — you configure capacity and TTL, and it acquires the underlying fabric memory and block leases for you.

use grafos_kv::KvBuilder;

let mut store = KvBuilder::new()
    .hot_buckets(256)          // hash map capacity in leased DRAM
    .default_ttl_secs(300)     // per-key TTL
    .cold_num_blocks(1024)     // enable block-backed persistence
    .build()?;

cold_num_blocks, from_leases, and from_cold require the grafos-kv persistence feature. Without that feature, the public API is the hot-tier FabricKvStore over leased memory:

use grafos_kv::{FabricKvStore, KvBuilder};

let mut store: FabricKvStore = KvBuilder::new()
    .hot_buckets(256)
    .default_ttl_secs(300)
    .build()?;

store.put(b"user:42", b"active")?;
let value = store.get(b"user:42")?;
let evicted = store.tick()?;
# let _ = (value, evicted);
# Ok::<(), grafos_std::FabricError>(())

build() does the following:

Acquires a fabric memory lease (via FBMU) sized for 256 hash buckets and creates a FabricHashMap — this is the hot tier.
Acquires a block storage lease (via FBBU) for 1024 blocks and creates a ColdTier append-only log — this is the cold tier.

The leases are owned by the FabricKvStore. They are not tied to the tasklet’s linear memory — the data lives on the fabric, accessed through lease-bound hostcalls.

Under the hood:

Hot tier: FabricHashMap in leased memory. Key lookups and writes go here first. Open addressing with linear probing, all I/O through FBMU hostcalls.
Cold tier: append-only log in leased block storage. Keys that survive a hot-tier eviction are spilled here. On startup, cold entries are promoted back to the hot tier on first access.

Step 2: Serve Requests Through Service Hostcalls

The tasklet accepts connections via native service hostcalls and dispatches key-value operations.

// svc_listen requires a cap_handle with RIGHTS_SVC_LISTEN for the port.
let listener = svc_listen(cap_handle, 8080, 64)?;

loop {
    let session = svc_accept(listener)?;
    if session < 0 { continue; } // no pending connection

    // svc_read / svc_write operate on session handles, not file descriptors.
    // See docs/grafos/service-abi-v0.md for the full hostcall table.
    let mut buf = [0u8; 1024];
    let n = svc_read(session, &mut buf)?;
    let request = &buf[..n as usize];

    let response = match parse_request(request) {
        Op::Put(key, value) => {
            store.put(&key, &value)?;
            b"OK".as_slice()
        }
        Op::Get(key) => {
            match store.get(&key)? {
                Some(val) => &val,
                None => b"NOT_FOUND",
            }
        }
        Op::Delete(key) => {
            store.delete(&key)?;
            b"DELETED"
        }
    };

    svc_write(session, response)?;
    svc_close(session)?;

    // Periodic maintenance: evict expired keys
    store.tick()?;
}

No filesystem. No POSIX bind() / accept() / read() / write(). The service hostcalls (svc_listen, svc_accept, svc_read, svc_write, svc_close) operate on leased listener and session handles.

Step 3: Survive a Tasklet Crash

Here is where fabric-backed storage pays off.

When a tasklet crashes, its compute lease expires. The hot tier’s memory lease is currently tied to the tasklet lifetime and expires with it. But the cold tier’s block lease can outlive the tasklet if its TTL has not expired — the append-only log on leased block storage is the durable state.

The scheduler detects the compute failure and starts a replacement:

1. Tasklet crashes on node A.
2. CPU lease expires → instance state = Fenced.
3. Scheduler places replacement on node B.
4. Replacement attaches to surviving leases by ID.
5. KvBuilder::from_leases() or from_cold() rebuilds the store.
6. Data is immediately accessible — no re-ingestion.

Two recovery paths, depending on which leases survived:

Both tiers survived (hot-tier memory lease TTL has not expired):

use grafos_std::mem::MemBuilder;
use grafos_std::block::BlockBuilder;
use grafos_kv::KvBuilder;

// The scheduler passes surviving lease IDs to the replacement tasklet.
let hot_lease = MemBuilder::attach(hot_lease_id)?;
let cold_lease = BlockBuilder::attach(cold_lease_id)?;
let mut store = KvBuilder::from_leases(hot_lease, cold_lease)?;

// All hot-tier keys are immediately accessible — no promote needed.
let val = store.get(b"user:42")?; // reads directly from recovered FabricHashMap

Cold tier only (hot-tier lease expired with the tasklet):

let cold_lease = BlockBuilder::attach(cold_lease_id)?;
let mut store = KvBuilder::from_cold(cold_lease, 64)?;

// Cold-tier keys are promoted to the fresh hot tier on first access.
let val = store.get(b"user:42")?; // scans cold log, promotes to hot

The cold-only path is the common case: memory leases typically have short TTLs (tied to the tasklet’s compute lease), while block leases can be set with longer TTLs to survive crashes.

Step 4: Run Independent Replicas For Service Availability

For availability, run multiple independent replicas under one service identity:

use grafos_core::ResourceKind;
use grafos_scheduler::{
    NodeConstraint, Priority, ReplicationMode, ResourceRequirement, ServiceId,
    ServiceSpec, Strategy, TenantId,
};

let spec = ServiceSpec {
    service_id: ServiceId {
        name: "kv-api".into(),
        tenant_id: TenantId(7),
    },
    version: "1.0.0".into(),
    module_hash: kv_module_hash,
    wasm: kv_module_bytes,
    replication: ReplicationMode::ActiveActive { replica_count: 2 },
    priority: Priority::Standard,
    strategy: Strategy::Spread,
    node_constraint: NodeConstraint::Any,
    anti_affinity_services: vec![],
    resources_per_instance: vec![
        ResourceRequirement { resource_type: ResourceKind::Cpu, capacity: 1 },
        ResourceRequirement { resource_type: ResourceKind::Mem, capacity: 32 * 1024 * 1024 },
    ],
    listener_port: 8080,
    max_sessions: 64,
    drain_deadline_secs: 30,
    required_rights: 0,
};

Each replica builds its own FabricKvStore (its own leased memory and block storage). The replicas are independent — there is no shared mutable state between them. Each replica owns its data.

This is the correct starting point for many services: independent replicas, each with their own storage, fronted by a resolver that routes clients to healthy instances. It is not the same thing as a shared replicated KV. Shared-state patterns require explicit fencing and, under cross-domain availability should use replicated map/log/object/checkpoint resources rather than independent per-replica stores.

When a replica dies:

The surviving replica continues serving its own data.
The scheduler starts a replacement.
The replacement attaches to the dead replica’s surviving cold-tier block lease via BlockBuilder::attach() and rebuilds via KvBuilder::from_cold(). If the hot-tier lease also survived, KvBuilder::from_leases() recovers both tiers with zero data loss.

Why This Is Different

Traditional service	grafOS native service
Data lives in process memory	Data lives in leased fabric memory / block storage
Persistence = write to disk or external DB	Persistence = cold tier in leased block storage
Crash = data lost unless replicated	Crash = compute lease expires; cold tier survives if TTL holds
Recovery = replay log or restore backup	Recovery = attach to surviving leases; cold tier promotes on access
Migration = copy data to new host	Migration = start new tasklet; cold tier block lease is fabric-reachable

The fundamental insight: the program’s durable state does not live inside the program. The cold tier lives on leased block storage, accessible to any authorized tasklet. Killing the compute does not kill the persisted data — as long as the block lease TTL has not expired.

The hot tier (leased fabric memory) can also survive a crash if its TTL has not expired. MemBuilder::attach() + KvBuilder::from_leases() reconnects a replacement tasklet to both tiers with zero data loss. If the hot-tier lease has expired, KvBuilder::from_cold() rebuilds from the surviving cold tier.

Failure Modes

Failure	Behavior
Tasklet crash	CPU lease expires; replacement attaches to surviving leases via `from_leases()` or `from_cold()`
Node dies	All leases on that node expire after TTL; replacement builds fresh store
Memory lease expires	Hot tier lost; cold tier promotes on next access in new store
Block lease expires	Cold tier lost; new store starts empty
All replicas down	Each replica’s leases tick toward expiry independently
Network partition	Lease renewal fails; after TTL, storage is fenced; partition heals → re-acquire
Cross-provider shared writes	Not provided by independent `FabricKvStore` replicas; use `ReplicatedMap` or a log-backed state machine

The Lease TTL Design Choice

Lease TTL is the knob that controls the tradeoff between safety and availability:

Short TTL (30s): storage reclaimed quickly after failure; less wasted capacity; but renewals are frequent and network hiccups can cause premature expiry.
Long TTL (5m): tolerates network blips; fewer renewal round-trips; but a dead service holds storage for longer before the fabric reclaims it.
Per-key TTL: put_with_ttl() lets hot keys live longer than cold keys, independent of the storage lease itself.

The right TTL depends on the workload. For a session store: 30s–60s. For a configuration cache: 5m–15m. For a durable ledger: as long as the tenant’s budget allows.

Testing This Recipe

In sim mode, the full lifecycle is testable without hardware:

use grafos_testkit::SimFabric;

let mut fabric = SimFabric::new(4);

// Place the KV service, submit tasklets
// PUT some keys, verify GET returns them
// Kill the tasklet's node
// Verify storage leases are still valid
// Start replacement, verify it can read the keys
// Let storage lease expire, verify data is gone