Storage backends

Integration API. Supported asynchronous storage implementations for custom engine wiring.

AsyncStore

AsyncStore(session, io_config=None)

The ArchetypeStore is a component that manages the storage and retrieval of archetype tables.

Since our Schema supports multiple simulations and runs, our namespace is simply "archetypes". This allows us to run multiple simulations across many worlds using the same catalog.archetypes, by definition, the exact set of components attached to an entity. So we don't really which simulation or run we're using, so long as we differentiate between the simulation and run.

Using Daft Sessions/Catalogs enables us to reference archetype tables without having to hold them in memory.

get_archetype_df async

get_archetype_df(
    sig,
    world_id,
    run_id,
    *,
    ticks=None,
    entity_ids=None,
    active_only=False,
    commit_tokens=None,
)

Get all archetypes that contain all of the specified component types.

commit_tokens is the reader-side visibility allowlist: current-generation rows must match a published manifest token. Legacy (v0.2) rows carry no commit identity and are implicitly epoch-0 visible — the allowlist never applies to them.

get_existing_table_schema async

get_existing_table_schema(table_id)

Return an existing table's Arrow schema without creating it.

get_existing_table_df async

get_existing_table_df(
    table_id,
    world_id,
    run_id,
    *,
    ticks=None,
    entity_ids=None,
    active_only=False,
)

Read an existing table by durable physical identity (never creates).

list_signatures async

list_signatures()

List all archetype signatures that have been registered via _ensure_table.

list_committed_signatures async

list_committed_signatures()

List only signatures that have been durably committed via append().

Unlike list_signatures(), this excludes signatures that were only auto-created by get_archetype_df (create-on-read). Use this when you need to distinguish 'sig was never written' from 'sig exists but has no rows at the queried tick'.

append async

append(sig, df)

Append a table with a new dataframe. Returns the durable batch receipt.

flush async

flush()

No-op: appends write through to Iceberg.

shutdown async

shutdown()

Shutdown the store.

AsyncCachedStore

AsyncCachedStore(async_store, cache_config)

This Store variant holds a cache.

get_archetype_df async

get_archetype_df(
    sig,
    world_id,
    run_id,
    *,
    ticks=None,
    entity_ids=None,
    active_only=False,
    commit_tokens=None,
)

Get all archetypes that contain all of the specified component types.

Reads are a union of already-flushed rows on disk and unflushed rows still in the memtable so that callers always see a coherent view of the signature's full history regardless of flush state. The commit-token allowlist applies to both layers, so staged rows are no more visible than durable rows.

list_signatures async

list_signatures()

Delegate to inner store.

list_committed_signatures async

list_committed_signatures()

Delegate committed-sigs check to inner store.

append async

append(sig, df)

Cache driven append with built in flush logic to underlying storage (super) a table with a new dataframe.

The receipt is staged (durable=False) unless this append tripped a flush: rows become durable at flush()/threshold/idle/shutdown. A commit coordinator must call flush() before publishing a manifest head — a head must never claim RAM-only rows are durable.

flush async

flush()

Drain every memtable to the inner store.

Called by the commit coordinator's owner before a manifest head is published, so visibility never outruns durability.

shutdown async

shutdown()

Stop background flushing and ensure all pending data is flushed to the inner store.

AsyncLancedbStore

AsyncLancedbStore(uri, namespace=None)

get_existing_table_schema async

get_existing_table_schema(table_id)

Return an existing table's Arrow schema without creating it.

get_existing_table_df async

get_existing_table_df(
    table_id,
    world_id,
    run_id,
    *,
    ticks=None,
    entity_ids=None,
    active_only=False,
)

Read an existing table by durable physical identity.

Unlike get_archetype_df, this never calls _ensure_table and therefore cannot turn a read into a committed-looking table.

get_archetype_df async

get_archetype_df(
    sig,
    world_id,
    run_id,
    *,
    ticks=None,
    entity_ids=None,
    active_only=False,
    commit_tokens=None,
)

Read one archetype's rows.

commit_tokens is the reader-side visibility allowlist: current-generation rows must match a published manifest token. Legacy (v0.2) rows carry no commit identity and are implicitly epoch-0 visible — the allowlist never applies to them.

list_committed_signatures async

list_committed_signatures()

List only signatures that have been durably committed via append().

Excludes signatures that were only auto-created by get_archetype_df (create-on-read).

flush async

flush()

No-op: appends write through to LanceDB.