ADR-00024: Store locks held while a store is in use, with the repository lock taken inside them¶

Field	Value
Status	accepted
Date	2026-10-07
Deciders	Mattias Jansson

Context and Problem Statement¶

Each local store, the immutable store and the mutable store, is protected against other processes by a lock on a file in the store directory. Before this decision, the store object owned that lock for its whole lifetime: the lock was taken when the store object was created and released when the last reference to the object was dropped.

Two components keep store objects alive without holding the repository lock. The client caches store objects between commands so that their in-memory state can be reused. A storage handle opened through lore_storage_open keeps its stores open until it is closed and never takes the repository lock. A repository command, on the other hand, took the repository lock first and opened the stores afterwards.

Two processes using the same repository could therefore each hold one of the two locks while waiting for the other. The lock wait has no timeout, so neither process returned. With three processes indexing one repository concurrently, this happened in 22 of 30 trials.

Three questions had to be answered. For how long should a store's lock be held? If it is not held for the lifetime of the store object, how can cached in-memory state be trusted when the store is used again? In which order should the store locks and the repository lock be taken?

Decision Drivers¶

The store's read and write operations are the most frequently executed paths in the system. They must not acquire a lock, make a system call or read a file on every call.
A process that has no store open must not block other processes. The client's cache of store objects is only useful if cached state does not hold a lock.
A store must never serve cached in-memory state that another process has changed on disk in the meantime, and a process that is terminated while writing must not leave other processes using state that predates its writes.
Both ways of reaching a store, a repository command and a storage handle, must be covered by the same exclusion.
There must be one lock order, enforced by the types rather than by review, because a violation is a permanent hang rather than a slow path.

Considered Options¶

How long the store lock is held:

For the lifetime of the store object, as before.
For the duration of each operation.
For the duration of a use of the store: from lore_storage_open to the matching close, for one repository command, for the creation of a repository, for as long as a virtual filesystem is mounted, or for the lifetime of a server. Concurrent uses in one process share one acquisition. A store with unwritten changes keeps the lock after the use ends, until a flush writes them.

How cached in-memory state is validated when a store is used again:

Reload the store from disk at the start of every use.
Compare file modification times in the store directory.
Keep a version marker in the store directory that every writer increments before its first write, and compare it against the value the store last saw.

Which lock is taken first:

The repository lock, then the store locks, as before.
The store locks, then the repository lock, with the holder of the repository lock owning the store lock acquisitions it was taken under.

Whether lock state belongs to the store object or to the directory:

One lock per store object, as before.
One lock per store directory, shared by every store object the process has open on that directory.

Decision Outcome¶

Chosen: option 3 for the duration, option 3 for validation, option 2 for the order and option 2 for the ownership.

Duration. The lock is held while the store is in use, where a use is a storage handle from open to close, one repository command, the creation of a repository, a virtual filesystem for as long as it is mounted, or the lifetime of a server. hold_for_command on ImmutableStore and MutableStore begins a use and returns a guard. The lock is taken by the first guard on a directory and released when the last guard is dropped. Operations performed during a use acquire nothing. When a use begins, the lock implementation stores one word that records the current version of the store's state and whether a write has occurred during this use. An operation reads that word and compares it with the version its store object last loaded, which is two relaxed atomic loads and no lock. A write operation also requires that the version marker has been incremented during this use, and the first write performs that increment. An operation performed outside any use, which only tests do, acquires the lock for itself, even while another operation's guard holds it: only a use stores the word, so an operation's own guard, which can be dropped before another operation ends, covers no other operation. Work spawned to outlive the operation that started it holds, until it ends, a guard that operation joined without waiting before spawning it, so the lock is held without a gap; where the operation cannot join, the work acquires a guard of its own before its first operation on the store. That guard keeps the lock for the work and covers no operation: once the use has ended, each of the work's operations acquires a guard of its own, as an operation outside any use does, so no operation outside the work runs on a guard that ends with it. A server's requests run in tasks that outlive the endpoints that accepted them, so the server releases its stores only once its runtime has shut down and dropped those tasks. Builds with the test-util feature, which every test of the stores uses, count each operation a use covers and fail when the use ends while one runs: the release of the lock fails if the operation counted itself first, and the operation fails if the release came first. Other builds, release included, count nothing. The method has no default implementation, so every store type states whether it has anything to lock. Stores reached over the network return nothing, and the composite store delegates to its local store. When the lock is released, each store object on the directory closes the packfiles it holds open, before the release, so that no handle of this process outlives the lock. The I/O driver runs a read, write or sync to its end after the future that issued it is dropped, so packfiles such I/O still holds keep the lock until it ends: a task tries the release again every millisecond, and the packfiles hold the directory's lock state, so the lock outlives a store object dropped with I/O in flight. A bucket or level file write holds a share of its group's writes in flight until its rename has landed, which keeps the lock as packfiles in use do, and the group's next flush waits for it before writing, so a write abandoned part way neither lands over a later write of the same file nor shares its temporary file. A compaction's truncate of a packfile holds a share of its group's writes in flight in the same way, so the lock is kept until a truncate left in flight has landed. An acquisition that joins a lock this process holds for a store object whose state is stale, which a store object opened after another on the directory was dropped always is, waits until no files on the directory are in use before its caller reloads, so the packfiles it sizes include every write a dropped object left in flight. The next operation that reaches a group reopens its packfiles. The size the store counted towards its compaction trigger is taken out as the files close and counted again as they reopen. A compaction pass takes its read claim before it measures the store, since measuring opens the packfiles.

Amendment, 2026-10-21: what test builds fail. Builds with the test-util feature fail when the lock is released while an operation a use covers runs, rather than when the use ends: the release fails if the operation counted itself first, and the operation fails if the release came first. A use that ends while a guard keeps the lock fails nothing. Work spawned to outlive an operation keeps the lock this way: its operations are covered by the use until the use ends, and one admitted before the end and counted after it runs on the work's guard, as in release builds, where the original rule failed it in test builds. Test builds publish, beside the word an operation reads, a second word that says whether this process holds the lock whatever the number of uses, and check an operation's count against it.

Validation. The version marker is a 16-byte file named epoch in the store directory. It holds a random value chosen when the file is first written and a counter. A writer increments the counter before its first write during a use. The increment is staged and renamed into place in one I/O job, which holds the lock until the new value is in place, so an acquisition dropped part way through it does not release the lock with the value still to land, and the marker cannot go back past an increment another process makes once it has the lock. The increment runs in a task of its own, which records the new value as the directory's current one and holds the acquisitions' gate until it has, so an acquisition dropped while the value is written leaves it recorded, and no second writer increments alongside it. Readers do not change the file. When a use begins after the lock was released, a store that reuses its state across uses compares the file against the value it recorded at its last use. If the values differ, if the file cannot be read, or if the store has never recorded a value, the store discards its cached buckets and re-reads the directory before serving anything. A store that does not reuse its state across uses does so whatever the file says. An absent file is a definite state and is treated as equal to an earlier absent file. The random value distinguishes a store directory that was deleted and recreated from the original, since the counter alone would restart from one. A store directory deleted while a store object on it was kept is recreated by the next acquisition, as opening a store creates its directory. That acquisition treats the store as stale whatever it recorded, and writes a marker with a new random value even when it only reads, since the absent file would otherwise equal an absence the store, or another process, recorded for the deleted directory. The recorded value is updated only after a reload has completed, so a reload that fails or is cancelled is retried at the next use. While one store object reloads, other acquisitions on the same directory wait.

Retention between uses. Reusing cached state across uses is opt-in, so that the version marker can be tested by those who opt in while every client writes it. ImmutableStoreSettings and MutableStoreSettings have reuse_state_across_holds, off by default, which the store passes to its lock. The client's disk stores take it from the environment variable LORE_STORE_RETAIN, set to 1 or true and read once per process. The server's stores have it set, since the server claims them for its lifetime once it has configured them and keeps what it loaded before that; other stores keep the default. Within one use the setting changes nothing. A store is read when it is opened under a lock acquisition of its own, which LocalImmutableStore::new_claimed and LocalMutableStore::new_claimed return with the store. The client's functions that open stores pass it on, and every caller keeps it until its own acquisition has joined it, so the lock is not released in between and the use finds the store current: a call loads each store at most once, and a kept store whose marker is unchanged not at all. With LORE_STORE_RETAIN set, the client keeps its stores after every repository call, and after lore_storage_close, for the keep-alive window, store_keep_alive_seconds or 10 seconds, without their locks, whether or not the call set store_keep_alive; the next use compares the version marker. Without it, a call that sets store_keep_alive keeps the stores and their locks for the window: the keep-alive joins the command's acquisitions, without waiting, before the command releases them, so the lock is held throughout and the marker decides nothing. A store whose acquisition cannot be joined is kept without one and reloads at its next use. A call that sets neither keeps nothing. Shutdown does not wait for the keep-alive window to end.

Order. A repository command acquires its store locks first and the repository lock second. A storage handle acquires store locks only. RepositoryLock owns a StoreHold containing the store guards it was acquired under and has no constructor without one. Its own lock field is declared first, so it is released before the store guards. A command that reuses a repository lock created by another command in the same process keeps its own store guards, and the flush spawned at the end of a command holds them until it completes. As a result, the repository's id, config and instance files are read before the repository lock is held; the one write to config happens after.

Ownership. Lock state is kept per store directory and shared by all store objects a process has open on that directory, because the operating system's file lock excludes two file descriptions from each other even within one process. Each store object keeps its own record of the version it last loaded. The store caches resolve a path to the directory it names, so different spellings of one directory reach one store object.

Release and unwritten changes. When the last guard on a directory is dropped, the lock implementation asks each store object on that directory whether any bucket is flagged as changed and not yet written. If one is, the lock is kept, and the flush that writes the remaining buckets releases it. A mutable store's unwritten buckets keep the lock of the immutable store it was opened beside as well, and its flush releases both. Every use acquires the immutable store before the mutable one, so a process keeping the mutable store's lock alone would acquire the immutable store's after it at its next use, and deadlock with a process holding the immutable store's lock and waiting for the mutable store's. A flush acquires only what it needs: a write use when changed buckets exist, a read use when only access timestamps are to be written, and nothing when nothing is flagged. The background eviction and compaction passes inspect in-memory state without acquiring the lock, acquire it only when they find work, and skip a store held by another process rather than waiting for it.

Amendment, 2026-10-23: the mutable store's lock is held inside the immutable store's. The order between the two store locks was kept by the callers, and a caller could reach the mutable store first. A command that found the immutable store in the client's store cache, where a read command's closing flush had kept it, and the mutable store not, opened the mutable store under a lock acquisition of its own and acquired the immutable store's lock after it. Against a process holding the immutable store's lock and waiting for the mutable store's, neither returned. The mutable store's lock is now nested in the lock of the immutable store it is opened beside, a local store or the local store a wrapper keeps: an acquisition that takes the mutable store's lock first takes a hold on the immutable store's, acquiring it if this process does not hold it, and the hold is released only after the mutable store's lock. The order holds whichever store a path reaches first, and whatever keeps the mutable store's lock, unwritten buckets or files in use, keeps the immutable store's with it, which replaces the rule above for a mutable store's unwritten buckets. The hold acquires through a lock that holds no store state, so it reloads nothing and never reports stale.

Supporting changes. Reads and writes of the version marker and the check for a legacy packfile directory go through lore_io::IoDriver rather than through blocking calls on a runtime thread. The driver's OpenOptions gains custom_flags on Unix so that the marker is read without following a link and without blocking on a FIFO, and the marker's staging file is created exclusively, so an entry placed at its path fails the creation rather than being opened. The streaming collector in lore_revision runs its producer as a task and drains the channel to the end on error, so an error does not cancel tasks in the middle of a reload. On Windows the packfiles are opened with read-only sharing, as before. The handles exist only while the lock is held, and no other process writes, truncates or removes a packfile while this one holds the lock, so an open the sharing mode refuses is one made outside the lock, and refusing it is preferable to letting the write through.

Consequences¶

Good: the deadlock is removed. With six processes running the lock interleave probe against one repository, 6 of 15 trials hung before the change and none of 15 after.
Good: a process with no store open does not block other processes, which makes the client's cache of store objects usable alongside other processes.
Good: operations during a use perform two atomic loads and no lock acquisition or I/O. The lock is acquired once per use.
Good: two store objects on one directory cannot block each other, and two spellings of one directory reach one store object.
Bad: every store directory gains the files epoch and epoch.new. A build without this change does not update the marker. If such a build writes to a store while a process with this change that retains stores has the store cached between uses, the newer process cannot detect the write. Running both builds against one store directory at the same time with LORE_STORE_RETAIN set is therefore not supported. Without it, and in sequential use, the newer process reloads at its first use after the lock was released.
Bad: a store that does not reuse its state across uses, which is every client store unless LORE_STORE_RETAIN is set, reloads at the first use after every release of its lock.
Bad: the repository's id, config and instance files are read before the repository lock is held. lore_storage_open already read config without the lock.
Bad: a flush that fails keeps the store's lock until a later flush succeeds, for as long as a bucket remains flagged. A process with a failing disk holds the store against other processes for as long as it runs.
Bad: correctness depends on every production code path beginning a use before operating on a store, because an operation during a use holds nothing of its own. The lock implementation cannot detect a path that begins no use, and only test builds detect a use that ends while an operation it covers is still running.
Amended 2026-10-21: test builds detect the lock released while an operation a use covered is still running, and no longer an operation that outlives its use while another guard keeps the lock until it ends. See the amendment to Duration.
Bad: acquiring the mutable store's lock when this process holds neither acquires the immutable store's first, so an operation on the mutable store alone, such as its delayed flush, waits for another process holding the immutable store, and whatever keeps the mutable store's lock, files in use included, keeps the immutable store's too. A build without the amendment still acquires the mutable store's lock first on the path it describes, and deadlocks against a build with it as against itself, so the order holds once every process using a store has it.
Added 2026-10-23, with the amendment to Release and unwritten changes.
Bad: each increment of the version marker causes every other process that has the store cached to re-read the directory, 256 groups, at its next use.
Bad: a store closes its packfiles whenever the lock is released and reopens them at the next use. A process that keeps a store between uses, such as the client's store cache, reads a group's packfile directory and opens its packfiles again at the first operation that reaches the group in a use. A periodic compaction check in a process that is otherwise idle does the same for every group, since it measures the store under a claim of its own.

More Information¶

Changes in this area should be validated by running processes against each other. The slow test in lore/tests/unit/storage/lock_interleave.rs creates a repository, then runs six copies of the test binary against it at once, each retaining stores and holding a storage handle open while it runs repository commands, under a deadline; a copy that does not complete is a hang. It is run on demand with cargo test -p lore --test unit -- --ignored --exact storage::lock_interleave::a_storage_handle_held_open_does_not_deadlock_repository_commands_across_processes. The unit tests in lore-storage/tests/unit/local/store_lock.rs cover the lock implementation, using a barrier available under the test-util feature to stop a marker write in progress. The store_retention target in lore-integration-tests covers how often a call loads each store and what a process keeps of its stores after a call and after a close, with and without LORE_STORE_RETAIN, each in a copy of the test binary. Its across_processes tests run three copies as store clients over one store directory, driven one request at a time over their stdin and stdout: one writes, one keeps its store objects between calls, and one opens them for every call. Each test checks the loads each call makes and what it reads after the writer puts, obliterates, copies, compacts, evicts, fans a group out or swaps a mutable value, and that a call waits for another process's unflushed change until a flush or the delayed flush writes it, for both stores where the change is to the mutable store alone. Its locks tests cover the order of the immutable store's, the mutable store's and the repository's locks for the calls a client makes, each in a copy of the test binary. With one of the three held through a descriptor of the test's own, which the lock excludes as it excludes another process, a read call and a write call, and for the two store locks a storage handle, wait holding every lock before it and none after it, whether the process kept nothing of its stores from the call before or kept only the immutable store, on stores of the repository's own, and for the immutable store's lock on a shared store too. A probe that takes the immutable store's lock whenever it is free finds the mutable store's free every time through a mix of calls, keep-alives, closing flushes, storage handles held open and calls at once, and the process holds neither lock once the mix has ended. Six copies running such mixes at once on one repository, and on repositories over one shared store, half of them retaining their stores, all finish. The integration tests in lore-integration-tests/tests/storage_test.rs check that a handle holds both stores until it is closed and that a command holds its stores before it waits for the repository lock. The store stress suite in scripts/store-stress runs the same comparisons longer. Its retention mode scripts calls between store client processes over one store directory, covering every store operation, a writer killed with SIGKILL in a call, with changes unflushed, in a fan-out commit of either store, or at each pause point a call reaches, and flushes their callers dropped part way, and checks the loads and the outcome of each call, that every process reads the same after a kill, and the store's files after each script. Its multi-process mode runs processes that keep their stores, some for a keep-alive window, and processes that open them for each call, at random, client or server shaped, some collecting the immutable store's garbage as clients do, some claiming their stores from several tasks at once as a service's concurrent commands do, every one dropping operations, the flushes that end its calls, openings and claims unfinished, killed at random or at a chosen file operation, some under simulated I/O errors on either store's files or with file operations slowed past the keep-alive window, so that store objects are dropped and opened again with writes in flight. It holds every read to what their shared journal settled and the loads of each call to the writes the journal records, increments shared counters by compare-and-swap from every process, holds the immutable store's bucket files to its memory after flushes, fails a store error that no simulated error or caller explains, and verifies the store against the journal afterwards, which records every claim, so that two processes found holding a store's claims at once fail it. Its scenarios cover a compaction dropped with a bucket write in flight and the next flush of the bucket, and a store object opened in a process after another on the directory was dropped with a packfile or bucket write in flight. Its single-process modes run operations both outside any span and in spans. Every kill under the file-layer test double, every simulated error and every cancellation is counted against the store operations running and the file operation it landed at, and the summaries list the operations and their own steps none reached, at which the sweeps and multi-process aim most of their kills and simulated errors.

Four details are easy to get wrong. The operating system's file lock excludes file descriptions, not processes, so a second lock on a file this process already holds waits forever. Resetting a bucket must keep its serialize_lock, because a writer may hold a clone of it. A reload must clear the store's record of having loaded every bucket and its loaded-size counters, or compaction, verification and the legacy packfile upgrade operate on a store they believe to be complete and which holds nothing. The recorded version must be updated only after a reload has completed, and before other acquisitions are allowed to proceed.

This decision should be revisited if a store has to be used by several processes at the same time, for example a server and a client sharing one directory. The file lock would then have to become a reader-writer lock. The version marker already supports that, since readers do not change it.