Storage
Object storage — an injectable Storage client whose URL picks the backend, s3:// or memory://, proved at boot, with presigned URLs.
Object storage shares the rest of the framework’s shape. Storage is a
regular provider: inject it, call a handful of async methods. One URL names the
store — s3://media in production, memory:/// in your tests — and the boot
proves the bucket answers before your app serves a request. The client hands
out short-lived presigned URLs, so bytes never transit your API, and reads
or writes objects server-side for workers that transform them.
The client ships in nest-rs-storage, built on
object_store — the object-store library
maintained under Apache Arrow. object_store already abstracts the vendor, so
nestrs adds no driver trait over it: the URL’s scheme picks the object_store
backend, the way a Redis URL’s scheme picks its topology.
Install
Section titled “Install”cargo add nest-rs --features storageWire it in
Section titled “Wire it in”StorageModule owns its StorageConfig (namespace storage, loaded
from <PREFIX>_STORAGE__*), opens the store at boot and registers Storage
as a singleton provider. Import it where a feature needs storage:
use nest_rs::core::module;use nest_rs::storage::StorageModule;
use crate::audio::AudioService;
#[module(imports = [StorageModule], providers = [AudioService])]pub struct AudioModule;Then name the store, in the file of the profile that runs beside it — the demo’s names the dev container’s RustFS:
NESTRS_STORAGE__URL=s3://nestrsNESTRS_STORAGE__ENDPOINT=https://rustfs:9000NESTRS_STORAGE__ACCESS_KEY=nestrsNESTRS_STORAGE__SECRET_KEY=nestrsA workspace nestrs new wrote has these lines already. Its compose.yml runs
a RustFS store on localhost:9000 and creates two buckets: the one
.env.development names (s3://<workspace>) and the one .env.test names
(s3://<workspace>-test), so what a test uploads never sits among what a
running app lists. It speaks plain HTTP there, which storage accepts under a
declared development or test profile only.
Name the store
Section titled “Name the store”The URL’s scheme picks the backend, and a path after the bucket confines every key under it:
| URL | Backend | Presigns |
|---|---|---|
s3://<bucket>[/<prefix>] | AWS S3, or a server speaking its API at ENDPOINT | yes |
memory:///[<prefix>] | a store in this process’s memory — development and tests | no |
s3://media/tenants/acme stores covers/a.png at tenants/acme/covers/a.png
in the bucket, and lists it back as covers/a.png. A prefix a URL would have to
encode — a space, %, a letter beyond ASCII — fails the boot, since keys would
land beside it. Any other scheme — file,
gs, az — fails the boot naming the scheme and what this release opens; the
URL itself is never quoted, since a pasted one may carry a credential.
memory:/// is the cheapest level to test storage at: a TestApp booting it
needs no server, and every call runs as in production but presigning, which
answers StorageError::Unsupported. A staging or production profile refuses
it, since a restart loses every object — and so does a process that declares
no <PREFIX>_ENV. Beside it, every S3 setting the deployment sets (ENDPOINT,
REGION, the keys, TLS_CA_CERT…) fails the boot naming that variable,
rather than being read by nothing — .env included, which a test reads too.
So a test whose .env names an S3 store opens memory either by blanking, in
.env.test, each S3 key .env sets (an empty value is unset), or by pinning
it: StorageModule::for_root(StorageConfig { url: "memory:///".into(), ..Default::default() }).
At boot
Section titled “At boot”The store is proved before any provider is built. An s3:// bucket answers one
page of its listing under the URL’s prefix, within OPERATION_TIMEOUT_SECS:
- a bucket that does not exist fails the boot at once, naming
<PREFIX>_STORAGE__URL; - a certificate the client does not trust fails it at once, saying which
authority to install or name in
TLS_CA_CERT; - a status every attempt would meet — a redirect, the sign of a wrong region,
or a
400— fails it at once, naming the status and the variables that shape the request:REGION,ENDPOINT,FORCE_PATH_STYLE; - a store that rejects the key — a
401, or S3’sInvalidAccessKeyId,SignatureDoesNotMatchorInvalidSecurity— fails it at once, namingACCESS_KEYandSECRET_KEY, since every call would fail the same way; - any other
403boots on with onewarnonnest_rs::storagenaming the credential variables. It is most often a policy withouts3:ListBucketon the prefix: a least-privilege policy is no reason to stop an app, and granting it silences the line. The line’scodesays what S3 answered when it isAccessDenied,AccountProblem,AllAccessDisabledorRequestTimeTooSkewed, so a skewed clock or a suspended account reads there; those last three fail every call. A valid key whose policies were all removed answersAccessDeniedtoo, so this line is all it gets until its first call fails; - anything else — no whole answer, a
5xx, a408or a429— is retried within the budget, each attempt awarn, then fails the boot as unreachable with its last cause.
memory:/// proves at once. The suite runs against RustFS 1.0 over TLS; AWS S3
and the other servers speaking its API answer the same requests, but no test of
ours runs against them.
On every readiness probe
Section titled “On every readiness probe”StorageModule attaches the store’s indicator, so an app importing
HealthModule gates /health/ready and /health/startup on one attempt of
that proof, under the key storage, with no import of its own:
what the boot refuses reads down, a key the store rejects included; what it
boots on reads up — a policy refusing the listing included, with the boot’s
warn filed by the probe that first meets the refusal, never by every probe —
and what it would retry reads down until the store answers again. One listing request per
probe; see
Indicators.
Uploads and downloads
Section titled “Uploads and downloads”A presigned URL keeps an upload’s bytes off your API, and a worker reads and writes objects itself, within a bound: Uploads and downloads.
| Method | Purpose |
|---|---|
connect(&config) | open and prove a store outside the container — a tool, a test |
url() -> &str | the store’s URL, shown without what a URL may carry beside its bucket and prefix |
presign_put(key, &put, expires) -> PresignedRequest | a signed PUT for direct client upload, binding put’s content type |
presign_get(key, expires) -> PresignedRequest | a signed GET for serving private originals |
head(key) -> Option<ObjectMetadata> | the object’s size, content type, ETag and last write; None if absent |
get_bytes(key) -> Bytes | download full bytes (server-side) |
get_stream(key) -> impl Stream<Item = Result<Bytes>> | download chunk by chunk, never buffering the whole object |
list(prefix) -> impl Stream<Item = Result<ObjectEntry>> | the keys under a prefix, streamed page by page; "" lists every key the URL reaches |
put_bytes(key, bytes, content_type) | upload bytes — takes anything Into<Bytes>, so the Bytes from get_bytes composes without a copy |
put_stream(key, content_type, stream) | upload from a stream of std::io::Result<Bytes> as a multipart upload, one 5 MiB part at a time |
delete(key) | remove an object; an absent key succeeds, so retention sweeps and failed-upload cleanup are idempotent |
list matches on path segments, so posts/cover returns posts/cover/a.png
and never posts/cover-2.png. put_stream aborts its multipart upload on any
failure — the store’s or your stream’s — so an interrupted upload leaves no
billable parts behind.
StorageError says no problem of its own: an infrastructure error does not
get to pick what a client reads, so wrap it in the feature’s own error, which is
where that is decided — see
audio/error.rs,
an error the client is owed no word of, answered with .opaque()? and its
chain filed at error. Its sentences
name the store and the operation, never a key: a key often carries user input,
so what object_store’s own errors add is said by their kind, without the key,
the URL or the store’s document they quote. The error holds that sentence and
nothing below it, so a {:?}, an anyhow report or a main returning it prints
no more than error_message says, and no line on nest_rs::storage names a key:
each names the store, and the trace names the call. The one failure a caller
decides on is its own variant: a download of a key holding nothing answers
StorageError::NotFound, where head answers None and delete succeeds. A
StorageError converts into
std::io::Error, so a get_stream feeds a streamed response body directly.
Configuration
Section titled “Configuration”Every field is settable via <PREFIX>_STORAGE__* env or a pinned
StorageConfig — the framework-wide dual-path config rule:
use nest_rs::storage::{StorageConfig, StorageModule};
StorageModule::for_root(StorageConfig { url: "s3://acme-media".into(), ..Default::default()}),Default is the one base, the bare import’s and a pin’s — no store, no
credential, and plain HTTP a
Surface left to the profile.
A declared development or test profile allows plain HTTP and names nothing more:
the store, its endpoint and its pair are the deployment’s, or the profile’s
.env file’s, in every profile, so an unset one fails the boot by name instead
of reaching a default.
| Key | Default | Meaning |
|---|---|---|
URL | none | the store, whose scheme picks the backend |
ENDPOINT | unset | S3 only: the server’s endpoint; unset ⇒ AWS |
TLS_CA_CERT | unset | S3 only: the authorities the endpoint’s certificate must chain to, replacing the system’s (TLS_CA_CERT_FILE for a path), read at the start: a renewed file takes effect at the next one |
TLS_CERT / TLS_KEY | unset | refused: object_store presents no client certificate, so setting one fails the boot |
REGION | us-east-1 | S3 only: the region requests are signed for |
ACCESS_KEY / SECRET_KEY | none | S3 only: static credentials |
FORCE_PATH_STYLE | true | S3 only: true ⇒ endpoint/bucket/key; false ⇒ virtual-hosted |
ALLOW_HTTP | Surface::ByProfile: open in dev/test, closed otherwise | S3 only: reach the endpoint over plain http:// — see the caution below |
OPERATION_TIMEOUT_SECS | 15 | the most a call, and the boot’s proof, wait for the store, retries included |
READ_TIMEOUT_SECS | 30 | the most a download waits for its next bytes while it is read |
So an s3:// store sets three keys at minimum — URL, ACCESS_KEY and
SECRET_KEY — and ENDPOINT unless it is AWS; memory:/// sets URL alone.
Limits
Section titled “Limits”- Two backends.
s3andmemoryopen in this release;file,gsandazare refused until someone needs them. - Static keys only. Ambient AWS credentials — an instance profile, IRSA,
EKS Pod Identity — are not read: object_store reads them only from
AWS_*variables, which would bypass the<PREFIX>_STORAGE__*namespace. - One store per app. Named stores,
Storage<I>beside the default, wait for a seam that keeps a generic module’s providers in the access graph (instances); the default store’s spelling stays when they arrive. - An upload is bound by its type, not its size.
object_storesigns noContent-Length, so a presignedPUTtakes any size up to S3’s 5 GiB: checkhead’sbyte_sizeonce it lands. - Redirects are object_store’s. The framework’s outbound clients follow no redirect; object_store follows S3’s, and offers no option to refuse them.
Reference
Section titled “Reference”crates/nest-rs-storage/—Storage,SignedPut,PresignedRequest,ObjectMetadata,ObjectEntry,StorageConfig,StorageModule, andStorageError; the store’s indicator under thehealthfeature.audioin the demo — mints the key, presigns an upload bound toaudio/mpeg, and the worker reads the stored object and writes a derived one.
Going further
Section titled “Going further”- Uploads and downloads — presigned
PUTandGET, server-side reads and writes, and how long a call waits. - Queue — background workers that read and transform stored objects.
- Database — persist the record a presigned upload finalizes.
- Configuration — the dual-path rule behind
StorageConfig.