Skip to content

Storage

Object storage — an injectable Storage client whose URL picks the backend, s3:// or memory://, proved at boot, with presigned URLs.

Object storage shares the rest of the framework’s shape. Storage is a regular provider: inject it, call a handful of async methods. One URL names the store — s3://media in production, memory:/// in your tests — and the boot proves the bucket answers before your app serves a request. The client hands out short-lived presigned URLs, so bytes never transit your API, and reads or writes objects server-side for workers that transform them.

The client ships in nest-rs-storage, built on object_store — the object-store library maintained under Apache Arrow. object_store already abstracts the vendor, so nestrs adds no driver trait over it: the URL’s scheme picks the object_store backend, the way a Redis URL’s scheme picks its topology.

Terminal window
cargo add nest-rs --features storage

StorageModule owns its StorageConfig (namespace storage, loaded from <PREFIX>_STORAGE__*), opens the store at boot and registers Storage as a singleton provider. Import it where a feature needs storage:

crates/features/src/audio/module.rs
use nest_rs::core::module;
use nest_rs::storage::StorageModule;
use crate::audio::AudioService;
#[module(imports = [StorageModule], providers = [AudioService])]
pub struct AudioModule;

Then name the store, in the file of the profile that runs beside it — the demo’s names the dev container’s RustFS:

.env.development
NESTRS_STORAGE__URL=s3://nestrs
NESTRS_STORAGE__ENDPOINT=https://rustfs:9000
NESTRS_STORAGE__ACCESS_KEY=nestrs
NESTRS_STORAGE__SECRET_KEY=nestrs

A workspace nestrs new wrote has these lines already. Its compose.yml runs a RustFS store on localhost:9000 and creates two buckets: the one .env.development names (s3://<workspace>) and the one .env.test names (s3://<workspace>-test), so what a test uploads never sits among what a running app lists. It speaks plain HTTP there, which storage accepts under a declared development or test profile only.

The URL’s scheme picks the backend, and a path after the bucket confines every key under it:

URLBackendPresigns
s3://<bucket>[/<prefix>]AWS S3, or a server speaking its API at ENDPOINTyes
memory:///[<prefix>]a store in this process’s memory — development and testsno

s3://media/tenants/acme stores covers/a.png at tenants/acme/covers/a.png in the bucket, and lists it back as covers/a.png. A prefix a URL would have to encode — a space, %, a letter beyond ASCII — fails the boot, since keys would land beside it. Any other scheme — file, gs, az — fails the boot naming the scheme and what this release opens; the URL itself is never quoted, since a pasted one may carry a credential.

memory:/// is the cheapest level to test storage at: a TestApp booting it needs no server, and every call runs as in production but presigning, which answers StorageError::Unsupported. A staging or production profile refuses it, since a restart loses every object — and so does a process that declares no <PREFIX>_ENV. Beside it, every S3 setting the deployment sets (ENDPOINT, REGION, the keys, TLS_CA_CERT…) fails the boot naming that variable, rather than being read by nothing — .env included, which a test reads too. So a test whose .env names an S3 store opens memory either by blanking, in .env.test, each S3 key .env sets (an empty value is unset), or by pinning it: StorageModule::for_root(StorageConfig { url: "memory:///".into(), ..Default::default() }).

The store is proved before any provider is built. An s3:// bucket answers one page of its listing under the URL’s prefix, within OPERATION_TIMEOUT_SECS:

  • a bucket that does not exist fails the boot at once, naming <PREFIX>_STORAGE__URL;
  • a certificate the client does not trust fails it at once, saying which authority to install or name in TLS_CA_CERT;
  • a status every attempt would meet — a redirect, the sign of a wrong region, or a 400 — fails it at once, naming the status and the variables that shape the request: REGION, ENDPOINT, FORCE_PATH_STYLE;
  • a store that rejects the key — a 401, or S3’s InvalidAccessKeyId, SignatureDoesNotMatch or InvalidSecurity — fails it at once, naming ACCESS_KEY and SECRET_KEY, since every call would fail the same way;
  • any other 403 boots on with one warn on nest_rs::storage naming the credential variables. It is most often a policy without s3:ListBucket on the prefix: a least-privilege policy is no reason to stop an app, and granting it silences the line. The line’s code says what S3 answered when it is AccessDenied, AccountProblem, AllAccessDisabled or RequestTimeTooSkewed, so a skewed clock or a suspended account reads there; those last three fail every call. A valid key whose policies were all removed answers AccessDenied too, so this line is all it gets until its first call fails;
  • anything else — no whole answer, a 5xx, a 408 or a 429 — is retried within the budget, each attempt a warn, then fails the boot as unreachable with its last cause.

memory:/// proves at once. The suite runs against RustFS 1.0 over TLS; AWS S3 and the other servers speaking its API answer the same requests, but no test of ours runs against them.

StorageModule attaches the store’s indicator, so an app importing HealthModule gates /health/ready and /health/startup on one attempt of that proof, under the key storage, with no import of its own: what the boot refuses reads down, a key the store rejects included; what it boots on reads up — a policy refusing the listing included, with the boot’s warn filed by the probe that first meets the refusal, never by every probe — and what it would retry reads down until the store answers again. One listing request per probe; see Indicators.

A presigned URL keeps an upload’s bytes off your API, and a worker reads and writes objects itself, within a bound: Uploads and downloads.

MethodPurpose
connect(&config)open and prove a store outside the container — a tool, a test
url() -> &strthe store’s URL, shown without what a URL may carry beside its bucket and prefix
presign_put(key, &put, expires) -> PresignedRequesta signed PUT for direct client upload, binding put’s content type
presign_get(key, expires) -> PresignedRequesta signed GET for serving private originals
head(key) -> Option<ObjectMetadata>the object’s size, content type, ETag and last write; None if absent
get_bytes(key) -> Bytesdownload full bytes (server-side)
get_stream(key) -> impl Stream<Item = Result<Bytes>>download chunk by chunk, never buffering the whole object
list(prefix) -> impl Stream<Item = Result<ObjectEntry>>the keys under a prefix, streamed page by page; "" lists every key the URL reaches
put_bytes(key, bytes, content_type)upload bytes — takes anything Into<Bytes>, so the Bytes from get_bytes composes without a copy
put_stream(key, content_type, stream)upload from a stream of std::io::Result<Bytes> as a multipart upload, one 5 MiB part at a time
delete(key)remove an object; an absent key succeeds, so retention sweeps and failed-upload cleanup are idempotent

list matches on path segments, so posts/cover returns posts/cover/a.png and never posts/cover-2.png. put_stream aborts its multipart upload on any failure — the store’s or your stream’s — so an interrupted upload leaves no billable parts behind.

StorageError says no problem of its own: an infrastructure error does not get to pick what a client reads, so wrap it in the feature’s own error, which is where that is decided — see audio/error.rs, an error the client is owed no word of, answered with .opaque()? and its chain filed at error. Its sentences name the store and the operation, never a key: a key often carries user input, so what object_store’s own errors add is said by their kind, without the key, the URL or the store’s document they quote. The error holds that sentence and nothing below it, so a {:?}, an anyhow report or a main returning it prints no more than error_message says, and no line on nest_rs::storage names a key: each names the store, and the trace names the call. The one failure a caller decides on is its own variant: a download of a key holding nothing answers StorageError::NotFound, where head answers None and delete succeeds. A StorageError converts into std::io::Error, so a get_stream feeds a streamed response body directly.

Every field is settable via <PREFIX>_STORAGE__* env or a pinned StorageConfig — the framework-wide dual-path config rule:

apps/api/src/module.rs
use nest_rs::storage::{StorageConfig, StorageModule};
StorageModule::for_root(StorageConfig {
url: "s3://acme-media".into(),
..Default::default()
}),

Default is the one base, the bare import’s and a pin’s — no store, no credential, and plain HTTP a Surface left to the profile. A declared development or test profile allows plain HTTP and names nothing more: the store, its endpoint and its pair are the deployment’s, or the profile’s .env file’s, in every profile, so an unset one fails the boot by name instead of reaching a default.

KeyDefaultMeaning
URLnonethe store, whose scheme picks the backend
ENDPOINTunsetS3 only: the server’s endpoint; unset ⇒ AWS
TLS_CA_CERTunsetS3 only: the authorities the endpoint’s certificate must chain to, replacing the system’s (TLS_CA_CERT_FILE for a path), read at the start: a renewed file takes effect at the next one
TLS_CERT / TLS_KEYunsetrefused: object_store presents no client certificate, so setting one fails the boot
REGIONus-east-1S3 only: the region requests are signed for
ACCESS_KEY / SECRET_KEYnoneS3 only: static credentials
FORCE_PATH_STYLEtrueS3 only: true ⇒ endpoint/bucket/key; false ⇒ virtual-hosted
ALLOW_HTTPSurface::ByProfile: open in dev/test, closed otherwiseS3 only: reach the endpoint over plain http:// — see the caution below
OPERATION_TIMEOUT_SECS15the most a call, and the boot’s proof, wait for the store, retries included
READ_TIMEOUT_SECS30the most a download waits for its next bytes while it is read

So an s3:// store sets three keys at minimum — URL, ACCESS_KEY and SECRET_KEY — and ENDPOINT unless it is AWS; memory:/// sets URL alone.

  • Two backends. s3 and memory open in this release; file, gs and az are refused until someone needs them.
  • Static keys only. Ambient AWS credentials — an instance profile, IRSA, EKS Pod Identity — are not read: object_store reads them only from AWS_* variables, which would bypass the <PREFIX>_STORAGE__* namespace.
  • One store per app. Named stores, Storage<I> beside the default, wait for a seam that keeps a generic module’s providers in the access graph (instances); the default store’s spelling stays when they arrive.
  • An upload is bound by its type, not its size. object_store signs no Content-Length, so a presigned PUT takes any size up to S3’s 5 GiB: check head’s byte_size once it lands.
  • Redirects are object_store’s. The framework’s outbound clients follow no redirect; object_store follows S3’s, and offers no option to refuse them.
  • crates/nest-rs-storage/ — Storage, SignedPut, PresignedRequest, ObjectMetadata, ObjectEntry, StorageConfig, StorageModule, and StorageError; the store’s indicator under the health feature.
  • audio in the demo — mints the key, presigns an upload bound to audio/mpeg, and the worker reads the stored object and writes a derived one.
  • Uploads and downloads — presigned PUT and GET, server-side reads and writes, and how long a call waits.
  • Queue — background workers that read and transform stored objects.
  • Database — persist the record a presigned upload finalizes.
  • Configuration — the dual-path rule behind StorageConfig.