OSW — Object Storage Wrapper

OSW — Object Storage Wrapper — is a small HTTP service, written in Rust, that stands between applications and S3-compatible object storage. Applications never hold storage credentials and never stream file bytes through their own servers: they ask OSW for short-lived signed links, the browser uploads or downloads directly against storage, and OSW manages what happens to a file afterwards — committing it, deleting it softly, restoring it, or copying it between buckets.

I built it once and reuse it across projects, including the backend of this website, because every one of them needs the same few things from file storage, and getting those few things right is harder than it looks. Talking to storage directly spreads credentials into every application, sends every file the long way through an application server, and leaves each application to reinvent the lifecycle a file needs.

At a glance

What it does

A. Hands out signed links instead of carrying files

A presigned URL is a temporary link, signed with the storage credentials, that allows one specific upload or download without handing over the credentials. OSW issues them for upload, download, or both, and names every new object itself, so the browser talks to storage directly and OSW never carries a byte of a file.

B. Gives each file a lifecycle

A file uploaded while a user fills in a form is not yet part of anything, and a deleted file should usually be recoverable for a while. OSW models both as a place on the path, inside the same bucket: a fresh upload lands in a temporary area, committing it moves it to its permanent path, a soft delete moves it to a deleted area, and a restore moves it back. Abandoned uploads are swept by the storage server itself, through a lifecycle rule that expires the temporary area.

C. Copies between buckets on the server

An application can move a file from a private bucket to a public one — for instance when an article's attachment becomes public along with the article — or copy it within one bucket, optionally removing the source. The copy happens inside the storage server; no byte passes through OSW or the application.

D. Returns many links at once

A page that lists many files can ask for all their download links in one request, optionally checking first that each file exists, and receives one row per requested path, in the same order.

E. Keeps one contract for every caller

Every response, success or failure, arrives in the same JSON envelope with a request id, every input error is reported at once, and the messages applications already parse stay word for word as they were.

Interface

OSW can be mounted under any route of a gateway. A batch request is the one that carries a JSON body:

{
  "bucket": "archive",
  "paths": ["reports/2026/q3.pdf", "reports/2026/q4.pdf"],
  "verify_exists": true
}

Standards

Inside

OSW follows Clean Architecture: storage rules in the domain, use cases grouped by resource, and S3, HTTP, configuration, and telemetry each behind a port or confined to their own layer. I rebuilt it into that shape layer by layer from an earlier version, in nine phases, and the last phase proved the published API specification byte-identical to the old one, so no application had to change. Clock and id generation are ports too, which makes every timestamp and name in a test predictable, and its 98 tests run without a storage server because each use case receives an in-memory storage that records every call.

Design decisions

Only OSW holds the keys, and links are signed for the address the browser will use. No application ever sees the storage credentials. A signature covers the host name, and the host OSW uses inside the network is usually not the one a browser can reach, so links are signed against the public origin the browser will use. The public origin is checked when the service starts, so a bad value fails at boot rather than on the first upload. The handles for the public origin are built once per bucket at start-up, because building one loads the system's trust store — about 305 microseconds of blocking disk work — while signing itself takes about 14, and a batch of 100 would have paid the first cost 100 times. The content type of an upload is placed inside the signed query rather than a signed header, so the browser does not have to repeat it exactly yet cannot change it, and path-style addressing carries over to the public handles so links stay correct behind a proxy. New objects are named {directory}/{uuidv7}.{extension} by OSW itself, so names never collide, sort by time, and cannot be chosen or overwritten by a client; the extension comes from a table of content types with a sensible fallback. A permission flag issues only the links a caller needs — upload, download, or both — and the link lifetime is configurable, with the expiry time returned so clients know when to ask again.

A file's state is visible in its path, and every change checks first. Uploads are temporary unless the caller says otherwise. Committing and restoring are a copy inside the storage server followed by a delete, so no bytes move however large the file. Paths are validated strictly and the object is looked up before any change, so a missing file is a clean 404 rather than a storage error. Deleting is soft by default and can be undone. OSW keeps no state and runs no sweeper: abandoned uploads are the storage server's job, done by its lifecycle rule.

Defaults are safe where they are harmless, and absent where they would hide a mistake. The buckets OSW may touch are exactly the buckets it is configured with, and it refuses to start with none. An unknown bucket is the caller's error, a 400, not a server fault. Single-file requests fall back to a default bucket, but a copy must name both buckets, because silently falling back would hide the caller's mistake.

A copy between buckets is a real server-side copy, and it never loses the source. The S3 client OSW uses can copy only within one bucket, so a cross-bucket copy presigns a PUT against the target with a signed x-amz-copy-source header and sends it through one reused HTTP client. That internal link never leaves the process. The header is sent as well as signed, or the signature would not match; the source key is encoded segment by segment, with spaces as %20 and never +, which storage would read as a literal plus. A copy that answers 200 with an error inside its body counts as a failure, because S3 can report a late copy failure that way. The source is never deleted when the copy failed; when the copy succeeded but deleting the source did not, the answer is still 200 with source_deleted: false — the backend team that asked for the endpoint wanted that, so a client does not repeat a copy that already happened. The target key defaults to the source key, copying within one bucket is allowed, and every validation message names the side it is about. TLS is rustls only, with the HTTP types pinned to the version the S3 client expects.

A batch fails only where it has to. The batch endpoint was added beside the single-file one after consumers worked around it by firing one request per file at once, which only moved the load onto OSW. A problem that affects every row — an unknown bucket, too many paths, an empty list — fails the whole request; a problem with one row fails only that row, and every other link is still returned, in the order requested, with duplicates echoed once per occurrence:

{
  "full_path_file": "reports/2026/q4.pdf",
  "download_url": null,
  "expired_at": null,
  "error": "not_found"
}

Every row carries the same four keys with explicit nulls, holds either a link or an error and never both, and uses a small vocabulary owned by the domain: not_found, invalid_path, and unavailable, the last kept apart so that a passing storage hiccup is retried rather than treated as a missing file. Whether each file is checked first has no default, because checking costs one storage request per file and skipping it loses the "missing" signal. Checks run with bounded concurrency, a limit kept separate from the cap on paths because the two exist for different reasons, so one caller cannot crowd out the others. One expiry time is computed for the whole batch, row errors are logged where they happen, and each row reuses the single-file path rules.

One contract, kept exactly. Every response is wrapped in one envelope, which replaced a dozen near-identical helpers. The JSON body limit is set on purpose, and malformed JSON still receives the envelope instead of the framework's plain-text error. Validation collects every problem before answering. Use-case errors are types that the HTTP layer maps to status codes, replacing an older trick that read prefixes of error strings, and a 500 says only which operation failed, with the cause in the log. The 401 body is the one deliberate exception, kept in its old shape for clients that already read it, as are the messages those clients parse. A missing object reported by the S3 client in its own inconsistent way is still recognised as a 404.

Identity comes from the gateway, and every request can be traced. OSW sits behind an API gateway that authenticates every caller and passes on only verified identity. A request id is accepted when the caller sends one and generated otherwise, then echoed back. Telemetry moved from writing straight to Elasticsearch to OpenTelemetry through a collector: incoming traces are continued, metrics use delta temporality because the collector's Elasticsearch exporter otherwise drops histograms and counters, HTTP metrics are labelled by route pattern rather than raw path, the client address comes from forwarded headers only when they arrive from trusted proxies, sensitive keys are redacted from logs, and telemetry can be switched off entirely, with a flush on shutdown when it is on. Configuration is read once, at start-up.

It ships small and runs unprivileged. The image is built in two stages, runs as a non-root user, and reads its configuration from a mounted file.

Specifications

Related

Before OSW is put in front of a storage server, that server is checked with this script.

Proving the storage behind OSW is ready before anything depends on it.

OSW is the case study of this article about code that the next reader can follow.

The template that brings up an object storage server together with OSW for a new project.

Where OSW's traces, logs, and metrics go once they leave the service.

Terms

Object storage

Storage that keeps files as whole objects in named buckets, reached over HTTP.

Presigned URL

A temporary link, signed with storage credentials, that allows one specific upload or download without giving the client those credentials.

Soft delete

Moving a file to a deleted area instead of destroying it, so that it can be restored.

Lifecycle rule

A rule configured on the storage server that expires or moves objects automatically, for example everything under a prefix after one day.

API gateway

A service in front of other services that authenticates callers and forwards only verified identity to them.

Port

An interface the inner layers define and the outer layers implement, so the rules never depend on a particular tool.