How I Prove an S3 Server Is Ready Before an Application Depends on It

An object store that answers a connection attempt has told you one thing: a process is listening. It has not told you that a browser will be able to upload directly to it, that a file nobody should see is actually unreadable, or that a hundred-megabyte upload will survive the proxy in front of it. Those are separate claims, and each one can fail on its own while everything around it looks healthy.

I learned to treat them as separate because the failures arrive late. The upload path breaks in a feature nobody has built yet; the access policy is wrong in a way that only becomes visible when the wrong person opens a link. So before a service depends on a storage server, I run a health check of my own against it — one that exercises every capability the application will eventually rely on, and reports which ones it actually proved.

Introduction

This article is about that health check and the reasoning inside it. It is not a guide to running it; the snippets exist to show why a decision was made. The interesting part is not the code but the gap between "the server responded" and "the server is ready", because that gap turned out to contain six distinct things worth testing.

A. What object storage is, and why "S3" is a protocol rather than a product

An object store keeps whole files — objects — addressed by a name inside a bucket, and hands them back over HTTP. It is not a filesystem: there are no directories, no partial writes in place, no renaming. An object is created, read, or deleted in one piece, and the flatness is what makes it scale.

Amazon published an HTTP interface for their object store, and over time that interface became a de facto standard. Today S3-compatible describes any server that speaks it — MinIO, Ceph, and a long list of others. The practical consequence is that an application written against the S3 API can, in principle, run against any of them. The phrase to hold on to is in principle, because compatibility is a spectrum and the edges are where this article lives.

Two mechanisms matter for everything below. A presigned URL is a link the server signs on a client's behalf: it carries a signature and an expiry, and it lets a browser upload or download directly without ever holding the storage credentials. And a bucket policy decides whether an object can be read anonymously — the difference between a public avatar and a private invoice.

B. What an application actually relies on

When a service says it uses object storage, it is quietly depending on several capabilities that have nothing to do with each other.

It relies on the browser being able to upload directly through a presigned URL, so that large files never pass through the application server. It relies on private objects being genuinely private, and public ones being genuinely reachable. It relies on large uploads being split into parts and reassembled correctly. And it relies on all of that working through whatever proxy sits between the internet and the storage server — which is usually where the size limits actually live.

Each of these is a separate promise. A server can keep three of them and break the fourth, and nothing in a connection test will notice.

C. Why storage failures surface far from their cause

The timing is the problem. A misconfigured upload path does not fail when the storage is provisioned; it fails the first time somebody attaches a file, which might be weeks into building the feature. A proxy body limit does not fail on the small test files used during development; it fails on the first real document a user uploads.

One class is worse than late — it is silent. If a bucket meant to be private is readable anonymously, nothing fails at all. Uploads work, downloads work, the application behaves correctly, and the only symptom is that a link which leaked is a link that works. That failure has no error message and no log entry, and it will not announce itself until somebody outside finds it.

D. Why a connection check does not answer this

Every storage server ships with a way to confirm it is alive, and administrative clients will happily list buckets and print server information. I use those, and they answer a narrower question than the one I have.

Listing buckets proves that credentials are valid and the service is running. It does not exercise a presigned URL, because listing goes through the SDK — the library an application uses to talk to S3 — with credentials attached — a completely different path from the one a browser takes. It does not test anonymous access, because the administrative client is never anonymous. And it certainly does not push enough data to discover that a reverse proxy will reject anything over a few megabytes.

What I wanted instead was an assertion about capabilities: not "the server is up", but "uploads through presigned URLs work, private buckets refuse anonymous readers, public buckets allow them, and a sixteen-megabyte file survives the round trip intact". That is a list of claims, and each claim needs its own test.

E. Six obstacles in the way

Stated as a goal: prove that a storage server is ready for an application to depend on, by exercising every capability the application will use and reporting which ones were actually demonstrated. Under one constraint that shapes the whole design — the check must leave the server exactly as it found it, because I run it against environments that hold real data.

Easy to state, awkward to build. Six obstacles stood in the way, and the first one is really a decision about what the word "ready" is allowed to mean.

The first is that "it connects" and "it is ready" are different claims, and only one of them is easy to test. A connection test is a single question with a single answer, which is why it is the default and why it is not enough. Readiness is a list: upload, download, integrity, access policy, large files, cleanup — and each item can fail while the others pass. The rejected alternative is the comfortable one, checking connectivity and listing buckets and calling the result healthy. What replaces it is a check per capability, each reported separately, so that a summary can say fourteen of fourteen rather than merely "OK".

The second is that testing an upload through the SDK does not test the path the browser will take. The SDK signs every request with the credentials it holds; a browser has no credentials and uses a presigned URL instead. These are different code paths on the server, they interact differently with the proxy in front of it, and one can work perfectly while the other is broken. Uploading through the SDK and assuming presigning follows is the rejected alternative — and it is tempting precisely because the SDK path is the one that is easiest to write. What replaces it is generating the presigned URL and then leaving the SDK entirely, issuing a plain HTTP request the way a browser would.

The third is that a bucket's access policy has to be proven rather than trusted. The configuration states which buckets are private and which are public, and the configuration is a statement of intent, not of fact. A private bucket whose policy is wrong will pass every functional test, because being readable by strangers breaks nothing that a test of my own credentials would notice. Trusting the configuration is the rejected alternative, and the cost of getting it wrong is not an outage but a leak. What replaces it is an explicit assertion from outside: fetch the object with no credentials at all, and require 403 where the configuration says private and 200 where it says public. Anything else is reported as a policy mismatch.

The fourth is that a large-file test must not hold the file in memory, while the upload library expects to rewind it. Generating sixteen megabytes — or several gigabytes, when the size is raised — and keeping it in a buffer is wasteful, and writing it to a temporary file makes the test depend on local disk. A generated stream solves both and introduces a subtler problem: the transfer manager calls seek() to replay a part when a request is retried, and if the replayed bytes differ from the originals, the checksum computed at completion will not match. Both obvious alternatives fail here — a temporary file, and a straightforward random generator that never repeats itself. What replaces them is a stream whose content is a deterministic function of its offset, so that reading position n twice always yields the same byte.

The fifth is that a multipart upload which reports success has not proven it was actually split. The transfer manager will quietly fall back to a single request when it decides chunking is unnecessary, and a proxy or gateway can consolidate parts in ways the client never sees. Trusting that the upload call returned without error is the rejected alternative, and it leaves the most interesting property untested. What replaces it is asking the server afterwards: a multipart object's entity tag carries a suffix naming how many parts it was assembled from, and the part count is only returned when the request explicitly asks about a part. Both are cheap, and together they turn an assumption into evidence.

The sixth is that the host a request is signed for must match the host it is sent to, exactly. Signature version 4 covers the host header, which makes several innocuous-looking normalisations destructive. Appending an explicit :443 to an HTTPS endpoint changes the signed host and invalidates every signature derived from it. Using an internal endpoint to build a URL that a browser will open produces a link the browser cannot resolve. The rejected alternatives are exactly those two conveniences; what replaces them is normalising the endpoint so that a default port is never attached, and rewriting only the scheme and host of a presigned URL while leaving the signed path and query untouched.

The order follows what each obstacle depends on. The first sets the shape — once readiness is a list rather than a question, everything else is an entry in that list. The second and third are a pair about who is making the request: one about the path a client without credentials takes, the other about what a stranger is allowed to see. The fourth and fifth are a second pair, both about large files, and the fifth only becomes answerable once the fourth makes the upload possible at all. The sixth is last because it is the one that breaks everything at once when it is wrong, and therefore the one that is diagnosed most easily by everything else failing together.

Method

A. A configuration that describes capabilities, not just an address

This is the answer to the first obstacle, and it decides what the rest of the tool can assert.

The configuration holds the connection details, and then — the part that matters — a list of buckets, each declaring what it is supposed to allow. Those two boolean flags are what turn a functional test into a policy test, because they give the tool an expectation to compare reality against.

{
  "s3": {
    "endpoint": "https://s3.example.com",
    "region": "us-east-1",
    "addressing_style": "path",
    "access_key": "example-access-key",
    "secret_key": "<secret>"
  },
  "public_origin": "https://s3.example.com",
  "presign_expiry_seconds": 900,
  "buckets": [
    { "name": "private", "upload_auth": true,  "download_auth": true  },
    { "name": "public",  "upload_auth": true,  "download_auth": false }
  ]
}

public_origin is separate from endpoint on purpose. The tool may reach the server over an internal address while the URLs it generates have to be openable from outside; keeping them apart is what makes the check honest about the address a browser will actually use.

Everything downstream reads from this. A bucket marked download_auth: true will be tested for refusal; one marked false will be tested for success. The configuration is not merely settings — it is the specification the server is being measured against.

B. Leaving the SDK to test the path a browser takes

This is the answer to the second obstacle. The presigned URL is generated with credentials and then used without them, and that separation is the entire point of the test.

The first half goes through the SDK, because signing requires the secret. The second half deliberately does not: the request is issued by an ordinary HTTP client, with no credentials anywhere, exactly as a browser would issue it.

# Signed with credentials...
presigned = client.generate_presigned_url(
    "put_object",
    Params={"Bucket": bucket.name, "Key": key},
    ExpiresIn=bucket.presign_expiry_seconds,
)

# ...then used without them, over the origin a browser would reach.
public_url = to_public_url(presigned, cfg.public_origin)
resp = requests.put(public_url, data=payload, timeout=timeout)

Two details are deliberate. The URL is rewritten to the public origin before it is used, so the test exercises the address real clients will hit rather than the internal one. And the response status is checked explicitly, because a proxy that rejects the body returns a perfectly well-formed HTTP error that a client library will not raise on its own.

There is a third detail I would not skip: the URL is masked before it is printed. A presigned URL contains a working signature, and a diagnostic tool that writes one into a log has created a credential leak while trying to prevent one.

C. Asserting the access policy from outside

This is the answer to the third obstacle, and it is the only test here whose failure is a security finding rather than an outage.

The check is deliberately naive: fetch the object with no credentials, and compare the result against what the configuration claimed. What makes it valuable is not the sophistication but the direction — it asks the question from where an outsider stands.

resp = requests.get(object_url(cfg.public_origin, bucket.name, key))

if bucket.download_auth:                 # config says: private
    ok = resp.status_code in (401, 403)  # a stranger must be refused
else:                                    # config says: public
    ok = resp.status_code == 200         # a stranger must succeed

Both directions are failures worth reporting, and they fail differently. A private bucket that answers 200 is a leak: the object is readable by anyone holding the link. A public bucket that answers 403 is an outage waiting to happen, and it will present later as broken images rather than as a storage problem.

Reporting these as a policy mismatch rather than a generic failure is intentional. The server is working; it is the agreement between the server and the configuration that is broken, and naming that distinction is what makes the output actionable.

D. A stream that generates itself, the same way twice

This is the answer to the fourth obstacle, and it is the one where the obvious implementations both fail for the same non-obvious reason.

The large-file test needs bytes without holding them. A generated stream provides that — but a transfer manager retries failed parts, and retrying means seeking backwards and reading the same range again. If the second read returns different bytes, the checksum computed when the upload completes will not match what the server received, and the failure appears as data corruption rather than as a flawed test harness.

So the content is derived from the offset rather than from a running generator. Block n is a hash of the seed and n, which means it can be recomputed at any time, in any order, as many times as the transfer manager wants.

def _block(self, index: int) -> bytes:
    # Content is a function of position, not of call order — a retry seeks
    # backwards and must read exactly the same bytes it read the first time.
    out = bytearray()
    counter = 0
    while len(out) < self._BLOCK:
        out.extend(hashlib.sha256(
            self._seed + index.to_bytes(8, "big") + counter.to_bytes(4, "big")
        ).digest())
        counter += 1
    return bytes(out[:self._BLOCK])

The same property makes the integrity check possible. Because the stream can be replayed from the beginning without storing anything, its checksum can be computed before the upload and compared against the object downloaded afterwards — end to end, with no temporary file anywhere in the process.

E. Proving the upload was actually split

This is the answer to the fifth obstacle. The upload reporting success is the beginning of the question, not the end of it.

A multipart object carries evidence of its own assembly. Its entity tag ends with a suffix naming the number of parts, which a single-request upload does not have — so the presence of that suffix is the proof that chunking happened rather than an assumption that it did.

# PartsCount is only returned when the request asks about a specific part.
head = client.head_object(Bucket=bucket.name, Key=key, PartNumber=1)

etag = str(head.get("ETag", ""))          # e.g. "…-4" for a 4-part object
match = _MULTIPART_ETAG_RE.match(etag)    # no suffix ⇒ server did a single PUT

The PartNumber=1 is not decoration. Without it the response omits the part count entirely, and the check would silently degrade into reporting "unknown" for every object — a test that cannot fail, which is the same as no test at all.

There is a second mode hiding in the same test, and it exists because of the proxy. Setting the part size equal to or larger than the file forces a single request, and then the assertion inverts: the absence of a multipart suffix becomes the expected result, and what is really being measured is whether the proxy in front of the server accepts a body that large. The same code answers two different questions depending on how the sizes relate.

F. Signing for the host that will actually be contacted

This is the answer to the sixth obstacle, and the failures it prevents are the most confusing ones in the whole tool.

Signature version 4 includes the host in what it signs. Two ordinary-looking conveniences therefore break it. Appending an explicit port to an endpoint that already uses the default for its scheme changes the host as the signer sees it, and every signature derived from that client is rejected — with an error about the signature, not about the port.

# Attach the port only when it is not the default for the scheme: SigV4
# signs the host, and ":443" on an https endpoint changes what is signed.
default_port = 443 if parts.scheme == "https" else 80
netloc = parts.hostname if port == default_port else f"{parts.hostname}:{port}"

The second convenience is the mirror image. When a presigned URL generated against an internal endpoint has to be opened from outside, only the scheme and host may be replaced — the path and the query string carry the signature and must survive untouched. Rebuilding the URL from its parts, rather than string-editing it, is what keeps that guarantee.

G. Leaving the server exactly as it was found

Every object the check creates is removed before it finishes, and the cleanup runs whether or not the tests passed. That matters because the check is most useful against environments that hold real data, and a diagnostic that litters the thing it inspects will not be run often enough to be useful.

The test objects are named with a recognisable prefix and a random suffix, so that anything left behind by an interrupted run is identifiable and obviously disposable. A flag exists to keep them deliberately — for the rare case where the object itself has to be examined — and the default is to delete.

try:
    ...                      # download, integrity, access policy
finally:
    # Runs even when a check above failed: a diagnostic that litters the
    # thing it inspects is one that stops being run.
    if uploaded and not keep_artifacts:
        client.delete_object(Bucket=bucket.name, Key=key)

Placing the deletion in a finally rather than after the assertions is the whole of it. The runs most likely to leave objects behind are the ones where something failed — which are also the runs most likely to be repeated several times in a row while the cause is being found.

Results and Discussion

A. What a passing run actually establishes

The output is a list of capabilities with a verdict on each, per bucket, and then a summary that counts them. A healthy private bucket reports fourteen checks passed; a public one reports thirteen, the difference being that a public bucket has no presigned download to test.

  Hasil per bucket:
    - private              14/14 check lolos   [OK]
    - public               13/13 check lolos   [OK]

  STATUS: HEALTHY  (2/2 bucket sehat)

The verdict has three states rather than two, and the middle one earns its place. Healthy means every bucket passed; unhealthy means none did; degraded means some did and some did not — which is the common case when a single bucket's policy is wrong and everything else is fine. Collapsing that into a plain failure would hide exactly the information that makes the report useful, and the exit code follows the same distinction so that the result can be acted on without reading the text.

What that sequence demonstrates is cumulative. The presigned upload succeeded, so signing and the proxy and the bucket permissions all work together. The download returned the same checksum, so the object was stored intact. The anonymous fetch was refused, so the policy matches the intent. The large file arrived with a four-part entity tag, so chunking genuinely happened and the parts were reassembled correctly.

That is the claim I wanted: not that the server is up, but that every capability the application is about to depend on has been exercised and observed to work. When something breaks afterwards, storage is no longer on the list of suspects.

B. The check that reports a security finding, not a failure

Of all the tests here, the access policy assertion is the one I would keep if I could keep only one — because it is the only one whose failure produces no symptom anywhere else.

Everything else in the list fails loudly and eventually. An upload that does not work will be noticed the first time somebody uploads. A proxy limit will be found by the first large file. But a private bucket that is readable by strangers behaves identically to a correctly configured one from every angle except the one nobody is looking from. There is no error, no slow request, no log line. The only way to discover it is to deliberately stand outside and try.

That asymmetry is the argument for building this kind of check at all. Tests that confirm things work have an alternative — wait, and find out. Tests that confirm things are not possible do not.

C. Where this stops being the right tool

It measures configuration and capability, not performance. The throughput it prints is a side effect of the transfer, useful for noticing that something is catastrophically slow and useless as a benchmark.

It tests the buckets it is told about, which means a bucket missing from the configuration is a bucket nobody checks. It proves the state of the server at one moment, and a policy changed next week needs the check run again — cheap, because the configuration file already exists.

And it is a single-run diagnostic rather than monitoring. It answers "is this ready to build against" well, and "has this stayed healthy for a month" not at all. Those are different questions and they deserve different instruments.

Conclusion

The pattern here is the same one I applied to identity providers, and the repetition is the point: an external dependency should be proven before anything is built on top of it, because the alternative is discovering its limits through the behaviour of half-finished code. Two hours spent asserting capabilities in advance removes an entire category of ambiguity later.

The narrower lesson is about what a check is allowed to assume. Every obstacle above came from the same mistake in a different disguise — trusting a report instead of demanding evidence. The upload said it succeeded; the configuration said the bucket was private; the transfer said it used multipart. Each was plausible, and each was worth a cheap question that turned it from a claim into a fact.

Terms

Object storage

A store that keeps whole files addressed by name within a bucket, served over HTTP. Flat by design: no directories, no in-place partial writes.

S3-compatible

Any server implementing the HTTP interface that became the de facto standard for object storage, allowing one client to work against several implementations.

Bucket

A named container for objects, and the level at which access policy is usually decided.

Presigned URL

A link signed on a client's behalf, carrying its own signature and expiry, so a browser can upload or download without ever holding credentials.

Bucket policy

The rule deciding whether objects can be read without credentials — the difference between a public asset and a private document.

Multipart upload

A large object sent as several parts and reassembled by the server, using three separate operations rather than one request.

ETag

An object's entity tag. For a multipart object it ends with a suffix naming how many parts it was assembled from, which makes it evidence that chunking occurred.

SigV4

The signing scheme that covers the host among other things, which is why an endpoint's host must match exactly what a request is sent to.

Addressing style

Whether a bucket appears in the path or in the hostname. Self-hosted servers usually need the path form.

Public origin

The address a browser will use, kept separate from the endpoint the tool connects to, so generated links are checked against the address real clients reach.