Design Records

• 1: Conditional DELETE: Why the Condition Must Be Evaluated Once
• 2: DSN-Only Database Notifications: A Compatibility Boundary for #53
• 3: Preview Text, Never Execute It: SILO Console Text Preview PRD
• 4: No I/O Before Auth, No Privilege From Headers
• 5: One Endpoint, Two Privileges: Separating User and Group Status
• 6: Config Environment Files Are Not Shell Scripts
• 7: Two SSE-C Keys, One CopyObject Response
• 8: Why CompleteMultipartUpload Must Return ChecksumType: Review of PR #57
• 9: When the Total Is Unknown: Folder Download Progress
• 10: A ListObjects Shortcut Must Not Turn a Missing Bucket into an Empty One
• 11: Read-Only Checksum Audit and Reliable CLI Output
• 12: Optional Checksums, Mandatory Failure: Repairing UploadPart and UploadPartCopy Compatibility
• 13: BadDigest, InvalidRequest, and the CompleteMultipartUpload Checksum Contract
• 14: Should SILO Fix ListMultipartUploads? Design Review of Issue #79

1 - Conditional DELETE: Why the Condition Must Be Evaluated Once

This document records the analysis, design discussion, and repair decision for SILO PR #12.

Status on 2026-08-26: PR #12 remains open at head 5b71a75e, 118 commits behind the latest main. Its commit has no DCO sign-off and GitHub reports no check runs. The improved design described here has been implemented, tested, and reviewed twice in an isolated local worktree, but it has not been committed, pushed, merged, or released.
Scope: correctly support If-Match for the single-object DeleteObject API, and fail closed instead of silently deleting when an unsupported per-object DeleteObjects ETag is received. Full batch-condition execution and bucket-policy enforcement remain separate deliverables.
Release boundary: local implementation, tests, review, commit, push, remote CI, merge, tag, image publication, and production deployment are independent gates.

Too Long; Didn’t Read (TL;DR)

The underlying problem is real. SILO currently ignores If-Match on DELETE, so a client can believe it is performing compare-and-delete while the server performs an unconditional deletion. PR #12 targets the right problem and correctly recognizes that the condition must be evaluated against fresh object state while holding a lock.

The original implementation puts the same HTTP callback into every erasure pool. Different pools can retain copies from different points in time, so each pool evaluates and mutates against its own ETag. A two-pool test reproduced both failures:

• the request ultimately returns 412 after the older matching copy has already been deleted;
• the request succeeds after deleting the current copy, while an older non-matching copy remains and becomes visible again.

The selected repair introduces no new condition framework. It follows the established multi-pool GET pattern: select the current object under the outer namespace lock, evaluate the condition exactly once, then clear the callback before calling lower pools. A false condition mutates no pool. A true condition allows the existing cleanup to run without reinterpreting the client condition per copy.

Why this is a real problem

Silently dropping the condition is unsafe

AWS conditional-delete documentation now defines the behavior for general-purpose buckets and both DeleteObject and DeleteObjects:

Ignoring the header is therefore not a harmless unsupported extension. It removes the concurrency guard the caller used to avoid deleting another writer’s newer object.

Not every S3 client sends conditional deletes, so prevalence is unknown. Severity for each relying caller is high: one silent downgrade can remove newly committed data.

A delete marker is not merely an ETag comparison case

PR #12 reuses the generic isETagEqual, which returns true whenever the right-hand value is *. Consequently, isETagEqual("", "*") is also true.

The more fundamental bypass occurs one layer above. erasureServerPools.DeleteObject returns success immediately when the current object is already a delete marker. That happens before the callback added by the PR. A diagnostic test observed zero callback calls and a successful result.

The impact needs precise wording:

• the delete-marker fast path did not remove a historical version or create another marker in the reproduced case; it bypassed the condition and falsely reported success;
• the multi-pool counterexamples do mutate storage on a failed request or leave a stale copy after success.

Changing only isETagEqual("", "*") cannot cross the outer fast path and risks altering a comparator shared by GET, PUT, and COPY.

What the original PR got right

Its high-level algorithm is sound:

1. detect If-Match in the handler;
2. read fresh ObjectInfo after acquiring the storage lock;
3. return before mutation when the condition is false;
4. encode the result as an S3 response.

This avoids the obvious TOCTOU window of a separate HEAD followed by DELETE. The PR also adds handler, helper, and erasure-layer tests. Its ordinary single-pool path correctly returns 412 and preserves the object for a wrong specific ETag.

The defect is not the decision to evaluate under a lock. It is choosing the wrong layer and therefore the wrong object state.

Where the atomicity boundary lives

The deletion path has two layers:

DeleteObjectHandler
    -> erasureServerPools.DeleteObject
         holds the namespace write lock
         selects the latest current pinfo across pools
         handles the delete-marker fast path
         selects a single-pool or all-pool deletion
             -> erasureSets / erasureObjects.DeleteObject

Only erasureServerPools.DeleteObject knows:

• which copy represents the current object;
• which pools still contain older copies or inconsistent metadata;
• whether the delete-marker fast path applies;
• whether multiple pools will be mutated concurrently.

The client condition therefore belongs at this layer. A single pool knows only its local copy and cannot reinterpret a condition on the logical current object.

Two-pool counterexamples

The test places an older object in pool 0 and a newer object with a different ETag in pool 1. Reads select pool 1 as current, while an unversioned delete cleans both pools.

Condition matches the old copy

The original PR lets pool 0 pass and delete its copy while pool 1 fails. The aggregate result follows the current pool and returns 412, even though storage changed.

Condition matches the current copy

Pool 1 passes and deletes the current copy. Pool 0 fails and retains the old copy. The request returns success, after which the old object becomes visible again.

The callback also captures a single http.ResponseWriter. Calling it concurrently from multiple pools can make multiple goroutines write the same HTTP response. Storage replicas should not concurrently decide wire-level output.

A pre-existing degraded-pool limitation

There is one related but inherited limitation outside this patch. If the selected current pool is readable and writable but an older, non-current pool is degraded, the existing all-pool delete path can return the selected pool’s success while an error from the older pool is not surfaced. That copy can remain and reappear after recovery.

The new condition does not create this behavior: it evaluates the readable current object correctly and then enters the same unversioned multi-pool cleanup used by an unconditional delete. Repairing error aggregation and recovery for partially degraded old pools should be tracked separately because it changes the guarantees of every unversioned multi-pool delete, not only conditional requests.

The selected minimal repair

1. Evaluate exactly once at the outer layer

After erasureServerPools.DeleteObject acquires the namespace write lock:

1. save opts.CheckPrecondFn;
2. remove it from options passed to lower layers;
3. inspect all pools and select the current pinfo;
4. if the current object cannot be read reliably, return a quorum error without calling the callback;
5. call the saved callback exactly once with pinfo.ObjInfo;
6. on success, continue through the existing deletion path with no lower-layer reinterpretation.

This pattern already exists in multi-pool GetObjectNInfo: save the callback, clear it below, select the latest object, and evaluate once. Reusing it limits the DELETE change to the real atomicity boundary.

2. Treat * as current-representation existence

The DELETE-specific check separates wildcard and ETag semantics:

specific ETag -> compare the client-visible ETag of the current object
*             -> require a non-empty object name and no delete marker

A missing key already returns Not Found during object selection. A current delete marker reaches the callback and returns 412. The generic isETagEqual remains unchanged.

3. Require read permission for a specific ETag

The handler first checks s3:DeleteObject. When the normalized condition is not a bare *, it additionally checks s3:GetObject:

• delete-only policy plus *: allowed;
• delete-only policy plus a specific ETag: 403 and no mutation;
• Get plus Delete and a matching ETag: allowed.

Authorization completes before any storage mutation.

4. Do not require SSE-C content decryption for DELETE

The original PR invokes the GET/PUT-oriented DecryptObjectInfo, which rejects an SSE-C object when SSE-C read headers are absent. Conditional DELETE needs the client-visible ETag, not plaintext content or decrypted size.

The selected implementation uses the established getDecryptedETag projection only for a specific ETag. Wildcard requests do not read the ETag. This reuses existing ETag behavior without imposing content-decryption requirements on DELETE.

5. Evaluate the current version

AWS specifies that conditional-delete evaluation applies to the current version. SILO’s outer pool selection already reads the current object, while preserving an explicit versionId for the eventual version deletion.

A regression test requests deletion of a historical version while matching that historical ETag rather than the current ETag. It must return 412 and preserve both versions.

6. Reject silent downgrades at unsupported edges

Two small guards keep the single-object feature from being bypassed:

• an empty or whitespace-only If-Match is rejected instead of becoming an unconditional delete;
• If-Match cannot be combined with the internal recursive x-minio-force-delete extension, whose prefix semantics cannot represent one object’s ETag condition; the HTTP handler rejects it and the storage layer also refuses any internal prefix-delete plus callback combination.

The batch XML decoder now also recognizes per-object <ETag> values. Until atomic per-item execution is implemented, any non-empty batch ETag rejects the entire request with NotImplemented before deletion begins. This is not batch conditional-delete support; it is a narrow data-safety guard against silently discarding a condition.

Rejected alternatives

Change only isETagEqual

It does not address the outer delete-marker fast path and risks changing several APIs that share the comparator.

Keep per-pool callbacks and aggregate the result

An aggregate error cannot roll back a copy already deleted by another pool. The condition applies to the logical current object, not independently to every physical copy.

Introduce a new condition object or transaction coordinator

The current feature has one If-Match condition, and CheckPrecondFn already expresses it. GET demonstrates the correct one-shot consumption pattern. A new DSL, state machine, or cross-pool transaction abstraction is unnecessary.

Complete every conditional-delete feature in one PR

DeleteObjects and policy conditions cross different API and repository boundaries. Combining XML parsing, per-item responses, IAM, quiet mode, and dependency publication with the core deletion repair would make the change harder to validate.

Test and acceptance contract

The minimally sufficient matrix is:

The original PR’s quorum test merely took 8 of 16 disks offline and asserted that some error occurred. Delete write quorum was already unavailable, so the same test passed on main without conditional DELETE. The replacement asserts the specific quorum result, zero callback calls, and object survival after restoring the disks.

Independent adversarial review

Two read-only local Claude Code reviews used the Fable model at xhigh effort against the exact server diff and both design records. Both verdicts were GO WITH NON-BLOCKING NOTES, with no P0, P1, or P2 findings after the first round’s changes were applied.

The first review found the conditional force-delete bypass, whitespace-only downgrade, silent batch-ETag discard, missing versioned-success coverage, and the inherited degraded-old-pool limitation. Those findings produced the guards, tests, and limitation text above. The second review confirmed the outer atomicity boundary, error handling, auth split, batch-field blast radius, response-writer behavior, bilingual parity, and minimality. Its remaining actionable P3 was a hypothetical internal caller combining prefix deletion with a callback; the storage layer now rejects that combination too.

One reviewer sentence suggested that SSE-C without customer-key headers would necessarily fail the condition. Direct inspection showed the opposite established behavior: getDecryptedETag projects the stored client-visible suffix without asking to decrypt object contents. A focused regression test now pins that behavior. Remaining non-blocking notes are multiple-header normalization and the deliberate 501-before-auth error-ordering nuance. A live-AWS differential check for specific ETag versus a current delete marker and versionId plus If-Match would still be useful before claiming byte-for-byte behavioral parity beyond the published contract.

Deliberate follow-up scope

Per-object conditions in DeleteObjects

The AWS DeleteObjects API accepts an <ETag> per <Object> and returns each outcome under <Deleted> or <Error> in the same 200 response.

The safety patch adds an ETag field to ObjectToDelete only so the handler can detect the condition and reject the entire request before mutation. This closes the previous silent unconditional-delete behavior, but it does not implement AWS’s required per-object evaluation or mixed <Deleted> / <Error> response.

Full compatibility remains a separate high-priority change: evaluate every item against the logical current object under the correct lock, apply the exact-ETag permission rule per item, preserve quiet-mode behavior, and report each failed condition without blocking unrelated items.

The s3:if-match policy condition key

AWS policies can enforce conditional deletes. SILO’s silo-pkg does not yet define s3:if-match. Full support requires:

1. the condition key and action map in silo-pkg;
2. a new silo-pkg release;
3. correct condition values for a single-delete header and batch per-item ETags;
4. a server dependency update and policy compatibility tests.

That is a separate cross-repository deliverable, not a prerequisite for making single-object execution correct.

Complexity, benefit, and cost

Production code remains small: one DELETE-specific condition helper, one extra authorization check, roughly a dozen lines that consume the callback once at the outer layer, and narrow fail-closed guards for malformed/recursive and as-yet unsupported batch conditions. Most complexity belongs in tests because deletion spans pools, versions, markers, quorum, and permissions.

The benefit exceeds the cost. It removes a dangerous silent unconditional delete and places the condition at an existing global consistency boundary. Reusing the current outer-lock/latest-object pattern is the minimal, sufficient, and necessary design.

Merge and release gates

The single-object repair becomes mergeable only after:

1. targeted condition, permission, versioning, quorum, and two-pool tests pass;
2. go test ./cmd, go vet ./cmd, formatting, and diff checks pass;
3. an independent adversarial review has no unresolved blocker;
4. the contribution is organized on current main with a valid author DCO sign-off;
5. DCO, Go CI, VulnCheck, and other required remote workflows are green;
6. the PR description distinguishes complete DeleteObject support from the batch fail-closed guard and links the full batch/policy follow-ups.

A merge is still not a release. Users can rely on the behavior only after a corresponding SILO release, package, docker.io/pgsty/minio image, deployment, and real-client verification have independently completed.

Conclusion

Conditional DELETE is worth implementing. PR #12 has the right goal and the useful insight that fresh state must be checked under a lock. The required correction is the boundary: a client condition belongs to the logical current object and cannot be interpreted independently by every physical copy.

The selected design moves one callback to the erasureServerPools layer that already selects the current object, preserves the generic comparator, handles wildcard/delete-marker semantics explicitly, and adds the specific-ETag read permission. It changes no storage format, dependency, or public condition framework. The batch change is deliberately limited to refusing an unsupported condition before mutation; full batch execution and policy support remain separate work.

That is the minimum complexity needed to make the feature sufficient and safe.

2 - DSN-Only Database Notifications: A Compatibility Boundary for #53

This document is the product requirements and final design record for SILO issue #53. It records the accepted compatibility boundary, implementation, and verification for PostgreSQL and MySQL bucket-notification targets.

Decision

SILO will retain PostgreSQL and MySQL notification targets, but support exactly one current configuration form for each:

• PostgreSQL requires a complete connection_string.
• MySQL requires a complete dsn_string.

The old five-field form — host, port, username, password, and database — remains unsupported by the current KV configuration system. SILO will not re-register those keys and will not synthesize a DSN from them during legacy migration.

The legacy migration contract is deliberately narrow:

This is a configuration-boundary decision, not removal of the database-notification feature.

Status: implemented in server commit f1ba68358; release pending.
Owner: SILO server repository.
Tracking: pgsty/silo#53.
Target: the next SILO patch release after implementation and verification.

Context

SILO inherited two generations of database-notification configuration from MinIO.

The pre-KV JSON configuration could describe a database connection either as a complete string or as five fields:

host
port
username
password
database

The current KV configuration exposes only the driver-native form:

notify_postgres  -> connection_string
notify_mysql     -> dsn_string

This direction is not new. MinIO deprecated the five discrete fields in RELEASE.2020-04-10T03-34-42Z and instructed operators to move to connection_string or dsn_string. SILO’s current help tables, environment-variable documentation, and examples already present the complete string as the supported interface.

SILO is a new community fork with an explicit migration step. Its compatibility contract prioritizes the S3 and Admin APIs, current MINIO_* settings, on-disk data, and current KV configuration. It does not need to perpetuate every pre-2020 configuration spelling when a supported canonical form has existed for years.

The defect

Before the fix, the legacy migration helpers, SetNotifyPostgres and SetNotifyMySQL, wrote both forms into the new KV configuration. Even when the old target already had a complete connection string, the helpers also emitted all five discrete keys, usually with empty values.

The new parser rejects those keys because neither DefaultPostgresKVS nor DefaultMySQLKVS registers them. Key validation checks key presence, not whether the corresponding value is empty. Both legacy source forms therefore fail:

old complete string -> canonical string + five empty unknown keys -> rejected
old discrete fields -> empty canonical string + five populated unknown keys -> rejected

The failure is amplified by notification initialization. FetchEnabledTargets is fail-fast across notification subsystems: the first invalid subsystem returns an error and a nil target list. The caller logs the error and continues starting the object server, leaving healthy Webhook, Kafka, NATS, and other targets unavailable as well.

Merely returning an error from the two migration helpers does not fix that behavior. The error propagates through readConfigWithoutMigrate and initConfig, but initConfigSubsystem currently logs non-retriable configuration errors as “some features may be missing” and returns success. The server then starts without assigning globalServerConfig; notification failure is only one consequence, because region, storage class, compression, identity, and other stored settings may also be absent. The implementation must therefore carry a typed database-migration error to the startup boundary and make that error fatal. Classifying it as retriable is also wrong because the server would retry forever without any state change that could repair the configuration.

The resulting behavior is especially dangerous because object I/O still works. Operators can see a healthy S3 service while every configured event pipeline has stopped. Targets are never constructed, so delivery or later replay of events produced during the outage must not be assumed.

There is also a diagnostic-exposure issue. The unregistered password key has no sensitivity metadata and may be copied verbatim into health or diagnostic material. The registered connection_string and dsn_string keys are already treated as sensitive values.

Why the first fix was reverted

The first repair registered the five discrete keys and taught the parser to read them. That made migrated targets pass CheckValidKeys, and it appeared attractive because the target argument structures and constructors still contain code for the old fields.

It also broke the documented connection-string path.

The shared mc admin config set tokenizer discovers field boundaries by looking for registered key names. It is not fully quote-aware. Once port became a registered key, this valid input contained what looked like a second top-level field:

connection_string="host=db port=5432 dbname=events user=app"

The tokenizer split at the port= inside the quoted value, truncated connection_string, and handed the remainder to the port parser. The command then failed with invalid port.

Under the current tokenizer, registering common words such as host, port, and password creates a direct conflict between the connection-string grammar and the top-level KV grammar. The attempted registration fix was therefore reverted. Re-registering those keys is not an acceptable solution.

Product judgment

Database notification targets are a specialized but useful capability. They provide a direct database-backed namespace view or access journal without requiring an external event bus. That remains valuable for small deployments and for users already operating PostgreSQL or MySQL.

The legacy spelling of their connection parameters has much less value. A five-field model cannot represent the useful range of driver options: TLS modes and certificates, connection timeouts, application names, Unix sockets, multi-host PostgreSQL settings, MySQL driver parameters, and future driver capabilities. Supporting both forms also creates precedence, merging, redaction, and testing questions that do not exist with one canonical value.

The complete string is the better abstraction boundary: SILO owns notification semantics, while the database driver owns connection syntax.

The product decision is therefore to keep the capability and remove the compatibility illusion. An unsupported legacy target must be rejected clearly; it must not be accepted and transformed into a configuration that later disables unrelated targets.

Goals

1. Establish connection_string and dsn_string as the only supported live configuration interfaces for database notifications.
2. Allow a legacy JSON target that already contains the canonical string to cross the migration boundary without modification to its connection semantics.
3. Reject enabled discrete-only legacy targets before a partial or invalid KV configuration is activated.
4. Replace the current silent runtime failure mode of #53 — healthy targets disabled while the server appears healthy — with an explicit startup-time failure that operators must resolve before the server runs.
5. Ensure no migration error, log line, health report, or diagnostic bundle exposes a database password.
6. Remove the ten Postgres/MySQL exceptions from the source-level unregistered-write audit.
7. Make the compatibility boundary and operator remediation explicit in release and migration documentation.

Non-goals

• Supporting both DSN and discrete database fields in the current KV interface.
• Automatically synthesizing a DSN from old discrete fields.
• Rewriting the shared KV tokenizer.
• Changing FetchEnabledTargets fail-fast semantics in this patch.
• Silently skipping an enabled database target and continuing with partial notification coverage.
• Removing PostgreSQL or MySQL notification targets.
• Deleting the legacy struct fields needed to decode and identify unsupported input. They remain on shared target argument structs that are also used by live constructors, whose discrete-field connection-string synthesis is unreachable from current KV configuration; those fields must not become supported configuration keys.
• Correcting ignored errors from the other eight legacy notification setters. Their pre-existing silent-skip behavior remains unchanged in this narrowly scoped database-migration patch and requires a separate audit and design decision.

Functional requirements

Current configuration

1. notify_postgres accepts connection_string; notify_mysql accepts dsn_string.
2. The five discrete keys remain unregistered and rejected by current configuration commands.
3. Existing full strings must continue to support the database driver’s syntax, including parameters whose names contain host, port, user, password, or database.
4. No new public environment variables or KV keys are introduced.
5. The declared legacy variables MINIO_NOTIFY_POSTGRES_HOST/PORT/USERNAME/PASSWORD/DATABASE and their MySQL equivalents are not wired into current parsing and remain unsupported. They must not be documented as working alternatives to the complete-string variables.

Legacy migration

1. SetNotifyPostgres must return without emitting a target when the legacy target is disabled.
2. For an enabled target, SetNotifyPostgres must require a non-empty ConnectionString and write only registered Postgres keys. If both a canonical string and discrete fields are present, the canonical string wins and every discrete value is discarded.
3. SetNotifyMySQL must apply the equivalent rule to DSN.
4. Neither helper may emit host, port, username, password, or database.
5. A missing canonical string must return a typed or wrapped migration error identifying the subsystem and target name.
6. cmd/config-migrate.go must check and propagate both helper errors. Ignoring them is forbidden.
7. No partially migrated configuration may be activated or persisted after either helper fails.
8. Error text may name the required key and remediation, but must not include any connection-field value.
9. The propagated typed migration error must abort server startup. It must not be downgraded to the non-fatal “some features may be missing” path in initConfigSubsystem, and it must not enter the retriable-error loop.
10. Validation errors for a supplied canonical string follow the same startup-fatal and secrecy rules; wrapping must add target context without repeating the DSN or its components.

Recommended error shape:

notify_postgres:archive uses unsupported legacy discrete connection fields;
set connection_string before migrating to SILO

Operator remediation

An operator encountering the error must choose an explicit remediation path. This applies both before an initial switch to SILO and when upgrading a deployment that is already running SILO: legacy migration output is not persisted, so the same old JSON source can re-enter migration on every start. A deployment that currently starts with notifications silently broken can therefore fail to start after this repair until the source configuration is corrected.

1. On a compatible intermediate MinIO release, replace the old fields with connection_string or dsn_string, verify the target, and then migrate to SILO.
2. Disable or remove the legacy database target, migrate the server, and recreate the target with the canonical string afterward.
3. For a fresh SILO installation, create the target directly with the canonical string; no legacy migration is involved.
4. For an existing SILO deployment that still reads a legacy JSON file, stop on the previous working release, back up the source configuration, then convert, disable, or remove the database target before starting the fixed release. Do not delete or rewrite unrelated configuration.

Documentation must not suggest that a discrete-only target will be converted automatically.

Availability trade-off

This decision intentionally turns one unsupported configuration from a degraded startup into a hard startup failure. The immediate availability cost is real: a server that previously served objects while all notifications were silently dead may refuse to start after the repair.

That cost is accepted because an object server that appears healthy while configured event sinks are absent creates silent, potentially unrecoverable downstream data loss. SILO is a new fork with an explicit migration boundary, and the discrete form has been deprecated since 2020. A fatal, actionable precondition is preferable to an upgrade that reports success with reduced notification coverage. The release note must make this startup behavior prominent; it must not be buried as an internal migration cleanup.

Security requirements

1. The unsupported-input error must never format the legacy argument structure or its values.
2. Tests must use a sentinel password and assert that it is absent from returned errors and captured logs.
3. Migrated output must contain the registered sensitive connection-string key and no standalone password key.
4. If a diagnostic bundle was exported from an affected deployment before this repair, operators should treat the database password as potentially disclosed and rotate it.

Alternatives considered

Register and parse the discrete fields

Benefit: preserves the old source form and uses already existing argument fields.
Rejected because: registration makes common field names visible to the shared tokenizer and corrupts quoted connection strings. It also expands the supported public configuration surface after the fields were deprecated in 2020.

Synthesize a canonical string during migration

Benefit: preserves discrete-only legacy installations.
Rejected because: it creates permanent code and test ownership for an obsolete input form, including PostgreSQL quoting, MySQL DSN formatting, socket and IPv6 behavior, defaults, and future driver drift. For a new fork with an explicit migration boundary, the benefit does not justify the continuing surface.

Skip only the unsupported target

Benefit: keeps the object server and other notification targets running.
Rejected because: silently discarding a configured event sink can cause unobservable and unrecoverable event loss. A clear migration failure is safer than an apparently successful upgrade with reduced notification coverage.

Change global notification fail-fast behavior

Benefit: limits the blast radius of future invalid targets.
Rejected for this change because: it neither repairs the database target nor closes the credential-exposure path, and it changes system-wide error semantics. It may be evaluated independently with its own operational contract.

Remove database notification targets

Benefit: removes the complete database-specific maintenance surface.
Rejected because: the targets remain useful and self-contained. The defect belongs to an obsolete configuration form, not to the notification capability itself.

Implementation scope

The server change should remain narrow:

1. Update internal/config/notify/legacy.go so the two database setters emit only canonical registered keys and reject enabled targets without a canonical string.
2. Update cmd/config-migrate.go to propagate the two database-helper errors with subsystem and target context.
3. Define a typed database-migration error and update cmd/server-main.go so initConfigSubsystem returns it as fatal instead of logging and ignoring it. It must remain non-retriable.
4. Leave ignored errors from the other eight legacy notification setters unchanged in this patch; record them for a separate audit rather than expanding #53 implicitly.
5. Remove all ten Postgres/MySQL entries from knownUnregisteredWrites; the ratchet should become empty unless another independently justified legacy exception exists.
6. Add focused migration, startup, validation, secrecy, and coexistence tests.
7. Update database-notification and migration documentation in silo.pgsty.com.

The patch must not register the old keys, change the generic tokenizer, or refactor unrelated notification targets.

Acceptance criteria

The implementation is complete only when all of the following are demonstrated:

1. A legacy PostgreSQL target with a complete connection string migrates, passes CheckValidKeys, and is returned by GetNotifyPostgres unchanged.

2. A legacy MySQL target with a complete DSN does the equivalent.

3. Discrete-only enabled targets for both databases fail before target initialization with an actionable error containing the subsystem and target name, and server startup aborts.

4. Missing-string and malformed-string errors contain none of the sentinel host, username, password, database, or DSN values.

5. Disabled discrete legacy targets do not create configuration entries and do not block migration.

6. Migrated KVS output contains none of the ten discrete keys, including empty ones.

7. When a legacy target contains both a canonical string and conflicting discrete values, only the canonical string is migrated and no discrete sentinel appears in any output KVS value.

8. A SetKVS regression test using the real DefaultPostgresKVS and DefaultMySQLKVS key sets accepts a quoted connection string containing port=, host=, or password=.

9. A configuration containing healthy Webhook, Kafka, or NATS targets cannot reach FetchEnabledTargets with an invalid migrated database target because readConfigWithoutMigrate fails without yielding, persisting, or activating a partial configuration, and startup aborts on that typed error.

10. initConfigSubsystem returns the typed migration error; it neither logs-and-continues nor enters the retriable loop.

11. knownUnregisteredWrites no longer contains Postgres or MySQL exceptions.

12. The following verification passes:

    go test ./internal/config/notify ./internal/config ./internal/event/target -count=1
    go test -v ./cmd -run 'Test(ReadConfigWithoutMigrate|InitConfigSubsystem)' -count=1
    git diff --check

    The verbose cmd output must show that tests with both prefixes actually ran; a zero-match warning is a failed acceptance check. The normal server CI suite must also pass. In the documentation checkout, run make check.

Implementation result

Server commit f1ba68358 implements the accepted design without expanding the public configuration surface:

• the two legacy database setters emit only connection_string or dsn_string plus registered target settings;
• disabled targets remain ignored, while enabled targets without a canonical string return a value-free LegacyDatabaseTargetError;
• only the two database migration errors are newly propagated;
• the typed error is non-retriable, escapes initConfigSubsystem, and is classified as fatal by serverMain before logger.FatalIf exits the process;
• the ten Postgres/MySQL exceptions were removed from knownUnregisteredWrites;
• focused tests cover complete-string round trips, canonical precedence, discarded discrete values, secrecy, failed-migration atomicity, startup classification, and the real tokenizer key sets.

The final local Claude Code review used Claude Fable 5 at max effort and returned GO with high confidence and no blocking findings. Verification included the focused package set, race tests, go vet ./cmd, and the complete go test ./cmd -count=1 suite. The review authorized only the six-file server commit; publication remains a separate gate.

Cross-repository review found no implementation changes are required in pgsty/mc, pgsty/silo-pkg, or pgsty/silo-console: the client forwards configuration text, the package repository owns no notification schema, and Console already serializes its form into the canonical connection_string or dsn_string. The public reference and compatibility documentation is updated with this record.

Release and compatibility statement

The release note must describe this as an enforced compatibility boundary:

SILO database notification targets require connection_string for PostgreSQL and dsn_string for MySQL. The pre-2020 discrete host/port/username/password/database form is not migrated. Convert or recreate such targets before switching the deployment to SILO.

Deployments already running SILO with an old-format source configuration are equally affected: after this release the server will not start until each enabled legacy database target is converted, disabled, or removed.

The issue should close only after the repair is present in a published server tag. A merged patch, a local site build, and a published release are separate completion gates.

Review record

Claude Fable 5 reviewed the first draft at xhigh effort on 2026-08-23 and returned approve with required changes. The required calibration was incorporated: startup-fatal propagation now extends through initConfigSubsystem; already-running SILO deployments are covered; the availability trade-off is explicit; canonical-string precedence, dead legacy environment variables, other ignored helper errors, and executable tests are specified.

The same model then completed a final source-backed verification pass. Final verdict: approve, with no blocking findings. It confirmed that the English and Chinese records are aligned, the requirements are implementable against the current server tree, and the acceptance criteria cover the startup, migration, parser-regression, and secrecy boundaries.

After implementation, a separate local Claude Code review using Claude Fable 5 at max effort traced the path through ExitFunc(1), inspected driver error behavior, ran the focused, race, vet, and full cmd suites, and returned GO with high confidence and no blocking findings.

3 - Preview Text, Never Execute It: SILO Console Text Preview PRD

Status: accepted design; implementation pending · Owner: pgsty/silo-console · Tracking: pgsty/silo#17 · Review: consensus of product, security, and frontend architecture reviews

SILO Console can preview images, PDFs, audio, and video, but not the small logs, text files, JSON documents, and XML documents that operators inspect every day. A correctly stored Content-Type does not help: these objects are classified as unsupported before the preview renderer is selected.

Restoring the old browser-native behavior would be easy. It would also be the wrong fix. An object in storage is controlled by the user who uploaded it. Loading that object as a same-origin HTML or XML document would turn a convenience feature into an execution boundary.

The accepted design therefore makes a stronger promise:

SILO previews eligible objects as bounded UTF-8 text. It never asks the browser to interpret their markup, MIME type, or file contents as a document.

This record fixes the product boundary, the resource limit, the security invariants, the implementation shape, and the evidence required before the feature can ship.

Decision

The first release will add a dedicated text preview type and a PreviewText component.

The contract is:

1. Preserve every existing image, PDF, audio, and video classification.
2. Only when the existing classifier returns none, consider a text fallback.
3. Admit the four target extensions or four exact passive text MIME types.
4. Fetch bytes through the ordinary authenticated download path, without preview=true.
5. Enforce a hard application read limit of 1 MiB.
6. Decode only strict UTF-8 and reject binary-looking content.
7. Render one React text node inside a scrollable <pre>.
8. Never use an iframe, HTML parser, XML parser, or HTML injection API.
9. Show the complete object or no object; do not show a truncated JSON or XML document.
10. Keep download available for files that are too large, invalidly encoded, or otherwise unavailable.

No Console API or S3 API change is required. The backend inline MIME allowlist is not expanded.

Current behavior

The defect is present in SILO Console v2.1.1, the version currently pinned by SILO when this design was written.

The frontend preview union contains only:

image | pdf | audio | video | none

Its extension table contains media formats, but not .log, .txt, .json, or .xml. Its MIME classifier likewise ignores text/plain, application/json, application/xml, and text/xml.

Runtime verification produced this split:

The object-detail action also uses the wrong conjunction when deciding whether Preview should be disabled. An authorized user can click Preview for an unsupported object and receive only the unavailable message; in other combinations, the UI can offer an action before the server rejects it.

The preview component still contains a generic same-origin iframe fallback. It is unreachable under the current type union, so the current defect is not an exploitable text-preview XSS. The dead branch is nevertheless hazardous: adding text to the union and letting it fall through would reactivate precisely the document-loading behavior this design rejects.

Root cause

This is contract drift across three independently evolved layers.

Classification drift

The browser code decides eligibility from filename and object metadata, but its closed type union has no text representation. Correct metadata cannot select a renderer that does not exist.

Response-policy drift

The Console server separately decides whether a response may be inline. It still treats plain text and JSON as safe passive MIME types, while XML and HTML remain attachments. That server decision is not reflected in the frontend classifier.

Renderer drift

The old generic iframe remains after the set of reachable preview types became media-only. The code therefore suggests a capability that the type system can no longer invoke.

The repair must realign the three layers without making MIME metadata a security boundary.

Why same-origin iframe preview is rejected

X-Frame-Options: SAMEORIGIN is not a sandbox. It controls who may embed a response; it does not limit what code inside a same-origin frame can do.

If uploader-controlled HTML, XHTML, SVG, or active XML were ever served as an inline same-origin document, it could act with the Console origin. An HttpOnly cookie would prevent direct cookie reads, but it would not prevent authenticated same-origin requests. A permissive or accidentally widened MIME rule would then turn stored content into stored application code.

nosniff, Content Security Policy, and Content-Disposition remain useful defense in depth, but none replaces the core invariant:

untrusted object bytes
        |
        v
strict text decoder
        |
        v
React textContent

never:
iframe / innerHTML / DOMParser / XML parser / executable document

Product contract

The feature is a read-only text viewer, not a web previewer and not an online editor.

The user should be able to:

• open a small eligible object from either the list or object-detail surface;
• read whitespace-preserving source text in the existing preview modal;
• select and copy text using browser-native behavior;
• understand whether a failure is caused by size, encoding, permission, object replacement, or network error;
• download the original bytes at any time.

The user must never be led to believe that:

• formatted JSON is the stored object;
• a partial XML document is complete;
• replacement characters are original bytes;
• an unsupported encoding has been decoded faithfully;
• an active HTML/XML document has been safely “sanitized” and executed.

Goals and non-goals

Goals

1. Preview small logs, text, JSON, and XML without a local download.
2. Keep object content inert regardless of extension, MIME, or payload.
3. Bound retained response bytes and rendered text to 1 MiB.
4. Preserve the stored text rather than silently reformatting it.
5. Keep list and detail actions consistent with permissions and type eligibility.
6. Support current object versions and explicitly selected historical versions.
7. Preserve anonymous-access and subpath-hosting behavior.
8. Ship the feature in Console first, then consume that exact Console revision in SILO.

Non-goals

• HTML or XHTML rendering.
• XML parsing, XSLT, external entities, or schema validation.
• Markdown rendering.
• JSON pretty-printing.
• YAML or CSV-specific behavior.
• Editing or saving.
• Syntax highlighting, line numbers, search, folding, ANSI rendering, or linkification.
• Head, tail, or truncated previews for large objects.
• Lossy decoding or automatic detection of GBK, UTF-16, Latin-1, or other encodings.
• A new backend text-preview endpoint.
• Changes to the existing SVG, media, PDF, download, share, or storage contracts.

An object such as notes.md may still be shown as raw text when its exact MIME type is text/plain. It does not gain Markdown semantics.

Eligibility contract

Eligibility is deliberately two-stage.

Stage 1: preserve the legacy media decision

Run the current image, PDF, audio, and video classifier unchanged. If it returns anything other than none, return that result.

This preserves historical behavior for conflicting filename and MIME combinations.

Stage 2: apply text fallback

Only after the legacy result is none:

1. Reject final extensions .html, .htm, and .xhtml.

2. Match the final filename extension case-insensitively against:

   • .log
   • .txt
   • .json
   • .xml

3. Normalize Content-Type by removing parameters, trimming whitespace, and lowercasing it.

4. Match the normalized MIME exactly against:

   • text/plain
   • application/json
   • application/xml
   • text/xml

An allowed extension or an allowed exact MIME is sufficient. Broad matches such as text/, substring tests, and application/+json are forbidden in this release.

The resulting matrix is normative:

Filename and MIME affect product eligibility only. They never select an executable rendering mode.

Resource contract

The binary limit is:

MAX_TEXT_PREVIEW_BYTES = 1,048,576

Exactly 1 MiB is eligible. 1 MiB plus one byte is not.

Known sizes

• If the selected version has a known size greater than the limit, do not request its body.
• If its known size is zero, show the empty-file state.
• If its known size is within the limit, begin a bounded request.
• An absent size is not the same as zero; it enters the bounded unknown-size path.

The current list-to-modal handoff must therefore preserve undefined rather than converting it to zero with a truthy fallback.

Bounded request

For a small or unknown size, request:

Range: bytes=0-1048576

The extra byte is an over-limit sentinel.

The client must:

1. Inspect Content-Range and Content-Length when present.
2. Read the response as a stream rather than calling response.text() or building a complete Blob.
3. Retain at most the limit plus the sentinel byte.
4. Cancel immediately when the sentinel byte is observed.
5. Enforce the same limit when the server ignores Range and returns 200.
6. Render only after end-of-stream proves that the complete object is within the limit.

An over-limit object opens an explanation state with its known size, the 1 MiB policy, and a Download action. It never shows a prefix fragment.

Request identity and cancellation

A preview request is identified by:

bucket + object name + version ID

The request must use the existing generated API client or an equivalent base-path-safe helper so that it preserves:

• same-origin credentials;
• the current Console subpath;
• version_id;
• anonymous-mode X-Anonymous: 1;
• current error handling and permission boundaries.

Close, object change, version change, bucket change, and component unmount must abort the active request and clear the old content.

Abort alone is insufficient. A generation token or invalidation flag must also prevent a response that already completed reading or decoding from updating a newer preview.

An aborted request is not an error and must not produce an error toast.

Encoding and fidelity

The first release supports strict UTF-8 only:

new TextDecoder("utf-8", { fatal: true })

Requirements:

• handle the UTF-8 BOM without displaying it;
• preserve Unicode text, emoji, tabs, LF, and CRLF;
• reject invalid UTF-8 rather than inserting replacement characters;
• reject decoded NUL characters as binary or unsupported content;
• do not guess another encoding;
• do not log or persist object text;
• always retain Download as the original-byte escape hatch.

The unsupported-encoding state should explain:

This object is not valid UTF-8 text or contains binary data. Download it to inspect the original bytes.

JSON and XML are displayed exactly as decoded source text. The first release must not run JSON.parse followed by JSON.stringify: that can alter unsafe integers, duplicate keys, whitespace, lexical forms, and the text users copy.

Safe renderer

The success state renders one text node:

<pre>{content}</pre>

The implementation must not use:

• iframe, object, or embed;
• dangerouslySetInnerHTML or innerHTML;
• DOMParser or an XML parser;
• Markdown or HTML rendering;
• an HTML data/blob URL;
• per-line or per-token spans;
• automatic links, ANSI escapes, or syntax markup.

One bounded text node keeps the DOM cost predictable and the security property inspectable.

The preformatted region uses a monospace font, preserves whitespace, defaults to no wrapping, owns both scrollbars, is keyboard focusable, and supports native selection and copy. No-wrap is intentional: it preserves aligned logs and avoids expensive layout of a single very long line.

UI states and permissions

The Preview action is enabled only when:

eligible preview type
AND object read permission
AND not a delete marker
AND not a prefix

The object-detail conjunction bug must be fixed, and list and detail surfaces must share the same eligibility function.

An eligible over-limit object still offers Preview. The modal explains why content is not loaded; disabling the button would leave the user unable to distinguish size, permission, and type failures.

The modal distinguishes:

HTTP error bodies must never be decoded and displayed as object content.

All new user-facing strings go through the existing translation layer and ship in English and Chinese together. The content region and controls must remain usable in light and dark themes and at narrow widths.

Functional and security requirements

Functional requirements

• FR1: Existing media and PDF classification remains unchanged.
• FR2: The text fallback follows the normative extension/MIME matrix.
• FR3: Eligible complete objects up to 1 MiB render as strict UTF-8 source.
• FR4: Over-limit objects render no partial content.
• FR5: Empty objects have a distinct successful empty state.
• FR6: Current and selected historical versions use the same version for metadata, size, and body.
• FR7: Anonymous access and subpath hosting retain their current request behavior.
• FR8: List and detail actions apply the same type and permission decision.
• FR9: Download, share, media, PDF, and storage behavior do not change.

Security requirements

• SR1: Object bytes can reach the DOM only through text content.
• SR2: Text Preview contains no document renderer or parser.
• SR3: At most 1 MiB plus one sentinel byte is retained.
• SR4: Closing or changing identity invalidates every previous response.
• SR5: Invalid UTF-8 and NUL content are not shown as faithful text.
• SR6: Errors, Redux, local storage, logs, and telemetry never retain preview text.
• SR7: Server authorization remains authoritative for direct requests.
• SR8: No CSP or backend inline MIME relaxation is introduced.

Implementation scope

Expected Console changes:

1. Refactor preview classification so the current media decision is preserved and text is an explicit fallback.
2. Add text to the preview type union.
3. Add a dedicated PreviewText component with streaming bounds, strict decode, request cancellation, and explicit states.
4. Route text objects explicitly to that component.
5. Remove the unreachable generic iframe fallback.
6. Fix the object-detail Preview disable expression and share eligibility logic with the list surface.
7. Preserve unknown size instead of coercing it to zero.
8. Add English and Chinese strings.
9. Add classification, component, resource, security, permission, version, and browser tests.

Expected unchanged areas:

• Console and S3 API paths;
• the backend safeMimeTypes list;
• Content Security Policy;
• object storage and metadata formats;
• image, PDF, audio, video, download, and share handlers;
• external frontend dependencies.

If a future product requires tailing, server-side transcoding, organization-wide policy, or reliable behavior through proxies that ignore Range, a dedicated server endpoint may be designed separately.

Rejected alternatives

Keep text preview disabled

Benefit: no new code or browser memory use.
Rejected because: logs and configuration objects are a routine object-storage workflow, and download-only inspection is an avoidable Console regression.

Reuse the same-origin iframe

Benefit: minimal code and browser-native presentation.
Rejected because: it turns uploader-controlled content and mutable MIME metadata into a same-origin document boundary. It also leaves resource use unbounded.

Add a backend preview API now

Benefit: central server-side limits and normalized text responses.
Rejected for the first release because: the user already has object-read permission, and the existing download endpoint provides versioning, authorization, and Range. A new API would duplicate contracts without establishing a new data-access boundary.

Show the first 1 MiB of a large object

Benefit: better large-log convenience.
Rejected because: partial JSON/XML is structurally misleading, UTF-8 boundaries need additional handling, and a single “preview” action would no longer mean complete content.

Decode invalid UTF-8 with replacement characters

Benefit: some damaged or legacy logs remain partially readable.
Rejected because: copied text would no longer faithfully represent the stored object. Lossy viewing and other encodings require a separate, explicit product mode.

Auto-format JSON

Benefit: more readable indentation.
Rejected because: parse/stringify can alter numbers, duplicate keys, lexical representation, and copied content. A future opt-in formatted view may sit beside, never replace, the raw default.

Add Monaco or another code editor

Benefit: line numbers, search, highlighting, and folding.
Rejected because: bundle, worker, CSP, and maintenance costs exceed the needs of a bounded read-only preview. A native <pre> is smaller and easier to audit.

Acceptance and test plan

Classification matrix

Automated tests must lock every normative matrix row, extension case handling, MIME parameter stripping, explicit HTML/XHTML denial, and unchanged media conflicts.

Resource tests

Cover:

• 0 bytes;
• 1 byte;
• exactly 1,048,576 bytes;
• 1,048,577 bytes;
• known over-limit size with zero body requests;
• unknown size;
• 206 with a revealing Content-Range;
• server ignores Range and returns 200;
• missing or false Content-Length;
• close and identity changes during streaming.

No case may retain or render more than the complete allowed object.

Encoding and fidelity tests

Cover UTF-8 Chinese, emoji, tabs, LF, CRLF, BOM, invalid byte sequences, NUL bytes, JSON unsafe integers, duplicate keys, original whitespace, XML declarations, DOCTYPE, CDATA, and stylesheet processing instructions.

The raw success view must preserve decoded text. Invalid and binary cases must show their dedicated state.

Security tests

Payloads containing <script>, event attributes, iframe tags, SVG handlers, XML stylesheets, external entities, and suspicious URLs must:

• appear literally in <pre>.textContent;
• create no corresponding DOM elements;
• execute no script or dialog;
• cause no object-content-originated request;
• encounter no iframe, object, embed, HTML parser, or XML parser in Text Preview.

Permission and race tests

Verify:

• no GetObject means no usable action and no retained body;
• historical versions require their corresponding permission;
• metadata and body use the same version ID;
• a late old response cannot replace a new object’s preview;
• 401, 403, 404, 416, and 5xx bodies never become preview content;
• anonymous access and Console subpaths do not regress.

Browser regression

Use a real SILO/Console test instance to inspect both English and Chinese routes, light and dark themes, and narrow and desktop widths. Media, PDF, download, share, and version workflows require smoke coverage alongside the new text states.

Delivery and completion gates

The change belongs to pgsty/silo-console, even though the user report is tracked in the SILO server repository.

Delivery is staged:

1. Merge the focused Console source and test change.
2. Pass TypeScript checking, production build, automated matrices, and real-browser security regression.
3. Update Console release notes and regenerate the actual embedded web assets.
4. Publish a Console version; a minor release is appropriate for the new visible capability.
5. Update SILO’s github.com/minio/console => github.com/pgsty/silo-console replacement to the exact new pseudo-version.
6. Build a SILO candidate from that exact dependency and repeat integration checks.
7. Publish the SILO binary and image, naming the first version that contains the feature.

These are separate states:

Issue #17 should not be described as fixed for users merely because a local preview or Console source PR exists.

Trade-off summary

The accepted design favors:

• explicit scope over a generic browser viewer;
• complete small files over partial large files;
• source fidelity over automatic formatting;
• strict UTF-8 over silent lossy decoding;
• one inert text node over a full editor;
• the existing download API over a new backend contract;
• a verifiable security invariant over convenient same-origin rendering.

The cost is real: large logs and legacy encodings still require download, and the first release has no search, line numbers, wrapping toggle, or highlighting. Those omissions are deliberate. They make the feature small enough to audit and strong enough to trust.

Review record

The design was independently reviewed from three perspectives:

• product scope, delivery, and acceptance;
• security and frontend architecture;
• compatibility and current-source verification.

The reviewers initially differed on MIME-only eligibility and lossy UTF-8 fallback. After cross-review they reached a single contract:

• existing media classification wins;
• text fallback accepts the four target extensions or four exact normalized MIME types;
• HTML/XHTML extensions are explicitly excluded;
• strict UTF-8 and NUL rejection are required;
• lossy viewing is deferred to a separate proposal.

No unresolved design question remains. Implementation may proceed against this record.

4 - No I/O Before Auth, No Privilege From Headers

This record describes the CORS hot-path and replication-request trust repair committed locally in SILO as 938603458.

Status on 2026-09-02: implementation, focused and race tests, the complete server package suite, object-lock tests, vet, build, two rounds of Fable 5 design review, repeated Opus 5 adversarial acceptance, and a real local TLS two-site replication run are complete. The candidate is pushed as pgsty/silo#101; remote CI, merge, tag, package, image, deployment, and production verification remain separate gates.
Scope: HTTP request interpretation before and inside the S3 handlers. No S3 wire field, object format, bucket metadata format, replication protocol, encryption format, or client command changes.
Security properties: pre-authentication CORS processing performs no object-layer I/O; a header never grants replication semantics by itself; SSE-C ciphertext paths and replica-only metadata require both authentication and the corresponding replication permission.

Too Long; Didn’t Read (TL;DR)

Two bugs looked unrelated:

1. an Origin header made the outermost CORS middleware treat the first URL segment as a bucket and synchronously load its metadata before authentication;
2. X-Minio-Source-Replication-Request made downstream code believe a request was internal replication merely because the header existed.

They shared the same design failure: untrusted request shape was allowed to acquire expensive or privileged internal meaning before an authorization boundary.

The repair establishes two invariants:

before authentication:  parse cheaply; never load bucket metadata
after authentication:   derive one trust decision; downstream code consumes it

For CORS, the outer middleware now reads only metadata already resident in memory. For replication, handlers authenticate the original signed request first, authorize the appropriate replication action, and then attach a private trust decision to the request context. Untrusted internal headers are stripped only after signature verification. The context decision—not header removal—is the authority used by option builders, encryption paths, object lock, event generation, and metadata persistence.

Failure A: pre-authentication CORS amplification

corsHandler wraps the complete server router. Any request carrying Origin reaches it before S3 authentication, request validity checks, and the normal API limiter.

The per-bucket CORS implementation originally called the normal bucket metadata getter:

Origin-bearing request
  -> first URL segment becomes "bucket"
  -> GetCorsConfig
  -> GetConfig cache miss
  -> read .metadata.bin
  -> probe ten legacy config paths
  -> cache a default BucketMetadata record

When .metadata.bin did not exist, the loader intentionally searched legacy configuration files. With none found, it returned a valid empty metadata record rather than NoSuchBucket. The generic getter then inserted that record into metadataMap.

An unauthenticated client could therefore vary otherwise plausible names and obtain two effects per distinct value:

• repeated erasure/object metadata reads before the normal request limiter;
• growth of the in-memory bucket metadata map.

Name validation alone cannot repair this. An attacker can generate an effectively unbounded sequence of syntactically valid, nonexistent bucket names. Distributed deployments eventually prune stale map entries during the 15-minute metadata refresh; single-node deployments do not start that refresh loop, so their synthetic entries persist until restart.

Failure B: a marker header became authority

SILO and its MinIO-compatible clients use internal headers to preserve source state during replication. The most important marker is:

X-Minio-Source-Replication-Request: true

Before this repair, several paths treated header presence—or its raw string value—as proof that the request was a replication request. That affected more than metadata extraction:

• GET of an SSE-C object could set NoDecryption and return ciphertext without the customer key to a caller holding only ordinary read permission;
• source ETag and modification time could replace server-generated values;
• source tagging, retention, and legal-hold timestamps could enter last-writer-wins comparisons;
• a past object-lock retention date could be accepted through a raw marker check;
• delete-marker identity and modification time could be supplied by the caller;
• successful object events could be suppressed;
• multipart actual size and encrypted checksum metadata could be injected at completion;
• X-Amz-Replication-Status could be persisted from ordinary PUT, COPY, or POST-policy metadata extraction.

The earlier CVE-2026-34204 repair correctly stopped ordinary PUT and COPY from importing the replication SSE metadata that could make objects unreadable. It did not yet provide one authority shared by every reader of the marker, source fields, event state, object-lock exceptions, or multipart completion metadata.

Selected design

One exact marker, two trust levels

The marker is accepted only when it appears exactly once and its value is exactly lowercase true. Duplicate values, mixed case, and any other value are untrusted.

The handler then derives two related decisions:

The split is required by the real wire protocol. Not every legitimate replication request repeats X-Amz-Replication-Status: REPLICA.

The receiver follows this matrix:

The explicit 403 for an unauthorized REPLICA request prevents a claimed replica write from being silently downgraded into a new ordinary object that may be replicated again.

Authenticate the original, then sanitize

SigV4 signs request headers. Removing an internal header before authentication would change the canonical request and turn a valid signature into SignatureDoesNotMatch.

The ordering is therefore mandatory:

original request
  -> existing signature/authentication path
  -> ordinary S3 action authorization
  -> replication action authorization
  -> derive trusted / replicaTrusted
  -> bind decision to request context
  -> clone and strip untrusted internal fields
  -> option parsing, encryption, object lock, storage, events

The audit logger retains the original request. The effective request clone retains public S3, SSE, checksum, object-lock, copy-source, proxy, and replication-validity headers. It strips only internal source/replication controls, including source ETag/MTime/delete-marker/timestamps, replication SSE state, actual object size, encrypted checksum transfer, and the request use of X-Amz-Replication-Status.

Header stripping is defense in depth. All privileged consumers use the private context decision or an explicit Boolean; they do not infer trust by looking at the clone.

Replica status is not generic user metadata

X-Amz-Replication-Status is an S3 response header that MinIO-compatible servers also use as an internal request control. It no longer belongs to the generic supported-request-metadata list.

Ordinary PUT, COPY, multipart initiation, Snowball/PAX extraction, and POST policy cannot persist it merely by submitting the field. The receiver sets REPLICA explicitly only in a replicaTrusted branch.

This closes a subtle POST-policy path: a form field could previously store REPLICA, causing the resulting object to evade normal replication scheduling even though the POST principal never held replication permission.

Object lock receives an explicit decision

The object-lock parser used to accept past retention dates when the raw marker header was present. That package now receives allowPastRetainDate explicitly from replicaTrusted state.

The surrounding handler also uses the same decision when deciding whether an existing compliance/legal-hold version may be overwritten by a replica. This removes an internal-header dependency from the reusable object-lock package.

Actual replication wire matrix

The design was checked against the silo-go v7.3.1 emitter selected by the server’s go.mod, not inferred from comments or upstream documentation.

s3:ReplicateDelete is the trust gate, not the receiver’s only permission. For compatibility with deployed target policies, a trusted replication delete also requires s3:DeleteObject; an explicit deny on s3:DeleteObjectVersion still blocks a named-version purge. Ordinary clients do not use this compatibility path: an explicit UUID or versionId=null requires an allow for s3:DeleteObjectVersion.

Requiring REPLICA for every trusted operation would break PutPart, multipart completion, and batch replication. Trusting every marker would recreate the vulnerability. Stored multipart provenance bridges the two requirements for encrypted raw parts.

CORS resident-only state machine

The outer CORS middleware must remain cheaper than the request it is about to route. It now calls a dedicated resident-only getter that takes one read lock and examines only in-memory state.

loadFailed is populated only from disk-derived bucket lists during startup or refresh. It cannot grow from a client path. Successful metadata load, Set, bucket removal, stale-bucket reconciliation, and subsystem reset clear the corresponding state.

Cold-cache residual boundary

There is one accepted edge: a real bucket can be absent from both metadataMap and loadFailed if a node misses the peer metadata-load notification. Until the next bucket refresh discovers and loads it, that node treats the name as unknown and uses global CORS.

CORS remains a browser response policy, not an authorization mechanism—normal S3 authentication and bucket policy still apply—but an operator relying on a restrictive per-bucket CORS document should understand the temporary relaxation. A follow-up can mark every disk-listed-but-nonresident bucket as load-failed before attempting its refresh, preserving fail-closed behavior without adding synchronous request I/O.

Alternatives rejected

Implementation boundary

The selected change is intentionally layered:

1. a small request-trust module defines exact marker parsing, replication authorization, private context state, and the post-authentication effective request;
2. object option builders parse source fields only when their caller provides trusted state;
3. DecryptObjectInfo, event request parameters, multipart completion, delete options, and object lock consume the same decision;
4. handlers calculate trust immediately after their existing authentication path;
5. multipart part handling combines current-request trust with stored MPU replica provenance;
6. generic metadata extraction does not accept replica status;
7. CORS middleware uses a separate resident-only metadata accessor and never calls the load-on-miss getter.

No object-layer API needs to infer HTTP trust. Programmatic internal callers that construct ObjectOptions{ReplicationRequest: true} remain unchanged.

Verification and adversarial review

Regression coverage includes:

• hundreds of distinct valid missing bucket names, both actual and preflight CORS requests, with zero metadata reads and no map growth;
• Console, reserved, invalid, startup, internal namespace, invalid stored CORS, and known load-failure paths;
• least-privilege SSE-C GET, HEAD, and GetObjectAttributes callers with correct, missing, wrong-case, and unauthorized markers;
• marker-only batch-style PUT preserving source ETag/MTime only with s3:ReplicateObject;
• unauthorized REPLICA PUT and DELETE returning 403;
• POST policy unable to forge replica status;
• object-lock past-date parsing with and without replica trust;
• marker-only CopyObject with SSE-C source headers copying plaintext rather than ciphertext;
• fake marker on an ordinary SSE-C MPU failing instead of storing raw bytes;
• a real in-process SSE-C multipart replication chain: encrypted source, raw ciphertext part, trusted replica initiation, marker-only PutPart and Complete, and exact plaintext recovery with the original key.

The final local tree passed focused and race tests, the complete cmd suite, object-lock tests, vet, build, and diff checks.

A separate black-box run started two TLS-enabled SILO instances built from the candidate and enabled real site replication. It verified:

• an SSE-C 4 KiB object;
• an SSE-C 12 MiB, three-part multipart object;
• an SSE-C CopyObject result;
• a replicated delete marker.

Source and target ETag, size, version ID, SSE-C key MD5, decrypted SHA-256, and delete-marker version ID matched; targets reported REPLICA.

Two Fable 5 review rounds first corrected the trust model for marker-only batch and multipart calls, then audited the implementation. A final independent Claude Code Opus 5 review reported GO, with no P0/P1 findings, and independently reran build, vet, race, object-lock, and full cmd tests.

Compatibility and operations

• Ordinary clients: no request change. Untrusted internal headers are ignored instead of acquiring internal semantics.
• Unauthorized claimed replica writes: requests carrying X-Amz-Replication-Status: REPLICA now return 403 where some multipart subpaths previously lacked a uniform check.
• Batch replication: destination credentials must include s3:ReplicateObject, as documented in the batch replication requirements. Without it, the receiver processes marker-only writes as ordinary writes and does not preserve source ETag/MTime.
• SSE-C: ordinary reads still require the customer key. Authorized replica reads may use the raw ciphertext path needed to preserve encrypted bytes.
• Events: only trusted replication suppresses replica creation/access events; a forged marker no longer silences them.
• Object lock: replica exceptions are permission-derived rather than header-derived.
• Performance: CORS removes pre-authentication backend work. Trusted writes add policy checks already required by the replication contract; no additional object pass is introduced.
• Rolling upgrade: wire and storage formats are unchanged. New receivers enforce the trust boundary; old receivers remain vulnerable to the old header semantics until upgraded. Per-bucket CORS behavior can therefore differ by node during the rolling window.
• Rollback: data written by the repaired version remains readable by the previous version, but rollback reopens both trust defects and restores pre-authentication metadata loads.

Residual risks and follow-ups

• Mark disk-listed but nonresident buckets fail-closed before refresh to narrow the cold-cache CORS window described above.
• Emit a rate-limited diagnostic when a marker-bearing request lacks replication permission; the safe ordinary fallback is otherwise easy to misdiagnose as an ETag/MTime mismatch.
• Replication validity probes retain their inherited permission-reporting behavior and should be audited separately rather than silently changed in this repair.
• This review covers the named source/replication headers. Other future internal controls must still answer the same question: which authenticated decision allowed this client value to acquire internal meaning?

Conclusion

An internal-looking header is still client input. A bucket-shaped URL segment is still attacker input. The durable repair is to stop either one from becoming authority by accident:

Before authentication, do no backend work. After authentication, derive trust once and pass the decision—not the claim—downstream.

That rule is broader than CORS or replication. It is the boundary future SILO handlers should preserve whenever inexpensive public request syntax meets expensive or privileged internal state.

5 - One Endpoint, Two Privileges: Separating User and Group Status

This document records the discussion, repair, and final authorization design for upstream issue minio/minio#21478 and SILO PR #73.

Status on 2026-08-26: SILO PR #73 was merged as 2e2377d1c, preserving the signed-off repair commit 58735ee38. All eight reported checks passed. Upstream issue #21478 and PR #21482 remain open, but minio/minio is archived and read-only, so no further issue comment or merge can be made there.
Group follow-up on 2026-08-28: final release review found the same fixed-action defect in set-group-status. Signed-off server commit d98250110 now selects admin:EnableGroup or admin:DisableGroup from the requested target state and adds a real four-way IAM authorization test. Local verification and independent review are complete; push, remote CI, merge, tag, and delivery remain pending.
Scope: authorize enabling and disabling a user with their respective existing Admin Actions. Do not change the route, status values, account storage, replication record, or client API.
Security property: possessing admin:DisableUser must not grant the ability to enable an account, and possessing admin:EnableUser must not grant the ability to disable one.
Release boundary: merge, tag, release package, container image, deployment, and production verification remain separate gates.

Too Long; Didn’t Read (TL;DR)

SILO exposes both admin:EnableUser and admin:DisableUser, but the shared set-user-status handler historically authorized every request with admin:EnableUser. A policy that granted only admin:DisableUser therefore could not disable an account. The workaround was to grant admin:EnableUser as well, which destroyed the least-privilege boundary that the two action names promised.

The selected repair derives exactly one required action from the requested target state before authorization:

The handler then calls validateAdminReq once. A four-way IAM test proves both positive operations and both denied cross-action operations. This is intentionally stricter than preserving the accidental historical behavior in which an Enable-only policy could also disable users.

The same rule now applies to group status:

Before the follow-up, an EnableGroup-only principal could disable a group, while a DisableGroup-only principal received AccessDenied for that exact operation. The group repair uses the same one-selector, one-authorization design rather than treating the two actions as aliases.

The reported defect

The Admin API uses one route for both state transitions:

PUT /minio/admin/v3/set-user-status
    ?accessKey=<target>
    &status=enabled|disabled

Before the repair, the handler checked one fixed action before reading the requested status:

objectAPI, creds := validateAdminReq(ctx, w, r, policy.EnableUserAdminAction)

The later call to SetUserStatus correctly received either enabled or disabled, but authorization had already treated both as Enable operations. admin:DisableUser existed in the policy vocabulary and documentation while being ineffective for this endpoint on its own.

Issue #21478 supplied the practical counterexample: an operator wanted a policy that could disable accounts during an incident without being able to restore them. A policy containing admin:DisableUser received AccessDenied; adding admin:EnableUser made the request work, but also gave the operator the more powerful recovery transition that the policy intentionally withheld.

This is not a missing convenience permission. It is a mismatch between the policy model and the enforcement point:

policy says:     DisableUser only
request says:    target state = disabled
handler checked: EnableUser
result:          legitimate disable denied
workaround:      grant an unwanted enable capability

Why two actions must mean two capabilities

An account state transition has direction. Disabling is commonly delegated to incident responders, fraud controls, compliance automation, or a break-glass process. Enabling restores access and may require a separate approver.

If either action authorizes both transitions, a policy author cannot express that separation. The server would publish two names while enforcing one combined capability. The design contract is therefore strict:

The built-in consoleAdmin policy grants admin:*, so full administrators retain both operations. The compatibility impact is limited to custom restricted policies that relied on the old accidental behavior.

The public PBAC reference now states the same contract for admin:EnableUser and admin:DisableUser.

Design goals and non-goals

Goals

1. Make both existing Admin Actions enforceable according to their names.
2. Preserve least privilege in both directions.
3. Perform one authorization decision and write at most one authorization error.
4. Preserve the route, request values, response format, self-mutation guard, IAM storage call, and site-replication hook.
5. Encode the contract in tests that fail if the two permissions are broadened or swapped again.

Non-goals

• split the endpoint into separate enable and disable routes;
• add a new combined action or change policy syntax;
• change user status persistence or replication;
• redesign Console permissions;
• infer release, image, deployment, or production delivery from a source merge.

Alternatives considered

Keep checking admin:EnableUser for both states

This preserves behavior but leaves admin:DisableUser unusable and forces over-privileged policies. It is the defect, not a compatibility contract worth retaining.

Require both actions for either transition

This makes the two labels decorative and prevents delegated disable-only operation. It is stricter in quantity but weaker in expressiveness and least privilege.

Try Enable authorization, then retry Disable authorization

Upstream PR #21482 attempted this shape for a disabled request. It first called validateAdminReq with EnableUser, then called it again with DisableUser if the first result was nil.

That helper has an important contract: when it returns a nil object layer, it has already written an error response. A Disable-only request can therefore commit a 403 response before the second authorization succeeds and the handler proceeds to mutate account state. Authorization fallback must never continue after an error response has been committed.

Accept either Enable or Disable for a disabled request

validateAdminReq already accepts multiple actions and succeeds if any one is allowed, so compatibility behavior could be implemented safely with one variadic call. That would let Disable-only policies work while preserving the historical ability of Enable-only policies to disable.

SILO rejected this option because the historical ability was the enforcement bug. It would solve the reporter’s positive case but retain a cross-action privilege that contradicts the two-action model. Operators who want both transitions can grant both actions explicitly.

Validate the status before authenticating

Rejecting unknown status values first would change error precedence: a caller that previously had to pass the Enable authorization gate could now receive a validation result before authorization. The repair does not need that broader behavioral change.

Unknown values therefore retain admin:EnableUser as the authorization default. Valid disabled is the only value that selects admin:DisableUser; the existing IAM layer remains responsible for rejecting invalid status values after authorization.

The selected implementation

The repair adds a pure selector:

func setUserStatusAdminAction(status string) policy.AdminAction {
    if madmin.AccountStatus(status) == madmin.AccountDisabled {
        return policy.DisableUserAdminAction
    }
    return policy.EnableUserAdminAction
}

The handler reads the route variables, selects the action, and authorizes exactly once:

vars := mux.Vars(r)
accessKey := vars["accessKey"]
status := vars["status"]

objectAPI, creds := validateAdminReq(ctx, w, r, setUserStatusAdminAction(status))
if objectAPI == nil {
    return
}

Everything after the gate remains unchanged:

• a caller still cannot enable or disable its own account;
• globalIAMSys.SetUserStatus validates and persists the requested status;
• site replication records the same status and timestamp;
• response and audit behavior use the existing path.

The selector depends only on the requested target state. It does not load the current user, infer a transition from stored state, or make authorization depend on whether the target exists. This keeps authorization deterministic and avoids a read-before-authentication dependency.

Why the repair is safe

The correctness argument consists of five invariants:

1. Every valid status maps to exactly one Admin Action.
2. validateAdminReq is invoked once, so a failed authorization cannot be followed by mutation.
3. The mutation call is reachable only after the selected action succeeds.
4. Invalid status values preserve the old Enable authorization boundary and are still rejected by the existing status-validation path.
5. No storage, replication, wire, or client contract changes; only the permission required to reach the existing mutation changes.

The change is a deliberate authorization tightening for Enable-only custom policies that used the disable operation. That tightening is the mechanism that makes admin:DisableUser a real independent capability.

Test design

Pure action mapping

The unit test fixes three selector cases:

Four-way IAM authorization matrix

The integration test creates separate users and policies, then exercises the real Admin API:

1. a Disable-only client successfully disables a target;
2. the same client receives AccessDenied when enabling it;
3. an Enable-only client successfully enables the target;
4. the same client receives AccessDenied when disabling it.

Positive assertions alone would not prove least privilege: both policies could accidentally authorize both states and still pass. The two negative cross-action assertions are the security regression tests.

The test removes every temporary user and policy after execution. It runs inside the existing IAM server suite, so it covers request signing, policy attachment, handler authorization, persistence, and Admin-client error decoding rather than testing only the helper.

Repair and verification record

The server checkout originally contained unrelated dependency, generated-credit, checksum-test, and security-document changes, while local main was behind the remote. The two user-status files were isolated into a clean worktree based on current origin/main; no unrelated file entered the repair commit.

Local verification passed:

go test ./cmd -run '^TestSetUserStatusAdminAction$' -count=1
go test ./cmd -run '^TestIAMInternalIDPServerSuite$' -count=1
git diff --check

The signed-off commit 58735ee38 was pushed in PR #73. Its eight remote checks all passed:

• DCO sign-off;
• format, build, and vet;
• lint and generated files;
• cmd/ tests;
• internal/ tests;
• race detector and S3 Select;
• cross compilation;
• vulnerability analysis.

The PR was merged with the repository’s normal merge strategy as 2e2377d1c. Local main was then fast-forwarded only after the two original working files were byte-for-byte and patch-ID identical to the merged result. The unrelated local changes remained intact, and the temporary worktree and task branch were removed after the code became recoverable from main and PR #73.

Least-privilege policy examples

Disable-only operator

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "admin:DisableUser",
        "admin:GetUser"
      ]
    }
  ]
}

This principal can inspect and disable another user, but cannot enable it.

Enable-only operator

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "admin:EnableUser",
        "admin:GetUser"
      ]
    }
  ]
}

This principal can inspect and enable another user, but cannot disable it. Grant both actions explicitly to roles responsible for the complete account lifecycle.

Group-status follow-up

The group endpoint has the same shape as the user endpoint:

PUT /minio/admin/v3/set-group-status
    ?group=<target>
    &status=enabled|disabled

It also publishes two existing actions, admin:EnableGroup and admin:DisableGroup. The inherited handler nevertheless authorized every request with EnableGroup before reading status. This was not merely a dead permission: it reversed least privilege in both directions. The wrong principal could disable a group, and the intended disable-only principal could not.

The follow-up adds setGroupStatusAdminAction, deliberately matching setUserStatusAdminAction:

func setGroupStatusAdminAction(status string) policy.AdminAction {
    if madmin.GroupStatus(status) == madmin.GroupDisabled {
        return policy.DisableGroupAdminAction
    }
    return policy.EnableGroupAdminAction
}

The integration test creates separate EnableGroup-only and DisableGroup-only administrators and a real target group. It proves:

1. DisableGroup-only can disable;
2. DisableGroup-only cannot enable;
3. EnableGroup-only can enable;
4. EnableGroup-only cannot disable.

The suite exercises signed Admin requests, policy attachment, handler authorization, IAM mutation, response decoding, and cleanup. Invalid status still selects the legacy Enable action before the existing validation error, so the change does not expose a new pre-authentication oracle. The successful site-replication hook remains after mutation and is not called for denied requests.

This follow-up changes no user behavior and introduces no new policy action. It makes the two already documented group actions enforce the same state-specific contract as their user counterparts.

Compatibility and migration

No client or API migration is required. The endpoint, query parameters, status strings, success response, and Admin-client method are unchanged.

Policy review is required for restricted administrative roles:

• a role that should only disable users needs admin:DisableUser;
• a role that should only enable users needs admin:EnableUser;
• a role that must do both needs both actions;
• consoleAdmin and other admin:* policies are unaffected;
• a legacy custom policy containing only admin:EnableUser can no longer use that permission to disable users and must add admin:DisableUser if both operations are intended.

The equivalent rules now apply to group-management roles:

• a role that should only disable groups needs admin:DisableGroup;
• a role that should only enable groups needs admin:EnableGroup;
• a role that must do both needs both actions;
• a legacy EnableGroup-only role can no longer disable groups.

This is a source-level compatibility change in authorization behavior, not a wire-protocol break.

Upstream disposition

As of this record, upstream issue #21478 and PR #21482 are still displayed as open. The upstream repository is archived and read-only. An attempt to leave the single-authorization analysis on the PR was rejected by GitHub because archived, locked discussions cannot accept comments.

The upstream artifacts remain useful provenance but are no longer an actionable delivery path. SILO owns its implemented semantics, tests, merge, release note, and eventual production verification.

Delivery state

Conclusion

The repairs make the authorization model tell the truth. Enabling and disabling users or groups are opposite state transitions with different operational risk, and SILO already exposes different policy actions for each direction. Each handler must therefore select the action from the requested target state and authorize once before mutation.

The code change is small because the design boundary is clear. The durable result is larger: an explicit permission matrix, rejected compatibility alternatives, an invalid-input rule, a four-way integration test, a clean merge record, migration guidance, and an honest release boundary.

6 - Config Environment Files Are Not Shell Scripts

This record defines the startup contract for MINIO_CONFIG_ENV_FILE and explains the compatibility repair committed in SILO as ce456dba0.

Status on 2026-08-28: implementation, focused tests, the complete cmd and internal suites, tagged tests, race tests, vet, lint, generated-file checks, rebrand guards, build, and an independent local Fable Max review are complete. The server commit exists locally; push, remote CI, merge, tag, package, image, deployment, and production verification remain separate gates.
Scope: environment-file parsing and named-target discovery only. No configuration key, subsystem, value, precedence, storage format, or client API changes.
Compatibility rule: the file is a SILO input format. Supporting an optional export prefix does not make it a POSIX shell program.

Too Long; Didn’t Read (TL;DR)

SILO can load startup variables from a file:

export MINIO_CONFIG_ENV_FILE=/etc/default/silo
silo server /data

The parser accepts assignments such as:

MINIO_ROOT_USER = silo-admin
MINIO_ROOT_PASSWORD = "  significant surrounding spaces  "
MINIO_NOTIFY_WEBHOOK_ENABLE_my-hook = off
MINIO_NOTIFY_WEBHOOK_ENDPOINT_my-hook = https://events.example.com/minio

The last two names are important. Multi-target configuration appends the target name verbatim after an underscore. The configuration subsystem does not restrict a target to a shell identifier; names containing -, ., :, digits, or printable Unicode can be discovered and resolved exactly.

A hardening change accidentally validated every key as [A-Za-z_][A-Za-z0-9_]*. It made my-hook invalid and stopped the server during restart even though the previous loader and the configuration target model accepted it. The repair validates what SILO actually needs instead:

• the name is non-empty, valid UTF-8, and made of visible non-whitespace characters;
• = and NUL are not allowed in a name;
• NUL is not allowed in a value;
• invalid input reports file and line without reporting the value;
• the complete file is parsed before any assignment is applied.

Why the regression was real

The environment-file loader calls os.Setenv after parsing. An operating-system environment is a list of strings, not a shell variable namespace. Shell assignment syntax is narrower because the shell must tokenize and expand variable names in its own language.

Named SILO configuration targets are built differently:

MINIO_<SUBSYSTEM>_<PARAMETER>_<target>

For example:

MINIO_NOTIFY_WEBHOOK_ENABLE_my-hook
MINIO_NOTIFY_WEBHOOK_ENDPOINT_my-hook

Target discovery lists variables by the fixed parameter prefix and treats the remaining suffix as the target name. Target lookup reconstructs the same name without uppercasing or sanitizing that suffix. Rejecting - in the file parser therefore broke a valid discover-to-resolve path; it did not protect a shell evaluation path because no shell evaluates the file.

The failure is operationally sharp. MINIO_CONFIG_ENV_FILE is loaded only at startup. A server can continue running with an old process environment, then fail on its next restart after the file or binary changes. Startup must fail on malformed input, but it must not invent a narrower target grammar than the configuration system.

The file grammar

Lines and comments

• blank lines are ignored;
• a line whose first non-whitespace character is # is ignored;
• an optional standalone export followed by whitespace is removed;
• exportFOO=value remains the key exportFOO; it is not mistaken for the prefix;
• the first = separates key and value, so additional = characters remain part of the value.

The file is not a shell. It does not perform variable expansion, command substitution, backslash processing, or inline-comment interpretation.

Keys

Surrounding whitespace around the key is removed. The remaining key must:

1. be non-empty valid UTF-8;
2. contain only Unicode graphic characters;
3. contain no whitespace, =, NUL, control, or invisible format characters.

This preserves OS-compatible names and multi-target suffixes while rejecting visually empty or structurally ambiguous keys. A key beginning with a digit or punctuation is accepted by the parser; SILO still reads only the exact names used by its configuration and runtime components.

Values and quoting

Unquoted values are trimmed. To retain leading or trailing spaces, quote the complete value with matching single or double quotes:

PLAIN = value
SPACED = "  value with significant spaces  "
TOKEN = scheme://user:password@example.com?a=b
EMPTY =

The parser removes one matching outer quote pair. It does not interpret escapes inside the quoted value. NUL is always rejected because it cannot be represented in an environment entry.

Failure and secrecy contract

Syntax errors stop startup. Diagnostics include the file path, line number, and the invalid key or error class, but never the value. A password on a malformed line must not be copied into logs.

Parsing is all-or-nothing: a syntax error returns no entries, and assignment starts only after the complete file has parsed. If the operating system rejects a validated assignment, SILO also stops startup and identifies the key and file. Since the process exits, it never serves requests with a partially loaded environment.

The file itself remains a privileged secret-bearing input. Operators must protect it with appropriate ownership and mode; parser validation is not a substitute for filesystem permissions.

Regression matrix

The committed tests cover:

• spaces and tabs around =;
• quoted values with significant spaces;
• standalone export, including Unicode whitespace after it;
• keys beginning with _, a digit, or punctuation;
• named targets using -, ., :, and Unicode;
• exact named-target discovery through the configuration subsystem;
• empty keys, whitespace, NUL, and invisible format characters;
• NUL values;
• multiple = characters in URLs and tokens;
• file-and-line diagnostics that redact values;
• all-or-nothing parse results.

The implementation passed the complete local server verification matrix and a read-only adversarial review. Windows-specific os.Setenv behavior has not been exercised on a Windows runner; unsupported platform rejection remains fail-fast rather than silent.

Compatibility and delivery

No configuration migration is required. Existing ordinary environment names behave unchanged. Files using shell-style whitespace become more predictable, and previously accepted named targets work again.

The visible compatibility changes are intentional:

• invalid or invisible names now fail instead of being silently ignored;
• unquoted surrounding value whitespace is trimmed; quote it when significant;
• malformed input stops startup with a redacted location-aware error;
• a valid punctuation-bearing target is no longer rejected merely because a shell could not assign it with NAME=value syntax.

This record describes a source commit, not a delivered release. Until the commit is pushed, tested remotely, merged, tagged, packaged, imaged, and deployed, operators must not assume a public SILO binary contains this parser contract.

Conclusion

Configuration compatibility depends on validating the format SILO actually consumes. MINIO_CONFIG_ENV_FILE borrows a small amount of dotenv-like syntax for operator convenience, but it is not executed by a shell. The repair restores named-target compatibility while retaining strict NUL, invisibility, redaction, and fail-fast guarantees.

7 - Two SSE-C Keys, One CopyObject Response

This record explains the CopyObject SSE-C checksum response repair committed in SILO as e37b0134a.

Status on 2026-08-28: implementation, encryption and key-rotation tests, complete server suites, race tests, static checks, build, and independent Fable Max acceptance review are complete. The commit exists locally; push, remote CI, merge, tag, package, image, deployment, and production verification remain separate gates.
Scope: the successful CopyObject XML and HTTP response after the destination object has committed. Stored object bytes, checksum metadata, encryption format, source decryption, federation, replication, and historical objects are unchanged.
Security property: source SSE-C headers may decrypt only source state; destination SSE-C headers may decrypt only committed destination state.

Too Long; Didn’t Read (TL;DR)

An SSE-C copy can use two independent keys:

SILO correctly wrote the destination with its destination key. However, after commit, both the XML generator and the generic PUT-response header helper received the complete CopyObject request. The checksum metadata decrypter intentionally prefers copy-source SSE-C headers when they are present. That priority is correct while reading the source, but wrong when interpreting the committed destination.

With source key A and destination key B:

source body       --decrypt A--> logical bytes
logical bytes     --encrypt B--> committed destination
destination csum  --seal B-----> stored checksum metadata
response decoder  --try A------> key mismatch, checksum omitted

The object and stored checksum were correct; only the successful response was incomplete. The repair constructs a destination response-header view by removing exactly the three copy-source SSE-C customer headers. It decrypts the destination checksum once, then reuses the resulting map for both XML and HTTP response headers.

Observable failure

The failure requires a checksum-bearing destination and distinct source/destination SSE-C contexts. A representative request supplies:

x-amz-copy-source: /bucket/source
x-amz-copy-source-server-side-encryption-customer-algorithm: AES256
x-amz-copy-source-server-side-encryption-customer-key: <key A>
x-amz-copy-source-server-side-encryption-customer-key-md5: <md5 A>
x-amz-server-side-encryption-customer-algorithm: AES256
x-amz-server-side-encryption-customer-key: <key B>
x-amz-server-side-encryption-customer-key-md5: <md5 B>
x-amz-checksum-algorithm: CRC32

Before the repair:

• CopyObject returned HTTP 200;
• reading the destination with key B returned the correct body;
• stored destination checksum metadata decrypted with key B and matched the logical bytes;
• the CopyObject XML and HTTP response omitted CRC32 and ChecksumType.

This is a response-contract defect, not evidence of corrupted object data.

The same ambiguity affects same-object SSE-C key rotation. After metadata has been resealed under key B, the request still carries source key A in the copy-source headers. Response generation must describe the post-rotation object, so it must use B.

Why the global decrypter must not change

The metadata decrypter’s copy-source priority is not itself a bug. Earlier in CopyObject, the server examines source checksum metadata to decide whether to preserve its algorithm, recompute a full-object value, or add the default CRC64NVME checksum. For an SSE-C source, that metadata is protected by the source object key and therefore requires the copy-source headers.

Changing the global priority to prefer destination SSE-C headers would fix the final response while breaking source checksum interpretation. The safe boundary is temporal and object-specific:

before commit: full request headers, source context
after commit:  destination-only SSE-C headers, destination context

The repair applies only at that post-commit boundary.

Selected implementation

Destination response view

The handler clones the request headers and removes exactly:

• X-Amz-Copy-Source-Server-Side-Encryption-Customer-Algorithm;
• X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key;
• X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key-MD5.

Regular destination SSE-C headers remain. SSE-S3 and SSE-KMS destination metadata needs no customer key and continues through the existing path.

Decrypt once, project twice

Before the repair, CopyObject called decryptChecksums once while building XML and again while writing success headers. For SSE-S3 or SSE-KMS this could repeat KMS unseal work.

The repaired flow is:

committed ObjectInfo
  -> decryptChecksums(destination headers) once
  -> CopyObjectResult XML fields
  -> x-amz-checksum-* and x-amz-checksum-type response headers

The generic setPutObjHeaders wrapper remains available to PutObject, CompleteMultipartUpload, and DeleteObject. CopyObject calls a narrow helper that accepts the already decrypted checksum map. ETag, VersionID, delete-marker, lifecycle prediction, and checksum header behavior remain in one shared implementation.

Regression matrix

The tests cover:

• plaintext source to SSE-C destination;
• compressed and uncompressed SSE-C destinations;
• SSE-C source key A to destination key B;
• checksum value and type in both CopyObject XML and HTTP headers;
• stored checksum decrypted with destination key B;
• destination body readable with B;
• same-object key rotation from A to B;
• checksum response after rotation;
• SSE-S3 source and destination combinations;
• all object-layer backends used by the API test harness.

The final combined tree passed focused encryption tests, the complete cmd and internal suites, the project’s tagged test configuration, full go test -race ./..., vet, lint, generated-file checks, rebrand guards, and a local build. A mirror Fable Max review reported no P0–P2 findings and independently confirmed that source decryption still receives the full request while destination response decryption receives the filtered view.

Compatibility and operational impact

• Successful CopyObject responses: checksum fields that were previously missing now appear when the committed destination has a checksum.
• Stored objects: no rewrite, migration, metadata-format, or encryption-format change.
• Existing objects: unaffected; the defect existed only in the one-time successful response.
• Clients: no request change. Clients already providing both source and destination SSE-C keys receive a more complete S3-compatible result.
• Performance: one metadata checksum decryption instead of two; no additional object read or hash pass.
• Rolling upgrade: old nodes may omit the fields while new nodes return them. Stored objects remain mutually readable.
• Rollback: restores response omission but does not damage objects created while the repair was present.
• Security: no key or digest value is added to logs or error responses. The response carries only the checksum already authorized for the successful write.

This repair does not resolve the separately deferred legacy federation CopyObject branch and does not audit or modify historical compressed-object checksums. Those questions have different data and operational boundaries.

Conclusion

CopyObject is one request with two object identities. Reusing the full request after commit erased that distinction: a source key was allowed to shadow the destination key while describing destination metadata. The durable repair is not a new encryption scheme; it is an explicit context boundary, followed by one decryption and two faithful response projections.

8 - Why CompleteMultipartUpload Must Return ChecksumType: Review of PR #57

This is the design, review, and decision record for SILO #47 and PR #57.

Status on 2026-08-26: PR #57 was approved and merged as a96116b1; #47 closed automatically. All nine checks on the tested PR head passed, followed by green Go CI and VulnCheck runs on main. No tagged release, package, container image, deployment, or production endpoint has yet been verified to contain the fix.
Scope: return the already-known checksum type from CompleteMultipartUploadResult; do not add new checksum algorithms.
Owner: pgsty/silo, the SILO server repository.
Release boundary: code review, merge, a green main, a tagged release, packages, container images, deployment, and production verification are separate gates.

Too Long; Didn’t Read (TL;DR)

SILO already computed and persisted the correct checksum type for a completed multipart object. HEAD, ListParts, and GetObjectAttributes could expose it. The completion response could not, because its Go response struct had checksum value fields but no ChecksumType field.

PR #57 adds that field, copies the existing value from the checksum map, registers the new exported symbol in the compatibility baseline, and tests FULL_OBJECT, COMPOSITE, and the no-checksum case. It does not recalculate data, change metadata, migrate objects, or weaken integrity checks.

The repair is correct and intentionally narrow. Maintainers approved the fork workflows, refreshed the stale PR branch onto current main, required every new check to pass, submitted an approving review, and merged while preserving the contributor’s signed-off commit. Repository integration is complete; release delivery remains a separate gate.

Where the defect came from

The defect was found while investigating #31, where a real boto3 client exposed several adjacent multipart-checksum incompatibilities. #31 was the data-path failure: a FULL_OBJECT CRC32 multipart upload could fail at completion. It was fixed independently by 0cff48f6c and 75859690b, then closed on 2026-08-04. That review deliberately split four adjacent findings into #46, #47, #48, and #50 instead of treating them as one checksum bug.

After the object completed successfully, another inconsistency remained:

complete_multipart_upload() -> ChecksumType: None
head_object()               -> ChecksumType: FULL_OBJECT

AWS S3 returned FULL_OBJECT in both places. SILO returned the checksum value in the completion XML, and the committed object retained the correct type, but the completion SDK result exposed a null type.

That observation became #47. It is a presentation defect, not a checksum-calculation or storage defect. It does not explain the earlier InvalidPart failure from #31, and repairing it does not replace the server-side part-checksum work tracked in #46, which later landed independently as 7fea6d5a5.

The S3 response contract

The AWS CompleteMultipartUpload API defines ChecksumType as an element of CompleteMultipartUploadResult. Its valid values are:

When an object has no additional S3 checksum, the element should be absent. A server must not invent a type with no checksum value.

This distinction matters to clients. The same Base64 field name can describe either a direct full-object checksum or a multipart composition. A client that validates the completion result needs the type to interpret the checksum correctly and to compare the response with the mode selected at CreateMultipartUpload.

What SILO did before the PR

The completion handler already passed the committed ObjectInfo to generateCompleteMultipartUploadResponse. That generator already called:

cs, _ := oi.decryptChecksums(0, h)

The checksum decoder returned a map containing both the algorithm value and the normalized object type:

CRC32                 -> "...Base64..."
x-amz-checksum-type    -> "FULL_OBJECT" or "COMPOSITE"

The response struct copied the values for CRC32, CRC32C, CRC64NVME, SHA1, and SHA256. It simply had nowhere to put the type:

committed ObjectInfo.Checksum
        -> decryptChecksums
        -> checksum values + x-amz-checksum-type
        -> CompleteMultipartUploadResponse
        -> checksum values copied, type discarded
        -> XML without <ChecksumType>
        -> SDK returns None / null

Other surfaces used the same state correctly. ListParts and GetObjectAttributes already returned ChecksumType; HEAD also reported the stored type. The loss was isolated to the success XML for CompleteMultipartUpload.

What PR #57 changes

The contributed diff contains one signed-off commit, three files, 60 added lines, and no deletions. Only two production lines change. A maintainer later merged current main into the contributor branch to refresh its CI context; that merge changed history, not the three-file product diff.

Add the response field

ChecksumType string `xml:"ChecksumType,omitempty"`

omitempty is part of the compatibility contract: checksum-free uploads retain the old XML shape.

Copy the existing normalized value

ChecksumType: cs[xhttp.AmzChecksumType],

The generator does not infer the type from an ETag, algorithm name, or part count. It uses the same decoded metadata that already supplies the checksum values.

Test the response surface

The added test covers:

• no checksum: the Go field is empty and <ChecksumType> is absent;
• a full-object checksum: the field is FULL_OBJECT and the tag is present;
• a multipart composite checksum: the field is COMPOSITE and the tag is present.

It checks the response value before XML encoding and separately checks omission/presence after encoding.

Record the exported compatibility symbol

CompleteMultipartUploadResponse.ChecksumType is an exported Go field. SILO’s rebrand guard performs an exact comparison of the exported compatibility surface, so the PR correctly adds the field to buildscripts/rebrand-guard/compat-baseline.json. This is an acknowledgement of an intentional public surface change, not a bypass of the guard.

Why the repair works

The correctness argument is a short chain of existing invariants.

1. ObjectInfo.Checksum is the committed checksum metadata. The completion response is generated only after the object layer returns the committed ObjectInfo.
2. decryptChecksums(0, h) uses the existing metadata-decryption path, including the request headers needed for SSE-C. No second decryption mechanism is added.
3. The checksum decoder writes x-amz-checksum-type only when it has decoded a non-empty checksum value.
4. Existing ChecksumType.ObjType() logic normalizes reachable states to FULL_OBJECT or COMPOSITE.
5. Indexing a nil or missing map entry returns the empty string.
6. XML omitempty removes the element for that empty string.

The resulting behavior is deterministic:

The change is therefore a missing projection from established state to the wire response. It does not create new checksum state and cannot make an incorrect checksum correct. It makes the response describe the state the server has already validated and committed.

Review and verification

The PR was reviewed after the contributor branch was refreshed onto current main. The update produced head c4b9d38d; the resulting tree hash, 39ec44c6b390c441413e490370f70fbacc4e6a91, exactly matched the isolated local no-commit merge. The result was clean and included the intervening checksum work on main.

Local verification on that exact merge result included:

targeted ChecksumType regression test
CGO_ENABLED=0 go test ./cmd/ -count=1 -timeout 30m
go vet ./cmd/
gofmt and git diff --check
rebrand compatibility guard
local DCO rule

The targeted regression completed in 2.174 seconds and the full cmd package test completed in 168.956 seconds. The commit author email matches its Signed-off-by trailer. Cryptographic Git commit signing is independent of DCO and is not required by this repository.

A separate read-only local Claude Code adversarial review inspected the merged diff, checksum serialization, XML path, current main, tests, DCO, and compatibility guard. Its verdict was COMMENT: the production change was correct and safe, but it preferred an additional HTTP-level completion test before merge. The maintainer agreed that such a test would improve fidelity, but disagreed that it was blocking: the handler delegates directly to the tested generator, while existing real MPU tests already cover persisted FULL_OBJECT and COMPOSITE states. The formal GitHub review therefore recorded APPROVED with the HTTP-level test as a follow-up.

Actions, branch refresh, and merge

The first four action_required runs had been created on 2026-08-09 against the PR’s old base. After approval, DCO passed but the old VulnCheck run used Go 1.26.5 and failed on newly published standard-library vulnerabilities fixed in Go 1.26.6. Current main had already moved to Go 1.27.0, and its latest VulnCheck was green. Treating the stale failure as either a product regression or an ignorable red check would both have been wrong.

The decision was to refresh the test context, not rerun or waive the stale result:

1. GitHub’s update-branch API merged current main (8d76a255c) into contributor head d014a12cf, producing c4b9d38d without conflicts.
2. GitHub created four new fork workflow runs for the refreshed head; all four were explicitly approved again.
3. All nine reported checks passed: DCO, VulnCheck, six jobs in Go CI, and the Test Release Pipeline. The release validation job completed in 11 minutes 26 seconds.
4. A formal approving review was submitted against c4b9d38d.
5. Merge used an expected-head guard and the repository’s normal merge strategy, producing a96116b1. This preserved the contributor’s signed-off commit rather than rewriting it through a squash. The PR’s Resolves #47 relationship closed the issue one second later.
6. The post-merge main VulnCheck and all six Go CI jobs also passed; cross-compilation, the slowest job, completed in 9 minutes 54 seconds.

This sequence matters because “the patch passed once” was not the acceptance criterion. The exact tree merged into current main had to be the tree reviewed and tested, and a stale CI environment could not substitute for that proof.

Evaluation of the PR

What is strong

• The scope matches the defect. Two production lines restore one missing response element.
• It reuses authoritative state. There is no duplicate type derivation and no new checksum algorithm branch.
• Backward compatibility is explicit. omitempty preserves checksum-free responses.
• The test covers both valid values and absence. A regression cannot silently restore the null result.
• The compatibility baseline is updated deliberately. CI is not weakened.
• DCO provenance is complete. The sole commit has a matching sign-off.

Non-blocking review notes

The test is correct for the changed generator but its fixtures are not byte-for-byte models of every production multipart metadata flag:

• the FULL_OBJECT fixture reaches the right value through a non-multipart checksum state rather than a completed multipart state carrying ChecksumMultipart, ChecksumIncludesMultipart, and ChecksumFullObject;
• the COMPOSITE fixture carries the multipart flag but omits the persisted per-part checksum block.

Existing API-level tests already exercise genuine FULL_OBJECT and COMPOSITE completion and verify their committed types. PR #57 tests the remaining projection from decoded state to the response field and XML. Adding an assertion to those full API tests would improve test fidelity, but it is not required for this two-line repair.

The PR places ChecksumType before the algorithm-specific fields, while AWS’s example response and SILO’s newer CopyObjectResponse place it after them. Mainstream S3 SDKs parse XML by element name, so this is a parity and style detail rather than a compatibility blocker. Moving the field is optional.

Finally, the contributor commit title says feat: even though the PR correctly marks itself as a bug fix. The final merge preserved that signed-off commit instead of rewriting it. This is a history/style imperfection, not a protocol or release blocker.

Why new algorithms do not belong in this PR

AWS now documents additional fields such as SHA512, MD5, and XXHASH variants. Adding those XML fields alone would create false compatibility.

SILO’s current checksum implementation supports CRC32, CRC32C, CRC64NVME, SHA1, and SHA256. A real new algorithm requires coordinated support across:

• request header parsing and validation;
• streaming checksum calculation;
• multipart FULL_OBJECT or COMPOSITE semantics;
• on-disk checksum encoding and decoding;
• UploadPart, UploadPartCopy, completion, copy, replication, HEAD, GET, ListParts, and GetObjectAttributes;
• SDK/client interoperability and a full encrypted/compressed/versioned test matrix.

PR #57 should not grow response-only placeholders for algorithms the server cannot calculate or persist. Each new algorithm family needs a separate compatibility decision, implementation, and review.

Compatibility and operational impact

• S3 clients: checksum-aware clients receive ChecksumType from future successful multipart completions instead of null.
• Wire format: one additive XML element appears only when an additional checksum exists. Clients that ignore unknown elements remain unaffected.
• Integrity: no checksum is recalculated or accepted differently. Existing validation semantics are unchanged.
• Stored data: no object, part, metadata, or erasure format changes. No migration or backfill.
• Existing objects: object state remains correct. A past completion response cannot be replayed; use HEAD or GetObjectAttributes to inspect an existing object’s type.
• Encryption: the response uses the established checksum metadata-decryption path. No key material or new secret is exposed.
• Performance: one map lookup and one optional XML element; no extra object read, hashing pass, or allocation proportional to object size.
• Rolling upgrade: old nodes omit the element and new nodes return it. Requests and stored objects remain compatible, but client-visible behavior stabilizes only after all serving nodes are upgraded.
• Rollback: rolling back removes the response element from future completions; it does not damage objects created while the fix was present.
• Other repositories: no server dependency, silo-pkg, MCLI, or Console change is required. Public documentation belongs in this site.

This is an additive compatibility repair, not a release feature that requires operators to rewrite data. Its only externally visible effect is a more complete success response.

Merge and release decision

The final decision had six parts:

1. accept the narrow projection fix without recalculating checksums or changing storage;
2. keep SHA512, MD5, and XXHASH families out of #57 until they have end-to-end server support;
3. record an HTTP-level completion test as useful follow-up work, not a blocker for the directly tested generator repair;
4. reject stale CI as merge evidence, update the branch to current main, and approve the newly created workflows;
5. merge only after the refreshed head was formally approved and every check was green, using an expected-head guard and a normal merge that preserved the DCO-signed contribution;
6. let Resolves #47 close the issue, then verify the resulting main workflows independently.

No dependency update, storage migration, or cross-repository implementation was required. That decision is now complete at the repository-integration gate.

A green main still does not prove that a SILO tag, release package, container image, deployment, or production endpoint contains the repair. Those delivery gates remain unverified and must be recorded separately when the next release ships.

Conclusion

PR #57 is a good example of a small compatibility fix whose correctness comes from respecting an existing source of truth. The checksum type was already calculated, validated, persisted, decryptable, and visible through other APIs. The completion response simply failed to project it into XML.

The accepted repair does exactly that projection and nothing more. It makes the wire response honest without touching user data, checksum mathematics, storage layout, or algorithm scope. The fork workflows, refreshed-head review, merge, automatic issue closure, and post-merge main verification are complete. What remains is delivery discipline: distinguish this merged fix from a tagged, packaged, imaged, deployed, and production-verified release.

9 - When the Total Is Unknown: Folder Download Progress

Status: Implemented and verified locally; commit, Console release, and Silo dependency update pending · Priority: P1 · Owner: pgsty/silo-console · Related issue: pgsty/silo#62 · PRD review: Claude Fable 5 (xhigh) — APPROVE · Implementation review: Claude Fable 5 (xhigh), 2026-08-23 — APPROVE, no P0/P1/P2 findings

SILO Console shows NaN% in Downloads / Uploads while downloading a folder. The ZIP normally keeps streaming and the stored objects are intact, but the progress bar has crossed from “unknown” into an invalid determinate state. Users see a full-looking bar, assume the transfer failed or finished, and retry it.

The proposed repair is intentionally narrow:

A download may enter determinate mode only when it has a finite, positive total measured in bytes applicable to that response. Without such a total, it remains indeterminate until completion, failure, or cancellation.

The server keeps streaming ZIPs. Ordinary files keep their percentages. The frontend gains one safe calculation boundary, reuses its existing indeterminate renderer, and closes one missing cancellation transition. This record defines why that is both sufficient and the smallest truthful fix.

The observed failure

The defect is present in the current silo-console v2.1.1, which is embedded by Silo RELEASE.2026-08-06T00-00-00Z.

Reproduction:

1. Put several objects below a prefix such as folder/.
2. Stay in the parent listing, select folder/, and click Download.
3. Open Downloads / Uploads before the transfer finishes.
4. The row displays NaN%; the ZIP request continues.

The runtime check used a prefix containing about 88.7 MiB and throttled Chromium to preserve the observation window. Two independent downloads produced the same NaN% state.

This is a frontend correctness bug. It is not evidence of corrupted objects, an altered disk format, or a failed S3 GET.

What is actually happening

The visible NaN% is the end of a contract mismatch across three layers.

A prefix has no object size

S3 folders are common prefixes, not stored directory objects. In the listing model, a prefix ends in / and carries size=0. The Console already renders that size as -, correctly treating it as not applicable.

The generated API model marks size as omitempty, so logical zeroes are absent from listing JSON. The single-selection thunk nevertheless passes object.size straight into the download helper: a prefix or zero-byte object therefore supplies undefined at runtime (while synthetic prefix records may supply 0). Neither value is a valid denominator.

A streamed ZIP has no known wire length

The server recognizes the trailing /, recursively lists the objects, then connects a zip.Writer to an io.Pipe. Objects are read, deflated, and copied to the HTTP response as the archive is produced.

That behavior is desirable: the server can send the first bytes without holding the complete archive in memory or on disk. Its consequence is equally deliberate: the final compressed byte length does not exist when headers are sent, so the response has Content-Type: application/zip and a filename, but no Content-Length.

The sum of source object sizes is not a substitute. Source sizes are uncompressed bytes; ProgressEvent.loaded counts response bytes after ZIP compression and framing. They are different units.

A progress event does not imply a computable percentage

The client currently computes every event as:

Math.round((event.loaded / fileSize) * 100)

For a prefix, the denominator is zero or absent. Depending on the value and event, JavaScript produces NaN (loaded / undefined or 0 / 0) or Infinity (positive bytes divided by zero).

The progress callback then writes that non-finite value into Redux and sets waitingForFile=false. That second operation is the decisive state error: the task leaves the existing indeterminate branch merely because an event arrived, not because the event contained a usable total. The determinate progress component receives the invalid value and renders an invalid label.

The complete chain is:

common prefix: size = 0
        |
        v
download(..., fileSize = 0)
        |
        v
streamed deflated ZIP, no Content-Length
        |
        v
event.loaded / 0 => NaN or Infinity
        |
        v
invalid percentage enters Redux; waitingForFile becomes false
        |
        v
determinate ProgressBar renders NaN%

Ordinary non-empty files avoid the defect because the server can stat the object, sets Content-Length, and the list size is positive. If the browser emits a progress event for an empty response, a zero-byte file reaches the same arithmetic boundary as a prefix even though it is a real object; it therefore belongs in the regression contract.

Product contract

The UI needs one honest distinction:

• Determinate means both transferred bytes and total bytes are known in the same unit.
• Indeterminate means the request is active but the total is unknown.

This yields four load-bearing invariants:

determinate  => total is finite and total > 0
determinate  => percentage is finite and 0 <= percentage <= 100
unknown total => indeterminate
terminal state => not indeterminate

These invariants are more general than objectPath.endsWith("/"): they cover prefixes, zero-byte files, malformed metadata, and any future unknown-length response without inventing object-type exceptions.

Goals and non-goals

Goals

1. A folder download never displays NaN%, Infinity%, or a fabricated percentage.
2. Unknown-length transfers use the existing indeterminate animation.
3. Known-length ordinary files retain their current percentage behavior.
4. Completion, failure, and cancellation always leave indeterminate mode.
5. A zero-byte file never produces a non-finite percentage and still reaches success.
6. No non-finite or out-of-range download percentage enters Redux.
7. The fix can ship in Console first and then be consumed by Silo as a dependency update.

Non-goals

• Do not pre-generate or buffer a complete ZIP on the server.
• Do not use the sum of uncompressed object sizes as network progress.
• Do not redesign the entire Object Manager state model.
• Do not route folders through the current immediately-completing BrowserDownload path.
• Do not solve the browser memory cost of XMLHttpRequest.responseType="blob" here.
• Do not change whether a cancelled row remains visible until the user clears it.
• Do not redesign mid-stream ZIP error signaling after HTTP headers have been sent.
• Do not modify the S3 API, Console API, object layout, or archive contents.

Those are legitimate follow-ups, but coupling them to this defect would enlarge risk without being necessary to restore truthful progress.

The decision

The minimum production repair has four parts.

D1. Calculate only from a valid total

Add a small pure function, separate from DOM and Redux side effects:

type DownloadProgressEvent = Pick<
  ProgressEvent,
  "loaded" | "lengthComputable" | "total"
>;

export const calculateDownloadPercent = (
  event: DownloadProgressEvent,
  objectSize: number,
): number | null => {
  let total: number | null = null;

  if (Number.isFinite(objectSize) && objectSize > 0) {
    total = objectSize;
  } else if (
    event.lengthComputable &&
    Number.isFinite(event.total) &&
    event.total > 0
  ) {
    total = event.total;
  }

  if (
    total === null ||
    !Number.isFinite(event.loaded) ||
    event.loaded < 0
  ) {
    return null;
  }

  return Math.min(
    100,
    Math.max(0, Math.round((event.loaded / total) * 100)),
  );
};

The source priority preserves compatibility:

1. A finite positive objectSize retains the current ordinary-file calculation.
2. If object size is unavailable but the browser declares the response length computable and supplies a finite positive event.total, use it.
3. Otherwise return null: no truthful percentage exists yet.

The helper’s output contract is complete: either null, or a finite number in [0,100].

D2. Keep unknown totals indeterminate

Change the XHR handler to dispatch only a real percentage:

req.addEventListener("progress", (event) => {
  const percent = calculateDownloadPercent(event, fileSize);

  if (percent !== null) {
    progressCallback(percent);
  }

  // No valid total: preserve waitingForFile=true so the existing UI remains
  // indeterminate instead of manufacturing a determinate value.
});

Download rows already start with waitingForFile=true, and ObjectHandled already renders that state with variant="indeterminate". There is no need to widen Redux to number | null, add another boolean, or change MDS.

When the first valid percentage arrives, the existing updateProgress action stores it and sets waitingForFile=false. When no valid percentage ever arrives, the row remains indeterminate until a terminal action.

D3. Make cancellation terminal

Completion and failure already clear waitingForFile. Cancellation does not. Add the missing transition in cancelObjectInList:

item.waitingForFile = false;

Without that line, the repaired prefix download would remain in the indeterminate rendering branch after abort, masking the Cancelled state. The row continues to follow the current product behavior: it remains as a cancelled record and can be removed manually. Automatic removal is not part of this change.

There is one event-order guard at the XHR boundary as well. abort() first produces readystatechange(DONE, status=0) and only then the abort event; without a status-zero return, the generic DONE branch marks the request failed before onabort can mark it cancelled. DONE/status zero is therefore left to the dedicated onerror or onabort handler, and onabort removes the stored request reference.

D4. Normalize an omitted zero-byte size

The single-selection thunk passes object.size || 0, matching the other download entry point. This restores the API model’s omitted logical zero before the helper checks Blob.size === fileSize, so an HTTP 200 zero-byte object completes at 100% instead of being reported as incomplete.

D5. Keep the server stream unchanged

The folder handler continues to generate a deflated ZIP through io.Pipe and omit Content-Length. No API, archive, storage, or resource-management contract changes.

State machine

The state does not move back from determinate to indeterminate. If a later event lacks a valid total after a valid percentage was observed, the handler simply retains the last valid value.

Failed and Cancelled both set done=true in the existing reducers. ObjectHandled uses done to change its close button from “abort request” to “remove record”; this repair preserves that behavior. The cancelled Redux value remains 0, while the existing ProgressBarWrapper renders a full orange terminal bar with a Cancelled label because ready=true. That established presentation is not part of this repair.

waitingForFile is not the ideal long-term name for “no computable progress.” Renaming it or replacing the booleans with a discriminated union would improve the model, but that is a separate refactor. In this repair, the field already expresses and renders the required state, so reusing it minimizes compatibility risk.

Why this is sufficient

The repair closes the bug by cases.

Ordinary non-empty file

objectSize > 0, so the helper uses the same denominator as today. The result is finite and clamped, updateProgress enters determinate mode, and completion still sets 100%.

Current streamed folder

objectSize is normalized to 0, while lengthComputable=false and event.total=0. The helper returns null; no invalid action is dispatched, so the row remains indeterminate. Completion sets waitingForFile=false, percentage=100, and done=true.

Future response with a real length

If a proxy or later server implementation provides a trustworthy response total, lengthComputable=true and event.total>0. The same code automatically produces a real percentage without another product change.

Zero-byte file

The omitted listing size is normalized to zero, and both totals are then zero, so an intermediate percentage is mathematically undefined. The row stays indeterminate for its usually brief lifetime; the zero-byte Blob now equals the normalized expected size, and the successful response transitions directly to 100%. 0/0 is never evaluated.

Failure and cancellation

Failure already exits indeterminate. The added cancellation transition does the same on abort. No terminal row can continue to look active merely because its total was unknown.

Mathematically, division occurs only when total belongs to (0, +infinity). The result is then clamped to [0,100]. Therefore neither NaN nor Infinity can cross the calculation boundary into Redux or the determinate renderer.

Rejected alternatives

Buffer the ZIP to obtain Content-Length

The server could generate the complete archive in memory or a temporary file, measure it, and then send it. That would provide an exact wire total, but at the cost of memory or disk pressure, delayed first byte, cleanup complexity, and worse concurrent-download behavior. An observability defect does not justify discarding streaming.

Sum the objects under the prefix

That sum is uncompressed logical data. event.loaded measures compressed response bytes plus ZIP framing. The units differ, so the bar could stop below 100%, exceed 100%, or move according to compression ratio rather than transfer completion. Reject.

Convert invalid progress to 0%

This hides the string but lies about the state: determinate 0% means the total is known and no portion has transferred. Users would still interpret the transfer as stalled. Unknown must remain unknown.

Special-case paths ending in /

That fixes the reported prefix but misses a real zero-byte object, invalid metadata, and other unknown-length responses. The correct boundary is denominator capability, not object type.

Send folders through BrowserDownload

The current large-file path creates an anchor and immediately calls the completion callback after clicking it. It cannot report true completion, console-managed cancellation, or a subsequent HTTP failure. It may be the basis of a later streaming-download design, but today it would replace one lie with another.

Sanitize inside ProgressBar

A generic component guard could be useful defense in depth, but it would leave invalid data in Redux and hide the broken state transition from every other consumer. The primary repair belongs where progress becomes application state.

Introduce percentage: number | null now

A discriminated progress state would be cleaner than the current booleans if the Object Manager were being redesigned. Adding null while retaining waitingForFile, done, failed, and cancelled would instead create more contradictory combinations. Removing the old fields is larger than this bug requires. Reuse the already-rendered indeterminate state now; redesign it separately.

Requirements and acceptance

Functional requirements

• FR1: An unknown total keeps the task indeterminate.
• FR2: A finite positive object size preserves ordinary-file percentages.
• FR3: A finite positive event.total is a fallback only when lengthComputable=true.
• FR4: Every dispatched percentage is finite and within [0,100].
• FR5: A zero-byte file never displays non-finite progress and reaches success.
• FR6: Completion, failure, and cancellation leave indeterminate mode.
• FR7: Versioned objects, anonymous downloads, previews, and long-filename entry points retain their existing call contract.

Non-functional requirements

• No new server CPU, memory, disk-buffer, or request cost.
• No new frontend dependency or build step.
• No change to the S3 API, Console API, ZIP content, or stored objects.
• The calculation must be testable without a DOM or live store.
• TypeScript typecheck and the production frontend build must pass.

Acceptance criteria

1. While a folder ZIP without Content-Length is active, its row shows an indeterminate animation and no percentage text.
2. On successful completion, the row reports success/100% and the ZIP can be opened.
3. A normal non-empty file continues to show finite determinate progress and completes at 100%.
4. A zero-byte file never shows NaN% or Infinity% and completes successfully.
5. Cancelling an unknown-total download aborts the request and shows Cancelled, not an active animation.
6. No download path can place a non-finite or out-of-range percentage in Redux.

Test plan

Pure calculation matrix

Use the existing @playwright/test runner for the pure module rather than adding a test framework. This needs one config-only addition in web-app/playwright.config.ts: a dependency-free unit project, for example with testMatch: /.*\.unit\.ts/. The existing chromium project depends on the auth setup against a live Console at localhost:9090; pure calculation and reducer tests must not be gated by that environment. No new dependency is introduced.

State tests

Cover the transition contract directly:

1. A new download starts with waitingForFile=true.
2. No valid progress action means it remains indeterminate.
3. Valid progress produces a finite value and waitingForFile=false.
4. Complete produces done=true, waitingForFile=false, percentage=100.
5. Failure produces failed=true, done=true, waitingForFile=false.
6. Cancel produces cancelled=true, done=true, waitingForFile=false, percentage=0.

Browser regression

Use the real Console test instance and Chromium:

1. Create a temporary bucket with several objects below folder/.
2. Select the prefix from its parent and start the download.
3. Apply CDP download throttling so the intermediate state is observable. Throttled runs must raise the default 30-second test timeout with test.setTimeout.
4. Open Downloads / Uploads and verify that the row exists, has no percentage label, and contains neither NaN% nor Infinity%.
5. Cancel it and verify the Cancelled terminal state.
6. Restore network conditions in finally.
7. Download again without throttling, wait for the browser download, and verify the ZIP.
8. Repeat the relevant assertions for one ordinary non-empty file and one zero-byte file.
9. Remove the bucket, objects, downloads, and temporary files in teardown.

The current Playwright project is Chromium-only, so CDP is an acceptable test mechanism. If Firefox or WebKit projects are later enabled, keep the pure and state tests cross-browser and gate only the throttled observation behind the Chromium project.

Implementation boundary

Expected Console changes:

1. Add downloadProgress.ts containing the pure calculation.
2. Change Objects/utils.ts to dispatch only a non-null percentage, let status-zero terminal events reach their dedicated handlers, and clean up an aborted request.
3. Normalize omitted zero sizes in the single-selection thunk.
4. Change cancelObjectInList to clear waitingForFile.
5. Add calculation, state, and browser regression coverage using existing dependencies, with a dependency-free unit project in playwright.config.ts.

Expected unchanged code and contracts:

• The Go folder-download handler and its streaming ZIP.
• ObjectHandled, ProgressBarWrapper, and MDS.
• IFileItem.percentage: number and the existing thunk callback types.
• S3 and Console API routes.
• Stored object and archive formats.

Delivery and rollback

The fix belongs in pgsty/silo-console, not the Silo server repository where the issue was reported.

Delivery order:

1. Transfer or cross-reference issue #62 to pgsty/silo-console.
2. Implement the bounded Console change.
3. Pass typecheck, production build, pure/state tests, and real browser regression.
4. Publish a new Console release.
5. Update Silo’s pinned Console pseudo-version or release dependency.
6. Build a Silo candidate and repeat folder, ordinary-file, zero-byte, cancel, and ZIP-integrity checks.
7. Publish Silo and record both affected and fixed versions on the issue.

There is no data migration. If the frontend change regresses, Silo can roll back only the Console dependency; server data and API behavior remain compatible.

Definition of done

• The calculation returns only null or a finite [0,100] number.
• Active unknown-total folder downloads render indeterminate.
• Ordinary files retain determinate progress.
• Zero-byte files never render invalid progress.
• Complete, failed, and cancelled rows all leave indeterminate mode.
• The streamed ZIP and server response contract remain unchanged.
• Typecheck, production build, and automated regressions pass locally.
• A Console release is published.
• Silo updates the Console dependency and passes candidate verification.

Follow-up work

Four adjacent improvements deserve separate design records:

1. Stream large folder downloads directly to the browser or filesystem instead of holding the full Blob in memory.
2. Replace the Object Manager’s boolean combination with a discriminated progress/terminal state.
3. Improve end-to-end integrity and error signaling for ZIP failures after headers have been sent.
4. Add a generic non-finite-value guard to shared progress components as defense in depth.
5. Repair the pre-existing Blob JSON error decoder and request-trace cleanup on HTTP failure paths.

None is required to stop the current UI from lying. The next maintenance iteration should first restore the smallest honest contract: known totals get percentages; unknown totals remain unknown.

10 - A ListObjects Shortcut Must Not Turn a Missing Bucket into an Empty One

This document records the problem analysis, design discussion, and repair decision for SILO #32 and PR #37.

Status on 2026-08-26: PR #37 was updated to the DCO-signed head e9c5340be, formally approved, and merged as 49c8aeac4; #32 closed automatically. DCO, VulnCheck, and all six Go CI jobs passed on the exact PR head; the post-merge main VulnCheck and all six Go CI jobs also passed. No tagged release, package, container image, deployment, or production endpoint has yet been verified to contain the repair.
Scope: verify bucket existence only for three listing shortcuts that bypass storage; do not restore the generic checkBucketExist, change the normal listing path, or introduce an existence cache.
Release boundary: local commit, push, remote CI, merge, tag, package, container image, deployment, and production verification are independent gates.

Too Long; Didn’t Read (TL;DR)

The problem is real and worth fixing. A normal ListObjects, ListObjectsV2, or ListObjectVersions request against a missing bucket reaches storage and receives BucketNotFound. Three inputs, however, return early:

• a marker outside the prefix;
• max-keys=0;
• a prefix beginning with /, including the Prefix="/" boto3 reproduction from #32.

Those branches return io.EOF directly. The caller treats EOF as a successful end of listing, so the client receives an empty 200 rather than S3’s 404 NoSuchBucket. The identity of the same missing resource changes from an error to success solely because the selection parameters differ. That breaks S3 compatibility and blocks a real user’s upgrade from the pre-regression release.

The repair must not put an expensive bucket check back in every listing. The selected design replaces only the three bare io.EOF returns with a small helper. The helper calls GetBucketInfo once: it returns the real error if the bucket is absent or cannot be confirmed, and preserves io.EOF when the bucket exists. The normal listing hot path is untouched. Only requests that would otherwise exit before storage pay the extra peer-and-disk fan-out.

That decision has now been executed: the strengthened repair passed local review, the exact PR head passed every remote check, and the expected-head-guarded merge entered a green main.

What is the problem?

One API exposes two bucket-existence semantics

#32 reproduces the defect by calling the following against a missing bucket:

s3.list_objects(Bucket="missing-bucket", Prefix="/")

AWS S3 raises NoSuchBucket; SILO returns a successful empty listing. The difference is not in authentication, routing, or XML serialization. It comes from the object-layer listPath control flow:

regular prefix
  -> enter listMerged
  -> consult storage
  -> missing volume/bucket becomes BucketNotFound
  -> HTTP 404 NoSuchBucket

shortcut input
  -> listPath returns io.EOF early
  -> storage is never consulted
  -> the caller treats EOF as normal completion
  -> HTTP 200 with an empty listing

/ is not the only trigger:

For an existing bucket, returning an empty listing from these branches is a reasonable optimization. For a missing bucket, the same EOF masks the resource error that should take precedence.

The regression has a known origin

The reporter confirmed correct behavior in RELEASE.2024-01-29T03-56-32Z and the regression beginning with RELEASE.2024-01-31T20-20-33Z. The corresponding upstream change is minio/minio#18917 / 80ca12008. It removed GetBucketInfo from generic argument checks and relied on actual Put, List, and Multipart storage operations to expose a missing bucket.

That optimization works on normal paths but leaves a gap: an early-return path never reaches the storage operation that is now responsible for producing the error. #32 does not require a broad rollback of the upstream optimization. It repairs the overlooked control-flow exits.

Why fix it?

The S3 contract explicitly requires NoSuchBucket

Both AWS ListObjects and ListObjectsV2 define NoSuchBucket as HTTP 404 when the specified bucket does not exist. prefix, marker, start-after, and max-keys select listing results; they must not turn a missing bucket identity into a successful request.

ListObjectVersions shares the same object-layer listing engine. Giving V1, V2, and version listings the same existence behavior on the same shortcut inputs prevents the three public APIs from diverging further.

An empty 200 changes client decisions

An empty 200 and a 404 are not interchangeable presentation details:

• 404 tells provisioning or test code to create the bucket, fix configuration, or stop;
• an empty 200 asserts that the bucket exists but has no matching objects;
• SDKs, synchronization tools, and integration tests continue down different branches;
• a test using SILO as an S3 substitute can pass locally and fail against AWS.

#32 also establishes a direct upgrade impact: an application relying on the older correct behavior cannot upgrade past the regression. The repair restores both S3 parity and upgrade compatibility.

The repair surface is narrow and testable

The bug is confined to three adjacent early returns. It does not involve object data, metadata formats, sorting, pagination-token encoding, permissions, or wire schemas. A very small production change can be pinned down with object-layer and HTTP-level contracts, so the benefit clearly exceeds the implementation risk.

Why not restore the global check?

Upstream did not remove generic GetBucketInfo as incidental cleanup. The motivation for #18917 states that checking the bucket before every Put, List, and Multipart operation fans out across servers; even after vectorization, the cost becomes visible beyond 100 nodes.

In current SILO, erasureServerPools.GetBucketInfo calls S3PeerSys.GetBucketInfo. That operation concurrently asks every peer and reduces quorum per pool, while each peer checks its local bucket state. It is not a cheap in-memory map lookup.

Two extremes are therefore unacceptable:

• never check: keep the incorrect empty 200;
• check before every List: restore semantics while undoing a critical large-cluster optimization.

The actual design question is whether the check can be confined to branches that never touch storage and therefore cannot discover the missing bucket naturally. It can.

How is it fixed?

Replace only three bare EOF returns

In cmd/metacache-server-pool.go, each shortcut previously executed:

return entries, io.EOF

It now executes:

return entries, z.listPathShortcutEOF(ctx, o.Bucket)

The helper has only two classes of outcome:

func (z *erasureServerPools) listPathShortcutEOF(ctx context.Context, bucket string) error {
    if _, err := z.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
        return err
    }
    return io.EOF
}

• existing bucket: preserve the previous empty-list behavior;
• missing bucket: pass BucketNotFound into the existing error mapping, producing HTTP 404 NoSuchBucket;
• state cannot be confirmed: propagate quorum, offline, timeout, or context errors instead of fabricating success.

The normal listMerged, metacache scan, sorting, pagination, and response-generation paths do not change.

Why the helper belongs here

The check must sit next to the shortcut for three reasons:

1. only this layer knows that it is about to bypass every storage access;
2. moving it into generic argument validation charges every call;
3. moving it into the scan layer cannot help because these branches never scan.

The name intentionally states the boundary. This is not a new generic checkBucketExist; it restores missing existence semantics immediately before a shortcut returns EOF.

Do not add a cache

A bucket-existence cache could reduce fan-out but immediately creates invalidation questions for create, delete, site replication, recovery, and expiry. Adding a second source of truth for three low-frequency shortcuts costs more complexity and consistency risk than it saves.

The selected implementation uses the existing GetBucketInfo source of truth. If future telemetry shows that large clusters receive frequent max-keys=0, slash-prefix, or disjoint-marker probes, the project can evaluate a dedicated metadata fast path, rate limiting, or a carefully invalidated cache using real data rather than speculative machinery in this compatibility patch.

Test and review evidence

Object-layer contract

The object-layer test runs against single-drive and multi-drive erasure setups and exercises four inputs:

• slash-prefixed prefix;
• zero limit;
• marker outside prefix;
• a regular prefix as a control that still receives the error naturally from storage.

Each case covers ListObjects, ListObjectsV2, and ListObjectVersions, using the typed isErrBucketNotFound predicate rather than brittle English error-string comparison.

HTTP contract

The handler test sends genuine signed requests for all three public APIs:

The HTTP test uses the real slash-prefix reproduction from #32. The other two shortcuts are enumerated at the object layer. This proves final wire behavior without repeating the full matrix in the slower handler fixture.

Local quality gates

The improved local commit passed:

go test ./cmd -count=1
the new object-layer and HTTP regressions (10 subcases)
focused go test -race
related existing listing tests
CGO_ENABLED=0 go build ./...
go vet ./...
CI-scope gofmt and git diff --check
post-commit focused regression rerun

The full local cmd test completed in 116.215 seconds. An independent local Claude Code review used the Fable model at Max effort to inspect the exact tree, call paths, error mapping, tests, performance boundary, and this decision. Its verdict was GO, with no mandatory pre-merge change.

The DCO-signed PR head e9c5340be then passed eight remote checks: DCO, VulnCheck, and six jobs in Go CI. After merge, the resulting main commit 49c8aeac4 independently passed VulnCheck and all six Go CI jobs. The slowest checks were PR cross-compile at 9 minutes 47 seconds and post-merge cross-compile at 9 minutes 30 seconds.

Can it introduce new problems?

Shortcut requests now fan out across the cluster

This is the most important and deliberately accepted cost. A shortcut on an existing bucket used to be little more than a local branch; it now calls GetBucketInfo. Directional local microbenchmarks observed:

These numbers show local relative cost only; they are not a latency prediction for a 100+ node deployment. Real distributed execution adds peer networks, quorum, and slowest-node tail latency, potentially making the gap much larger. That is precisely why the check must not expand into the normal listing path.

The risk concentrates in malformed or probe-style traffic. A misconfigured client polling max-keys=0, a slash prefix, or disjoint markers at high frequency can amplify what was a cheap request into peer-and-disk work. After merge, the actual frequency of these inputs should be observed through S3 traces or metrics; rate limiting or optimization should follow evidence.

A degraded cluster exposes more real errors

Previously, a shortcut could return an empty 200 while peers were offline or bucket quorum was unavailable because it never consulted cluster state. The repair can return quorum, timeout, or service errors in those conditions.

That is more honest behavior, not an availability regression: if the server cannot establish that the bucket exists, it must not assert a valid empty bucket. Clients depending on unconditional empty success will nevertheless observe a behavior change.

Bucket create/delete races are not linearizable

GetBucketInfo and returning the empty result are two actions. The bucket can be deleted immediately after the check, or created immediately after a missing-bucket result is formed. This patch does not and should not add a transaction spanning bucket lifecycle to a listing shortcut.

This is the same concurrency class as other APIs that validate a resource before acting. The repair guarantees that the request no longer succeeds with no existence evidence at all; it does not promise a cross-node, cross-lifecycle linearizable snapshot of an empty listing.

Clients relying on the bug will receive 404

Some clients may have adopted the missing bucket’s empty 200 as fact. They will now enter an error branch. This is a visible compatibility change, but it restores the documented S3 contract and the pre-regression behavior. Preserving the bug merely transfers upgrade cost to clients that correctly rely on 404.

Two adjacent edges remain out of scope

The adversarial review recorded two non-blocking P3 boundaries:

1. When resuming a metacache continuation, the c.fileNotFound branch still returns bare io.EOF. A stale or crafted continuation token used after bucket deletion could theoretically receive an empty 200. Adding GetBucketInfo there would affect normal continuation traffic and needs a separate performance and error-precedence design.
2. Some V1 and version-list marker/prefix combinations return NotImplemented during HTTP handler validation before reaching the object layer; the V2 start-after route can reach it. This patch fixes storage shortcuts masking a missing bucket; it does not redefine precedence between malformed parameters and resource errors.

Neither blocks merge. The first is outside #32’s ordinary initial-list reproduction; the second is inherited handler behavior. Recording them prevents “all three shortcuts are covered” from being overstated as byte-for-byte AWS parity for every possible parameter combination.

Alternatives considered

Keep upstream behavior

This has zero performance change and minimizes fork divergence. It also keeps a documented S3 incompatibility, a regression with a known release boundary, and a misleading result when SILO is used as an integration-test substitute. For a narrow and well-tested compatibility repair, that tradeoff is no longer justified.

Restore generic checkBucketExist

This covers every path at once but reintroduces peer fan-out into every Put, List, and Multipart operation, directly undoing the large-cluster optimization from #18917. The cost is disproportionate and the option is rejected.

Fix only Prefix="/"

That passes the single issue reproduction but leaves the same root defect in max-keys=0 and marker-outside-prefix. The branches are adjacent and share the same semantics, so one helper is simpler and less likely to regress.

Add a bucket-existence cache

This makes shortcuts cheaper but requires semantics for create, delete, replication, recovery, and stale TTL windows. There is no telemetry showing enough shortcut traffic to justify that complexity, so it is not selected.

Complexity and cost-benefit

The overall cost-benefit is favorable. The reason is not that GetBucketInfo is cheap—it is not—but that its cost is strictly limited to three shortcuts that otherwise cannot discover the missing bucket. A narrow performance cost in exchange for explicit protocol correctness is better than either a global rollback or indefinitely preserving the incorrect behavior.

Acceptance decision and remaining gates

The final decision was: accept and merge the strengthened PR #37 revision without expanding the production scope.

The accepted sequence was:

1. replace the old fork head with the current-main, DCO-signed revision while preserving Jason Lin as a co-author;
2. retain typed error predicates, V1/V2/version-list object-layer coverage, and HTTP-level 404 / NoSuchBucket assertions;
3. update the PR description with the shortcut fan-out cost and unchanged normal-path boundary;
4. approve the fork workflows and require all eight reported checks to pass on exact head e9c5340be;
5. submit a formal approving review against that head;
6. merge with an expected-head guard, producing 49c8aeac4, automatically close #32, and require the resulting main Go CI and VulnCheck to pass independently.

No cache, feature flag, additional abstraction, or continuation-token redesign was required. High-frequency shortcut traffic and large-cluster tail latency remain observability follow-ups, not reasons for speculative code expansion.

Repository integration is complete. A tag, package, docker.io/pgsty/minio image, deployment, and real S3-client verification must still complete before the repair can be described as delivered to users.

Conclusion

The issue is not merely “a slash prefix reports the wrong error.” The listing engine uses io.EOF to mean two different things: an empty result from an existing bucket and an early exit that never established whether the bucket exists. Removing generic existence checks for large-cluster performance was a sound upstream optimization, but the shortcuts violate its premise that a real storage operation will naturally surface a missing bucket.

The selected repair restores that premise by calling the existing GetBucketInfo only at three storage-bypassing exits. It makes those requests more expensive and exposes real errors on degraded clusters; both are explicit costs. In return, SILO restores S3’s 404 semantics, upgrade compatibility, and test fidelity while preserving the upstream optimization on the normal listing hot path.

This worthwhile, controlled compatibility fix is now merged and green on main; release delivery remains a separate gate.

11 - Read-Only Checksum Audit and Reliable CLI Output

This is the design and implementation record for MCLI’s read-only checksum verification workflow and pgsty/mc#5, the non-TTY output defect found during release review.

Status: shipped in mcli 20260901. The command merged to main through pull requests #8 and #13, is exercised against a real SILO server in hosted CI, and pgsty/mc#5 is closed. Bundling the client into the Server image remains a separate gate.
Owner: pgsty/mc.
Tracking: pgsty/mc#5.
Safety boundary: verification is read-only; repair is not part of this command.

Too Long; Didn’t Read (TL;DR)

Historical CopyObject implementations could calculate a stored additional checksum over transformed storage bytes instead of the logical bytes returned by S3. mcli checksum verify inventories objects and independently streams the logical body through the recorded algorithm. Each candidate becomes MATCH, MISMATCH, NO_CHECKSUM, WOULD_VERIFY (dry run), one of ten UNKNOWN_* classifications, or one of three SKIPPED_* results.

The first implementation worked in a terminal but printed nothing when stdout was redirected. MCLI automatically marked non-TTY execution as quiet to disable progress UI, and the new command accidentally treated that internal state as a user request to suppress audit records. The repair separates semantic output from progress suppression without changing global quiet behavior or enabling progress bars in CI.

Command and scope

mcli checksum verify ALIAS/BUCKET/OBJECT
mcli checksum verify --recursive ALIAS/BUCKET[/PREFIX]
mcli checksum verify --manifest candidates.jsonl ALIAS

Version one supports CRC32, CRC32C, CRC64NVME, SHA1, and SHA256 checksums marked as FULL_OBJECT. It can select one object, an exact VersionID, current objects under a prefix, all versions, or exact entries from a JSON Lines manifest. It also supports SSE-C key mappings, time and size filters, dry-run estimation, bounded workers, download limits, JSON output, and an optional JSON Lines report.

It does not verify COMPOSITE checksums, infer type from an ETag, inspect xl.meta, identify the historical writer with certainty, or repair metadata. The endpoint must report the checksum type (x-amz-checksum-type) alongside the checksum; on one that does not, every checksummed object is classified UNKNOWN_CHECKSUM_TYPE rather than guessed at.

Read-only data path

For every selected object, MCLI:

1. sends HEAD with checksum mode enabled and retains every supported checksum plus ChecksumType;
2. rejects unsupported or ambiguous states as UNKNOWN_* instead of guessing;
3. streams GET logical bytes through bounded hashers without writing the body to disk;
4. uses VersionID pinning, or If-Match plus a second HEAD for mutable unversioned/null objects;
5. compares independently calculated values with the stored values.

The S3 boundary allows LIST, HEAD, and GET only. Tests fail if a write method reaches the mock endpoint.

Result and exit contract

Every candidate produces one stable result:

The summary carries objects, a verified count, the count of every result status, and incomplete. verified is MATCH plus MISMATCH: the only results that actually streamed a body through a hasher. A run that enumerated many objects and verified none is visible as such.

--fail-on accepts mismatch, unknown, no-checksum, any, or none. The default any returns exit 1 for mismatches and incomplete verification. no-checksum returns exit 1 when any object carries no checksum or when nothing was verified at all, so an empty prefix or a stale manifest cannot pass as a clean audit. Dry-run does not apply --fail-on. Argument, authentication, enumeration, and report-write failures remain command failures rather than object classifications.

In particular, SKIPPED_TOO_LARGE makes the default any return exit 1 because the size cap leaves the audit incomplete. Time-filter and delete-marker skips do not fail by themselves.

Output and automation contract

Object records and the final summary are semantic output:

• Unless the caller explicitly sets --quiet, -q, or MC_QUIET=true, stdout receives every object record and the final summary in both TTY and non-TTY execution.
• Non-TTY --json emits exactly one compact JSON value per line. TTY JSON keeps MCLI’s existing pretty presentation.
• Global flags work at the app, checksum, and verify levels.
• --report is independent of stdout. It still writes object records and the final summary as JSON Lines when explicit quiet suppresses stdout.
• Output transport does not change --fail-on decisions.

The distinction matters because MCLI’s historical globalQuiet has two inputs: an explicit quiet flag and an automatic non-TTY state used to disable progress UI. Changing that global would risk re-enabling progress output across copy, get, put, mirror, and other commands.

The selected repair is command-local. It walks the full CLI context chain for explicit quiet/JSON flags because the CLI library’s GlobalBool stops at the nearest ancestor flag set. It also restores JSON Lines mode inside the checksum action because nested Before hooks can reset it after an app-level --json. No other command’s progress or output behavior changes.

Report, secrets, and operational cost

Report files are created with mode 0600, must not already exist, and contain metadata/results rather than object bodies or SSE-C keys. The manifest likewise contains only bucket, key, and optional VersionID.

Verification downloads every supported object body. Operators should use --dry-run, --max-size, time filters, --max-workers, and the global download limit to bound cost and load. NO_CHECKSUM and UNKNOWN_* counts must remain visible; neither may be presented as successful verification.

What a mismatch proves

A mismatch proves only that the additional checksum returned at verification time does not describe the logical bytes returned at verification time. It does not prove that a particular historical compression defect created the object, and it is not an external source-of-truth comparison.

Do not overwrite checksum metadata in place. Audit and classify first. For a confirmed, operationally relevant mismatch, prefer a new key or new version, verify the replacement, then switch consumers deliberately. Leave UNKNOWN_* objects out of automatic repair.

Verification record and release boundary

The local acceptance matrix covers TTY human/JSON, non-TTY pipes, regular-file redirects, app/parent/leaf JSON and quiet flags, environment quiet, report under quiet, report-write failure, and MISMATCH/UNKNOWN exit status. It also includes real historical MATCH, MISMATCH, and unsupported-composite objects on a local S3 server.

The command shipped in mcli 20260901 from a signed tag at the tip of main, with the functional suite - including a checksum verification run against a real SILO server - green for that commit, and pgsty/mc#5 is closed. Bundling the client into the Server image and a production audit remain later, separately evidenced gates.

12 - Optional Checksums, Mandatory Failure: Repairing UploadPart and UploadPartCopy Compatibility

This is the complete design and implementation record for SILO #46. The repair was not merely a changed if statement. One apparently optional S3 header reached into multipart completion semantics, copy responses, compression and encryption pipelines, compatibility baselines, and release verification.

Status: server implementation and local verification complete; commit, PR, remote CI, release, and production verification pending.
Owner: pgsty/silo, the SILO server repository.
Tracking: #46.
Independent follow-ups: #63 CopyObject + compression checksum, #64 federated UploadPartCopy checksum.
Adversarial review: local Claude Code, Fable 5, --effort max; final verdict GO, with no blocking findings.

Too Long; Didn’t Read (TL;DR)

A multipart upload splits a large file into smaller parts. A client may attach a checksum to each part so the server can verify the transfer, but AWS defines that checksum as optional. SILO used to treat it as mandatory: an ordinary UploadPart failed without one, and UploadPartCopy could never work because it has no part-body checksum to provide.

After the repair, SILO still validates a checksum when the client sends one. When the client omits it, SILO computes the checksum while reading the original bytes and saves the result. This happens before compression and encryption, requires no second read, and changes no on-disk format. The result is AWS-compatible behavior without weakening data integrity.

Decision

When a multipart upload declares a checksum algorithm in CreateMultipartUpload, SILO applies this contract:

1. If the client supplies a part checksum, the server continues to validate it. A wrong value or algorithm fails and is never hidden by fallback computation.
2. If the client omits the part checksum, the server computes it in one pass with the MPU algorithm over the logical plaintext stream, before compression and encryption, and persists the result.
3. A normal UploadPart echoes a checksum response header only when the client supplied the checksum. A server-computed fallback is not echoed.
4. UploadPartCopy has no client part-body checksum, so the server computes the value and returns it in CopyPartResult.
5. ListParts returns the persisted part checksum.
6. FULL_OBJECT completion continues to linearize the full checksum from stored part checksums. COMPOSITE completion continues to require a checksum for every part; clients can recover those values with ListParts.
7. Computation occurs during the existing read. Completion never re-reads the entire object merely to manufacture missing state.

In one sentence:

The optional input is the client-provided checksum value, not the server’s responsibility to maintain a consistent checksum-enabled MPU.

How we found it

The defect surfaced while investigating a different multipart checksum issue, #31.

#31 concerned CompleteMultipartUpload: for FULL_OBJECT, a client can complete with part numbers, ETags, and an optional full-object checksum without retaining every part checksum in the completion XML. Tracing that path backward exposed a stronger, earlier condition in erasureObjects.PutObjectPart:

if cs := fi.Metadata[hash.MinIOMultipartChecksum]; cs != "" {
    if r.ContentCRCType().String() != cs {
        return InvalidArgument{/* checksum missing */}
    }
}

Once an MPU declared a checksum algorithm, every UploadPart had to carry the matching x-amz-checksum-* value. Omitting it returned:

400 InvalidArgument:
checksum missing, want "CRC32", got ""

API-level probes reproduced the behavior on both the single-drive and erasure backends.

Reviewing CopyObjectPartHandler raised the severity from a client-configuration incompatibility to P0. UploadPartCopy has no request body for the caller to checksum. The handler reads the source object, constructs an internal reader, and eventually enters the same PutObjectPart implementation. There is no client header and no SDK setting that can repair the request. Every checksum-enabled MPU therefore rejected UploadPartCopy by construction.

What AWS requires

This cannot be decided by saying that MinIO has historically behaved a certain way. The S3 protocol is the authority.

The AWS UploadPart API describes each algorithm-specific checksum header as something that “can be used as a data integrity check.” More importantly, its response fields say that the checksum is present only when it was provided in the request.

The AWS UploadPartCopy API is different: when the MPU was created with an algorithm, the copy result contains that part checksum. There is no copy request body, so this is necessarily a server-computed value.

The AWS ListParts API is the standard way to recover checksums for parts in an upload that is still in progress.

The algorithm/type matrix also rules out treating the repair as one Boolean flag:

FULL_OBJECT is limited to CRCs that can be linearized, but SHA1 and SHA256 still need correct per-part digests for COMPOSITE completion.

SDK configuration makes the gap practical. Current AWS SDKs usually calculate request checksums when an operation supports them, but users can choose request_checksum_calculation = when_required, and low-level callers can initiate an algorithm without repeating it on every part. S3 accepts those requests; SILO did not.

Why removing the check is not a fix

The most tempting patch is to delete the comparison and allow a checksum-less part to proceed. That only moves the failure to completion.

SILO does not reconstruct and re-read all object bytes during MPU completion. It reads ObjectPartInfo.Checksums from each part.N.meta:

• a missing entry immediately becomes InvalidPart;
• FULL_OBJECT calls Checksum.AddPart, combining digests with their part lengths;
• COMPOSITE concatenates the raw digest bytes and hashes them into the object checksum.

The actual invariant is therefore:

checksum-enabled MPU
        => every committed part has a checksum for the MPU algorithm

Deleting the upload check without filling the metadata would make UploadPart appear successful, leave ListParts incomplete, omit the UploadPartCopy response value, and fail later during completion. A delayed failure is harder to diagnose than the original immediate one.

Alternatives considered

What the commercial edition taught us

We downloaded and verified the then-current MinIO AIStor RELEASE.2026-08-07T18-34-35Z. Without a commercial license the server enters offline mode and denies S3 operations, so the evidence came from Go pclntab and ARM64 disassembly, not a black-box compatibility run.

The static analysis showed that AIStor already:

• installs a server hasher when the client checksum is absent;
• persists the result in part metadata;
• exposes checksum fields in CopyPartResult.

It nevertheless applies fallback only to CanMerge() algorithms—CRC32, CRC32C, and CRC64NVME. SHA1/SHA256 COMPOSITE still follows the old checksum missing path. More importantly, the hasher is attached in the object layer to the current r.Reader; under compression or encryption that reader may already represent transformed storage bytes.

AIStor validated the general direction—compute and store—but not an implementation that SILO could copy mechanically.

How adversarial review overturned the first design

The first plan tried to centralize every decision inside erasureObjects.PutObjectPart: read the MPU metadata in the object layer and install a server hasher when the incoming reader had no client checksum. It looked attractive because all internal callers would share one rule.

The first Fable 5 Max adversarial review found that this design was wrong for compression.

newS2CompressReader is not a lazy wrapper. Construction immediately launches a goroutine:

go func() {
    _, err := io.Copy(comp, r)
    // ...
}()

The S2 writer also reads several blocks concurrently. After constructing the compressor, the handler still performs option parsing, encryption preparation, and the object-layer call. By the time PutObjectPart installed a hasher, the plaintext reader could already have lost several MiB:

• a large part would get a checksum with a missing prefix;
• a small part could reach EOF before installation and produce no result;
• mutating ServerSideHasher concurrently with Read would be a data race.

That finding changed the responsibility split:

The handler installs the hasher before any eager transform starts; the object layer validates the algorithm, requires a result, and persists it atomically.

This was the decisive turn in the design. Putting logic in the lowest layer may look more uniform, but stream correctness depends equally on when bytes begin moving and which representation of those bytes a layer can see.

Final implementation

A dedicated logical checksum reader

PutObjReader originally distinguished two concepts:

• Reader, the stream sent to storage, possibly compressed or encrypted;
• rawReader, used by older ETag and checksum code.

Under compression, even rawReader may not directly see plaintext; it can merely carry an ETag through an etag.Tagger chain. The repair therefore did not overload it. It added an unexported field:

checksumReader *hash.Reader

This reader always represents the logical S3 part bytes. WithEncryption can replace the storage Reader, but it must preserve checksumReader.

Unexported accessors on PutObjReader then:

• return the effective client or server checksum type;
• prefer the client value whenever it exists;
• otherwise return the server result finalized at EOF.

Keeping the mechanism unexported minimizes public Go API growth and gives #63 a shared internal path without prematurely changing ordinary CopyObject behavior.

Preparing the hasher before transformations

prepareMultipartChecksumReader loads the algorithm and checksum type saved with the MPU:

1. no declared algorithm means no work;
2. an existing client checksum is compared by base algorithm;
3. a wrong algorithm preserves the InvalidArgument rejection;
4. an omitted client checksum installs the corresponding server hasher on the plaintext reader.

For normal UploadPart:

• the compressed path prepares actualReader after request-checksum parsing but before newS2CompressReader;
• the uncompressed path prepares the request hash reader before the encryption reader is constructed.

For UploadPartCopy:

• a checksum-enabled MPU first gets an inner hash reader over the logical source range;
• a range copy hashes only the selected bytes;
• compression and destination encryption start only after that reader is ready.

The object layer remains authoritative

Early handler preparation does not replace the storage invariant. erasureObjects.PutObjectPart still:

• re-parses the expected MPU algorithm;
• requires an effective checksum type that matches;
• obtains the checksum map after erasure encoding finishes;
• reports an internal error instead of committing if an enabled algorithm has no result;
• writes the checksum with the ETag, sizes, and index into part.N.meta, then atomically renames the part.

An internal caller that bypasses the HTTP handler without preparing a valid checksum is therefore rejected just as before. It cannot silently commit a part that violates the MPU invariant.

CopyPart response shape

CopyObjectPartResponse gained the five algorithms supported by this source tree:

ChecksumCRC32
ChecksumCRC32C
ChecksumCRC64NVME
ChecksumSHA1
ChecksumSHA256

All are omitempty, so an MPU without checksums produces the old XML. Normal UploadPart still uses the existing TransferChecksumHeader and echoes only a client request value; fallback computation does not alter that response.

Why it works

After the repair, the data flow is:

logical plaintext part
        |
        +--> client checksum verifier (if supplied)
        |         or
        +--> server-side hasher (if omitted)
        |
        v
compression (optional)
        |
        v
encryption (optional)
        |
        v
erasure encode / storage
        |
        v
persist ETag + size + logical part checksum atomically

This satisfies four requirements that previously appeared to conflict:

1. Protocol compatibility: omitting an optional header succeeds.
2. No integrity downgrade: a supplied client value is still checked end to end and is never hidden by server fallback.
3. Correct object semantics: the checksum covers logical S3 bytes, not compressed data or ciphertext.
4. Controlled cost: hashing shares the existing read and adds CPU, not a second disk or network pass.

EOF has a precise role. hash.Reader finalizes ServerSideChecksumResult only when it reaches EOF. Closing the compression pipe synchronizes the compressor goroutine with the storage read; the object layer reads the result only after encoding returns. Targeted -race tests verified that concurrency boundary.

The compatibility-baseline blocker

The five new CopyObjectPartResponse fields are exported Go API. SILO’s buildscripts/rebrand-guard rescans imports, environment variables, headers, routes, storage markers, and exported symbols, then compares them in both directions with buildscripts/rebrand-guard/compat-baseline.json. An unacknowledged symbol makes CI fail.

After recording the five #46 fields, the guard still reported two additions:

internal/config/notify:notify:type:LegacyDatabaseTargetError
internal/config/notify:notify:method:LegacyDatabaseTargetError.Error

They did not come from #46. They belong to the earlier database-notification repair f1ba68358 on the local main branch. The cmd startup path intentionally needs the exported type for errors.As, but that earlier commit had not updated the compatibility baseline. Every later change based on that HEAD would therefore fail the CI guard.

We chose “option A”: acknowledge the two notification symbols as part of their original repair while retaining the five #46 fields. The final baseline diff is exactly seven additions and zero deletions, and the guard reports:

exported=9021
Silo rebrand compatibility baseline is unchanged

This does not disable the check. Exact set equality means that acknowledging a nonexistent symbol also fails. The change explicitly records two intentional compatibility-surface additions.

golangci-lint has not yet run locally; it remains a remote go.yml gate. Green local go test, go vet, race, and rebrand-guard results do not substitute for green remote CI.

Verification evidence

The new tests execute 76 subtests across:

• CRC32, CRC32C, and CRC64NVME FULL_OBJECT;
• CRC32, SHA1, and SHA256 COMPOSITE;
• correct client checksums, wrong algorithms, and wrong values;
• absence of a server-computed checksum in normal UploadPart responses;
• server values in UploadPartCopy responses and ListParts;
• a real 5 MiB + 1 KiB two-part full-object merge;
• zero-length parts and overwriting the same part number;
• a range copy whose SHA256 covers only the copied interval;
• single-drive and 16-drive erasure backends;
• default, versioned, compressed, encrypted, and compressed-plus-encrypted modes;
• explicit SSE-C and SSE-S3.

Local validation included:

go test -race ./cmd -run '^TestAPIUploadPartServerSideChecksum' -count=1
go test ./cmd -count=1
go test ./... -count=1
go vet ./cmd
git diff --check
go run ./buildscripts/rebrand-guard

All passed. Two subsequent Claude Code Fable 5 Max implementation reviews and the final acceptance review returned GO with no blocking findings.

Cost, risk, and release boundary

When a client omits its value, the server performs one additional hash over the part. CRC cost is small; SHA costs more CPU. Both share the read that already had to occur, without buffering an entire part in memory or adding a completion-time second pass.

During a rolling upgrade, old and new nodes may answer the same checksum-less request differently: a new node accepts it while an old node returns 400. ObjectPartInfo.Checksums did not change format, so stored data remains downgrade-readable, but client-visible behavior stabilizes only after all serving nodes have upgraded. The release note must call that out.

This record describes a local main worktree. The implementation has not been committed, pushed, run through remote CI, or packaged into a release. SILO documentation belongs to silo.pgsty.com; a successful local Hugo build does not mean that the product in the wider pgsty.com ecosystem has shipped.

Why two follow-ups remain separate

Adversarial review found two related but independent issues.

#63: CopyObject + compression

Ordinary CopyObject can also attach a server-side checksum to a transformed stream. It shares the root cause and the new checksumReader mechanism, but it is a different API with a different test matrix and rollback boundary. We chose a separate repair and require that PR to reuse this plaintext-reader contract instead of inventing a second abstraction.

#64: legacy federation

Legacy etcd federation turns UploadPartCopy into an ordinary remote UploadPart. Under the AWS response semantics preserved here, that remote request does not return a server fallback value, so the proxy may still lack the checksum required for CopyPartResult. A follow-up must independently choose between a remote-returned value and an ETag-verified ListParts fallback. It must not make all external UploadPart responses non-compliant merely to simplify an internal proxy.

Separating them does not abandon consistency. Consistency is maintained through one shared rule:

Every server-computed S3 checksum binds to the logical plaintext stream, is installed before any eager transform, and is validated and persisted by the object layer that owns the storage invariant.

Lessons retained

The repair leaves lessons more durable than its individual lines of code:

1. An optional header does not make internal state optional. If the protocol lets the client omit a value, the server must produce the state its own completion path needs.
2. Request acceptance and response disclosure are separate contracts. A normal UploadPart may compute internally and still omit the value; UploadPartCopy must return it.
3. Stream layers are defined by byte semantics. The lowest layer is not automatically correct if it no longer sees logical bytes, and an eager goroutine turns “install later” into a race.
4. A commercial implementation is evidence, not the specification. AIStor showed the direction and the boundary that could not be copied.
5. A compatibility guard is a change-acknowledgment mechanism. compat-baseline.json exists to assign every new compatibility surface, not merely to make CI quiet.
6. Independent defects should ship independently while sharing invariants. #63 and #64 remain separate, but both must cite and obey the checksum-reader contract established here.

The final result is not a broad relaxation. It is a stricter and more accurate boundary: clients may omit optional information; the server may not omit correctness.

13 - BadDigest, InvalidRequest, and the CompleteMultipartUpload Checksum Contract

This is the design, investigation, and verification record for SILO #48, with the decision boundary for the related SILO #50.

Status: pgsty/silo#74 merged as 590aeaa7d, and pgsty/silo.pgsty.com#6 merged as 9805dd7; full local verification, remote CI, and independent Opus 5 Max acceptance review completed on the linked changes. Tag, release, package, image, deployment, and production verification remain separate pending gates.
2026-08-28 follow-up: signed-off server commit f7bc725d8 closes the remaining type-only and invalid-token bypass without changing CRC64NVME canonicalization. Complete local, tagged, race, static, build, and Fable Max verification passed; push, remote CI, merge, tag, and delivery remain pending.
Owner: pgsty/silo, the SILO server repository.
Implementation scope: CompleteMultipartUpload error semantics only; no storage-format, checksum-math, dependency, Console, package, or client change.
Independent decision: #50 remains probe-gated and is not part of this repair.

Too Long; Didn’t Read (TL;DR)

Issue #48 is valid and should be fixed, with two corrections to the original report.

First, the checksum-type comparison is worse than the issue states. SILO used bitmask containment instead of equality. An upload created as FULL_OBJECT and completed as COMPOSITE failed, but the reverse COMPOSITE to FULL_OBJECT direction could pass the type check. The repair must compare the base algorithm and normalized multipart object type independently and symmetrically.

Second, the missing-part-checksum row originally lacked a direct AWS capture. That evidence now exists in the official boto/s3transfer project: issue #241 records a real S3 InvalidRequest response naming sha256 and missing part 1, and PR #242 repaired the client and added tests. This is strong enough to implement the response contract without a new AWS account probe.

The accepted behavior is:

The repair uses completion-specific error types. It deliberately does not change the global mapping of hash.ChecksumMismatch, so PutObject, UploadPart, streaming trailers, and other operations retain their existing XAmzContentChecksumMismatch contract.

Issue #50 is a separate question. AWS documents that CRC64NVME is full-object only, but the available sources do not prove that S3 rejects an explicit CRC64NVME + COMPOSITE initiation instead of canonicalizing it. Upstream MinIO intentionally implemented canonicalization and exposes FULL_OBJECT in the initiation response, so the behavior is not silent. A raw AWS probe is required before changing it.

Scope and decision

This record answers two different questions:

1. Are the #48 error-code deviations real, externally observable compatibility defects with enough evidence to repair?
2. Does the same evidence authorize changing the CRC64NVME canonicalization described by #50?

The decisions are:

• #48: accept with corrections and implement. The error codes are part of the S3 wire contract. Returning a different code makes SDK behavior and operator diagnosis diverge even when the request is rejected in both systems.
• #50: do not implement yet. The capability matrix proves the resulting checksum must be full-object. It does not establish whether an invalid requested type is rejected, ignored, or canonicalized. Those are different wire contracts.

The repair is intentionally narrow. It does not add algorithms, recalculate stored data, reinterpret successful uploads, or change the optionality rules repaired for #31 and #46.

Evidence ledger

Not all evidence has the same authority. The implementation decision uses the following hierarchy.

This distinction matters. The original #48 comment correctly downgraded the third row while it was supported only by secondary reports. The boto transcript and the merged client repair close that evidence gap.

The observable contract

Object checksum mismatch

For a FULL_OBJECT multipart upload, SILO combines stored part checksums and compares the result with the optional object checksum supplied on completion. The old code returned hash.ChecksumMismatch. A global API mapping converted that type to:

400 XAmzContentChecksumMismatch

AWS explicitly documents BadDigest for the corresponding completion integrity failure. Reusing the existing generic ErrBadDigest code without a custom message would still be misleading because its static text says Content-MD5; CRC32, CRC32C, and CRC64NVME are not Content-MD5.

The new response is therefore operation-specific:

400 BadDigest
The CRC32 checksum you specified did not match the calculated checksum.

The response does not disclose the expected or supplied digest.

Checksum type mismatch

The checksum type saved by CreateMultipartUpload is part of the upload’s contract. A completion may not switch between COMPOSITE and FULL_OBJECT.

The old test was:

!provided.Type.Is(expectedType)

ChecksumType.Is is a containment operation over a bitmask, not equality. For CRC32:

created FULL_OBJECT + completed COMPOSITE => rejected
created COMPOSITE   + completed FULL_OBJECT => containment succeeds

The second request could proceed using the persisted composite rules. If the caller supplied the composite checksum value under a FULL_OBJECT declaration, completion could even succeed. This is a protocol validation bypass, not merely the wrong error label.

The repair normalizes both values into multipart checksum types, then compares:

1. base algorithm equality; and
2. object type equality (COMPOSITE versus FULL_OBJECT).

For algorithms whose two object-type forms are both syntactically accepted—currently CRC32 and CRC32C—both mismatch directions now return 400 BadDigest. SHA1 and SHA256 with FULL_OBJECT are rejected earlier as InvalidArgument; CRC64NVME is the canonicalized special case discussed under #50 below. Base-algorithm mismatch remains a separate InvalidArgument path because #48 and the cited AWS type contract do not authorize broadening that behavior.

Type-only assertions and invalid tokens

The first #48 repair remembered whether x-amz-checksum-type was present, but its object-layer comparison was still nested under WantChecksum != nil. WantChecksum is populated only when completion carries a checksum value. A caller could therefore send a type assertion without a whole-object checksum:

CreateMultipartUpload:   CRC32 + COMPOSITE
CompleteMultipartUpload: x-amz-checksum-type: FULL_OBJECT
                         no x-amz-checksum-crc32 value

The server returned success and persisted the initiated composite state. It did not corrupt the object, but it accepted an explicit integrity assertion that contradicted the upload contract.

There was a second parser asymmetry. In the header-without-algorithm path used by completion, an unknown value such as NOT_A_TYPE could be ignored when a checksum header was also present. Relying on ChecksumType.ObjType() after creating an invalid bitmask would not be safe: an invalid non-multipart value can fall through to the full-object default. Raw enum validation must happen first.

The follow-up stores the explicit raw type string in ObjectOptions, accepts only COMPOSITE or FULL_OBJECT, and compares it with the initiated multipart type independently of WantChecksum. The order is deliberate:

1. reject every unknown non-empty token as InvalidArgument;
2. compare the base algorithm when a checksum value is supplied;
3. compare the explicit object type whenever the upload recorded a checksum algorithm;
4. report an explicit type mismatch as BadDigest even when no object checksum value was supplied.

CRC64NVME remains a deliberate exception. A raw COMPOSITE token is normalized to FULL_OBJECT before comparison, preserving the inherited behavior pending the #50 AWS probe. A legal type-only header on an upload that recorded no checksum algorithm remains outside the comparison because there is no initiated checksum type to assert against; its exact AWS error semantics remain unproven and were not expanded into this repair.

Missing composite part checksum

For a composite upload, the completion XML must include the selected checksum for every listed part. SILO previously compared an empty client value with the stored part checksum and returned InvalidPart.

That conflated three different states:

• the part or ETag does not exist;
• a checksum was supplied but has the wrong value or algorithm;
• the required checksum element is absent.

The third state now has a dedicated error. Its wire message follows the AWS response captured by boto/s3transfer:

400 InvalidRequest
The upload was created using a sha256 checksum. The complete request must include
the checksum for each part. It was missing for part 1 in the request.

The error is emitted for the first missing part and includes its actual number. FULL_OBJECT behavior is unchanged: a completion may omit per-part checksum elements, while any supplied part checksum must still be valid.

Root cause in the upstream lineage

The behavior is inherited rather than a SILO-specific redesign.

• MinIO PR #15433 introduced extended checksum handling and the global hash.ChecksumMismatch to XAmzContentChecksumMismatch mapping. That mapping is suitable for streaming upload validation but too broad for completion semantics.
• MinIO PR #20855 added full-object checksums and CRC64NVME. It introduced the checksum-type comparison and intentionally canonicalized CRC64NVME to full-object with the comment that AWS appears to ignore the supplied mode.
• MinIO PR #20953 tightened invalid algorithm/type combinations but retained the CRC64NVME special case. That is evidence of deliberate upstream behavior, not an accidental missing branch.
• MinIO issue #20944 reported an AWS BadDigest versus MinIO InvalidPart difference. The divergence was acknowledged but not repaired.

The upstream repository is now archived. SILO therefore owns the compatibility decision, tests, and maintenance burden rather than waiting for an upstream correction.

Repair design

Operation-scoped errors

Changing the global hash.ChecksumMismatch mapping would alter every operation that uses it. That would be a larger, weakly evidenced compatibility change.

The repair adds three package-private, sentinel-backed error helpers in the server command package. Keeping the helpers and the request-header-presence flag private avoids expanding SILO’s exported Go compatibility surface:

• completeMultipartChecksumMismatch, mapped to BadDigest with a checksum-aware description;
• completeMultipartChecksumTypeMismatch, mapped to BadDigest with provided and initiated types;
• missingPartChecksum, mapped to InvalidRequest with algorithm and part number.

Only CompleteMultipartUpload produces these types. The global mapping remains:

hash.ChecksumMismatch => XAmzContentChecksumMismatch

This preserves PutObject and UploadPart behavior and makes the compatibility boundary visible in code.

Symmetric type validation

Both persisted and supplied types are normalized with the multipart flags before comparison. This is necessary because a bare CRC checksum type describes a non-multipart full-object checksum through ObjType(), while the same base value means composite after multipart context is applied. Object type is compared only when the completion request explicitly contains x-amz-checksum-type; omitting an optional header does not synthesize a COMPOSITE assertion.

The resulting invariant is:

provided base algorithm == initiated base algorithm
AND
provided multipart object type == initiated multipart object type

The second condition applies only to an explicitly supplied type. This comparison is symmetric and remains compatible with the existing CRC64NVME canonicalization. It fixes #48 without silently deciding #50.

Precise missing-value detection

For each part, the server already builds a map of all checksum fields supplied in the completion XML. The repair distinguishes:

COMPOSITE + no checksum field supplied => missingPartChecksum / InvalidRequest
expected field present but wrong value => InvalidPart
only a different algorithm supplied    => InvalidPart
FULL_OBJECT + no checksum field         => allowed
FULL_OBJECT + any checksum field        => validate it

This is intentionally narrower than converting every part checksum failure to InvalidRequest. Only the state demonstrated by AWS evidence changes.

The same edit corrects the internal InvalidPart expected/actual field order. The generic S3 InvalidPart wire response did not expose those digest values, but internal error text and logs should still describe them correctly.

Regression and detection matrix

The API-level tests exercise signed HTTP requests through both the single-drive and erasure object-layer backends.

The committed type-mismatch regression uses CRC32, while an independent acceptance probe covered CRC32C as well. The follow-up additionally covers type-only, unknown, lowercase, matching, and checksum-bearing invalid-token cases. The same matrix confirms that SHA1/SHA256 FULL_OBJECT requests stop earlier at the existing invalid-combination check and that CRC64NVME still canonicalizes an explicit COMPOSITE token. Those distinctions are protocol boundaries, not untested claims that every algorithm reaches the same error mapper.

Focused verification command:

go test ./cmd -run 'TestAPIErrCode$|TestAPICompleteMultipart(FullObjectChecksumMismatch|CompositeStillRequiresPartChecksums|CompositeChecksumMismatch|ChecksumTypeMismatch)$|TestAPIUploadPartServerSideChecksumDoesNotMaskClientErrors$' -count=1

Observed result on 2026-08-27:

ok  github.com/minio/minio/cmd

The complete local package gate was then rerun after the review-driven additions:

go test ./cmd ./internal/hash -count=1

ok  github.com/minio/minio/cmd           121.462s
ok  github.com/minio/minio/internal/hash   0.566s

git diff --check also passed. Independent review of the final diff remains a separate gate. A local pass is not remote CI, a merged commit is not a release, and a release is not production deployment.

Independent adversarial review

The first review of the actual server diff was performed with local Claude Code in read-only safe mode. Its verdict was GO with no blocking findings. It independently confirmed the operation-scoped mapping, symmetric bitmask normalization, per-part missing-value detection, both object-layer backends, and preservation of UploadPart behavior.

The review identified four useful gaps that were incorporated before the second full test run:

• distinguish an omitted optional type header from an explicit COMPOSITE assertion;
• separate value-mismatch and type-mismatch error types;
• exercise the composite checksum-of-checksums mismatch path;
• pin missing part 2, algorithm mismatch, and unchanged CRC64NVME canonicalization.

One first-review concern was rejected by primary evidence: it questioned whether checksum type mismatch should return InvalidRequest. The AWS CompleteMultipartUpload reference and AWS CLI reference explicitly specify BadDigest when the completion type differs from initiation.

The final first-round re-review verdict was FINAL GO, no blockers. It explicitly withdrew the earlier error-code concern, agreed with accepting #48 and deferring #50, verified that the new guards preserve the intended non-changes, and found no English/Chinese drift.

A subsequent independent acceptance run used Claude Code claude-opus-5 with maximum effort. It returned ACCEPT, no blocking findings, reproduced the old composite-as-FULL_OBJECT bypass end to end against the pre-fix code, verified that the new API assertions fail against that code, and probed all five checksum algorithms in both type directions.

The 2026-08-28 follow-up received a separate local Fable Max mirror review over the complete uncommitted release-review diff. It returned GO, with no P0–P2 findings. The primary review independently checked its seven P3 observations: five were non-blocking boundaries, while two proposed causes were disproved by the actual config and key-rotation call paths. The review confirmed that raw invalid types are rejected before normalization, type-only mismatch is enforced, source-side checksum decryption still receives the full request, and CRC64NVME canonicalization remains untouched.

Five pre-existing or deliberately deferred, non-blocking observations remain outside these repairs:

• SHA1/SHA256 FULL_OBJECT combinations are rejected by the existing parser as InvalidArgument before the new type-mismatch mapper; only CRC32/CRC32C reach both mismatch directions;
• CRC64NVME treats any type value as full-object state, so completion with an explicit COMPOSITE token is still accepted through canonicalization pending the #50 AWS probe;
• when initiation recorded no checksum algorithm but completion supplies an object checksum, SILO returns BadDigest; AWS documentation says such a value is accepted and ignored, so this should be triaged as a separate compatibility issue;
• composite part-count and value mismatches both become BadDigest with the same description;
• a full-object checksum carrying a -N suffix has that suffix ignored while its digest is still validated.

None is introduced by these patches, and none changes the #48 decision. They should be triaged separately if strict message or invalid-header parity becomes a maintenance priority.

Why #50 is not included

Issue #50 says CRC64NVME + COMPOSITE should be rejected at initiation. Three facts are confirmed:

1. AWS’s algorithm matrix supports CRC64NVME only as a full-object checksum.
2. SILO and upstream MinIO canonicalize the request to full-object state.
3. The server returns x-amz-checksum-type: FULL_OBJECT from CreateMultipartUpload, so the substitution is externally visible rather than silent.

What is not confirmed is the decisive wire behavior: does AWS reject the explicit invalid combination, or accept it and return/carry full-object state? A capability matrix does not answer that question.

The upstream history also argues against guessing. PR #20855 added the canonicalization intentionally, and PR #20953 preserved it while tightening other invalid combinations. That may be based on an AWS observation, but the comment is not a reproducible transcript.

The same representation also affects completion: FullObjectRequested treats every CRC64NVME checksum as full-object state, so a stored FULL_OBJECT upload completed with the raw header value COMPOSITE is accepted as full-object rather than rejected as a type mismatch. This completion-side residue falls under the same raw-token-versus-canonical-state evidence question. It is explicitly not claimed fixed by #48.

PutObject must not be bundled into this decision. Its API reference does not define x-amz-checksum-type, so accepting, rejecting, or ignoring that header is a separate undocumented-header question.

Required AWS probe

Before changing #50, capture a raw SigV4 request and response against a general-purpose AWS S3 bucket:

1. send CreateMultipartUpload with x-amz-checksum-algorithm: CRC64NVME and x-amz-checksum-type: COMPOSITE;
2. record the HTTP status, error code/message, request ID, and all checksum response headers;
3. if accepted, upload one part and complete it, recording whether S3 requires per-part values and which type HeadObject reports;
4. repeat with FULL_OBJECT as the control;
5. probe PutObject separately, explicitly labeling it as an undocumented-header experiment.

Only a captured rejection authorizes replacing canonicalization with validation. If AWS accepts and canonicalizes, #50 should be corrected or closed rather than implemented.

Compatibility and operational impact

• Successful requests: checksum semantics are unchanged, except that omitting the optional x-amz-checksum-type header is no longer misclassified as an explicit COMPOSITE assertion. That intentional interoperability relaxation changes the old erroneous 400 into success.
• Rejected requests: apart from that omitted-header case, HTTP status remains 400; the affected S3 error code and message become AWS-compatible. Explicit type-only mismatch is now enforced, and an unknown non-empty type is rejected as InvalidArgument before bitmask normalization.
• Integrity: unchanged or stronger. The reverse type-bypass is closed; no failed completion commits an object.
• Stored data: no format, checksum encoding, metadata, erasure layout, migration, or backfill change.
• Performance: constant-time comparisons and error construction only; no additional data reads or hashing passes.
• Security/privacy: digest values are not returned in the new messages. Bucket and object names are not added to them.
• Rolling upgrade: nodes may return different error codes until all serving nodes are upgraded, but successful objects remain compatible.
• Rollback: restores the old error codes and asymmetric check; it does not require data rollback.
• Other repositories: no Console, shared-package, MCLI, or SDK change is required. This public design record is the only cross-repository deliverable.

Merge and release gates

The follow-up must keep #50 out unless a raw AWS transcript changes the decision, run remote DCO/Go CI/vulnerability/release-pipeline checks on its final commit, and merge from the current SILO main. Repository integration, release artifact, image, deployment, and production probe remain independent gates; none can be inferred from a local test or documentation build.

Conclusion

#48 is a correct compatibility issue, and the evidence now covers all three rows. The safest repair does not relabel checksum failures globally. It teaches CompleteMultipartUpload to report its own protocol errors, compares checksum types symmetrically, and identifies a genuinely missing composite part checksum without confusing it with a missing part or a wrong value.

#50 is related by discovery history, not by proof. The server’s current CRC64NVME canonicalization is deliberate and visible. Until AWS’s exact response is captured, changing it would replace one unverified assumption with another.

That boundary is the central design decision: implement what the official contract and tests establish, test the hidden consequence found in the code, and leave the remaining policy question behind an explicit, reproducible evidence gate.

14 - Should SILO Fix ListMultipartUploads? Design Review of Issue #79

This is the problem, design, and decision record for SILO issue #79.

Status on 2026-08-30: confirmed compatibility defect; design proposal only. No server implementation, release artifact, deployment, or production verification is claimed by this record.
Recommendation: fix it as a planned P1 compatibility project, not as a small cache patch. If the project declines full compatibility, reject unsupported requests explicitly instead of returning a successful response that did not honor them.
Scope: ListMultipartUploads for S3 general-purpose buckets. This record does not add directory-bucket behavior or AbortIncompleteMultipartUpload lifecycle support.
Owner: pgsty/silo, the SILO server repository.
Release boundary: design, specification capture, prototype, implementation, source QA, commit, release artifact, documentation, deployment, and live verification are separate gates.

The problem in plain language

Imagine that four large files are still being uploaded:

tables/a/part-1
tables/a/part-2
tables/b/part-1
other/file

An S3 client asks, “show me every unfinished upload below tables/.” AWS S3 returns the first three. SILO currently treats tables/ as if it were the complete name of one object, looks for exactly that object, and returns an empty list.

If the client removes the prefix and asks for every unfinished upload in the bucket, SILO takes a different shortcut: it reads a process-local memory cache. That cache may contain all four uploads on the node that created them, but it does not survive a restart and is not authoritative across nodes. The upload data is still on disk; the list is wrong.

This is why the defect is more serious than one ignored query parameter. Cleanup tools can receive 200 OK, conclude that no unfinished uploads exist, and report success while uploads remain on disk. The server is not losing committed objects, but it is giving callers a false view of unfinished work.

Executive decision

SILO should fix this behavior if it intends to keep advertising practical S3 compatibility.

The repair is justified because the current endpoint silently claims success, behaves differently after restart or node switching, and breaks standard prefix-based cleanup and pagination. The default 24-hour stale-upload collector limits storage accumulation on default configurations, but it does not make the API result truthful.

The repair is not a small change. Existing upload directories contain only a one-way hash of the bucket and object key, and the original key is not stored in their xl.meta. A correct implementation must begin persisting that identity for new uploads, discover candidates with erasure-aware quorum rules, apply S3 semantics globally across pools and sets, and handle legacy uploads during a rolling upgrade.

The recommended direction is therefore:

1. record the bucket and object key in the upload’s existing quorum-written metadata;
2. build a bounded on-demand scan as the durable correctness path;
3. keep any cache only as a rebuildable optimization;
4. enable strict S3 behavior only after every writer has upgraded and all keyless legacy uploads have drained;
5. consider a durable secondary index only if measurements prove that scanning cannot meet a product-approved service-level objective.

Sources and provenance

The problem statement and proposed design are grounded in five kinds of evidence.

The S3 contract

The AWS ListMultipartUploads API defines the public contract for general-purpose buckets:

• prefix selects every upload whose key starts with that string;
• delimiter groups matching keys into CommonPrefixes;
• max-uploads limits a page, with 1,000 as the documented maximum;
• key-marker and upload-id-marker continue a truncated listing;
• upload-id-marker is ignored when key-marker is absent;
• results are ordered by object key, then by initiation time for uploads with the same key.

AWS documentation does not settle every implementation edge unambiguously. Equal timestamps, invalid or out-of-range max-uploads, URL encoding, marker boundaries, and the way CommonPrefixes consume a page should be captured once against AWS and stored as fixtures before implementation.

The reported defect

Issue #79 supplied a self-contained signed reproducer against pgsty/silo:latest and compared SILO with AWS, RustFS, SeaweedFS, and Garage. Its four central observations reproduce:

The issue correctly identifies a compatibility failure, but its statement that max-uploads is always ignored and IsTruncated is always false is broader than the implementation. Those claims hold on the empty-prefix cache path used by the reproducer; the exact-object path can honor max-uploads and upload-id-marker and can set IsTruncated.

The upstream design history

The behavior was inherited rather than invented by SILO:

• MinIO PR #5248 deliberately removed prefix-based listing from the erasure backend in 2017 “to simplify” multipart support.
• MinIO PR #20407 added the empty-prefix multipart cache in 2024, mainly for Alluxio tests.
• A 2025 report of the same exact-key behavior, MinIO issue #20989, was closed as working as intended.
• SILO’s current S3 compatibility reference already records the exact-object-name divergence, although it did not explain the cache, pagination, marker, delimiter, or restart limitations before this design record.

This history explains why the code looks deliberate. It does not make the endpoint compatible with the AWS contract.

Source review

The current source has two mutually exclusive listing paths:

erasureServerPools.ListMultipartUploads
  prefix == ""  -> return entries from node-local mpCache
  prefix != ""  -> hash prefix as a complete object key
                    -> select one set
                    -> list one sha256(bucket/object) directory

The important locations are:

• cmd/erasure-server-pool.go: empty-prefix mpCache, per-pool concatenation, and the internal exact-object lookup used by NewMultipartUpload;
• cmd/erasure-multipart.go: exact-object listing, upload directory construction, stale-upload cleanup, and the quorum write for a new upload’s xl.meta;
• cmd/erasure-sets.go: hashing a supplied object name to one erasure set;
• cmd/bucket-handlers.go: public request validation, including a 501 NotImplemented guard when key-marker does not share the request prefix;
• cmd/object-api-multipart_test.go: a large expected-results table whose final assertion block checks only echoed scalar fields, not the returned uploads, prefixes, markers, or truncation state.

Independent reproduction and adversarial review

The issue scenario was independently reproduced against the reviewed SILO source with a single-node server and SigV4 requests. Additional probes established that:

• an exact object key can paginate its own uploads;
• upload-id-marker currently affects that exact-key path even without key-marker, contrary to AWS;
• NextKeyMarker remains empty on an exact-key truncated page;
• max-uploads=0 behaves as unlimited in the current path;
• a plain server restart empties the bucket-wide view while exact-key lookup still finds the on-disk uploads.

A second adversarial architecture review challenged the storage, quorum, migration, suspended-pool, mixed-version, and performance assumptions. The corrections from that review are incorporated below; this record does not treat an AI review as a substitute for code tests or an AWS conformance capture.

What the code actually does

Empty prefix: a volatile node-local view

With no prefix, the pool layer returns every MultipartInfo for the bucket from mpCache, sorted only by initiation time. It does not apply max-uploads, key-marker, upload-id-marker, or delimiter, and it does not compute continuation markers or IsTruncated.

The cache is initialized empty at process startup. Creation populates only the node that handles the request. Completion and abort delete cache entries, including peer notifications in some paths, but creation has no equivalent durable cluster-wide population or startup rebuild. Consequently:

• a restart can change a non-empty listing into an empty one;
• two nodes can return different answers for the same bucket;
• a successful response is not evidence that the server has enumerated durable upload state.

Non-empty prefix: an exact object lookup

With a non-empty prefix, the string is passed through object hashing as though it were a complete object name. One erasure set is selected, and listing reads the directory derived from sha256(bucket/object).

This path can enumerate multiple upload IDs for that exact object. It sorts them by initiation time, applies its upload-id-marker, stops at max-uploads, and sets IsTruncated. It still does not implement lexical prefix matching, CommonPrefixes, general key-marker semantics, or NextKeyMarker.

Multiple pools make pagination less correct

For a non-empty request on a multi-pool deployment, the pool layer invokes each active pool with the same maximum and concatenates the results. It does not perform a global ordered merge or recompute page boundaries and next markers. A request for N items can therefore collect up to N from each pool.

Suspended pools are skipped by listing and by the other public multipart verbs. In-progress uploads left on a suspended or decommissioning pool are therefore inaccessible, not merely unlisted. That is a related lifecycle defect, but listing alone must not advertise handles that PutObjectPart, ListParts, CompleteMultipartUpload, and AbortMultipartUpload cannot use. Pool drain or forced abort should be designed as a separate cross-verb change.

Why current uploads cannot be backfilled

The multipart namespace is flat:

.minio.sys/multipart/<sha256(bucket/object)>/<upload-id>/xl.meta

The hash is one-way. The original bucket and key are not encoded in the path. They are also not stored as a name field in the current multipart xl.meta; the supplied object name only influences the erasure distribution during newFileInfo construction.

Therefore an all-directory scan can discover that an upload exists, but it cannot determine which bucket or key it belongs to. The current node-local cache cannot repair this reliably because it is incomplete across nodes and disappears on restart.

This rules out a tempting “small” fix: scanning every existing xl.meta and applying prefix filters. New identity metadata or a durable index is required, and old keyless uploads need an explicit migration policy.

Complexity assessment

The semantic algorithm is not the hardest part. The hard part is obtaining a complete, quorum-valid, globally ordered input set without turning a listing call into an uncontrolled cluster-wide metadata storm.

Overall, this is a high-complexity compatibility project with medium wire-compatibility risk and high implementation-correctness risk. It is not a destructive object-format migration: the recommended design adds internal metadata for new incomplete uploads and leaves the existing directory scheme in place.

Compatibility and operational impact

Wire behavior changes

A correct implementation deliberately changes observable results:

• prefix=foo will match foo, foobar, and foo/..., not only the exact key foo;
• bucket-wide results will be ordered by key and initiation time rather than only initiation time;
• max-uploads will actually limit a page;
• the default and maximum will move from SILO’s current 10,000 constant toward the AWS limit of 1,000, subject to the captured edge-case contract;
• clients must follow NextKeyMarker and NextUploadIdMarker instead of assuming one response contains everything;
• delimiter requests will return CommonPrefixes;
• the current handler-side 501 for a marker outside the prefix will be replaced by the captured AWS semantics.

These are compatibility fixes, but they can break software that accidentally depends on SILO’s old non-S3 behavior. In particular, a client that ignores pagination may see fewer entries after the repair. Strict behavior should therefore be introduced through an explicit release and rollout contract, not silently slipped into an unrelated patch.

Storage-format compatibility

The recommended write path adds the bucket and object key as reserved internal metadata inside the new upload’s existing quorum-written xl.meta. It does not rename multipart directories or create a second transactional write.

Before CompleteMultipartUpload renames upload metadata into the completed object, the new upload-only fields must be removed alongside the multipart checksum fields that are already stripped there.

An old binary completing an upload created by a new binary will not know to remove the new internal keys. They would remain inert and hidden from S3 user metadata, but persist in the completed object’s internal metadata. Rolling-upgrade tests must prove that unknown reserved keys do not disturb healing, replication, metadata comparison, or downgrade reads. The product must then choose between tolerating that residue and adding a scrubber; it must not assume the keys disappear.

Operational cost

Because all buckets share one flat hash namespace, an on-demand scan is O(all active multipart uploads in the cluster), not O(uploads in the requested bucket). Bounded parallelism, cancellation, memory limits, and failure behavior are part of correctness, not optional tuning.

Default SILO configuration expires stale multipart uploads after 24 hours and runs cleanup every 6 hours. Once the last old writer has been upgraded, the keyless population should normally drain within roughly 30 hours. Operators with a larger custom expiry have a longer migration window. A zero value is mapped back to the 24-hour default in the current code; no supported “disabled” expiry value was identified in this review.

The collector bounds default storage accumulation, but does not repair a false listing response. It also does not remove the need to test sustained legitimate multipart activity, failure modes, and custom expiry settings.

Severity

The recommended classification is P1 / high compatibility, not P0:

• no committed object data loss was demonstrated;
• no security boundary is bypassed;
• unfinished uploads remain on disk until completed, aborted, or collected;
• default stale-upload cleanup bounds accumulation in the ordinary configuration.

It remains high rather than medium because the server returns fabricated success, the answer changes after restart or node switching, and cleanup or quiescence tooling can be misled into false confidence.

Options considered

Option 0: leave the current behavior unchanged

This has no engineering cost and preserves every accidental behavior. It also preserves false 200 OK responses, node-local inconsistency, restart volatility, broken prefix cleanup, and an inaccurate impression of S3 support.

This option is acceptable only if SILO deliberately downgrades the public compatibility claim and treats the endpoint as unsupported. Even then, silently returning an incomplete success is inferior to explicit rejection.

Decision: reject as a long-term position.

Option 1: explicit documented divergence

Reject combinations that SILO cannot honor with a stable NotImplemented-class error and document the exact supported subset. This is operationally honest and much smaller than full compatibility.

It is still a breaking change: tools that currently receive an empty or unbounded 200 OK may begin failing jobs. It also does not produce an S3-compatible endpoint. The error behavior and default release policy must be deliberate.

Decision: acceptable short-term containment if full compatibility is declined or deferred; not a compatibility fix.

Option 2: persist identity, scan durable state, optionally cache

For each new upload, store the bucket and key in reserved internal metadata in the upload’s existing xl.meta. For listing, discover upload directories across accessible pools and sets, validate candidates with erasure read quorum, then run one global S3 semantic layer. A cache may accelerate this path only if it can be rebuilt and reconciled from durable state.

This avoids a second write transaction and keeps the directory layout stable. Its principal cost is the cluster-wide scan.

Decision: recommended, subject to a performance and failure-mode spike.

Option 3: durable bucket-scoped ordered index

Maintain a secondary index ordered by bucket, key, and upload identity. Listing becomes scalable and naturally paginable, but create, complete, abort, healing, rollback, and reconciliation must keep two locations consistent across failures. The design resembles multipart index structures that upstream MinIO deliberately removed while simplifying this subsystem.

Decision: no-go unless measurements show that Option 2 cannot meet the product-approved service-level objective.

Rejected variant: repair only mpCache

Filtering, sorting, paginating, broadcasting creates, or rebuilding the current cache would improve symptoms but would not by itself establish a durable quorum-valid source of truth. A cache-only patch risks producing a more convincing but still incorrect answer.

Decision: reject. A cache can optimize a correct read path, never define it.

Recommended design

1. Freeze the public contract first

Create a recorded AWS fixture suite for general-purpose buckets covering:

• ordering across keys and multiple uploads of one key;
• equal initiation times and a deterministic total-order tie-break;
• prefix and exact-key overlap;
• key-marker with and without upload-id-marker;
• upload-id-marker without key-marker;
• delimiter, CommonPrefixes, and page accounting;
• max-uploads omitted, 0, 1, 1,000, and greater than 1,000;
• encoding-type=url;
• empty pages, final pages, and next-marker values.

The captured responses should become repository fixtures. CI should not depend on live AWS access.

2. Persist recoverable identity in the existing write

At NewMultipartUpload, add reserved internal metadata for the canonical bucket and object key before the existing writeAllMetadata quorum write. The exact key names are an implementation detail, but they must be versioned, unambiguous, size-bounded by the existing object-key limits, and excluded from client-visible metadata.

At successful completion, delete those upload-only keys before copying fi.Metadata into the final object metadata and before renameData. Abort and stale cleanup already delete the entire upload directory and need no separate index operation.

3. Separate discovery from validation

Candidate discovery and candidate validity are different questions.

For every accessible, non-suspended pool and set:

1. list candidate hash and upload directories from all online drives required by the configured list-quorum policy;
2. union and deduplicate those names;
3. read the candidate xl.meta through the normal erasure metadata machinery;
4. include the upload only when its metadata is quorum-valid and contains a valid bucket/key identity;
5. tolerate a candidate disappearing during abort, completion, or stale cleanup;
6. under strict list quorum, fail the request rather than return a partial 200 OK when a required set cannot be evaluated.

Using the first healthy disk for discovery is insufficient: that disk may have been offline when a still-quorum-valid upload was created.

4. Apply semantics once, globally

Feed the validated candidates from all pools and sets into a pure semantic layer. The layer owns bucket filtering, prefix, delimiter grouping, ordering, markers, maximum-page accounting, URL encoding, IsTruncated, and next markers.

Pool-local limits and markers must not be applied before the global merge. The result should be deterministic under duplicate discovery and independent of which node handles the request.

5. Preserve the internal exact-object operation

erasureServerPools.NewMultipartUpload currently calls ListMultipartUploads(bucket, object, ...) to keep another upload for the same object in the same pool. If the public function starts treating that argument as a lexical prefix, foo could match foobar and select the wrong pool.

Introduce a narrowly named internal helper such as FindMultipartUploadPool or ListMultipartUploadsExact. It should use the existing object hash path and must not share the public prefix semantics.

6. Treat cache as an optimization

The existing mpCache may be removed. If retained, it must satisfy all of the following:

• durable state remains authoritative;
• startup can rebuild it;
• create, complete, and abort updates are propagated consistently;
• reconciliation detects missed events and stale entries;
• a cold or divergent cache falls back to the quorum-valid scan;
• correctness tests pass with the cache disabled.

7. Gate strict behavior through rolling migration

Legacy upload records lack bucket/key identity and cannot be reconstructed reliably. Use two externally meaningful modes:

• legacy mode, the initial upgrade default: new writers persist identity; keyless uploads are counted and drained; the documented response policy for a mixed keyed/keyless population must be selected explicitly;
• strict mode: activation requires every writer node to advertise the new metadata capability and the observed keyless count to be zero. Discovering a keyless upload afterward is an error with anomaly telemetry, not a silent omission.

A short shadow comparison can help validate the new scanner, but a permanent third operating mode is unnecessary unless the spike finds a need. With default expiry, the expected legacy drain is about one day plus one cleanup interval after the last old writer stops.

There is one unresolved product choice in legacy mode:

This choice belongs in the ADR. Strict mode has no such ambiguity: it must fail loud if its precondition is violated.

8. Keep suspended-pool lifecycle separate

Listing should initially mirror the accessibility contract of the other multipart verbs and scan non-suspended pools. Adding suspended-pool entries to listing alone would expose uploads that cannot be extended, completed, or aborted.

Open a separate lifecycle design for in-progress uploads when a pool drains: either keep all multipart verbs available until the uploads finish, migrate them, or force-abort them under a documented policy. Do not hide that problem inside #79.

Performance spike and decision rule

Option 2 is preferred because it has one durable write location, but its scan cost must be measured rather than assumed.

Generate 1,000, 10,000, and 100,000 active uploads across a matrix of pools, sets, and drive counts. Measure:

• cold and warm p50/p95/p99 latency;
• total and per-drive ListDir operations;
• metadata-read and internode RPC counts;
• peak memory and allocation volume;
• cancellation latency;
• behavior with slow, offline, healing, and intermittently disappearing drives;
• simultaneous create, complete, abort, and stale cleanup;
• first-page and deep-page cost with selective and empty prefixes.

The acceptance threshold is a product decision and must be recorded before interpreting the result. A guessed one- or two-second target is not evidence. If the scan meets the approved target with bounded resource use, reject Option 3. If it does not, use the measurements to design the smallest durable index that solves the demonstrated bottleneck.

Test and release gates

Semantic and unit tests

• pure table tests generated from recorded AWS fixtures;
• ordering, marker, delimiter, encoding, truncation, and maximum-edge coverage;
• property tests ensuring pagination returns each logical upload exactly once;
• deterministic behavior with duplicate candidates and equal timestamps.

Object and handler tests

• strengthen the existing object-layer table to assert uploads, common prefixes, markers, and truncation;
• parse and validate handler XML bodies instead of checking only status codes;
• verify default and invalid max-uploads handling;
• test exact-helper pool selection independently of public prefix semantics.

Distributed and failure tests

• restart equivalence and node-switch equivalence;
• multiple sets and pools with a single global page boundary;
• candidate missing from one drive but present at quorum;
• minority ghost after partial abort;
• concurrent completion and GC rename-to-trash;
• unavailable set under every supported list_quorum policy;
• rolling upgrade, old-writer reintroduction, downgrade completion, and strict-mode gating;
• unknown internal metadata under healing and replication.

Delivery gates

1. approve the ADR, including product mode and performance SLO;
2. commit the captured conformance fixtures;
3. complete and review the storage spike;
4. implement and pass focused, full, race, and failure QA;
5. update the S3 compatibility reference and operational guidance;
6. commit and merge the source change;
7. build and identify the release artifact or container image;
8. canary a rolling upgrade and observe keyless-drain telemetry;
9. enable strict mode only after its gates hold;
10. verify the live endpoint before closing #79.

Passing an earlier gate is not evidence that a later gate happened.

Final recommendation: fix it, but do not rush it

Leaving the current endpoint indefinitely is the wrong trade-off. This is not an obscure response-field mismatch: it affects discovery and cleanup of unfinished data, returns successful but false answers, and changes behavior across nodes and restarts. Those properties undermine the practical meaning of S3 compatibility.

At the same time, a direct implementation patch is also the wrong trade-off. The current disk layout cannot identify legacy uploads, a correct scan needs erasure-aware discovery and quorum, and wire-correct pagination changes observable client behavior.

The balanced decision is:

• GO for the ADR, AWS fixture capture, metadata-plus-scan prototype, and performance/failure spike;
• GO conditionally for Option 2 after the product SLO and legacy response policy are approved;
• NO-GO for a cache-only repair, an immediate durable secondary index, strict-by-default behavior in a patch release, or closing the issue before rolling-upgrade reachability is demonstrated;
• if implementation capacity is unavailable, GO for an explicit documented divergence and stable error behavior rather than continuing to fabricate successful listings.

This preserves compatibility discipline without pretending that a high-risk distributed listing change is a two-line bug fix.