LuxorItPro

Sheet 10/16Storage infrastructureIndependent project

S3-compatible storage synchronisation

A sync utility that shows a plan before it moves anything, and can resume a run that failed half way.

Fig. 01 How a sync run is planned before it writes
How a sync run is planned before it writes How a sync run is planned before it writes. Dry run (Both endpoints compared). Then Planned operations (Listed before any write). Then Executing run (Manifest written as it goes). Then Resume or stop (Continue or trip the breaker). Arrows show the write path between components; a dashed arrow marks data derived from another component. Dry run BOTH ENDPOINTS COMPARED Planned operations LISTED BEFORE ANY WRITE Executing run MANIFEST WRITTEN AS IT GOES Resume or stop CONTINUE OR TRIP THE BREAKER

Drawn from the Approach section. The dry run is the default first question: what would this actually do?

Problem

Copying data between buckets is easy to start and hard to trust. A command that takes an hour gives no useful answer until it finishes, a failure in the middle leaves an unknown half-state, and nobody can say afterwards exactly what moved. The usual response is to run it again and hope, which for storage means duplicating objects and paying for the privilege.

Our role

Design and implementation, including the failure model.

Approach

The tool separates planning from execution. A dry run compares two endpoints and produces the operations it would perform, so the first question — what would this actually do — is answered before anything is written.

An executing run writes a manifest, which makes it resumable: a run that stops records where it stopped, and the next invocation continues from the manifest rather than from a full re-scan. Failure handling is explicit, with a circuit-breaker pattern so a failing endpoint is not hammered by the remainder of a large batch. A Cloudflare Worker entry point allows the same logic to run close to the storage instead of on a workstation.

Outcome

The planning, manifest, and resumable execution paths are implemented. Reliability and data-protection claims need restore testing against each supported provider first, so this stays tooling rather than a backup guarantee.

Notes

No bucket names, credentials, or endpoints are published. The patterns are the part worth describing; the configuration is not.

All case studies Start a project