data-copy.md

Data copy of google_storage_bucket.media to stackit_objectstorage_bucket.media

Generated by StackToIT. StackToIT moves no data and opens no connection: this is the preparation, run by a person. The document is written from the steps and the checks of the execution plan (execution-plan.json): each section is one step, in the order of the plan, and every sentence, command and check in it is that step's own, so the document holds nothing the plan does not. A command is a template: a placeholder in angle brackets stands for a name, an endpoint or a credential the plan does not hold, and a password is read from the environment, never pasted. StackToIT runs none of them.

Source: google_storage_bucket.media (google_storage_bucket). Target: stackit_objectstorage_bucket.media.

Steps

Preflight

Step MAN/data/google_storage_bucket.media/preflight (data preflight, phase DATA); owner: Data Architect.

What. Close what the plan lists as UNKNOWN, from the move host, before any object moves. The credentials of the target are the environment names TARGET_ACCESS_KEY_ID and TARGET_SECRET_ACCESS_KEY, set in the shell of the move host at run time from the credential of a credentials group; the source uses the Google identity of that shell (the service account the host runs as, or a login you made for the move; rclone's env_auth takes it from the environment). No value is written into a file, a ticket or this plan. In the shell of the move host configure the two remotes from the environment: export RCLONE_CONFIG_SRC_TYPE="google cloud storage" RCLONE_CONFIG_SRC_ENV_AUTH=true RCLONE_CONFIG_DST_TYPE=s3 RCLONE_CONFIG_DST_PROVIDER=Other RCLONE_CONFIG_DST_ENDPOINT=https://object.storage.eu01.onstackit.cloud RCLONE_CONFIG_DST_ACCESS_KEY_ID="$TARGET_ACCESS_KEY_ID" RCLONE_CONFIG_DST_SECRET_ACCESS_KEY="$TARGET_SECRET_ACCESS_KEY" RCLONE_CONFIG_DST_ACL=private. Source: rclone lsd SRC:, rclone size SRC:<source-bucket> --json (the count and the bytes), gcloud storage buckets describe gs://<source-bucket> (versioning, retention policy, default key, requester pays, soft delete policy, storage class), and for the objects the owner names rclone lsjson --metadata SRC:<source-bucket> --files-only -R --max-depth 1. Target (after the draft is applied, the bucket exists and a key of a credentials group is issued): rclone lsd DST: and rclone size DST:<target-bucket> --json: the bucket is empty. Record the number of objects and the bytes beside the size the customer stated (the size of the data is not provided); the plan has no input for the number of objects. Check that the move host reaches both endpoints.

Why. the checks of the plan that are UNKNOWN are closed by these calls before any object moves

Needs. the checks that are UNKNOWN and what closes each: target-bucket-and-credentials (closed by: preflight on the target: rclone lsd DST: and rclone size DST:<target-bucket> --json with that key (the bucket exists and is empty)); source-read-access (closed by: preflight on the source: rclone lsd SRC: and rclone size SRC:<source-bucket> --json with the identity the move host uses, and a read of a sample of the objects the owner names); object-count-and-size (closed by: preflight on the source: rclone size SRC:<source-bucket> --json for the count and the bytes; the customer states the size under source.dataset.size); encryption (closed by: preflight on the source: gcloud storage buckets describe gs://<source-bucket> --format='value(default_kms_key)' and the key of a sample of the objects); object-metadata (closed by: the rehearsal: rclone lsjson --metadata of a sample on both sides, compared field by field); requester-pays (closed by: preflight on the source: gcloud storage buckets describe gs://<source-bucket> --format='value(billing.requester_pays)'); soft-deleted-objects (closed by: preflight on the source: whether the owner needs deleted objects restored on the target (they are restored in the source first, then copied)); copy-cannot-catch-up (closed by: the customer states source.dataset.size, source.dataset.bandwidth and source.dataset.write_rate; the rehearsal measures the real throughput)

Done when. every UNKNOWN check of the plan is HOLDS or FAILS, the number of objects and the bytes are recorded, and both buckets answer

Risks.

• the key for the move reaches every bucket of the project until it is removed at the cleanup
• check acl-and-policy FAILS: the source states IAM bindings on the bucket: bucket configuration, not object data, so a copy of the objects brings none of it; it is settled where the draft names it, and the consumers' keys reach every bucket of the project until a policy narrows them

Rollback. nothing changed on either side: read-only calls and, on the target, an empty bucket

Rehearsal

Step MAN/data/google_storage_bucket.media/rehearsal (data rehearsal, phase DATA); owner: Data Architect.

What. Rehearse the copy with a representative part of the bucket: the objects of one prefix and the largest and the smallest objects, into the prefix stacktoit-rehearsal/ of the target bucket. Note the elapsed time and the bytes a second for small and for large objects, compare rclone lsjson --metadata of a sample on both sides (content type, custom metadata, storage class, modification time), and delete the prefix stacktoit-rehearsal/ when the rehearsal is done.

Why. the transfer lower bound is the wire time only; the cost of each request for small objects, the retries and the throughput towards STACKIT are not computed, only a rehearsal measures them

Needs. the customer's statements in inputs.yaml: source.dataset.size, source.dataset.bandwidth

Done when. the elapsed times are recorded, the metadata of the sample is compared and the window of the final pass is agreed from them

Rollback. delete what the rehearsal wrote; the real bucket and the source are untouched

Initial copy (rclone-copy)

Step MAN/data/google_storage_bucket.media/initial-copy/rclone-copy (data migration, phase DATA); owner: Data Architect.

What. While the writers keep writing to the source, copy every live object to the target. The credentials of the target are the environment names TARGET_ACCESS_KEY_ID and TARGET_SECRET_ACCESS_KEY, set in the shell of the move host at run time from the credential of a credentials group; the source uses the Google identity of that shell (the service account the host runs as, or a login you made for the move; rclone's env_auth takes it from the environment). No value is written into a file, a ticket or this plan. With the remotes of the preflight: rclone copy SRC:<source-bucket> DST:<target-bucket> --s3-upload-cutoff 0 --progress --log-file=<log-file> (the form STACKIT's tutorial uses). Read the log for every error before the next step.

Why. the copy runs while the application is live, so the freeze later only has to cover the objects that changed since the last pass

Done when. the command ends without an error and rclone size SRC:<source-bucket> --json and rclone size DST:<target-bucket> --json differ only by what was written after the copy started

Risks.

• a copy of the live objects carries no noncurrent version
• the log names every object key

Rollback. delete the copied objects from the target bucket; the source is untouched

Delta sync (rclone-copy)

Step MAN/data/google_storage_bucket.media/delta-sync/rclone-copy (data delta sync, phase DATA); owner: Data Architect.

What. Run the same rclone copy again, as often as the business needs the target to be close to the source: each pass skips the objects that are identical (rclone compares size and modification time; --checksum compares size and hash where there is one, and Cloud Storage stores the MD5 natively) and copies the ones that are new or changed. It deletes nothing on the target, so an object deleted in the source stays there until the final pass. Note how many objects and bytes each pass transferred.

Why. each pass is shorter than the one before while the write rate stays below the bandwidth; the last pass is the one that runs inside the freeze

Needs. the customer's statements in inputs.yaml: source.dataset.write_rate

Done when. the passes get shorter until a pass is as short as the freeze window the business accepts

Risks.

• a pass that does not get shorter means the copy does not catch up at this write rate

Rollback. as the initial copy: delete the copied objects from the target; the source is untouched

Write freeze (rclone-copy)

Step MAN/data/google_storage_bucket.media/write-freeze/rclone-copy (data write freeze, phase CUTOVER); owner: Migration Owner.

What. List who writes to the bucket: the units the README lists for it, and every job, partner and signed URL outside the scan (if Cloud Audit Logs record data writes for the bucket, its object writes and deletes name the principals; the Terraform does not show whether they are). Record the endpoint, the bucket name and the credential settings of every client where they can be restored from. Then stop the writers at the application (services, jobs, uploads). Do not change the source bucket (no read-only policy, no retention lock): it stays read-write, so the way back needs nothing on the source.

Why. the source stays untouched, so the way back needs nothing on the source

Needs. the customer's statements in inputs.yaml: source.dataset.downtime

Done when. no client writes to the source

Rollback. start the writers again against the source: nothing was changed

Final sync (rclone-copy)

Step MAN/data/google_storage_bucket.media/final-sync/rclone-copy (data final sync, phase CUTOVER); owner: Data Architect.

What. Inside the freeze run the rclone copy of the initial copy once more and read how many objects it transferred; run it a second time: it transfers nothing. Then see what the target holds that the source no longer has: rclone sync SRC:<source-bucket> DST:<target-bucket> --dry-run prints what a sync would delete; read the list, and only a person who has read it runs the sync without --dry-run (the target must equal the source at the cutover, and an object deleted in the source after the initial copy is otherwise still served).

Why. the pass inside the freeze copies the last changes; the dry run is the only place where the deletions of the source after the initial copy are seen

Needs. the customer's statements in inputs.yaml: source.dataset.downtime

Done when. the second pass transfers nothing and every object that exists only on the target is settled

Risks.

• the freeze lasts as long as the last pass and the settlement of the deletions

Rollback. the source is current to the freeze and untouched: put the writers back on it

Reconcile (rclone-copy)

Step MAN/data/google_storage_bucket.media/reconcile/rclone-copy (data reconcile, phase CUTOVER); owner: Data Architect.

What. Both sides, with the writers stopped: the number of objects and the bytes (rclone size SRC:<source-bucket> --json, rclone size DST:<target-bucket> --json); the list of keys (rclone lsf -R --files-only SRC:<source-bucket> | sort > source-keys.txt, the same for the target, diff source-keys.txt target-keys.txt is empty); and the content: rclone check SRC:<source-bucket> DST:<target-bucket> --one-way --combined=<check-file> compares size and hash where both sides have one and marks every difference, so read the file and the log; an object for which the two sides share no hash (the target's ETag of a multipart upload is not an MD5) is at best compared by size, which is not its content, so run rclone check ... --download for the prefixes that matter and for a sample of the rest. One read that the application itself does on its most important objects through the STACKIT endpoint with a key of a credentials group: <application-read>, the query and the answer are the customer's, the source does not state them. Record GO or NO-GO.

Why. the move is accepted only when the inventory and the content agree on both sides; a mismatch is a NO-GO

Done when. the counts, the key lists and the checks agree, the application read works: GO

Risks.

• a hash that rclone does not have for an object leaves that object compared by size only

Rollback. on NO-GO take the rollback without switching anything: the clients still point at the source, which is current to the freeze

Verification: the checks after the transfer

These are the checks the owner makes with the writers stopped before recording GO; the reconcile step in the execution plan states the same checks, with the same ids. A check with no command says why: the plan states no command it cannot back.

• Count on both sides (object-count-and-bytes, command): Count the objects and the bytes of both buckets with the writers stopped. Expected: the number of objects and the bytes are equal on both sides.

~~~text rclone size SRC:<source-bucket> --json rclone size DST:<target-bucket> --json ~~~

• Count on both sides (key-lists, command): List the keys of both buckets, sort them and compare the lists. Expected: the difference of the two lists is empty.

~~~text rclone lsf -R --files-only SRC:<source-bucket> | sort > source-keys.txt rclone lsf -R --files-only DST:<target-bucket> | sort > target-keys.txt diff source-keys.txt target-keys.txt ~~~

• Content on both sides (content, command): Compare size and hash where both sides have one, and read the combined file and the log for every difference; an object for which the two sides share no hash is at best compared by size, which is not its content, so run the check with --download for the prefixes that matter and for a sample of the rest. Expected: the combined file marks no difference.

~~~text rclone check SRC:<source-bucket> DST:<target-bucket> --one-way --combined=<check-file> ~~~

• Not provided, so left as a placeholder or for the owner to fill in: the prefixes that matter; the file the result is written to (<check-file>).

• Application check (application-check, manual): Read the objects the application itself reads on its most important data through the STACKIT endpoint with a key of a credentials group, and compare them with the source; then write one object from a test copy of the application and read it back. Expected: the reads return the same objects, the write succeeds and the read returns it. - No command: the read and the write are the application's, and the source states neither the keys nor the answer. - Not provided, so left as a placeholder or for the owner to fill in: the read the application makes on its most important objects (<application-read>); the write the test copy of the application makes.
• Rollback limit (rollback-limit, manual): Record the point after which a rollback loses data. Until the first of the uploads that the target bucket accepts after the switch, a rollback is only the connection settings put back: the source bucket was not changed after the freeze and is current to it. From that write on, what the target bucket holds exists only there: it is copied back to the source bucket with a method rehearsed before the window, or accepted as lost. Once the source bucket is retired (its retirement step, after the GO and the retention the owner decides) there is nothing to go back to. Expected: before the window the owner records the date until which the source bucket is kept, the method that copies the writes of the target bucket back or the decision that they are lost, and who calls the rollback. - No command: how long the source is kept and whether the writes made on the target are copied back or lost are decisions of the owner of the application: the source states neither, so the plan names the limit and no date. - Not provided, so left as a placeholder or for the owner to fill in: the date until which the source bucket is kept; the method that copies the writes of the target bucket back, or the decision that they are lost.

Clean-up (rclone-copy)

Step MAN/data/google_storage_bucket.media/cleanup/rclone-copy (data cleanup, phase DECOMMISSION); owner: Data Architect.

What. Only after the target has run in production for the period the customer decides and the way back is no longer wanted: delete the key lists and the logs of the move (they name every object key); delete the credential made for the move, the Google login or key made for it, and take the environment names out of the shell history; keep the source bucket, with its versions, for the retention the customer decides before any object of it is deleted, and do not delete it in the same hour as the cutover.

Why. the move leaves key lists, logs, two credentials and an environment behind; none of them stays

When. after the owner records GO at the end of the verification, and once the way back is no longer wanted

Done when. no key list or log is left, the credentials of the move are removed, and the retention of the source is recorded

Risks.

• the source is not deleted in the same hour as the cutover

Rollback. after the cleanup the way back is the retained source bucket; there is none once it is deleted