# Data copy of google_storage_bucket.media to stackit_objectstorage_bucket.media Generated by StackToIT. StackToIT moves no data and opens no connection: this is the preparation, run by a person. The document is written from the steps and the checks of the execution plan (execution-plan.json): each section is one step, in the order of the plan, and every sentence, command and check in it is that step's own, so the document holds nothing the plan does not. A command is a template: a placeholder in angle brackets stands for a name, an endpoint or a credential the plan does not hold, and a password is read from the environment, never pasted. StackToIT runs none of them. Source: `google_storage_bucket.media` (google_storage_bucket). Target: `stackit_objectstorage_bucket.media`. ## Steps | # | Step | Method | Owner | |---|---|---|---| | 1 | Preflight | every method | Data Architect | | 2 | Rehearsal | every method | Data Architect | | 3 | Initial copy | rclone-copy | Data Architect | | 4 | Delta sync | rclone-copy | Data Architect | | 5 | Write freeze | rclone-copy | Migration Owner | | 6 | Final sync | rclone-copy | Data Architect | | 7 | Reconcile | rclone-copy | Data Architect | | 8 | Clean-up | rclone-copy | Data Architect | ## Preflight **Step** `MAN/data/google_storage_bucket.media/preflight` (data preflight, phase DATA); owner: Data Architect. **What.** Close what the plan lists as UNKNOWN, from the move host, before any object moves. The credentials of the target are the environment names TARGET_ACCESS_KEY_ID and TARGET_SECRET_ACCESS_KEY, set in the shell of the move host at run time from the credential of a credentials group; the source uses the Google identity of that shell (the service account the host runs as, or a login you made for the move; rclone's env_auth takes it from the environment). No value is written into a file, a ticket or this plan. In the shell of the move host configure the two remotes from the environment: `export RCLONE_CONFIG_SRC_TYPE="google cloud storage" RCLONE_CONFIG_SRC_ENV_AUTH=true RCLONE_CONFIG_DST_TYPE=s3 RCLONE_CONFIG_DST_PROVIDER=Other RCLONE_CONFIG_DST_ENDPOINT=https://object.storage.eu01.onstackit.cloud RCLONE_CONFIG_DST_ACCESS_KEY_ID="$TARGET_ACCESS_KEY_ID" RCLONE_CONFIG_DST_SECRET_ACCESS_KEY="$TARGET_SECRET_ACCESS_KEY" RCLONE_CONFIG_DST_ACL=private`. Source: `rclone lsd SRC:`, `rclone size SRC: --json` (the count and the bytes), `gcloud storage buckets describe gs://` (versioning, retention policy, default key, requester pays, soft delete policy, storage class), and for the objects the owner names `rclone lsjson --metadata SRC: --files-only -R --max-depth 1`. Target (after the draft is applied, the bucket exists and a key of a credentials group is issued): `rclone lsd DST:` and `rclone size DST: --json`: the bucket is empty. Record the number of objects and the bytes beside the size the customer stated (the size of the data is not provided); the plan has no input for the number of objects. Check that the move host reaches both endpoints. **Why.** the checks of the plan that are UNKNOWN are closed by these calls before any object moves **Needs.** the checks that are UNKNOWN and what closes each: target-bucket-and-credentials (closed by: preflight on the target: `rclone lsd DST:` and `rclone size DST: --json` with that key (the bucket exists and is empty)); source-read-access (closed by: preflight on the source: `rclone lsd SRC:` and `rclone size SRC: --json` with the identity the move host uses, and a read of a sample of the objects the owner names); object-count-and-size (closed by: preflight on the source: `rclone size SRC: --json` for the count and the bytes; the customer states the size under source.dataset.size); encryption (closed by: preflight on the source: `gcloud storage buckets describe gs:// --format='value(default_kms_key)'` and the key of a sample of the objects); object-metadata (closed by: the rehearsal: `rclone lsjson --metadata` of a sample on both sides, compared field by field); requester-pays (closed by: preflight on the source: `gcloud storage buckets describe gs:// --format='value(billing.requester_pays)'`); soft-deleted-objects (closed by: preflight on the source: whether the owner needs deleted objects restored on the target (they are restored in the source first, then copied)); copy-cannot-catch-up (closed by: the customer states source.dataset.size, source.dataset.bandwidth and source.dataset.write_rate; the rehearsal measures the real throughput) **Done when.** every UNKNOWN check of the plan is HOLDS or FAILS, the number of objects and the bytes are recorded, and both buckets answer **Risks.** - the key for the move reaches every bucket of the project until it is removed at the cleanup - check acl-and-policy FAILS: the source states IAM bindings on the bucket: bucket configuration, not object data, so a copy of the objects brings none of it; it is settled where the draft names it, and the consumers' keys reach every bucket of the project until a policy narrows them **Rollback.** nothing changed on either side: read-only calls and, on the target, an empty bucket ## Rehearsal **Step** `MAN/data/google_storage_bucket.media/rehearsal` (data rehearsal, phase DATA); owner: Data Architect. **What.** Rehearse the copy with a representative part of the bucket: the objects of one prefix and the largest and the smallest objects, into the prefix `stacktoit-rehearsal/` of the target bucket. Note the elapsed time and the bytes a second for small and for large objects, compare `rclone lsjson --metadata` of a sample on both sides (content type, custom metadata, storage class, modification time), and delete the prefix `stacktoit-rehearsal/` when the rehearsal is done. **Why.** the transfer lower bound is the wire time only; the cost of each request for small objects, the retries and the throughput towards STACKIT are not computed, only a rehearsal measures them **Needs.** the customer's statements in inputs.yaml: source.dataset.size, source.dataset.bandwidth **Done when.** the elapsed times are recorded, the metadata of the sample is compared and the window of the final pass is agreed from them **Rollback.** delete what the rehearsal wrote; the real bucket and the source are untouched ## Initial copy (rclone-copy) **Step** `MAN/data/google_storage_bucket.media/initial-copy/rclone-copy` (data migration, phase DATA); owner: Data Architect. **What.** While the writers keep writing to the source, copy every live object to the target. The credentials of the target are the environment names TARGET_ACCESS_KEY_ID and TARGET_SECRET_ACCESS_KEY, set in the shell of the move host at run time from the credential of a credentials group; the source uses the Google identity of that shell (the service account the host runs as, or a login you made for the move; rclone's env_auth takes it from the environment). No value is written into a file, a ticket or this plan. With the remotes of the preflight: `rclone copy SRC: DST: --s3-upload-cutoff 0 --progress --log-file=` (the form STACKIT's tutorial uses). Read the log for every error before the next step. **Why.** the copy runs while the application is live, so the freeze later only has to cover the objects that changed since the last pass **Done when.** the command ends without an error and `rclone size SRC: --json` and `rclone size DST: --json` differ only by what was written after the copy started **Risks.** - a copy of the live objects carries no noncurrent version - the log names every object key **Rollback.** delete the copied objects from the target bucket; the source is untouched ## Delta sync (rclone-copy) **Step** `MAN/data/google_storage_bucket.media/delta-sync/rclone-copy` (data delta sync, phase DATA); owner: Data Architect. **What.** Run the same `rclone copy` again, as often as the business needs the target to be close to the source: each pass skips the objects that are identical (rclone compares size and modification time; `--checksum` compares size and hash where there is one, and Cloud Storage stores the MD5 natively) and copies the ones that are new or changed. It deletes nothing on the target, so an object deleted in the source stays there until the final pass. Note how many objects and bytes each pass transferred. **Why.** each pass is shorter than the one before while the write rate stays below the bandwidth; the last pass is the one that runs inside the freeze **Needs.** the customer's statements in inputs.yaml: source.dataset.write_rate **Done when.** the passes get shorter until a pass is as short as the freeze window the business accepts **Risks.** - a pass that does not get shorter means the copy does not catch up at this write rate **Rollback.** as the initial copy: delete the copied objects from the target; the source is untouched ## Write freeze (rclone-copy) **Step** `MAN/data/google_storage_bucket.media/write-freeze/rclone-copy` (data write freeze, phase CUTOVER); owner: Migration Owner. **What.** List who writes to the bucket: the units the README lists for it, and every job, partner and signed URL outside the scan (if Cloud Audit Logs record data writes for the bucket, its object writes and deletes name the principals; the Terraform does not show whether they are). Record the endpoint, the bucket name and the credential settings of every client where they can be restored from. Then stop the writers at the application (services, jobs, uploads). Do not change the source bucket (no read-only policy, no retention lock): it stays read-write, so the way back needs nothing on the source. **Why.** the source stays untouched, so the way back needs nothing on the source **Needs.** the customer's statements in inputs.yaml: source.dataset.downtime **Done when.** no client writes to the source **Rollback.** start the writers again against the source: nothing was changed ## Final sync (rclone-copy) **Step** `MAN/data/google_storage_bucket.media/final-sync/rclone-copy` (data final sync, phase CUTOVER); owner: Data Architect. **What.** Inside the freeze run the `rclone copy` of the initial copy once more and read how many objects it transferred; run it a second time: it transfers nothing. Then see what the target holds that the source no longer has: `rclone sync SRC: DST: --dry-run` prints what a sync would delete; read the list, and only a person who has read it runs the sync without `--dry-run` (the target must equal the source at the cutover, and an object deleted in the source after the initial copy is otherwise still served). **Why.** the pass inside the freeze copies the last changes; the dry run is the only place where the deletions of the source after the initial copy are seen **Needs.** the customer's statements in inputs.yaml: source.dataset.downtime **Done when.** the second pass transfers nothing and every object that exists only on the target is settled **Risks.** - the freeze lasts as long as the last pass and the settlement of the deletions **Rollback.** the source is current to the freeze and untouched: put the writers back on it ## Reconcile (rclone-copy) **Step** `MAN/data/google_storage_bucket.media/reconcile/rclone-copy` (data reconcile, phase CUTOVER); owner: Data Architect. **What.** Both sides, with the writers stopped: the number of objects and the bytes (`rclone size SRC: --json`, `rclone size DST: --json`); the list of keys (`rclone lsf -R --files-only SRC: | sort > source-keys.txt`, the same for the target, `diff source-keys.txt target-keys.txt` is empty); and the content: `rclone check SRC: DST: --one-way --combined=` compares size and hash where both sides have one and marks every difference, so read the file and the log; an object for which the two sides share no hash (the target's ETag of a multipart upload is not an MD5) is at best compared by size, which is not its content, so run `rclone check ... --download` for the prefixes that matter and for a sample of the rest. One read that the application itself does on its most important objects through the STACKIT endpoint with a key of a credentials group: ``, the query and the answer are the customer's, the source does not state them. Record GO or NO-GO. **Why.** the move is accepted only when the inventory and the content agree on both sides; a mismatch is a NO-GO **Done when.** the counts, the key lists and the checks agree, the application read works: GO **Risks.** - a hash that rclone does not have for an object leaves that object compared by size only **Rollback.** on NO-GO take the rollback without switching anything: the clients still point at the source, which is current to the freeze ### Verification: the checks after the transfer These are the checks the owner makes with the writers stopped before recording GO; the reconcile step in the execution plan states the same checks, with the same ids. A check with no command says why: the plan states no command it cannot back. - **Count on both sides** (`object-count-and-bytes`, command): Count the objects and the bytes of both buckets with the writers stopped. Expected: the number of objects and the bytes are equal on both sides. ~~~text rclone size SRC: --json rclone size DST: --json ~~~ - **Count on both sides** (`key-lists`, command): List the keys of both buckets, sort them and compare the lists. Expected: the difference of the two lists is empty. ~~~text rclone lsf -R --files-only SRC: | sort > source-keys.txt rclone lsf -R --files-only DST: | sort > target-keys.txt diff source-keys.txt target-keys.txt ~~~ - **Content on both sides** (`content`, command): Compare size and hash where both sides have one, and read the combined file and the log for every difference; an object for which the two sides share no hash is at best compared by size, which is not its content, so run the check with --download for the prefixes that matter and for a sample of the rest. Expected: the combined file marks no difference. ~~~text rclone check SRC: DST: --one-way --combined= ~~~ - Not provided, so left as a placeholder or for the owner to fill in: the prefixes that matter; the file the result is written to (). - **Application check** (`application-check`, manual): Read the objects the application itself reads on its most important data through the STACKIT endpoint with a key of a credentials group, and compare them with the source; then write one object from a test copy of the application and read it back. Expected: the reads return the same objects, the write succeeds and the read returns it. - No command: the read and the write are the application's, and the source states neither the keys nor the answer. - Not provided, so left as a placeholder or for the owner to fill in: the read the application makes on its most important objects (); the write the test copy of the application makes. - **Rollback limit** (`rollback-limit`, manual): Record the point after which a rollback loses data. Until the first of the uploads that the target bucket accepts after the switch, a rollback is only the connection settings put back: the source bucket was not changed after the freeze and is current to it. From that write on, what the target bucket holds exists only there: it is copied back to the source bucket with a method rehearsed before the window, or accepted as lost. Once the source bucket is retired (its retirement step, after the GO and the retention the owner decides) there is nothing to go back to. Expected: before the window the owner records the date until which the source bucket is kept, the method that copies the writes of the target bucket back or the decision that they are lost, and who calls the rollback. - No command: how long the source is kept and whether the writes made on the target are copied back or lost are decisions of the owner of the application: the source states neither, so the plan names the limit and no date. - Not provided, so left as a placeholder or for the owner to fill in: the date until which the source bucket is kept; the method that copies the writes of the target bucket back, or the decision that they are lost. ## Clean-up (rclone-copy) **Step** `MAN/data/google_storage_bucket.media/cleanup/rclone-copy` (data cleanup, phase DECOMMISSION); owner: Data Architect. **What.** Only after the target has run in production for the period the customer decides and the way back is no longer wanted: delete the key lists and the logs of the move (they name every object key); delete the credential made for the move, the Google login or key made for it, and take the environment names out of the shell history; keep the source bucket, with its versions, for the retention the customer decides before any object of it is deleted, and do not delete it in the same hour as the cutover. **Why.** the move leaves key lists, logs, two credentials and an environment behind; none of them stays **When.** after the owner records GO at the end of the verification, and once the way back is no longer wanted **Done when.** no key list or log is left, the credentials of the move are removed, and the retention of the source is recorded **Risks.** - the source is not deleted in the same hour as the cutover **Rollback.** after the cleanup the way back is the retained source bucket; there is none once it is deleted