data-move.md

Data move of module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this to stackit_opensearch_instance.dependencies_catalog_opensearch_this

Generated by StackToIT. StackToIT moves no data and opens no connection: this is the preparation, run by a person. The document is written from the steps and the checks of the execution plan (execution-plan.json): each section is one step, in the order of the plan, and every sentence, command and check in it is that step's own, so the document holds nothing the plan does not. A command is a template: a placeholder in angle brackets stands for a name, an endpoint or a credential the plan does not hold, and a password is read from the environment, never pasted. StackToIT runs none of them.

Source: module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this (aws_opensearch_domain). Target: stackit_opensearch_instance.dependencies_catalog_opensearch_this.

Steps

Write freeze (client-import)

Step MAN/data/module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this/write-freeze/client-import (data write freeze, phase CUTOVER); owner: Migration Owner.

What. Freeze the writers of module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this: list who writes, record the connection settings of every client, then stop the services, jobs and migrations that write; do not change the source

Why. the last copy and the comparison of the two sides are made with the writers stopped, and the source stays untouched so the way back needs nothing on the source

Done when. no client writes to the source

Method. This step belongs to the method export-and-import-with-the-engine-clients of the decision DEC-GEN-CONVERT-DATA-MOVEMENT: the plan carries it out only when that decision is answered with it; until it is answered, the steps of every open method are listed.

Rollback. start the writers again against the source: nothing was changed

Final sync (client-import)

Step MAN/data/module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this/final-sync/client-import (data final sync, phase CUTOVER); owner: Data Architect.

What. Inside the freeze, copy the indices of module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this to stackit_opensearch_instance.dependencies_catalog_opensearch_this with the reindex operation of the service, as the OpenSearch how-to "Migrate data from Elasticsearch to OpenSearch 2" describes

Why. the draft creates the instance empty and the service documents the reindex operation as the way in; the plan does not know the indices, their size or the access between the source and the target

Needs. the indices to move, their size and the access of the target to the source

Done when. every index of module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this is in stackit_opensearch_instance.dependencies_catalog_opensearch_this and the reindex reported no failure

Risks.

• the how-to of the service has an Elasticsearch instance as the source of the reindex: an OpenSearch domain as the source of the same operation is an inference, not a statement of the page

Method. This step belongs to the method export-and-import-with-the-engine-clients of the decision DEC-GEN-CONVERT-DATA-MOVEMENT: the plan carries it out only when that decision is answered with it; until it is answered, the steps of every open method are listed.

Rollback. the writers are stopped, not removed, and a reindex reads the source, so the source is the way back until the reconcile records GO: put the writers back on it; before another attempt delete the indices the reindex wrote into stackit_opensearch_instance.dependencies_catalog_opensearch_this

Reconcile (client-import)

Step MAN/data/module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this/reconcile/client-import (data reconcile, phase CUTOVER); owner: Data Architect.

What. Reconcile module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this and stackit_opensearch_instance.dependencies_catalog_opensearch_this with the writers stopped (the number of documents of every index on each side, a checksum or a sample of their content, and one read the application does) and record GO or NO-GO

Why. the copy is made by a tool the plan knows nothing of, so the plan states no check of that tool; the comparison of the two sides with the writers stopped is the check that does not depend on it

Done when. the records and the sample match on both sides, the read the application does returns the same on both, and the owner records GO

Method. This step belongs to the method export-and-import-with-the-engine-clients of the decision DEC-GEN-CONVERT-DATA-MOVEMENT: the plan carries it out only when that decision is answered with it; until it is answered, the steps of every open method are listed.

Rollback. NO-GO: start the writers again against the untouched source; the target holds only a copy

Verification: the checks after the transfer

These are the checks the owner makes with the writers stopped before recording GO; the reconcile step in the execution plan states the same checks, with the same ids. A check with no command says why: the plan states no command it cannot back.

• Count on both sides (counts, manual): Count what the target store holds and what the source holds, with the writers stopped: the number of records, objects or keys, in the unit the store counts in. Expected: the two counts are equal, or every difference is explained and recorded. - No command: the plan holds no command of this store's own tools that it can state for the comparison (the store is moved by a tool of your own or has no family in the plan), so it states what is compared and not how.
• Content on both sides (content, manual): Compare the content of the target store with the source with the writers stopped: a checksum or a sample of the records that matter, and the ones written last. Expected: the checksums or the sampled records are equal on both sides. - No command: the plan holds no command of this store's own tools that it can state for the comparison (the store is moved by a tool of your own or has no family in the plan), so it states what is compared and not how.
• Application check (application-check, manual): Run one read the application itself makes on its most important data against the target store and compare its answer with the answer the application gets from the source; then, from a test copy of the application, make one write against the target store and read it back. Expected: the two answers are equal, the write succeeds and the read returns it. - No command: the read and the write are the application's, and the source states neither the query nor the answer. - Not provided, so left as a placeholder or for the owner to fill in: the read the application makes on its most important data and its expected answer; the write the test copy of the application makes.
• Rollback limit (rollback-limit, manual): Record the point after which a rollback loses data. Until the first of the writes that the target store accepts after the switch, a rollback is only the connection settings put back: the source was not changed after the freeze and is current to it. From that write on, what the target store holds exists only there: it is copied back to the source with a method rehearsed before the window, or accepted as lost. Once the source is retired (its retirement step, after the GO and the retention the owner decides) there is nothing to go back to. Expected: before the window the owner records the date until which the source is kept, the method that copies the writes of the target store back or the decision that they are lost, and who calls the rollback. - No command: how long the source is kept and whether the writes made on the target are copied back or lost are decisions of the owner of the application: the source states neither, so the plan names the limit and no date. - Not provided, so left as a placeholder or for the owner to fill in: the date until which the source is kept; the method that copies the writes of the target store back, or the decision that they are lost.

Write freeze (own-tool)

Step MAN/data/module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this/write-freeze/own-tool (data write freeze, phase CUTOVER); owner: Migration Owner.

What. Freeze the writers of module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this: list who writes, record the connection settings of every client, then stop the services, jobs and migrations that write; do not change the source

Why. the last copy and the comparison of the two sides are made with the writers stopped, and the source stays untouched so the way back needs nothing on the source

Done when. no client writes to the source

Method. This step belongs to the method migrate-with-a-tool-of-my-own of the decision DEC-GEN-CONVERT-DATA-MOVEMENT: the plan carries it out only when that decision is answered with it; until it is answered, the steps of every open method are listed.

Rollback. start the writers again against the source: nothing was changed

Final sync (own-tool)

Step MAN/data/module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this/final-sync/own-tool (data final sync, phase CUTOVER); owner: Data Architect.

What. Copy the data of module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this to stackit_opensearch_instance.dependencies_catalog_opensearch_this with the migration tool or the replication you chose: write its steps, the checks before the move, the freeze window it needs and the way back

Why. the draft creates the instance empty and the plan knows nothing of a tool of your own, so it holds the step and the person who chose the tool writes the steps

Needs. the tool, its steps, the checks before the move, the freeze window and the way back, from the person who runs it

Done when. the data of module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this is in stackit_opensearch_instance.dependencies_catalog_opensearch_this and the checks of the tool hold

Method. This step belongs to the method migrate-with-a-tool-of-my-own of the decision DEC-GEN-CONVERT-DATA-MOVEMENT: the plan carries it out only when that decision is answered with it; until it is answered, the steps of every open method are listed.

Rollback. the writers are stopped, not removed, and nothing the plan does changes the source, so the source is the way back until the reconcile records GO: put the writers back on it; before another attempt remove the indices the tool wrote into stackit_opensearch_instance.dependencies_catalog_opensearch_this (the draft created it empty); check that the tool you chose does not change the source, since the plan knows nothing of it

Reconcile (own-tool)

Step MAN/data/module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this/reconcile/own-tool (data reconcile, phase CUTOVER); owner: Data Architect.

What. Reconcile module.dependencies.module.catalog_opensearch.aws_opensearch_domain.this and stackit_opensearch_instance.dependencies_catalog_opensearch_this with the writers stopped (the number of documents of every index on each side, a checksum or a sample of their content, and one read the application does) and record GO or NO-GO

Why. the copy is made by a tool the plan knows nothing of, so the plan states no check of that tool; the comparison of the two sides with the writers stopped is the check that does not depend on it

Done when. the records and the sample match on both sides, the read the application does returns the same on both, and the owner records GO

Method. This step belongs to the method migrate-with-a-tool-of-my-own of the decision DEC-GEN-CONVERT-DATA-MOVEMENT: the plan carries it out only when that decision is answered with it; until it is answered, the steps of every open method are listed.

Rollback. NO-GO: start the writers again against the untouched source; the target holds only a copy

Verification: the checks after the transfer

These are the checks the owner makes with the writers stopped before recording GO; the reconcile step in the execution plan states the same checks, with the same ids. A check with no command says why: the plan states no command it cannot back.

• Count on both sides (counts, manual): Count what the target store holds and what the source holds, with the writers stopped: the number of records, objects or keys, in the unit the store counts in. Expected: the two counts are equal, or every difference is explained and recorded. - No command: the plan holds no command of this store's own tools that it can state for the comparison (the store is moved by a tool of your own or has no family in the plan), so it states what is compared and not how.
• Content on both sides (content, manual): Compare the content of the target store with the source with the writers stopped: a checksum or a sample of the records that matter, and the ones written last. Expected: the checksums or the sampled records are equal on both sides. - No command: the plan holds no command of this store's own tools that it can state for the comparison (the store is moved by a tool of your own or has no family in the plan), so it states what is compared and not how.
• Application check (application-check, manual): Run one read the application itself makes on its most important data against the target store and compare its answer with the answer the application gets from the source; then, from a test copy of the application, make one write against the target store and read it back. Expected: the two answers are equal, the write succeeds and the read returns it. - No command: the read and the write are the application's, and the source states neither the query nor the answer. - Not provided, so left as a placeholder or for the owner to fill in: the read the application makes on its most important data and its expected answer; the write the test copy of the application makes.
• Rollback limit (rollback-limit, manual): Record the point after which a rollback loses data. Until the first of the writes that the target store accepts after the switch, a rollback is only the connection settings put back: the source was not changed after the freeze and is current to it. From that write on, what the target store holds exists only there: it is copied back to the source with a method rehearsed before the window, or accepted as lost. Once the source is retired (its retirement step, after the GO and the retention the owner decides) there is nothing to go back to. Expected: before the window the owner records the date until which the source is kept, the method that copies the writes of the target store back or the decision that they are lost, and who calls the rollback. - No command: how long the source is kept and whether the writes made on the target are copied back or lost are decisions of the owner of the application: the source states neither, so the plan names the limit and no date. - Not provided, so left as a placeholder or for the owner to fill in: the date until which the source is kept; the method that copies the writes of the target store back, or the decision that they are lost.