data-move.md

Data move of module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this to stackit_mongodbflex_instance.dynamodb_tables

Generated by StackToIT. StackToIT moves no data and opens no connection: this is the preparation, run by a person. The document is written from the steps and the checks of the execution plan (execution-plan.json): each section is one step, in the order of the plan, and every sentence, command and check in it is that step's own, so the document holds nothing the plan does not. A command is a template: a placeholder in angle brackets stands for a name, an endpoint or a credential the plan does not hold, and a password is read from the environment, never pasted. StackToIT runs none of them.

Source: module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this (aws_dynamodb_table). Target: stackit_mongodbflex_instance.dynamodb_tables.

Steps

Write freeze (own-tool)

Step MAN/data/module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this/write-freeze/own-tool (data write freeze, phase CUTOVER); owner: Migration Owner.

What. Freeze the writers of module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this: list who writes, record the connection settings of every client, then stop the services, jobs and migrations that write; do not change the source

Why. the last copy and the comparison of the two sides are made with the writers stopped, and the source stays untouched so the way back needs nothing on the source

Done when. no client writes to the source

Method. This step belongs to the method migrate-with-a-tool-of-my-own of the decision DEC-GEN-CONVERT-DATA-MOVEMENT: the plan carries it out only when that decision is answered with it; until it is answered, the steps of every open method are listed.

Rollback. start the writers again against the source: nothing was changed

Export (own-tool)

Step MAN/data/module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this/export/own-tool (data migration, phase CUTOVER); owner: Data Architect.

What. Inside the freeze, export the table retail-store-ecs-carts (module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this) to S3 with the DynamoDB export to S3, which requires point-in-time recovery, and the source does not state point-in-time recovery for this table (the provider's default is off): a request for a table without it fails with PointInTimeRecoveryUnavailableException. Switch recovery on first (a change to the AWS table that this plan does not make) and request the export only after it is on, using aws dynamodb export-table-to-point-in-time --table-arn <table ARN> --s3-bucket <bucket> --s3-prefix <prefix> --export-format DYNAMODB_JSON; keep the manifest-summary.json (its itemCount is the count the comparison uses). Without the export, read the table in full with Scan, which consumes the read capacity of the table, and count the items you write out

Why. the writers are stopped, so the export holds the last write; the export to S3 reads the table's continuous backups and consumes none of its read capacity, and the plan holds no other way to read a table in full

Needs. the S3 bucket the export is written to and the account that may write to it, the table ARN, the size of the table and the time the business accepts the freeze to last

Done when. the export of the table retail-store-ecs-carts (module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this) is COMPLETED and its manifest-summary.json states the item count that the comparison uses

Risks.

• no service level agreement guarantees the completion time of an export (AWS: How export to S3 works), so the plan cannot compute how long the freeze lasts: it lasts at least as long as the export, the conversion and the import together
• an export is not transactionally consistent: a transaction can be torn between two exports (AWS: Requesting a table export); with the writers stopped before the export no transaction is in flight, which is why the export is inside the freeze and not before it

Method. This step belongs to the method migrate-with-a-tool-of-my-own of the decision DEC-GEN-CONVERT-DATA-MOVEMENT: the plan carries it out only when that decision is answered with it; until it is answered, the steps of every open method are listed.

Rollback. nothing changed on either side: the writers are stopped, not removed, the table is untouched and the export is a read; delete the exported objects from the bucket before another attempt

Conversion (own-tool)

Step MAN/data/module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this/transform/own-tool (data transform, phase CUTOVER); owner: Data Architect.

What. Convert the export of the table retail-store-ecs-carts (module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this) into documents of the collection of stackit_mongodbflex_instance.dynamodb_tables with a script of your own: unwrap the type descriptors of the DynamoDB JSON (S, N, B, BOOL, NULL, M, L, SS, NS, BS) into BSON values; make the key id the _id (or a unique index on id); keep numbers in the BSON type that holds them: a DynamoDB number has up to 38 digits of precision and a decimal128 has 34, so a value of more than 34 significant digits cannot be a decimal128 without rounding (keep it as a string, as AWS advises beyond 38 digits, or report it); turn a string set, a number set and a binary set (SS, NS, BS) into an array: a set holds unique values in no order and an array does not, so the application code that relies on set operations changes; plan the secondary indexes global secondary index idx_global_customerId (customerId) as MongoDB indexes on the collection (a local index shares the key of the table, so its index leads with the table's partition key); projections are not carried (a MongoDB index finds whole documents); the item size needs no check in this direction: DynamoDB holds an item of up to 400 KB and a BSON document holds up to 16 mebibytes; the consistency the application needs is its own decision: a read of DynamoDB is eventually consistent unless the request asks for a strongly consistent read, and a global secondary index is eventually consistent only; MongoDB's read concern and read preference are chosen by the application's driver

Why. a DynamoDB item is not a document: the export wraps each value in a type descriptor and DynamoDB has types, keys, indexes and expiry that MongoDB holds differently; nothing in a copy tool names these, so the plan states each one this table has

Needs. the script and the MongoDB driver it uses, the collection name, and for each attribute that holds a number the largest number of digits it ever holds

Done when. every item of the export of the table retail-store-ecs-carts (module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this) is converted into a document, or is reported with the reason it was not (a number of more than 34 digits, an item the script cannot read), and the count of documents plus the count of reports equals the itemCount of the export

Risks.

• a DynamoDB number of more than 34 significant digits does not fit a decimal128: the plan cannot see the values, so the script counts such values and either keeps them as strings or reports them
• a set (SS, NS, BS) becomes an array: unique values and no order are guarantees of a set that an array does not give
• the conversion is the script of a person; the plan names what it must handle and runs nothing
• a global secondary index of DynamoDB is eventually consistent and may be sparse; a MongoDB index is updated with the write, and an item without the index's attributes is left out of a partial or sparse index only if the person builds it so

Method. This step belongs to the method migrate-with-a-tool-of-my-own of the decision DEC-GEN-CONVERT-DATA-MOVEMENT: the plan carries it out only when that decision is answered with it; until it is answered, the steps of every open method are listed.

Rollback. the conversion only reads the export and writes files: delete them, the table and the instance are untouched

Final sync (own-tool)

Step MAN/data/module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this/final-sync/own-tool (data final sync, phase CUTOVER); owner: Data Architect.

What. Load the documents of the table retail-store-ecs-carts (module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this) into the collection of stackit_mongodbflex_instance.dynamodb_tables (STACKIT MongoDB Flex) with a MongoDB client of your choice (mongoimport, or a driver script), then create its indexes

Why. the draft creates the instance empty; the STACKIT documentation read documents no import of a dump into MongoDB Flex (the service has no restore-from-source attribute), so the plan names no tool of the service and writes no command of its own: the instance is a MongoDB instance and a MongoDB client reaches it at the host and port of its user

Needs. the converted documents, the user and the access list of the instance (the draft's acl variable lists the networks that may connect), and the indexes the conversion named

Done when. every document of the conversion of the table retail-store-ecs-carts (module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this) is in the collection and the indexes the conversion named exist: the keys, then the index of the global secondary index idx_global_customerId (customerId)

Risks.

• build the indexes after the load, not before it: each index is built once over loaded documents (MongoDB allows at most 64 indexes on a collection)
• the draft lists the clients that may connect as the variable acl and gives no user: the person creates the user of the instance and the clients (the load and the applications) come from networks that list contains

Method. This step belongs to the method migrate-with-a-tool-of-my-own of the decision DEC-GEN-CONVERT-DATA-MOVEMENT: the plan carries it out only when that decision is answered with it; until it is answered, the steps of every open method are listed.

Rollback. the writers are stopped, not removed, and nothing the plan does changes the source, so the source is the way back until the reconcile records GO: put the writers back on it; before another attempt remove what the tool wrote into stackit_mongodbflex_instance.dynamodb_tables (the draft created it empty); check that the tool you chose does not change the source, since the plan knows nothing of it

Reconcile (own-tool)

Step MAN/data/module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this/reconcile/own-tool (data reconcile, phase CUTOVER); owner: Data Architect.

What. Reconcile the table retail-store-ecs-carts (module.dependencies.module.dynamodb_carts.aws_dynamodb_table.this) and the collection of stackit_mongodbflex_instance.dynamodb_tables with the writers stopped: the number of documents in the collection must equal the itemCount of the export (manifest-summary.json), less the items the conversion reported, a sample of keys must read the same items on both sides, and the application's read of one item must return the same on both; record GO or NO-GO

Why. the copy is made by a tool the plan knows nothing of, so the plan states no check of that tool; the comparison of the two sides with the writers stopped is the check that does not depend on it

Done when. the records and the sample match on both sides, the read the application does returns the same on both, and the owner records GO

Method. This step belongs to the method migrate-with-a-tool-of-my-own of the decision DEC-GEN-CONVERT-DATA-MOVEMENT: the plan carries it out only when that decision is answered with it; until it is answered, the steps of every open method are listed.

Rollback. NO-GO: start the writers again against the untouched source; the target holds only a copy

Verification: the checks after the transfer

These are the checks the owner makes with the writers stopped before recording GO; the reconcile step in the execution plan states the same checks, with the same ids. A check with no command says why: the plan states no command it cannot back.

• Count on both sides (document-count, command): Read the itemCount the export states and count the documents of the collection, with the writers stopped. The count of the collection equals the itemCount less the items the conversion reported (and less the documents the TTL index already removed, where time to live is on); the ItemCount of describe-table is not used, because DynamoDB updates it about every six hours. Expected: the documents of the collection equal the itemCount less the items the conversion reported and the documents the TTL index removed.

~~~text aws s3 cp s3://<bucket>/<prefix>/AWSDynamoDB/<export-id>/manifest-summary.json - mongosh "<connection-string>" --eval 'db.<collection>.countDocuments({})' ~~~

• Not provided, so left as a placeholder or for the owner to fill in: the bucket, prefix and id of the export; the connection string of the instance (it holds a password: read it from the environment); the collection.

• Content on both sides (key-sample, command): Read a sample of keys from the table with a strongly consistent read and the same keys from the collection, and compare the items. Expected: every sampled item reads the same on both sides, with the conversion's types (numbers, sets, dates) applied.

~~~text aws dynamodb get-item --table-name <table> --key '<key as DynamoDB JSON>' --consistent-read mongosh "<connection-string>" --eval 'db.<collection>.findOne({ _id: <key> })' ~~~

• Not provided, so left as a placeholder or for the owner to fill in: the keys to sample; the connection string of the instance.

• Application check (application-check, manual): Run one read the application itself makes on its most important data against the collection and compare its answer with the answer the application gets from the table; then, from a test copy of the application, make one write against the collection and read it back. Expected: the two answers are equal, the write succeeds and the read returns it. - No command: the read and the write are the application's, and the source states neither the query nor the answer. - Not provided, so left as a placeholder or for the owner to fill in: the read the application makes on its most important data and its expected answer; the write the test copy of the application makes.
• Rollback limit (rollback-limit, manual): Record the point after which a rollback loses data. Until the first of the writes that the collection accepts after the switch, a rollback is only the connection settings put back: the DynamoDB table was not changed after the freeze and is current to it. From that write on, what the collection holds exists only there: it is copied back to the DynamoDB table with a method rehearsed before the window, or accepted as lost. Once the DynamoDB table is retired (its retirement step, after the GO and the retention the owner decides) there is nothing to go back to. Expected: before the window the owner records the date until which the DynamoDB table is kept, the method that copies the writes of the collection back or the decision that they are lost, and who calls the rollback. - No command: how long the source is kept and whether the writes made on the target are copied back or lost are decisions of the owner of the application: the source states neither, so the plan names the limit and no date. - Not provided, so left as a placeholder or for the owner to fill in: the date until which the DynamoDB table is kept; the method that copies the writes of the collection back, or the decision that they are lost.