Simulate failures

Recover from a server crash

Unlike many modern applications, Temporal automatically preserves the state of your Workflow even if the server is down.

1. Make sure your Worker is stopped. Press CTRL+C in the Worker terminal.
2. Switch to the terminal where your Workflow ran. Start the Workflow again with npm run client.
3. Verify the Workflow is running in the UI.
4. Shut down the Temporal Server with CTRL+C.
5. After it stops, restart it with the same database file you used before.

Your Workflow is still listed:

If the Temporal Cluster goes offline, you can pick up where you left off when it comes back online.

Recover from an unknown error in an Activity

This demo makes a call to an external service in an Activity. If that call fails due to a bug, the Activity produces an error.

To test this out, simulate a bug in the deposit Activity function. Let your Workflow continue to run but don't start the Worker yet.

Open activities.ts and switch out the comments on the return statements so the deposit function calls bank2.depositThatFails.

Save your changes and start the Worker again:

npm run worker

You'll see the Worker complete withdraw but error on deposit - and keep retrying:

2023-10-11T19:03:25.778Z [INFO] Worker state changed { state: 'RUNNING' }
Withdrawing $400 from account 85-150.

Depositing $400 into account 43-812.

2023-10-11T19:03:29.445Z [WARN] Activity failed {
  attempt: 1,
  activityType: 'deposit',
  taskQueue: 'money-transfer',
  error: Error: This deposit has failed
}
Depositing $400 into account 43-812.

2023-10-11T19:03:30.455Z [WARN] Activity failed {
  attempt: 2,
  activityType: 'deposit',
  error: Error: This deposit has failed
}

...

View more in the Web UI:

Your Workflow is running, but only withdraw has succeeded. With Temporal, you can debug and fix the issue while the Workflow is running.

Pretend that you found a fix. Switch the comments back on the deposit return statements and save.

Cancel the Worker with CTRL+C, then restart:

npm run worker

The Worker picks up right where the Workflow was failing:

2023-10-11T19:17:18.918Z [INFO] Worker state changed { state: 'RUNNING' }
Depositing $400 into account 43-812.

Switch back to where npm run client was running:

...

Transfer complete (transaction IDs: W3436600150, D9270097234)

Visit the Web UI again and you'll see the Workflow has completed:

You have just fixed a bug in a running application without losing the state of the Workflow or restarting the transaction.

Conclusion

You now know how to run a Temporal Workflow and understand some of the value Temporal offers. You explored Workflows and Activities, you started a Workflow Execution, and you ran a Worker. You also saw how Temporal recovers from failures and retries Activities.

Further exploration

1. Change the Amount to 1000000 in client.ts. Run npm run client and see the withdraw Activity fail.
2. Change the targetAccount to an empty string. See the Activity fail and the money go back to the original account.
3. Change the retry policy in workflows.ts so it only retries 3 times. Does the Workflow place the money back into the original account?