Loading and writing of statefiles for outputs is not threadsafe #330
Labels
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
Checklist
My Environment
Expected Behaviour
When running stacks concurrent in a batch with
terraspace all upwe expect it to be able to pull all remote states used for outputs in parallel without running into errors.Current Behavior
When pulling and reading remote states for using with terraspace outputs it sometimes runs into an error when two stacks in the same batch are dependent on the same stack. In this case you get the error message as the actual outptut.
(Output <output> was not found for the <dependend_stack> tfvars file. Either <stack_with_output> stack has not been deployed yet or it does not have this output: <output>. Also, if local backend is being used and has been removed/cleaned, then it will also result zero-byte state.json with the 'terraform state pull' used to download the terraform state and output will not be found.)This has lead to unwanted deletion of resources, as the attributes for resources changed from the actual value to the error message. As there was no assertion of this value the resources were deleted and terraform only failed when the cloud API asserted the value as false. This left us with the deleted resources.
Step-by-step reproduction instructions
This seems to be a race condition of rubys subprocess handling, so we could only reproduce it of running pipelines more often to force the triggering of the issue / bug. We determined it by checking the source code of terraspace for the error message in the value and followed the program stack. From this we could validate that it had to occur somewhere between terraform (called by terraspace) loading the statefile and terraspace reading it. We quickly had the suspicion that it is caused by two or more of the subprocesses created by the
terraspace allcommand (see: deploy_batch.rb).When executing a command like
terraform state pull > /path/to/fileit will empty the file as soon as the terraform command starts and flushes the output into the file when it is done. During the command the file is empty (zero-byte) when read by another process.To verify this we added log statements including timestamps before and after the command execution in the fetcher.rb file:
The next time the issue / bug occured we were able to extract the following log:
Log Output
There are annotations at the line where the subprocesses call what.
Code Sample
Solution Suggestion
This could probably be handled with a lock. Either with a thread safe variable which can be check by all other subprocesses that resembles the availability of the state.json. Another way could be to create a lock file before any write operation which can also be checked by other subprocesses. If whatever used locking mechanism fails the program falls into a retry loop.
Also it would be helpful to fail the terraspace operation when the output could not be loaded instead of continuing with the error message as the value.
Other Issues
We found another issue referencing the same problem:
#300
The text was updated successfully, but these errors were encountered: