Lately we’ve had some questions about whether it’s possible to recover data from full VMs that are unreachable for reasons such as:
- Kernel bug described here
- Lost SSH keys
- Other failures that prevent a VM from booting
While this is possible with deployments created through Terraform, it has not yet been possible for deployments created in the Playground. So, I decided to whip up a tool that could help users to recover their data.
Prerequisites
- A full VM deployment on a live node (we can’t help if the node is down)
- Full VM is the old style that uses a raw disk image (the tool will give an error otherwise)
- A Linux environment to run the tool in (it might work on other OS but isn’t tested yet)
- Seed phrase used to deploy the original VM (again, can’t help if the seed is lost)
Outcome
- A new VM that has the old VMs disk(s) attached as data disks
- Your original VM will not be restored to working condition—this tool is only designed to give a chance to recover data
Before you continue, please note that this tool is experimental and provided with limited support. It has the potential to affect your deployments in undesired ways if used without caution.
If you’re reading this, please also consider backing up any important data you have, whether it’s stored in a VM on the ThreeFold Grid or anywhere else.
Steps
To start, download, extract, and run the patched grid-cli binary from my fork of the tfgrid-sdk-go repository:
wget https://github.com/scottyeager/tfgrid-sdk-go/releases/download/v0.16.0-recovervm2/tfgrid-sdk-go_Linux_x86_64.tar.gz
tar xf tfgrid-sdk-go_Linux_x86_64.tar.gz
./tfcmd recover
This will start a series of interactive prompts that will take you through the process of recovering the VM disk. You’ll have a chance to enter a new SSH key if desired.
Once the process is completed, you’ll see the info for your recovery VM on screen. Connect to the new VM and issue the following command:
mount /dev/vdb1 /mnt
After that’s done, you’ll be able to access all files from your original VM under the /mnt directory in the new VM. You can continue using the recover VM if you want, with the extra disk attached, or copy the files into a new VM using scp for example, if you’d rather not continue paying for the extra disk.
If you have any questions or issues, please respond in this thread or feel free to message me directly on Telegram (@scottyeager). Hope this helps!