Create Instance from backup on another Zone (DRaaS use case) #11560

abh1sar · 2025-09-02T10:54:14Z

Description

In continuation of #10140 , this PR adds the functionality that a new Instance can be created on a different Zone from which the backup was taken.
Doc PR: apache/cloudstack-documentation#556

Supported backup Providers are NAS and Dummy.
The two zones have to be on the same CloudStack Installation
The backup repository still needs to be added only for the Zone where backup will be taken.
A new parameter DRaaS Enabled should be checked while creating the backup repository if it is to be used for creating instance in other zones.
(The operator should ensure that the backup repository is accessible in other the Zones as well.)
Added updateBackupRepository api with parameters name, address, mountopts and draasenabled.

Other changes done:

Throw relevant errors to the end user. i.e, if backup repository is not accessible or the backup files are not present or corrupted.
Added a timeout to mount operation so that restore fails early if backup repository is not accessible. Current behaviour is that mount will block indefinitely until libvirt times out the command after 1 hour.
Changes to decrease the RTO: Earlier, the new Instance had to be started (to create volumes), stopped, restored and started again. The start-stop cycle takes a long time. The whole process took around 200s on my local env with 15 GB of volume size.
I have added a new param VirtualMachineProfile.Param.ReturnAfterVolumePrepare which creates the volume but doesn't start the VM in orchestrateStart. So one Stop and one Start is avoided during the Instance creation. The whole process finished within 60s with this change.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Error thrown when tmp directory was not present in the kvm host

Error thrown when the repository was not accessible and mount failed

Error thrown when backup file was not present

Error thrown when the backup file was not a valid qcow2 disk

How did you try to break this feature and the system with this change?

codecov · 2025-09-02T11:02:15Z

Codecov Report

❌ Patch coverage is 49.40239% with 127 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.39%. Comparing base (3d6ec29) to head (0df7f8d).

Files with missing lines	Patch %	Lines
...r/backup/repository/UpdateBackupRepositoryCmd.java	0.00%	35 Missing ⚠️
...rg/apache/cloudstack/backup/NASBackupProvider.java	62.16%	12 Missing and 2 partials ⚠️
...rg/apache/cloudstack/backup/BackupManagerImpl.java	53.57%	12 Missing and 1 partial ⚠️
...n/java/com/cloud/vm/VirtualMachineManagerImpl.java	0.00%	9 Missing ⚠️
...g/apache/cloudstack/backup/BackupRepositoryVO.java	10.00%	9 Missing ⚠️
...ce/wrapper/LibvirtRestoreBackupCommandWrapper.java	82.00%	5 Missing and 4 partials ⚠️
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	63.15%	3 Missing and 4 partials ⚠️
...udstack/api/response/BackupRepositoryResponse.java	0.00%	6 Missing ⚠️
...apache/cloudstack/backup/RestoreBackupCommand.java	0.00%	6 Missing ⚠️
...che/cloudstack/backup/NetworkerBackupProvider.java	0.00%	5 Missing ⚠️
... and 5 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11560      +/-   ##
============================================
+ Coverage     17.36%   17.39%   +0.02%     
- Complexity    15237    15277      +40     
============================================
  Files          5888     5889       +1     
  Lines        525741   525906     +165     
  Branches      64164    64186      +22     
============================================
+ Hits          91274    91458     +184     
+ Misses       424167   424140      -27     
- Partials      10300    10308       +8

Flag	Coverage Δ
uitests	`3.63% <ø> (-0.01%)`	⬇️
unittests	`18.43% <49.40%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ed UpdateBackupRepositoryCmd api.

github-actions · 2025-09-04T02:22:04Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

…global setting

abh1sar · 2025-09-04T04:01:19Z

plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java

        } else {
+            clusterId = getClusterIdFromRootVolume(vm);


fyi - hostId won't be set for create Instance from Backup. But we can get the clusterId from the root volume which is already created.

abh1sar · 2025-09-04T04:02:55Z

.../main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapper.java

+            mount += " -o " + mountOptions;
+        }
+
+        int exitValue = Script.runSimpleBashScriptForExitValue(mount, mountTimeout, false);


fiy - runSimpleBashScript doesn't return any exception. just a return value. So no need to keep it inside try-catch. So, checking the return value instead.

abh1sar · 2025-09-04T04:03:22Z

.../main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapper.java

-        } catch (Exception e) {
-            throw new CloudRuntimeException(String.format("Failed to unmount backup directory: %s", backupDirectory), e);
+        String umountCmd = String.format(UMOUNT_COMMAND, backupDirectory);
+        int exitValue = Script.runSimpleBashScriptForExitValue(umountCmd);


fyi - same here. runSimpleBashScript doesn't throw any exception.

abh1sar · 2025-09-04T04:07:32Z

server/src/main/java/org/apache/cloudstack/backup/BackupManagerImpl.java

+            String error_msg = String.format("Failed to create Instance [%s] from backup [%s] due to: %s.", vm.getInstanceName(), backupDetailsInMessage, result.second());
+            logger.error(error_msg);
+            processRestoreBackupToVMFailure(vm, backup, eventId);
+            throw new CloudRuntimeException(error_msg);


fyi - this is done to pass on the error passed by the provider.

harikrishna-patnala

overall looking good, can you please clarify me with few questions I've raised @abh1sar thank you

...java/org/apache/cloudstack/api/command/user/backup/repository/UpdateBackupRepositoryCmd.java

engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java

plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java

server/src/main/java/com/cloud/vm/UserVmManagerImpl.java

ui/src/components/view/DeployVMFromBackup.vue

harikrishna-patnala · 2025-09-08T09:53:22Z

@abh1sar also can you please check the failures in simulator CI run

abh1sar · 2025-09-09T03:59:52Z

@abh1sar also can you please check the failures in simulator CI run

Thanks @harikrishna-patnala. I have fixed the issue and addressed other comments.

harikrishna-patnala

code LGTM

harikrishna-patnala · 2025-09-09T07:31:38Z

@blueorangutan package

abh1sar · 2025-09-09T07:31:53Z

@blueorangutan package

blueorangutan · 2025-09-09T07:32:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-09T09:27:14Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 14912

abh1sar · 2025-09-10T09:46:16Z

@blueorangutan test

blueorangutan · 2025-09-10T09:48:04Z

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-09-11T03:19:52Z

[SF] Trillian test result (tid-14275)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 59571 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11560-t14275-kvm-ol8.zip
Smoke tests completed. 146 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

draas initial changes

2588ee4

boring-cyborg bot added component:api component:backup component:compute component:kvm component:orchestration component:UI labels Sep 2, 2025

harikrishna-patnala added this to the 4.22.0 milestone Sep 2, 2025

harikrishna-patnala added this to Apache CloudStack 4.22.0 Sep 2, 2025

harikrishna-patnala moved this to In Progress in Apache CloudStack 4.22.0 Sep 2, 2025

abh1sar added 2 commits September 2, 2025 16:46

Fix UT failure in UserVmManagerImplTest

8f0ce46

Added option to enable disaster recovery on a backup respository. Add…

4eac151

…ed UpdateBackupRepositoryCmd api.

github-actions bot added the status:has-conflicts label Sep 4, 2025

abh1sar added 2 commits September 4, 2025 08:49

Added timeout for mount operation in backup restore configurable via …

090c222

…global setting

Merge remote-tracking branch 'upstream/main' into draas

b6b8a0f

github-actions bot removed the status:has-conflicts label Sep 4, 2025

abh1sar changed the title ~~Create Instance from backup on another Zone~~ Create Instance from backup on another Zone (DRaaS use case) Sep 4, 2025

abh1sar commented Sep 4, 2025

View reviewed changes

abh1sar mentioned this pull request Sep 7, 2025

Create Instance from Backup on another Zone apache/cloudstack-documentation#556

Draft

harikrishna-patnala reviewed Sep 8, 2025

View reviewed changes

abh1sar added 4 commits September 8, 2025 23:01

Addressed review comments

6c03978

fix for simulator test failures

a58f6d4

Added UT for coverage

dae4f3b

Fix create instance from backup ui for other providers

7449646

Added events to add/update backup repository

0df7f8d

harikrishna-patnala approved these changes Sep 9, 2025

View reviewed changes

Create Instance from backup on another Zone (DRaaS use case) #11560

Are you sure you want to change the base?

Create Instance from backup on another Zone (DRaaS use case) #11560

Uh oh!

Conversation

abh1sar commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Other changes done:

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Sep 4, 2025

Uh oh!

abh1sar Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

abh1sar Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

abh1sar Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abh1sar Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harikrishna-patnala commented Sep 8, 2025

Uh oh!

abh1sar commented Sep 9, 2025

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

harikrishna-patnala commented Sep 9, 2025

Uh oh!

abh1sar commented Sep 9, 2025

Uh oh!

blueorangutan commented Sep 9, 2025

Uh oh!

blueorangutan commented Sep 9, 2025

Uh oh!

abh1sar commented Sep 10, 2025

Uh oh!

blueorangutan commented Sep 10, 2025

Uh oh!

blueorangutan commented Sep 11, 2025

Uh oh!

Uh oh!

abh1sar commented Sep 2, 2025 •

edited

Loading

codecov bot commented Sep 2, 2025 •

edited

Loading

abh1sar Sep 4, 2025 •

edited

Loading

abh1sar Sep 4, 2025 •

edited

Loading