Hey Checkyourlogs Fans,

In this scenario, we encountered a critical RAID failure on a SuperMicro Windows Backup target, where both drives in the RAID array were lost. Thankfully, we had been using Veeam Agent for Windows, which allowed us to create physical backups and prepare for Bare Metal Restores. However, we faced an unexpected challenge involving the Recovery Media during the recovery process. This post details how we addressed the issue using a combination of Veeam’s tools and resources like PXE boot, updated ISO media, and proper driver injection.

A computer screen with white text Description automatically generated

Initial Problem: RAID Failure and Recovery Media Issues

When both drives in the RAID array failed, the initial approach was to use Veeam’s Bare Metal Restore functionality. Unfortunately, we encountered the following error when attempting to use the recovery media:

Error Message:

“Exception has been thrown by the target of an invocation. Failed to establish connection: no valid IP address found. SSL Connection exception. Unable to establish client-server connection. The required security package does not exist.”

Veeam Knowledge Base (KB) 4512: Connection Issues

The error led us to Veeam’s KB 4512, which outlines a common issue with older versions of Windows ISO media that may not be compatible with the necessary recovery procedures. This KB explains that the error is related to outdated recovery media created without certain security patches and network drivers.

Resolution Steps:

  1. Rebuild a VM using the latest October 2024 Windows Server 2022 ISO from the Volume License Key (VLK) download site. The old ISOs had compatibility issues, and the updated one provided the necessary fixes.
  2. Install Veeam Agent for Windows on the newly created VM.
  3. Take a fresh backup and create new Recovery Media from this environment.

With the updated ISO and new recovery media, we were able to move forward.

PXE Boot Setup: KB4574 for Veeam Agent Recovery Media

Next, we decided to configure Windows Deployment Services (WDS) to PXE boot the Veeam Agent for Windows Recovery Media. This allowed us to avoid physical media and work directly over the network.

Steps:

  1. Boot.wim file from the Veeam Recovery Media was moved to the WDS server.
  2. Intel RAID drivers were injected into the Boot.wim to ensure smooth operation and recognition of the hardware.
    A screenshot of a computer Description automatically generated

This PXE boot setup worked flawlessly and allowed us to quickly boot into the recovery environment, but we noticed that RAID drivers were missing from the default Boot.wim file.

RAID Driver Injection and Recovery

After booting from PXE, we had to manually inject the Intel RAID drivers into the Boot.wim to properly recognize the RAID array.

  1. Reboot the system into the Veeam Recovery Environment (VeeamRE).
  2. Open the command prompt using Shift + F10.
  3. Use DiskPart to check for the presence of RAID volumes:

diskpart

list disk

A computer screen shot of a black screen Description automatically generated

  1. Once the RAID volume was visible, we ran the Bare Metal Recovery process from VeeamRE.

Bare Metal Recovery Process

With the RAID recognized and the recovery environment functioning, we followed the standard Bare Metal Recovery steps:

  1. PXE Boot into the Veeam Recovery Media

    A screenshot of a computer recovery program Description automatically generated

  2. Select network Storage

    A screenshot of a computer Description automatically generated

A screenshot of a computer Description automatically generated

  1. Choose Network Storage as the location and select Veeam Backup Repository as the source.
  2. Enter the IP address of the Veeam Backup & Replication (VBR) server.
  3. Use a Recovery Token generated from the Veeam backup repository.

    A screenshot of a computer Description automatically generated

  4. Select the backup and begin the restore process
    A screenshot of a computer error Description automatically generated

The recovery process continued successfully from this point, and we restored the system to its previous state.

KB4512 Fixes in Detail

As mentioned earlier, KB4512 identified the need for recovery media created using an updated version of Windows Server 2022 (from September 2023 or later). The steps provided in the KB involve ensuring that:

  • The recovery media is built using the latest ISO.
  • The Windows Recovery Environment (WinRE) has applied the appropriate security packages and network driver updates.

We initially attempted using an Evaluation Edition of Server 2022, but the ISO was outdated. The newer ISO from Visual Studio Subscriptions (under the version tagged “Updated Sep 2023 or later”) was critical in resolving the issue.

Lessons Learned

Here are some of the critical takeaways from this process:

  1. Always keep ISO media updated: Recovery media created with older ISOs can lead to connection and driver issues. Make sure you use up-to-date ISOs for both system installs and recovery.
  2. PXE Boot is a powerful tool: PXE booting Veeam Recovery Media via WDS allowed us to avoid physical media and provided a flexible recovery option, especially in large environments.
  3. Driver injection is key: For systems using RAID, be prepared to inject drivers into the recovery environment to ensure that the RAID array is recognized during recovery.

Thanks,

Dave