Proxmox reboots randomly after installing a PCIe SSD adapter including 2 SSDs

Chat
1

Hi friends,

I installed a Startech PEX8M2E2 Dual M.2 PCIe SSD adapter card (recommended here) in an HP Microserver Gen10 Plus to use 2 M-Key SSDs.
After some time Proxmox reboots.

Following error is logged before the reboots:

Feb 23 08:21:48 pve3 kernel: NMI: PCI system error (SERR) for reason b0 on CPU 0.
Feb 23 08:21:48 pve3 kernel: Dazed and confused, but trying to continue
Feb 23 08:21:48 pve3 kernel: pcieport 0000:08:0c.0: can't change power state from D3cold to D0 (config space inaccessible)
Feb 23 08:21:48 pve3 kernel: pcieport 0000:08:04.0: can't change power state from D3cold to D0 (config space inaccessible)

I already updated Proxmox and the System ROM, changed the power management settings, used different kernels and enabled/disabled bifurcation without success.

I also asked HPE and Startech support (no solution yet) but maybe the Proxmox users here have an idea?

1 Like
2

Hi @mrmarkuz

I’ve used about 5-6 Startech Dual NVME Adapters (last was also PEX8M2E2, but also used other models from Startech), alwayys on HP Servers (Different Models), never had any issues so far.
The oldest is now running for nearly two years now…

Maybe also important: What disks are you using there?
Any wear shown on the disks? (I had a couple of Samsung (970?) with faulty Disk-BIOS, the BIOS was updated, which fixed the issue. The wear: 35 % in 2 weeks!

I actually have the same HPE Microserver Gen10 Plus at home, I’m also following the same dok to use better CPU / More RAM. But this one I didn’t use the NVME, as I had 2 TB SSDs (new) lying around, also 2 x 128 GB SSDs as system…

And I had spare 2.5"->3.5" Adapters for the HPE Microserver…

I need the PCI for increased Networking, I’m using a 2x2.5 GBE NIC from Delock. The box is working well with 32 GB RAM, will upgrade to 64 GB RAM soon.

My 2 cents
Andy

PS:

You did read this (From said report by serverthehome.com you included…)

These PCIe switches use power, so we generally suggest avoiding them given how power-constrained the MSG10+ is…

This CAN easily be an issue, these boxes allow really max 180W all in all!

2 Likes
3

I’ve never handled these kinds of problems, but I’ll hope to find a way to work around the problem.
If pcieport 0000:08:0c.0 is where the adapter is connected you may try to disable pcie power management with pcie_port_pm=off on boot args.

But I fear Andy is correct, you have power supply problems.

1 Like
4

Side hint: verify consumption from a metered power socket.
Which is not reliable enough as DC amperometer, but whatever.

1 Like
5

Thanks for the hints, replacing the SSDs seems to have solved the issue.

3 Likes
6

from what SSD to what SSD?

7

does the microserver can do pcie bifurcation ?

EDIT

seem yes :smiley:

1022Ă—800 75.9 KB

do you have enbaled raid on it ?

8

Who would want a RAID, when you can use ZFS on Proxmox?
That’s what I use on the same Box…
.

Bildschirmfoto 2023-03-06 um 15.33.44
Bildschirmfoto 2023-03-06 um 15.33.441920Ă—902 92.1 KB

9

whatever the raid system, if you have two ssd nvme, you need a raid I think, either for redundancy or (for a bad purpose) speed with raid0

as a side note, I am not a big fan of ZFS

10

I don’t need ZFS on a Workstation or an average server, but for Hypervisor use? Nothing better so far!

ZFS has mirroring (=ZFS-mirror, =RAID1)

1 Like
11

I am curious because I just tossed two months ago a ssd nvme with PCIE 4X card in my old proxmox (10years old, pcie 2.0) and I must admit I should have done it before, really fast

1 Like
12

Seagate Exos X16 - 14TB - 3.5", SATA, 7.2k, 256MB, 512e

For backups (Mass-Storage) these are fantastic, but only current Mid-Class Synologys accept them:

EG:

DS1621+
DS1621xs

both accept these disks.

The DS1522+ will not accept these EXOS disks.

13

AFAIK no PCI bifurcation is needed when using the Startech PEX8M2E2 card.
RAID and ZFS are working without having it enabled.

From Crucial to Samsung 970 Evo

1 Like
14

fun, because without bifurcation you should see only one ssd, maybe the auto setting in the bios is the master key

1 Like
15

I also use auto in the bios PCIe settings.

There are options using PCIe switches such as the StarTech PEX8M2E2 adapter which offers two drive capability in a single PCIe slot without requiring bifurcation support.

From the servethehome guide

2 Likes