Hi Marcel and welcome!
Due to this
I’m guessing that traffic is not correctly routed properly due to conntrack keeping connections active as we’ve experienced a few posts away.
My suggestion is to wipe the conntrack table through the UI when doing such changes. We’re not willing to do such thing automatically due to potentially in progress connections that must be kept active (I.E. a telephone call).
Please try and come back if some other issue arise