Here is my final config with my ISP router basically in bridge mode (MODEM only). If ever you would like to keep your router in Modem + Firewall + Router, you will need to do as @mrmarkuz suggested and define yourself a static route from your ISP’s router to point to your VPN network. Unfortunately mine didn’t have that and they didn’t foresee doing it in the future leaving me with no other choice but to change it in bridge mode which solved all my problems.
The config is as follow :
