Upcoming change to default Let's Encrypt chain

On 30 September 2021 the DST Root CA X3 has expired.
This certificate is part of Let’s Encrypt full chain and it’s used to support old devices with Android <= 7. According to this chart, around 7% of existing Android devices.
To make things work, Let’s Encrypt leverages on a never-fixed bug of old Android versions.

This choice has been proven to raise more issues even on recent devices and software which, of course, have their own bugs (see here and here).

As Nethesis, we had to face the problem because many customers were not ready for such change.
We saw issues with VoiP phones, mail clients and even Wordpress installations.
Enterprise installations quickly received a workaround for this scenario, which has been proven working well on more than 5K machines.

So the hard choice is: should we break the compatibility with very few known Android devices or we should break the compatibility of many unknown products (mail clients, IOT devices, etc)?
We had very mixed feelings on this choice mostly because we didn’t want to betray our own rule: “New features/enhancements and bug fixes must not alter the behavior of existing systems” (see developer manual).
One choice could have been to keep the current default and change it for NethServer 7.10.
Still, we do not know when CentOS 7.10 will be out and in the meanwhile the problem could hit more devices on next few months.

In the end, we decide to disobey our own rule and change Let’s Encrypt default chain.
After issue 6584 will be released, Let’s Encrypt certificates will not contain DST Root CA X3 certificate anymore.
We are confident that the new behavior will have almost no impact on existing installations, but if you have any clients that could be affected by the change, you can easily switch back to the old chain:

config setprop pki LetsEncryptShortChain disabled
/usr/libexec/nethserver/letsencrypt-certs -f

Please note that existing certificates will not be changed.

The package will be released on Monday 25 October, 2021.

More in-depth analysis available at following links:

7 Likes

Uninstalling Mattermost from my 7.0 Android smartphone…

You do not need to. Please read the post carefully: you just need to change a prop.

But, if you still want to switch to the short chain and retain Android 7 compatibility, you could try to load the CA certificate to the device, see this article (courtesy of @Amygos, we didn’t test it).

2 Likes

First of all, thanks for the resources.
Currently, this Android 7 phone is used for accessing “without browser” to a Mattermost installation. “Guarded” by a Let’sencrypt certificate.
Android 7 is old, and I bought an used phone for cheap only for DGC (otherwise, my phone, which is not my smartphone, would be an Android 6 Samsung Xcover3. I love hit-resistant devices).
Using Mattermost is a plus.

I don’t want to endorse use of older OSes, even more if without any security updates (latest provided to this device is September 2019). And users should not use it.

In this direction the “loss” of the older certificate CA is a nice push to customers to use updated OSes.

Nevertheless, when the setup will go online again, I’ll try to install the app again and try to use the article about certificates. A “decent” new Android 11 phone (don’t rely on that for photos) is still priced not that much (140 euros), sometimes with greater ram and bigger capacity. It won’t last that long (silicon shortage, SoC war) but… still good time to upgrade.

3 Likes

My solution would be: install LineageOS 18.1 (android 11) on my Android 7 Xiaomi mi5…

Still I find it disturbing that old(er) systems are left behind by a service that has become so important for a safe(r) internet. It forces ordinary people, often with little to no money to buy new devices, to buy (in this case) a new phone.

I don’t disagree with your approach/solution, @robb; but some caveats can suggest that “a good idea” which i stand for, is “not a good idea” for a final user device.

Some apps don’t accept to be installed on custom ROMS or with a rooted device.
Some cooked ROMS are often far from being optimally cooked for every hardware, sometimes lacking of a set of features that some people want. And polishing of glitches and bugs more than sometimes i still rough (being kind), even in huge projects like LineageOS. I used a Galaxy SIII cooked with Lineage for few months before the power up button gave up. And i still have a laying aroung Moto G (the first one).
Well… With recent Androids, phones sometimes go from stumbling performances (with the stock rom) … to falling.
Moreover… It’s a matter of make the right buy at the first time. Maybe soon I’ll cook a newer rom into my Redmi 4 (still trying to unlock the bootloader). Which unfortunately don’t manage that right LTE B10.

I don’t see that the situation’s under their control. Certificates, including root CA certificates, will expire. When that happens, they’ll need to be replaced or updated. If the underlying OS doesn’t handle that, that isn’t under LE’s control.