Persistent storage not working with multimedia (SD)cards

[mark_nl (Mark Verlinde)]

Note upfront, as you probably know I’m working on arm64 but pretty sure this is applicable to X86_64 too.

As per subject it is not possible to setup persistent storage for logs on multimedia cards (mmcblk devices) such as SD-cards or onboard eMMC.
Partitions on these devices are numbered like nvme devices with a leading p (i.e. p1, p2 etc.). Note the partition numbers pX of the eMMC (mmcblk1) and SD-Card (mmcblk0) on my setup.

# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0          7:0    0 253.4M  0 loop /overlay
mmcblk1      179:0    0  28.9G  0 disk
├─mmcblk1p1  179:1    0    16M  0 part
└─mmcblk1p2  179:2    0   300M  0 part /rom
mmcblk1boot0 179:32   0     4M  1 disk
mmcblk1boot1 179:64   0     4M  1 disk
mmcblk0      179:96   0  29.7G  0 disk
└─mmcblk0p1  179:97   0  29.7G  0 part /mnt/data

Moreover, an embedded MMC (eMMC) typically has 2 hardware partitions ( mmcblk1boot0 and mmcblk1boot1 ) which can be used for early bootloaders. parted can’t handle this very well and fails with the one-liner used to create an 3th partition for persistent storage:

# parted -s --fix -a optimal /dev/mmcblk1 unit MiB mkpart ext4 464MiB 100%
parted: invalid token: ext4
Error: Expecting a partition type.

However, it worked if you are explicit concerning the partition Type (primary / extended)

# parted -s --fix -a optimal /dev/mmcblk1 unit MiB mkpart primary ext4 464MiB 100%

Resulted in an 3th partition on the eMMC

# parted /dev/mmcblk1
GNU Parted 3.6
Using /dev/mmcblk1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: MMC A3A551 (sd/mmc)
Disk /dev/mmcblk1: 31.0GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      33.6MB  50.3MB  16.8MB  primary  ext2         boot
 2      67.1MB  382MB   315MB   primary
 3      487MB   31.0GB  30.6GB  primary

My question is: Are you receptive to finding a solution for this?

TL;DR:
Looking into this I overlooked the preferred shell script is sh. So found out the default shell (sh) provided by busybox is actually ash with bash-isms..

This works with busybox (but it shouldn’t)

#!/bin/sh

number=$1

if [[ $number == 1 ]]; then # this is bash NOT sh
  message="True"
else
  message="False"
fi

echo $message

[Tbaile (Tommaso Bailetti)]

Hi Mark!

I’m thinking that the issue is the same on both arm and amd.

We get the device list from the following:

lsblk --json -b --output-all

And then we filter by type of device, so maybe that’s the issue with the missing SD from the UI. This is done due to avoid people setting USBs as hard drives. While I find the fix non-trivial, I’m actually a bit reluctant on doing so. On high logging firewalls you could fry the SD pretty quickly.

Please run the command above and post me the result! So we can dig a bit better on the issue.

@mark_nl

Looking into this I overlooked the preferred shell script is sh. So found out the default shell (sh) provided by busybox is actually ash with bash-isms..

That’s the OpenWRT way of doing things , stick to sh when you can, otherwise you’ll have no clue if it would work or not. They rewrote and re-implemented much of the busybox implementations, so beware of that!

[mark_nl (Mark Verlinde)]

@Tbaile

And then we filter by type of device, so maybe that’s the issue with the missing SD from the UI.

The SD-card and the (free space) on the eMMC show up in the UI.
On de SD-card mkfs fails because the script assumes the new created partition is <device>1 and on a (e)MMC’s it is <device>p1. Pretty sure using an separate (2nd) nvme for logging fails too for same reason.

This works for me over here, not sure if it breaks something:

NS-storage: handle multimedia cards

@Tbaile

I’m actually a bit reluctant on doing so. On high logging firewalls you could fry the SD pretty quickly.

SD-card have evolved a lot. For sure, one needs to use an high quality card of sufficient size to give the level-wearing algorithms a chance. I would not be that concerned.

[mark_nl (Mark Verlinde)]

Maybe even better avoiding making assumptions on the partition naming scheme,
rather pick the last one?

# lsblk /dev/mmcblk1 -o KNAME | sed -n \$p
mmcblk1p3

# lsblk /dev/mmcblk0 -o KNAME | sed -n \$p
mmcblk0p1

[mark_nl (Mark Verlinde)]

@Tbaile

Please run the command above and post me the result

Sorry overlooked this:

# lsblk --json -b --output-all

# lsblk --json -b --output-all
{
   "blockdevices": [
      {
         "alignment": 0,
         "id-link": null,
         "id": null,
         "disc-aln": 0,
         "dax": false,
         "disc-gran": 4194304,
         "disk-seq": 13,
         "disc-max": 46137344,
         "disc-zero": false,
         "fsavail": 207192064,
         "fsroots": [
             "/"
         ],
         "fssize": 263585792,
         "fstype": "f2fs",
         "fsused": 56393728,
         "fsuse%": "21%",
         "fsver": null,
         "group": "root",
         "hctl": null,
         "hotplug": false,
         "kname": "loop0",
         "label": "rootfs_data",
         "log-sec": 512,
         "maj:min": "7:0",
         "maj": "7",
         "min": "0",
         "min-io": 512,
         "mode": "brw-------",
         "model": null,
         "mq": "  1",
         "name": "loop0",
         "opt-io": 0,
         "owner": "root",
         "partflags": null,
         "partlabel": null,
         "partn": null,
         "parttype": null,
         "parttypename": null,
         "partuuid": null,
         "path": "/dev/loop0",
         "phy-sec": 512,
         "pkname": null,
         "pttype": null,
         "ptuuid": null,
         "ra": 128,
         "rand": false,
         "rev": null,
         "rm": false,
         "ro": false,
         "rota": false,
         "rq-size": 128,
         "sched": "none",
         "serial": null,
         "size": 265682944,
         "start": null,
         "state": null,
         "subsystems": "block",
         "mountpoint": "/overlay",
         "mountpoints": [
             "/overlay"
         ],
         "tran": null,
         "type": "loop",
         "uuid": "76194b9b-8783-4b71-ac9f-83e14ae2edb6",
         "vendor": null,
         "wsame": 0,
         "wwn": null,
         "zoned": "none",
         "zone-sz": 0,
         "zone-wgran": 0,
         "zone-app": 0,
         "zone-nr": 0,
         "zone-omax": 0,
         "zone-amax": 0
      },{
         "alignment": 0,
         "id-link": null,
         "id": null,
         "disc-aln": 0,
         "dax": false,
         "disc-gran": 4194304,
         "disk-seq": 9,
         "disc-max": 46137344,
         "disc-zero": false,
         "fsavail": null,
         "fsroots": [
             null
         ],
         "fssize": null,
         "fstype": null,
         "fsused": null,
         "fsuse%": null,
         "fsver": null,
         "group": "root",
         "hctl": null,
         "hotplug": false,
         "kname": "mmcblk1",
         "label": null,
         "log-sec": 512,
         "maj:min": "179:0",
         "maj": "179",
         "min": "0",
         "min-io": 512,
         "mode": "brw-------",
         "model": null,
         "mq": "  1",
         "name": "mmcblk1",
         "opt-io": 0,
         "owner": "root",
         "partflags": null,
         "partlabel": null,
         "partn": null,
         "parttype": null,
         "parttypename": null,
         "partuuid": null,
         "path": "/dev/mmcblk1",
         "phy-sec": 512,
         "pkname": null,
         "pttype": "dos",
         "ptuuid": "df8f0e08",
         "ra": 128,
         "rand": false,
         "rev": "0x8",
         "rm": false,
         "ro": false,
         "rota": false,
         "rq-size": 128,
         "sched": "mq-deadline",
         "serial": "0xea7541f0",
         "size": 31037849600,
         "start": null,
         "state": null,
         "subsystems": "block:mmc:mmc_host:platform",
         "mountpoint": null,
         "mountpoints": [
             null
         ],
         "tran": "mmc",
         "type": "disk",
         "uuid": null,
         "vendor": null,
         "wsame": 0,
         "wwn": null,
         "zoned": "none",
         "zone-sz": 0,
         "zone-wgran": 0,
         "zone-app": 0,
         "zone-nr": 0,
         "zone-omax": 0,
         "zone-amax": 0,
         "children": [
            {
               "alignment": 0,
               "id-link": null,
               "id": null,
               "disc-aln": 0,
               "dax": false,
               "disc-gran": 4194304,
               "disk-seq": 9,
               "disc-max": 46137344,
               "disc-zero": false,
               "fsavail": null,
               "fsroots": [
                   null
               ],
               "fssize": null,
               "fstype": "ext4",
               "fsused": null,
               "fsuse%": null,
               "fsver": null,
               "group": "root",
               "hctl": null,
               "hotplug": false,
               "kname": "mmcblk1p1",
               "label": "kernel",
               "log-sec": 512,
               "maj:min": "179:1",
               "maj": "179",
               "min": "1",
               "min-io": 512,
               "mode": "brw-------",
               "model": null,
               "mq": "  1",
               "name": "mmcblk1p1",
               "opt-io": 0,
               "owner": "root",
               "partflags": "0x80",
               "partlabel": null,
               "partn": 1,
               "parttype": "0x83",
               "parttypename": null,
               "partuuid": "df8f0e08-01",
               "path": "/dev/mmcblk1p1",
               "phy-sec": 512,
               "pkname": "mmcblk1",
               "pttype": null,
               "ptuuid": null,
               "ra": 128,
               "rand": false,
               "rev": null,
               "rm": false,
               "ro": false,
               "rota": false,
               "rq-size": 128,
               "sched": "mq-deadline",
               "serial": null,
               "size": 16777216,
               "start": 65536,
               "state": null,
               "subsystems": "block:mmc:mmc_host:platform",
               "mountpoint": null,
               "mountpoints": [
                   null
               ],
               "tran": "mmc",
               "type": "part",
               "uuid": "84173db5-fa99-e35a-95c6-28613cc79ea9",
               "vendor": null,
               "wsame": 0,
               "wwn": null,
               "zoned": "none",
               "zone-sz": 0,
               "zone-wgran": 0,
               "zone-app": 0,
               "zone-nr": 0,
               "zone-omax": 0,
               "zone-amax": 0
            },{
               "alignment": 0,
               "id-link": null,
               "id": null,
               "disc-aln": 0,
               "dax": false,
               "disc-gran": 4194304,
               "disk-seq": 9,
               "disc-max": 46137344,
               "disc-zero": false,
               "fsavail": 0,
               "fsroots": [
                   "/"
               ],
               "fssize": 49020928,
               "fstype": "squashfs",
               "fsused": 49020928,
               "fsuse%": "100%",
               "fsver": null,
               "group": "root",
               "hctl": null,
               "hotplug": false,
               "kname": "mmcblk1p2",
               "label": null,
               "log-sec": 512,
               "maj:min": "179:2",
               "maj": "179",
               "min": "2",
               "min-io": 512,
               "mode": "brw-------",
               "model": null,
               "mq": "  1",
               "name": "mmcblk1p2",
               "opt-io": 0,
               "owner": "root",
               "partflags": null,
               "partlabel": null,
               "partn": 2,
               "parttype": "0x83",
               "parttypename": null,
               "partuuid": "df8f0e08-02",
               "path": "/dev/mmcblk1p2",
               "phy-sec": 512,
               "pkname": "mmcblk1",
               "pttype": null,
               "ptuuid": null,
               "ra": 128,
               "rand": false,
               "rev": null,
               "rm": false,
               "ro": false,
               "rota": false,
               "rq-size": 128,
               "sched": "mq-deadline",
               "serial": null,
               "size": 314572800,
               "start": 131072,
               "state": null,
               "subsystems": "block:mmc:mmc_host:platform",
               "mountpoint": "/rom",
               "mountpoints": [
                   "/rom"
               ],
               "tran": "mmc",
               "type": "part",
               "uuid": null,
               "vendor": null,
               "wsame": 0,
               "wwn": null,
               "zoned": "none",
               "zone-sz": 0,
               "zone-wgran": 0,
               "zone-app": 0,
               "zone-nr": 0,
               "zone-omax": 0,
               "zone-amax": 0
            }
         ]
      },{
         "alignment": 0,
         "id-link": null,
         "id": null,
         "disc-aln": 0,
         "dax": false,
         "disc-gran": 4194304,
         "disk-seq": 10,
         "disc-max": 46137344,
         "disc-zero": false,
         "fsavail": null,
         "fsroots": [
             null
         ],
         "fssize": null,
         "fstype": null,
         "fsused": null,
         "fsuse%": null,
         "fsver": null,
         "group": "root",
         "hctl": null,
         "hotplug": false,
         "kname": "mmcblk1boot0",
         "label": null,
         "log-sec": 512,
         "maj:min": "179:32",
         "maj": "179",
         "min": "32",
         "min-io": 512,
         "mode": "brw-------",
         "model": null,
         "mq": "  1",
         "name": "mmcblk1boot0",
         "opt-io": 0,
         "owner": "root",
         "partflags": null,
         "partlabel": null,
         "partn": null,
         "parttype": null,
         "parttypename": null,
         "partuuid": null,
         "path": "/dev/mmcblk1boot0",
         "phy-sec": 512,
         "pkname": null,
         "pttype": null,
         "ptuuid": null,
         "ra": 128,
         "rand": false,
         "rev": null,
         "rm": false,
         "ro": true,
         "rota": false,
         "rq-size": 128,
         "sched": "mq-deadline",
         "serial": null,
         "size": 4194304,
         "start": null,
         "state": null,
         "subsystems": "block:mmc:mmc_host:platform",
         "mountpoint": null,
         "mountpoints": [
             null
         ],
         "tran": "mmc",
         "type": "disk",
         "uuid": null,
         "vendor": null,
         "wsame": 0,
         "wwn": null,
         "zoned": "none",
         "zone-sz": 0,
         "zone-wgran": 0,
         "zone-app": 0,
         "zone-nr": 0,
         "zone-omax": 0,
         "zone-amax": 0
      },{
         "alignment": 0,
         "id-link": null,
         "id": null,
         "disc-aln": 0,
         "dax": false,
         "disc-gran": 4194304,
         "disk-seq": 11,
         "disc-max": 46137344,
         "disc-zero": false,
         "fsavail": null,
         "fsroots": [
             null
         ],
         "fssize": null,
         "fstype": null,
         "fsused": null,
         "fsuse%": null,
         "fsver": null,
         "group": "root",
         "hctl": null,
         "hotplug": false,
         "kname": "mmcblk1boot1",
         "label": null,
         "log-sec": 512,
         "maj:min": "179:64",
         "maj": "179",
         "min": "64",
         "min-io": 512,
         "mode": "brw-------",
         "model": null,
         "mq": "  1",
         "name": "mmcblk1boot1",
         "opt-io": 0,
         "owner": "root",
         "partflags": null,
         "partlabel": null,
         "partn": null,
         "parttype": null,
         "parttypename": null,
         "partuuid": null,
         "path": "/dev/mmcblk1boot1",
         "phy-sec": 512,
         "pkname": null,
         "pttype": null,
         "ptuuid": null,
         "ra": 128,
         "rand": false,
         "rev": null,
         "rm": false,
         "ro": true,
         "rota": false,
         "rq-size": 128,
         "sched": "mq-deadline",
         "serial": null,
         "size": 4194304,
         "start": null,
         "state": null,
         "subsystems": "block:mmc:mmc_host:platform",
         "mountpoint": null,
         "mountpoints": [
             null
         ],
         "tran": "mmc",
         "type": "disk",
         "uuid": null,
         "vendor": null,
         "wsame": 0,
         "wwn": null,
         "zoned": "none",
         "zone-sz": 0,
         "zone-wgran": 0,
         "zone-app": 0,
         "zone-nr": 0,
         "zone-omax": 0,
         "zone-amax": 0
      },{
         "alignment": 0,
         "id-link": null,
         "id": null,
         "disc-aln": 0,
         "dax": false,
         "disc-gran": 4194304,
         "disk-seq": 12,
         "disc-max": 3741319168,
         "disc-zero": false,
         "fsavail": null,
         "fsroots": [
             null
         ],
         "fssize": null,
         "fstype": null,
         "fsused": null,
         "fsuse%": null,
         "fsver": null,
         "group": "root",
         "hctl": null,
         "hotplug": true,
         "kname": "mmcblk0",
         "label": null,
         "log-sec": 512,
         "maj:min": "179:96",
         "maj": "179",
         "min": "96",
         "min-io": 512,
         "mode": "brw-------",
         "model": null,
         "mq": "  1",
         "name": "mmcblk0",
         "opt-io": 0,
         "owner": "root",
         "partflags": null,
         "partlabel": null,
         "partn": null,
         "parttype": null,
         "parttypename": null,
         "partuuid": null,
         "path": "/dev/mmcblk0",
         "phy-sec": 512,
         "pkname": null,
         "pttype": "gpt",
         "ptuuid": "5d1b9d5a-458e-475e-b6c8-07df63969ad9",
         "ra": 128,
         "rand": false,
         "rev": null,
         "rm": false,
         "ro": false,
         "rota": false,
         "rq-size": 128,
         "sched": "mq-deadline",
         "serial": "0x137e0d42",
         "size": 31914983424,
         "start": null,
         "state": null,
         "subsystems": "block:mmc:mmc_host:platform",
         "mountpoint": null,
         "mountpoints": [
             null
         ],
         "tran": "mmc",
         "type": "disk",
         "uuid": null,
         "vendor": null,
         "wsame": 0,
         "wwn": null,
         "zoned": "none",
         "zone-sz": 0,
         "zone-wgran": 0,
         "zone-app": 0,
         "zone-nr": 0,
         "zone-omax": 0,
         "zone-amax": 0,
         "children": [
            {
               "alignment": 0,
               "id-link": null,
               "id": null,
               "disc-aln": 3145728,
               "dax": false,
               "disc-gran": 4194304,
               "disk-seq": 12,
               "disc-max": 3741319168,
               "disc-zero": false,
               "fsavail": 29613047808,
               "fsroots": [
                   "/"
               ],
               "fssize": 31231758336,
               "fstype": "ext4",
               "fsused": 6262784,
               "fsuse%": "0%",
               "fsver": null,
               "group": "root",
               "hctl": null,
               "hotplug": true,
               "kname": "mmcblk0p1",
               "label": "ns_data",
               "log-sec": 512,
               "maj:min": "179:97",
               "maj": "179",
               "min": "97",
               "min-io": 512,
               "mode": "brw-------",
               "model": null,
               "mq": "  1",
               "name": "mmcblk0p1",
               "opt-io": 0,
               "owner": "root",
               "partflags": null,
               "partlabel": "primary",
               "partn": 1,
               "parttype": "0fc63daf-8483-4772-8e79-3d69d8477de4",
               "parttypename": null,
               "partuuid": "e2b54a48-b663-43e6-b2b4-502cf1b468e9",
               "path": "/dev/mmcblk0p1",
               "phy-sec": 512,
               "pkname": "mmcblk0",
               "pttype": null,
               "ptuuid": null,
               "ra": 128,
               "rand": false,
               "rev": null,
               "rm": false,
               "ro": false,
               "rota": false,
               "rq-size": 128,
               "sched": "mq-deadline",
               "serial": null,
               "size": 31913410560,
               "start": 2048,
               "state": null,
               "subsystems": "block:mmc:mmc_host:platform",
               "mountpoint": "/mnt/data",
               "mountpoints": [
                   "/mnt/data"
               ],
               "tran": "mmc",
               "type": "part",
               "uuid": "07ec5302-5f30-4578-8470-36658e1d78e0",
               "vendor": null,
               "wsame": 0,
               "wwn": null,
               "zoned": "none",
               "zone-sz": 0,
               "zone-wgran": 0,
               "zone-app": 0,
               "zone-nr": 0,
               "zone-omax": 0,
               "zone-amax": 0
            }
         ]
      }
   ]
}

[Tbaile (Tommaso Bailetti)]

@mark_nl

SD-card have evolved a lot. For sure, one needs to use an high quality card of sufficient size to give the level-wearing algorithms a chance. I would not be that concerned.

True, but my PI keeps killing one every 3-4 years

Spoke yesterday with the team, we are probably gonna plan a fix for that. Thank you for the output!