DM-VDO for Nethserver

I must admit, I didn’t try it (yet) but what I read in a few minutes makes this a very nice feature for NethServer as a fileserver: VDO was originally developed by Permabit Technology Corp. as a proprietary set of kernel modules and userspace tools. This software and technology has been acquired by Red Hat, has been relicensed under the GPL (v2 or later), and this repository begins the process of preparing for integration with the upstream kernel.

VDO installs within the Linux device mapper framework, where it takes ownership of existing physical block devices and remaps these to new, higher-level block devices with data-efficiency capabilities. In other words: it adds deduplication to a basic RH/CentOS server.

It might be a (very) nice feature to add once NS 7.5 is available…

More info on RH site: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/vdo

As far is i was able to understand from your link this is a quite huge slice between the system and the filesystem, and without a partition tool/management available from admin interface, is quite a “go/no-go” setup during the first install. Just like ZFS.
Which one is more reliable, scalable and solid? And also, easier to implement and manage?

I know that cockpit is on the way…

tnx it is also in my “to see” list , these are the other links that I have marked

i must only find the time :pensive:
but I do not think it can be included already in 7.5, but I think, however, we can see how it works and how to manage it with cockpit

1 Like

Hello there,

Any info in this implementation?

AFAIK currently no.

Well, i install this:

yum install kmod-kvdo vdo

and it appear on Nethserver Cockpit gui :slight_smile:

Enought to me :slight_smile:

image

If it is that simple, then maybe a little pointer in the wiki can be enough to add it.
Is it also functional from Nethserver-cockpit? Is the de-dup active?

Yes, it working well on Ns-Cockpit. I set the compression and deduplication, seems to working.

[root@hun25-09v ~]# vdostats --all --human-readable
/dev/mapper/vdo0 :
  version                             : 28
  release version                     : 131337
  data blocks used                    : 21796645
  overhead blocks used                : 1283912
  logical blocks used                 : 131730836
  physical blocks                     : 104857600
  logical blocks                      : 157286400
  1K-blocks                           : 419430400
  1K-blocks used                      : 92322228
  1K-blocks available                 : 327108172
  used percent                        : 22
  saving percent                      : 83
  block map cache size                : 134217728
  write policy                        : async
  block size                          : 4096
  completed recovery count            : 0
  read-only recovery count            : 0
  operating mode                      : normal
  recovery progress (%)               : N/A
  compressed fragments written        : 4713675
  compressed blocks written           : 728729
  compressed fragments in packer      : 1
  slab count                          : 198
  slabs opened                        : 58
  slabs reopened                      : 30
  journal disk full count             : 0
  journal commits requested count     : 19205
  journal entries batching            : 0
  journal entries started             : 16969951
  journal entries writing             : 0
  journal entries written             : 16969951
  journal entries committed           : 16969951
  journal blocks batching             : 0
  journal blocks started              : 453611
  journal blocks writing              : 0
  journal blocks written              : 453611
  journal blocks committed            : 453611
  slab journal disk full count        : 0
  slab journal flush count            : 2699
  slab journal blocked count          : 0
  slab journal blocks written         : 11702
  slab journal tail busy count        : 0
  slab summary blocks written         : 11562
  reference blocks written            : 4900
  block map dirty pages               : 15999
  block map clean pages               : 16769
  block map free pages                : 0
  block map failed pages              : 0
  block map incoming pages            : 0
  block map outgoing pages            : 0
  block map cache pressure            : 0
  block map read count                : 16456518
  block map write count               : 8484322
  block map failed reads              : 0
  block map failed writes             : 0
  block map reclaimed                 : 0
  block map read outgoing             : 0
  block map found in cache            : 23719201
  block map discard required          : 323801
  block map wait for page             : 865070
  block map fetch required            : 32768
  block map pages loaded              : 304681
  block map pages saved               : 188068
  block map flush count               : 152100
  dedupe advice valid                 : 972242
  dedupe advice stale                 : 88
  concurrent data matches             : 10745
  concurrent hash collisions          : 0
  invalid advice PBN count            : 0
  no space error count                : 0
  read only error count               : 0
  instance                            : 0
  512 byte emulation                  : off
  current VDO IO requests in progress : 1
  maximum VDO IO requests in progress : 2000
  dedupe advice timeouts              : 0
  flush out                           : 2402102
  write amplification ratio           : 0.4
  bios in read                        : 7972461
  bios in write                       : 10886425
  bios in discard                     : 0
  bios in flush                       : 2402102
  bios in fua                         : 2402264
  bios in partial read                : 0
  bios in partial write               : 0
  bios in partial discard             : 0
  bios in partial flush               : 0
  bios in partial fua                 : 0
  bios out read                       : 8941891
  bios out write                      : 2786414
  bios out discard                    : 0
  bios out flush                      : 0
  bios out fua                        : 0
  bios meta read                      : 312764
  bios meta write                     : 1553917
  bios meta discard                   : 0
  bios meta flush                     : 374493
  bios meta fua                       : 202760
  bios journal read                   : 0
  bios journal write                  : 453948
  bios journal discard                : 0
  bios journal flush                  : 203096
  bios journal fua                    : 202759
  bios page cache read                : 304681
  bios page cache write               : 340203
  bios page cache discard             : 0
  bios page cache flush               : 152135
  bios page cache fua                 : 0
  bios out completed read             : 8941891
  bios out completed write            : 2786414
  bios out completed discard          : 0
  bios out completed flush            : 0
  bios out completed fua              : 0
  bios meta completed read            : 312764
  bios meta completed write           : 1398707
  bios meta completed discard         : 0
  bios meta completed flush           : 219283
  bios meta completed fua             : 202760
  bios journal completed read         : 0
  bios journal completed write        : 453611
  bios journal completed discard      : 0
  bios journal completed flush        : 202759
  bios journal completed fua          : 202759
  bios page cache completed read      : 304681
  bios page cache completed write     : 188103
  bios page cache completed discard   : 0
  bios page cache completed flush     : 35
  bios page cache completed fua       : 0
  bios acknowledged read              : 7972461
  bios acknowledged write             : 10886425
  bios acknowledged discard           : 0
  bios acknowledged flush             : 2402102
  bios acknowledged fua               : 2402264
  bios acknowledged partial read      : 0
  bios acknowledged partial write     : 0
  bios acknowledged partial discard   : 0
  bios acknowledged partial flush     : 0
  bios acknowledged partial fua       : 0
  bios in progress read               : 0
  bios in progress write              : 0
  bios in progress discard            : 0
  bios in progress flush              : 0
  bios in progress fua                : 0
  read cache accesses                 : 0
  read cache hits                     : 0
  read cache data hits                : 0
  KVDO module bytes used              : 533809920
  KVDO module peak bytes used         : 534878864
  KVDO module bios used               : 37286
  KVDO module peak bio count          : 37574
  entries indexed                     : 37627487
  posts found                         : 972330
  posts not found                     : 7499739
  queries found                       : 0
  queries not found                   : 0
  updates found                       : 4713759
  updates not found                   : 0
  current dedupe queries              : 0
  maximum dedupe queries              : 2000
3 Likes

Hello Guys,

have you did other test or uses of VDO on NS?

I have made some test in lab.
Does anyone use it in prod?

Hello @Serhiy_Hryhoretskyy. Welcome to the NethServer community.
I have not heard before of anyone using VDO on NethServer Can you tell a bit more about your lab test? Besides the compression, what do we gain with VDO that is not available for ZFS (often used with proxmox) or even BTRFS (I think the default fs for Centos8)? This with the dedup in mind…
Here is presentation video by RH at DevConf, last january 2020. I think it clarifies quite well what VDO is and what it does.

1 Like