MooseFS and MooseFS PRO for awesome data replication around the world. It’s a perfect fit for NethServer 8 - run the Metadata service on the NS8 Leader, chunkservers on the NS8 Workers. Distributed storage with a GUI 
I’m keen to help spearhead development of that as someone that writes Ansible playbooks for MooseFS and manages automation for it. (Not affiliated with the project officially, just as a hobbyist and user of it).
Hi @wings
MooseFS Pro seems NOT open-source… It’s commercial.
I anyway prefer CEPH. It’s fully open source, no “paid” versions with BLOBs…
My 2 cents
Andy
Yes, Pro is closed source. However, MooseFS itself is a community project, open source and does everything NethServer would need it to do. Pro simply brings some additional features, and as I’d be using the MooseFS containers I’d love to have support for Pro built in (as an optional extra).
Ceph is great too, and ideally we’d support both - but Ceph is MUCH heavier in terms of resource requirements, much more fragile, and much more complicated I know because I run MooseFS, GlusterFS, LizardFS and CephFS in different environments/past jobs and in my homelab…
…further, MooseFS is much more suited to running on “one” server and then expanding out to more servers than Ceph, and I’ve experienced that personally.
I use CEPH on Proxmox, the best platform to run CEPH - and NethServer.
If you like using closed source, that’s everyone’s free choice.
In my long experience, all software with a “paid” version can’t use the same code as the open / free version. There has to be at least a BLOB which is not open for the functionality, but also for license verification et al.
And from there, the development difers. Sure some may “flow” back, but it’s not in the real interest.
And Proxmox is rock solid!
Probably you’re running it all on UNRAID…
That gets the medal for the slowest performing system - on top of it closed source.
My 2 cents
Andy
No, I run it on Proxmox as well as bare metal (Ubuntu) and VMs (Debian), and touch Ceph clusters as large as 60 petabytes…
I work in HPC. Distributed storage has been a passion of mine for nearly a decade, across half a dozen different forms of distributed storage including Ceph.
I get what you’re saying, but MooseFS is open source, and I run both the open source and closed source versions alongside each other. Everything I develop that supports MooseFS supports both because I hate vendor lock-in and prefer open source.
Ceph works at scale and it’s great, I’m not knocking it - but for a project like NethServer 8, which is designed to scale from one machine to many, something like MooseFS GPLv3 is much lighter weight, much more suited to running on low power devices like Raspberry Pis, much more suited to starting at one node and scaling up, much more resilient to failures, high latency, low bandwidth connections… the list of reasons really does go on and on I’d be happy to take this to DMs if this is off-topic at this point…
I wouldn’t dream of putting CEPH together with NethServer, in that sense MooseFS may be a better fit.
As you, I hate vendor lock in. And I can easily do without - beacuse I can!
I just looked at two comparisons of CEPH / MooseFS, but also their web pages and especially Wikipedia entries. The talk on Wikipedia is always interesting.
As for CEPH, there’s only one critic: One link is not working.
As to MooseFS, there are a few points. MooseFS Wikipedia was created by an affiliated person.
No comparisons.
Somewhat obscure with stuff like this:
What NFS like authentication? NFS has no authentication.
← not true, unless you only use public exports. smartasses of the world, unite.
How stable is MooseFS, are there any reports, well known organisations using it?
A file system needs to be 101% Rock Solid in my Opinion!
And needs really usable tools, not like ReiserFS, which I took one look at and mentally trashed!
Sure enough, the guy had to admit that the “dancing b-trees” could lose data…
It does look interesting, but is it worth it in sense of stability?
My 2 cents
Andy
MooseFS suffers from obscurity, for sure.
It’s got password based authentication which is optional, not on by default.
There’s a few big installations, plenty around 10PiB with hundreds of servers from discussions with the developers, but it’s not really talked about very much by customers.
In terms of stability, in my opinion it’s the most stable distributed filesystem I’ve used, but I’m self-admittedly kind of a “fanboy/fangirl” of it.
It has some really nice features like instant copy-on-write snapshots of folders, which can be very handy for snapshotting the state of something before making changes to it, and per-folder redundancy settings so you can have less- and more- important data and treat it as such. For example, my music collection exists on every disk in my clusters.
I’m used to ZFS, which already comes with a whole backpack of features and is among the stablest I’ve seen or used in my lifetime.
However: It is NOT a replicated file system, and synching ZFS does not make it one…
CEPH with ZFS, but that’s certainly not lightweight and suitable for running on a RPI.
I use Raspberries a lot (At least while they were available, at the moment they aren’t !), especially as a independent NUT server for virtual environments.
Yeah, it has a lot of ZFS-y features. ZFS snapshots and MooseFS snapshots work very similarly, but the difference is you can snapshot individual folders which creates a perfect clone, as opposed to full filesystems (I forget if you can do that with ZFS?).
In terms of non-distributed FS, I’d put ZFS at the top of my list in terms of “I’d trust this with family photos” security. MooseFS running on top of ZFS bricks is especially nice 
two-feet brake stop.
Choose to support a FS or use it as standard (IMVHO) si quite FAR from the goals of NethServer8. I don’t want to say “no talking about exotic/specialized/cool filesystems”… but at current state NethServer is “distro-agnostic” (mostly) so both Fedora and Debian are option but… The talk about supporting that or other filesystem (any type, any source) IVMHO is in the wrong place here.
If someone is willing to advocate any “puppy” they love… go for it! But especially in this space, seems quite out of place.
Considering also than NethServer8, colliding any way with ProxMox, ESXi, XEN and VirtualBox, has no way to compete.
Is MooseFS even something that can run in a Docker-ish container? Regardless of its merits, the last dozen or so posts sound like they belong in their own topic.
Does a filesystem run into a contaner or underlying the OS?
That’s what I thought. Moved to a new topic.
Or close it. Unless it become a FS-chat topic.
Because if MooseFS driver won’t be developed for debian or Fedora, IDK if devs are gonna implement it for both.
Packages are already there for both distros.
Hi
As both of you guys know, on most Linux distros changing the used file system is little more than loading it.
NethServer itself doesn’t use ext4, the defacto standard for Linux, but uses XFS.
Proxmox lets you choose from ext4, xfs, zfs to ceph and more.
Drivers are available.
It seems this MooseFS is very sleek, and is very low in resources, low enough that it will run comfortably on a Raspberry. CEPH is to heavyweight to run well on a Raspberry.
I’m a sceptic by nature, but this does look interesting.
And yes, we did switch to PM…
@danb35 Thx for splitting this off…
My 2 cents
Andy
As CentOS 6 and 7 propose as default…
Hi @pike
Point is: it makes no or hardly any difference…
True that XFS can’t do repairs when booting, like ext4. However, it’s also much robuster than ext4, so hardly ever get’s into a needed “repair” condition.
ZFS even less!
The stabler and better a filesystem is, the better for users/sysadmins - the server is generally much more stabler.
Note:
I have NO experience (yet) with MooseFS, so I’m not advocating it.
I will, however, test this on a VM, maybe on a Raspberry to see how much Load this generates and how stable it is. Most likely even this weekend.
As MooseFS is a distributed filesystem, that’s not intended or suitable as a generic filesystem for a server. However, for the storage space, it can make BIG sense!
And hearing about a stable 72 PB storage with MooseFS, that makes it doubly interesting!!!
My 2 cents
Andy
I think that makes difference…
As far as i can guess, devs read what CentOS provided, thought about it some time… and go for the default. And IMVHO unless specific conditions/cases, no gain of any means using MooseFS.
Currently Fedora use BTRFS as default. ext4 for Debian. By any means I feel confident that something different will rise as “default FS” from devs. And if Neth8 wont’ be a distro-agnostic toolbox, due to current arrangement (container, not virtualized disk) no gain of any kind of something different from a well known and polished FS.