I think we should split the problem/solution into 2 parts:
1.) Repair view
If we set a custom quota, then we need to see this on the page for quotas. At them moment we usually do not see it there, because the user set quota gets changed to default as soon as a mail is received for his mailbox. Hope you can reproduce this. The quota page does not show much useful information at the moment.
2.) Setting a quota must have an effect
Usually I do this to force users to clean up their mailboxes. What we could do is:
a) send user a mail if his quota is nearly reached (e.g. <=10% capacity left)
b) block him from sending further messages if he reached his quota or before he reached it.
c) block incoming messages if mailbox is over quota
d) block incoming messages if mailbox over quota but notify admin or user, too
(a-b-c) is what exchange does (exchange knows a quota for not allowed to send any more and a higher quota where no messages are received any more) and this method does it's job out in the field.
I think it's not good to receive mails if mailbox is over the quota!! This would sooner or later fill up your entire HD.
And it's also not good to assume the max quota found in the system as general max. This would be too much and again your HD would be out of storage. Let's look to a setup with 100 Mailboxes most 1GB Limit some 5GB Limit. Let's assume we have a 250GB HD. Mails would come in until Mailboxes filled up with 100*5GB =500GB, but your HD is 250GB only....
So my recommendation - keep it simple for the beginning:
* repair view of quota page
* do not accept mail from user or from outside if quota is over it's limit (guess that's how it is now and as it is done by most mail servers out)
* we need to give user a chance to know about his quota. Sogo und Roundcube will show it anyway, but what if they are working with Thunderbird or pure imap? Thats why the should get a notification best before their box is filled up. To keep it simple the notification could be the warning sent out to sender if quota is reached.