Howto install Calibre Content Server


(Marc) #1

A Work In Progress howto to install Calibre Server in NethServer, as requested in this thread: Education applications wishlist

WARNING: Not yet ready to use on a production server

The calibre Content server allows you to access your calibre libraries and read books directly in a browser by downloading the book in an off-line cache.

Install Calibre Server:

sudo -v && wget -nv -O- https://download.calibre-ebook.com/linux-installer.sh | sudo sh /dev/stdin

By default Calibre Content Server will be installed under /opt/calibre path. To use a different location check the installation options.

Create a directory to store a library. If you want to use more than one library a better naming would be /var/lib/nethserver/calibre/libraries/default and so on. For simplicity here we will use a single library.

mkdir -p /var/lib/nethserver/calibre/library

Add books to the library. You can upload some e-books to your server, or download them form Project Gutenberg, for example. Calibre identifies file types by its extension, make sure the uploaded e-books have the appropriate file extension. For simplicity here we create an empty book:

calibredb add --empty "test.epub" --title "Test" --with-library "/var/lib/nethserver/calibre/library"

Calibre can be set to work in read-only mode and open access, or work with user credentials. You can add users and set permissions using:

#add users and set permissions (interactive)
calibre-server --userdb "/var/lib/nethserver/calibre/users.sqlite" --manage-users

To auto-start calibre-server you can create a service unit:

cat << EOF > /etc/systemd/system/calibre-server.service
[Unit]
Description=calibre content server
After=network.target

[Service]
Type=simple
User=calibre
Group=calibre
ExecStart=/opt/calibre/calibre-server \
	--access-log=/var/log/calibre-server-access.log \
	--enable-auth \
	--log=/var/log/calibre-server.log \
	--port=8008 \
	--url-prefix=/calibre \
	--userdb=/var/lib/nethserver/calibre/users.sqlite \
	"/var/lib/nethserver/calibre/library"

[Install]
WantedBy=multi-user.target
EOF

More options for calibre-server command are described in the manual.

As we have set the service to run as calibre user, we create that user:

useradd -r -s /sbin/nologin -d /var/lib/nethserver/calibre -c "Calibre User" calibre

Before we forget, set permissions for calibre’s database, libraries and e-books:

chown -R calibre:calibre /var/lib/nethserver/calibre

Prepare log files:

touch /var/log/calibre-server.log /var/log/calibre-server-access.log
chown calibre:calibre /var/log/calibre-server*.log

Enable a port for calibre-server:

config set fw_calibreserver service status enabled TCPPort 8008 access green
signal-event firewall-adjust

Enable and start calibre-server service:

systemctl enable --now calibre-server

Access calibre server from a web browser going to http://FQDN-or-IP:8008 or http://FQDN-or-IP:8008/calibre and login with any user created through calibredb command.

You can browse the library, upload e-books (if user have no restrictive permissions), read/convert/download them.

To log out you have to either clean session and cookies or close the browser.


To consider:

  • Use a Reverse Proxy (no additional port exposed)
  • Use of a Virtual Host
  • https access (current howto settings are not secure)
  • logrotate
  • No AD/LDAP integration. Consider an import script or, IF it’s even possible, append some action to user create/modify events (only for local accounts provider, won’t work with remote providers).
  • Security
  • Consider other e-book management options.

Things to test

  • http(s) with and without authentication
  • books upload (testing each supported file type and small/big files). Multi-upload supported.
  • read epub, pdf and other formats
  • convert from/to many formats (most problematic is PDF which might be missing some library, occasionally had problems converting some big epub to other formats)
  • login / read from mobile devices
  • …

We Are NethServer - Community Overview - Sep 18
Community SPRINT
Education applications wishlist
(Markus Neuberger) #2

Very nice! Working as expected. :sunglasses:

You didn’t restart httpd after creating the zz_calibre.conf file but it’s working without it, at least in my first tests.

grafik


(Rob Bosch) #3

I have been asking around on the calibre forums about LDAP/AD authentication. https://www.mobileread.com/forums/showthread.php?t=309507
The answer to my question:

Use a reverse proxy, then you can use whatever authentication scheme you like in the proxy server. The calibre server uses http auth set it to use basic http uath instead of digest and pass the Authorization header from the proxy server to the calibre server.

Would this be an option/possible to implement in NS? And would this be an option for other NON LDAP/AD applications to configure authentication? (like Gibbon for instance)


(Mark Verlinde) #4

I saw fedora has a package for calibre and tried to rebuild it for centos without succes:

Error: No Package found for python-cssutils >= 0.9.9
Error: No Package found for python-html5-parser
Error: No Package found for python-mechanize
Error: No Package found for python-qt5
Error: No Package found for python-qt5-devel
Error: No Package found for python-qt5-webkit

Stopped my experiment here, if this thread evolves further:
here my promise to have a go to build it for centos/nethserver.
(someone succeeded two years ago why shouldn’t we?
https://copr.fedorainfracloud.org/coprs/seuvitor/calibre/packages/)


(Marc) #5

Removed some settings from the howto, mainly due to problems with apache reverse proxy to a subdirectory (/calibre/) after the server address:

  • using HTTP and no Authentication: able to read books
  • using HTTP and Authentication: able to read and upload books, unable to convert them: ajax sends a POST request to http://FQDN-or-IP/conversion/start/3?library_id=library&sort=timestamp.desc that ends in 404 error (it’s missing the proxied subdirectory /calibre/, or the request should go to calibre server).
    The same without using the reverse proxy, directly accessing calibre server (http:FQDN-or-IP:8008/calibre/), works so the problem is in the reverse proxy settings.
Details
#Test Settings
## /etc/httpd/conf.d/zz_calibre.conf
AllowEncodedSlashes On
RewriteEngine on
RewriteRule ^/calibre/(.*) http://127.0.0.1:8008/calibre/$1 [proxy]
RedirectMatch permanent ^/calibre$ /calibre/

## /etc/systemd/system/calibre-server.service
[Unit]
Description=calibre content server
After=network.target

[Service]
Type=simple
User=calibre
Group=calibre
ExecStart=/opt/calibre/calibre-server --userdb=/var/lib/nethserver/calibre/users.sqlite --enable-auth --auth-mode=basic --access-log=/var/log/calibre-server-access.log --log=/var/log/calibre-server.log --port=8008 --url-prefix=/calibre "/var/lib/nethserver/calibre/library"

# Errors
Failed to communicate with "/conversion/start/3?library_id=library&sort=timestamp.desc", with status: [404] Not Found
The requested URL /conversion/start/3 was not found on this server.
[core:info] [pid 3714] [client 192.168.0.11:41600] AH00128: File does not exist: /var/www/html/conversion/start/3, referer: http://192.168.0.2/calibre/

# Request headers (excerpt)
POST /conversion/start/3?library_id=library&sort=timestamp.desc HTTP/1.1
Host: 192.168.0.2
Referer: http://192.168.0.2/calibre/
Content-Type: application/json;charset=UTF-8
Content-Length: 1991
Connection: keep-alive

# Response headers
HTTP/1.1 404 Not Found
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16
Content-Length: 217
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

Have been trying many apache directives, installed mod_proxy_html to try some of its directives too, but I don’t know apache enough to make it work. If someone knows about Apache reverse proxy and is able to help is more than welcome.

Looking at the bright side, a quick test of VirtualHost config seems to work but it’s missing SSL Certificate directives, at least, and some additional polishing:

Virtual Host
# Remember to create a record for the virtual host domain
# --url-prefix on calibre-server.service should not be necessary
# fw_calibreserver can be set to be accessed only from localhost
# --listen-on=127.0.0.1 can be added to calibre-server.service

<VirtualHost *:80>
    IncludeOptional conf.d/default-virtualhost.inc
</VirtualHost>

<VirtualHost *:80>
    ServerName calibre.domain.tld
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [L,R=301]
</VirtualHost>

<VirtualHost *:443>
    ServerName calibre.domain.tld
    SSLEngine On
    AllowEncodedSlashes NoDecode
    RewriteEngine On
    RewriteRule ^/(.*) http://127.0.0.1:8008/$1 [proxy]
</VirtualHost>

Once we find a better config we can add it to the howto.


Have not done any test and we must consider security implications. Here is some useful info:


(Thorsten) #6

Hi Marc,

Very nice … I did not dare that so far but I have a different approach:

I did something similar on July 25th (one day before vaccation) for my wife: I installed an Ubuntu-VM on Proxmox next to Nethserver. This machine is dedicated to Calibre and got a Virtual Host / Reverse Proxy entry from Nethserver. Requests are redirected from

https://books.mydomain.tld -> https://internal_IP_of_calibre_server:8080

On Nethserver I have a file share folder called ebooks. I installed @stepandl NFS module and mounted this share on the Ubuntu-VM

To be consideres

  • solution does not interfere with nethserver
  • solution allows access on port 80 (standard http)
  • solution is independend … if something goes wrong, it goes wrong …
  • I use the share from windows machines from which I do an upload of ebooks
    -> this requires a restart of the calibre instance each time a new ebook is uploaded
  • browsers on ebook readers and smart phones are not very happy with the redirection
  • to the best of myknowledge calibre does not allow LDAP / AD

…


(Thorsten) #7

I remember that I did something like that on Zentyal - but I did not save the config files of apache … however this will cause manual alteration of apache config files …


(Rob Bosch) #8

Hi @thorsten
If it can be changed manually, it should also be possible to use a custom template…


(Stéphane de Labrusse) #9

for apache authentication I love to use pam https://wiki.nethserver.org/doku.php?id=developer:authentication_through_apache#mod_authnz_pam
It works either for openldap or for samba AD, for NS7 it is really a pleasure to play with :slight_smile:


(Marc) #10

Just giving it a try these days but long road ahead. I’ve many doubts on how to proceed with the module (…at this hour, even questions I’m unaware of how to formulate in my mind :sweat_smile:).

TL/DR:

  • how to store properties for book libraries and handle paths collisions and naming of paths (which restrictions must be enforced).
  • rpm packaging options, installation/update method and how to handle uninstallation.
  • request for a working apache configuration to use calibre-server behind a reverse proxy from a subdirectory (http(s)://domain.tld/calibre). Must allow user authentication, upload of ebooks and conversion of ebooks, being the latter the main problem.
  • What about user authentication?
  • suggest default library name: now it’s the english word Books
  • port number for the service: picking 8008 (http alternate) was a temporary choice due to default port clashing with one from onlyoffice/collabora module/howto, IIRC.

Haven’t thought yet nor done anything with users, and nothing on GUI side.

What is more or less done...
  • db entries
  • service (templated, as it requires parameters from db)
  • apache virtual host template, via FQDN (…but reverse proxy to subdir not working); DNS record has to be set manually
  • update/save events and conf action (evolving creatures)
  • creation of a default library
  • logrotate (untested)
  • spec file able to build an rpm (but will need amendments)
Working on...
  • script to add/disable libraries (left aside for now)
  • samba share where to drop books to be automatically added to libraries via cron (not sure about using inotify). Great part of it based on the work done by @stephdl for transmission module.
    The idea was to have one network share with directories on its root matching the directory name of each library, but…

Main doubt is about libraries of books in relation to nethserver db. Calibre Server is able to manage multiple libraries (though users might be used to have just one).
ATM I’ve a Libraries db property (config show calibre Libraries) with a single field containing absolute paths to the libraries, using ; as separator:

/var/lib/nethserver/calibre/libraries/Books;/var/lib/nethserver/calibre/libraries/New Library

That can be managed but isn’t very comfortable and prone to error. ATM only need absolute paths but, if need for more options arise, we could consider having something like this (even with more fields):

LIBRARY_ID LIBRARY_PATH SAMBA_SHARE
MyBooks /var/lib/nethserver/calibre/libraries/MyBooks /mountpoint/MyBooks

Here I can do with your suggestions…

Was also thinking of having libraries with absolute paths and libraries that only specify the directory name:

  • libraries stored in db without slashes / (absolute path build from code later on):
    • Books
    • Comics, magazines and whatever
    • Ma bibliothèque personnelle
  • libraries with absolute path (starting with /):
    • /var/lib/nethserver/calibre/libraries/Comics
    • /another/path/where/sysadmin/set/permissions/for/calibre/user

Other entries wouldn’t be processed.

Some possible issues: symbols and international characters, libraries with same directory name (Books, books, /path/to/other/Books) and samba share.

The idea was to let sysadmin:

  • manually copy/upload existing calibre libraries registering them in the db for calibre-server to manage them.
  • use libraries stored in other paths
  • use provided or custom mounted samba share
  • auto add books from samba share: shall books be removed once added to calibre? (samba share could have a trash bin)

If you expect them to cause issues comment on it. If some feature is superfluous it can be dropped. Forgetting about custom library paths and custom samba mountpoints would ease it significantly.


About RPM package(s) and Calibre installation… ATM the update event checks if calibre is installed. If it is not, goes online to download the upstream installer script and runs it (pre-compiled binaries). Kind of a sacrilege for packagers.

Other options would be to integrated the installer script inside the rpm OR use a calibre-installer.rpm OR build from sources as @mark_nl intended. First two options suppose less work and maintenance, third option would be the only one for packagers but requires more work and could lead to fight with the removal/replacement of multiple components (due to licensing, package separation…) to comply with the fedora/redhat guidelines.

Kindly expecting your ideas and suggestions. Even answers to unasked questions, from mind-readers out there. Time to sleep on it.

EDIT: forgot about https://github.com/kovidgoyal/build-calibre


(Marc) #11

From localhost and user authentication books can be added while calibre-server is running (IIRC):

calibredb add --recurse "/path/to/books dir/" --with-library http://127.0.0.1:8008/#library_id --username theuser --password thepassword

To see the changes multiple browser refresh page could be required.
Haven’t tried yet but should be possible to do the same from CLI from a remote computer with calibre 3.x, by specifying the server FQDN or IP.


(Mark Verlinde) #12

As mentioned before, i’m old school, so would prefer to have a (s)rpm. As promised will try to build it and report back.

It’s a no-go if:

  • if the package is not stable.
  • if it is a maintaining nightmare, not easy to build : prerequisite is different versions and it’s dependency’s have to build in mock without throwing issues (ie pushing them to build service like copr should be easy)

We can be light-hearted regarding replacement of components due to licensing, RHEL / fedora are quite strict at is this. IIRC it dealt with non-opensource licensed fonts which could be free to use in the public domain.

Feedback on other topics follows as soon as getting more in to this package.

From a non- developer to a (self-claimed) non-developer: nice job: looks really clean :+1:


(Mark Verlinde) #13

Installed it bluntly using your work in progress package and completed it with some commands from your how to and it is up and running.

Imported 3000+ books and despite it threw quite some errors regarding book covers and unrecognized fonts (probably fault of the quality of the input-data rather than calibre-server) the vast majority looks to be oke.

Need to play with it more to give usefull feedback.

Recently i had a look at the mattermost package for something completely different;
however the usr/sbin/mattermost-bulk-user-create could be a inspiration for adding users to the userdb.
this in combination with
@rob https://www.mobileread.com/forums/showthread.php?t=309507
and
@stephdl https://wiki.nethserver.org/doku.php?id=developer:authentication_through_apache#mod_authnz_pam


(Marc) #14

Thanks for commenting on it, apprecited. Code on github doesn’t have the latest changes yet, because it’s not tested and was thinking to change structure of libraries on nethserver db.

Yep, have been using mattermost module as template.


(Mark Verlinde) #15

first no-go: do not know yet, did not install it (yet) because of second no-go;

It’s not easy to build :disappointed_relieved: code can be reveling: setup.py clearly says python > 2.7.9.

Centos is on 2.7.5, needed to build python >= 2.7.9 or build al calibre’s python dependency’s (it are a lot!) as a scl package. Opted for the first, which implied getting rid of centos python2-rpm-macros.

Although there is a Calibre package build against epel 7 in the result directory…
Won the battle but lost war. :rage:

To be continued…


(Marc) #16

Which one of the next options do you recommend, option 2?

1.) creating a new type property (calibrelib) inside configuration database:

# config set key type [prop1 val1] [prop2 val2] ...
config set library1 calibrelib [prop1 val1] [prop2 val2] ...
database key type properties/values
configuration library1 (name or id) calibrelib …

2.) creating a separate db, so calibre service config remains inside configuration database, and put libraries on calibrelib database:

# db dbfile set key type [prop1 val1] [prop2 val2] ...
db calibrelib set library1 library [prop1 val1] [prop2 val2] ...
database key type properties/values
calibrelib library1 (name or id) library …

(Mark Verlinde) #17

First some thoughts to completely confuse all of you :grinning:

Over the years kindoff learned not to try to organize big chunks of data in to directory’s according to a well-crafted scheme which at the end never seem to fit.
With other words store the data uncategorized and let computers do the filtering / querying afterwards.
Calibre desktop seems to have a virtual library mechanism, although it seems to be more a search utility rather than a mechanism to restrict access to certain sources. Moreover no clue if the content-server has this ability.

Just some thoughts

IMHO option 2 (separate) is most versatile, for those who do not agree with the above can go completely wild splitting up their libraries.

IIUC library are defined by their path and the (json) db lives in the root of this path. IF SO:

To keep is simple I would suggest to have a fixed prefix ( ie /var/lib/nethserver/calibre/libraries/ ) and on a fresh install one library with an arbitrary name without sub categories. (ie lib starts at the root)

If subcategories are desired you can make your well-crafted scheme and call /store them as (i.e.)
(key) library1 with an array of catogories (prop)'books','comics'
or
(key) libary2 (prop) '/books/computing','books/cooking','commics' (you get the drill)

These properties can be concatenated to the absolute path to the libraries (i.e.) /var/lib/nethserver/calibre/libraries/libary2/books/computing.

possible caveat can be creation of multiple library entries (record) for the same library.


(Mark Verlinde) #18

A heads up on the calibre rpm package;

Once the build environment is setup, dependencies and in-place python-2.7.9 update, the build process is not hard to do.
Successfully build a updated version of calibre (3.19.0 > 3.28.0) without issues and minor adjustments to the original FC29 spec file.

Did not do much testing, the content server runs and serves the “empty test book”.
Disadvantage: this package is aimed to be the desktop client and therefor pulls in stuff we do not need. (full installed dependency list here).
Not difficult to solve, once we know the dependencies of calibre-server the package can be split up into (meta) packages {common, calibre, server} to manage dependencies/requirements. The challenge is to find out the exact dependencies…and… the manual suggest some x11 components are needed.
(on the bottom of the page)

Note

The calibre server does not need a running X server, but it does need the X libraries installed as some components it uses link against them.

Not sure if the install script takes care of this.

For those who want to test this early WIP (= not for production !!!) package ; add this repo and yum install calibre

[markvnl-calibre]
name=Copr repo for calibre owned by markvnl
baseurl=https://copr-be.cloud.fedoraproject.org/results/markvnl/calibre/epel-7-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=1
gpgkey=https://copr-be.cloud.fedoraproject.org/results/markvnl/calibre/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1

Note: you have to edit the systemd unit to fit this how-to

EDIT:

  • updated calibre rpm to 3.29.0
  • To be able to read a book (offline) in the browser needed extra setting--auth-mode=basic in systemd-unit…and…
  • extra settings SetEnv force-proxy-request-1.0 1 and SetEnv proxy-nokeepalive 1 in apache conf NOTE: “copied” this apache setting from a forum still need to examine it in detail.

Still not convinced it’s the way to go (rpm package vs install script) :thinking:
A bit more optimistic it could be an option :grinning:


(Thorsten) #19

Just for completion purpose, I started my old Zentyal server for the last time and colleced the old apache config file (just the user part). Do not worry about the random LDAP-Password - the server is formatted now :slight_smile:

<Directory "/srv/www/bib.myname.dyndns.org">
AllowOverride All
</Directory>

ProxyRequests On
ProxyVia On
ProxyPreserveHost On
 
<Proxy *>
Order deny,allow
Allow from 172.17.0
</Proxy>

SSLProxyEngine On  
ProxyPass / https://172.17.0.31:8080
ProxyPassReverse / https://172.17.0.31:8080

<Location />

Order deny,allow
Deny from all
Allow from 172.17.0
Allow from 127.0.0.1
Allow from localhost

AuthType Basic
AuthBasicProvider ldap

AuthName "bib.myname.dyndns.org"
AuthLDAPURL "ldap://localhost:390/dc=myname,dc=dyndns,dc=org?uid"
AuthzLDAPAuthoritative on
AuthLDAPBindDN "cn=zentyalro,dc=myname,dc=dyndns,dc=org"
AuthLDAPBindPassword "2WwvTTSAQx9RyvIM/6FY"
Require valid-user
Require ldap-group cn=calibre,ou=groups,dc=myname,dc=dyndns,dc=org

# Satisfy any
</Location> 

Some notes:

  • Zentyal ldap was running on port 390
  • I created a group “calibre” populated with allowed Zentyal users