Gitea A painless self-hosted Git service

danb35 · August 23, 2018, 10:08am

Yes, that did the trick–now able to clone and push via SSH.

danb35 · August 23, 2018, 11:51am

It’ll be a couple of days before I’m able to test backup/restore, but I just want to make sure how it’s supposed to work:

Run data backup of server with gitea installation
Do a fresh Neth installation
Restore from the data backup
Install nethserver-gitea
???
Profit?

If I’m following this thread correctly, I shouldn’t need to do anything extra after restoring and installing nethserver-gitea, right? Repos and all other data should be visible as they were before the backup?

mark_nl · August 23, 2018, 12:13pm

the only scenario tested now is: deleted all files from /var/lib/nethsever/gitea and the mysql database > restored them for backup.

Before claiming this package is any good I want to test:

bluntly overwrite all existing data with restore from backup
remove data of one repository > restore repo from backup
remove database > restore db from backup
http://docs.nethserver.org/en/latest/backup.html#disaster-recovery

The latter would fit your test-case the best.
i’ll keep you posted if something pops up

Edit: numbered topics for ease of reference

danb35 · August 23, 2018, 12:27pm

I’d think this should be possible with restoring that directory–can test this.

That sounds problematic, unless each database is dumped separately during the pre-backup event.

Sounds like a few scenarios to test–I should be able to bang on this a bit, but it will likely be a few days.

mark_nl · August 23, 2018, 12:47pm

Never did this before
looking in the backup data set found:
duplicity-full.20180822T060014Z.vol1.difftar/snapshot/var/lib/nethserver/backup/mysql/gitea.dump

Hence the idea for the test-case ;
the character set of the db worries me if someone goes wild on emoticons.

@all_others are there more critical test-cases regarding backup and restore?

danb35 · August 23, 2018, 12:59pm

That does look like they dump the databases individually, so yes, this case could work.

mark_nl · August 23, 2018, 7:04pm

With a small data set (3 small repo’s)

survived this
(A) deleted data from one repo . put it back and restarted gitea : oke
(B) removed repo (nethserver-gitea) in web-gui > danger zone; albeit all warnings this can’t be undone
created new empty repo in web gui (=nethserver-gitea)
restored the nethserver-gitea repo files to /tmp/nethserver-gitea

git clone /tmp/nethserver-gitea nethserver-gitea
git remote remove origin
git remote add origin https://_hostname_/git/_user_/nethserver-gitea.git
git push -u origin master

all commits and code resorted, missing is other data such as issues

rm -rf /var/lib/mysql/gitea/* ; gitea dies
systemctl stop gitea
restored gitea.dump (/var/lib/nethserver/backup_2018-08-23/mysql)
need to drop (remove) the database and recreate it
mysql --defaults-file=/root/.my.cnf -e "DROP DATABASE gitea"
mysql --defaults-file=/root/.my.cnf -e "CREATE DATABASE IF NOT EXISTS gitea;"
mysql --defaults-file=/root/.my.cnf gitea < /var/lib/nethserver/backup_2018-08-23/mysql/gitea.dump
systemctl start gitea
up and running again
still to do

danb35 · August 24, 2018, 11:47pm

Hmmm. Rolled back the test VM to a clean state, built nethserver-gitea-0.0.4, and installed that (along with dependencies). Now can’t log in at all. Not with gitea/gitea, not with a local user on the system.

Edit: Yeah, must be some difference in how authentication is handled between 0.0.3 and 0.0.4. When I rolled the VM back to a clean state and reinstalled 0.0.3, I could log in with gitea/gitea. When I repeated, but installed 0.0.4, I get this when I try:

Not seeing anything in the logs yet, but I’m sure I haven’t looked everywhere. Authentication attempts don’t seem to hit /var/log/gitea/gitea.log, even when I set the log level to Debug in app.ini.

mark_nl · August 25, 2018, 6:15am

Cannot reproduce it yet, will try to do so!

Biggest change between 0.0.3 and 0.0.4 is how the database is populated with tables.
A bit of background: The workflow of a “normal” non-nethserver install is to configure it in the web-gui on first run. This means the user need to provide the the password of the database, create a admin user and configure a login source.
To face this chalange the database is populated with tables up front, a (local) admin user is added and a record for PAM login source is directly written in the database.

Until 0.0.3 the data base was populated with a dump of the structure of the db from a (once) running instance. In 0.0.4 during setup (conf) gitea is started up for 2 seconds to populate the db with tables. The advantage is the “0.0.3” method is a maintaining nightmare, for new installs the version of provided db structure must be inline with the current gitea version.

In short: most likely at your install the db is not properly populated with tables the following commands have failed:

# populate database with tables on the fly by running gitea for 2s
# and add a local admin user

pushd /var/lib/nethserver/gitea
timeout 2s su gitea -c \
"GITEA_WORK_DIR='/var/lib/nethserver/gitea' /usr/bin/gitea web --config /etc/gitea/app.ini"
su gitea -c \
"GITEA_WORK_DIR='/var/lib/nethserver/gitea' /usr/bin/gitea admin create-user --name gitea --password 'gitea' --email gitea@example.com --admin --config /etc/gitea/app.ini"
popd

# add PAM login source
epoch=$(date +%s)
/usr/bin/mysql --defaults-file=/root/.my.cnf -e \
"INSERT INTO gitea . login_source (id, type, name, is_actived, is_sync_enabled, cfg, created_unix, updated_unix) 
VALUES ('1', '4', 'Nethserver', '1', '0', '{\"ServiceName\":\"password-auth\"}', '$epoch', '$epoch');"

EDIT(2): ~~thinking out loud now; to speed up testing i use an image with nethserver-mysql already installed. maybe that’s why it works for me, will tests this!~~ Nope can’t reproduce yet.

Thanx for the report!

mark_nl · August 25, 2018, 7:40am

clean install with only ldap account provider all is fine, journalctl also shows the above mentioned provisioning succeed:

... usermod[2865]: change user 'gitea' home from '/var/lib/gitea' to '/var/lib/nethserver/gitea'
...  esmith::event[2858]: /var/lib/nethserver/gitea ~
... su[2886]: (to gitea) root on none
... su[2886]: pam_unix(su:session): session opened for user gitea by root(uid=0)
... esmith::event[2858]: 2018/08/25 09:19:43 [T] AppPath: /usr/bin/gitea
... esmith::event[2858]: 2018/08/25 09:19:43 [T] AppWorkPath: /var/lib/nethserver/gitea
... esmith::event[2858]: 2018/08/25 09:19:43 [T] Custom path: /var/lib/nethserver/gitea/custom
... esmith::event[2858]: 2018/08/25 09:19:43 [T] Log path: /var/log/gitea/
... esmith::event[2858]: 2018/08/25 09:19:44 Serving [::]:3000 with pid 2887
... su[2886]: pam_unix(su:session): session closed for user gitea
... esmith::event[2858]:
... esmith::event[2858]: Session terminated, killing shell...2018/08/25 09:19:45 Exiting pid 2887.
... esmith::event[2858]:  ...killed.
... su[2905]: (to gitea) root on none
... su[2905]: pam_unix(su:session): session opened for user gitea by root(uid=0)
... esmith::event[2858]: 2018/08/25 09:19:47 [I] XORM Log Mode: File(Warn)
... esmith::event[2858]: New user 'gitea' has been successfully created!
... su[2905]: pam_unix(su:session): session closed for user gitea
... esmith::event[2858]: ~
... esmith::event[2858]: Action: /etc/e-smith/events/nethserver-gitea-update/S30nethserver-gitea-conf SUCCESS [4.508758]
... systemd[1]: Reloading.

usermod “home dir” oke
start and killing gitea after 2s (this is the “hack” to populate the db with tables) oke
tables are in place, otherwise New user 'gitea' has been successfully created! would have failed.

“Inserting” of record for PAM login source is not logged ( it’s on my todo list now) but did work for me.

Strange

danb35 · August 25, 2018, 11:40am

I don’t know systemd well at all, so didn’t know about journalctl. So, back to baseline on the test VM, scp over the gitea and nethserver-gitea RPMs, and yum install *.rpm while running journalctl -f in another terminal. It looks like this is the relevant portion:

Aug 25 07:33:31 neth-test.familybrown.org esmith::event[2873]: Action: /etc/e-smith/events/nethserver-gitea-update/S00initialize-default-databases SUCCESS [0.380144]
Aug 25 07:33:31 neth-test.familybrown.org esmith::event[2873]: expanding /etc/gitea/app.ini
Aug 25 07:33:31 neth-test.familybrown.org esmith::event[2873]: expanding /etc/httpd/conf.d/default-virtualhost.inc
Aug 25 07:33:31 neth-test.familybrown.org esmith::event[2873]: expanding /etc/httpd/conf.d/zz_gitea.conf
Aug 25 07:33:31 neth-test.familybrown.org esmith::event[2873]: expanding /etc/pki/gitea/certs/gitea.crt
Aug 25 07:33:31 neth-test.familybrown.org esmith::event[2873]: expanding /etc/pki/gitea/private/gitea.key
Aug 25 07:33:31 neth-test.familybrown.org esmith::event[2873]: expanding /etc/sysconfig/gitea
Aug 25 07:33:31 neth-test.familybrown.org esmith::event[2873]: Action: /etc/e-smith/events/actions/generic_template_expand SUCCESS [0.34837]
Aug 25 07:33:31 neth-test.familybrown.org usermod[2880]: change user 'gitea' home from '/var/lib/gitea' to '/var/lib/nethserver/gitea'
Aug 25 07:33:31 neth-test.familybrown.org esmith::event[2873]: /var/lib/nethserver/gitea ~
Aug 25 07:33:32 neth-test.familybrown.org su[2901]: (to gitea) root on none
Aug 25 07:33:32 neth-test.familybrown.org su[2901]: pam_unix(su:session): session opened for user gitea by root(uid=0)
Aug 25 07:33:32 neth-test.familybrown.org esmith::event[2873]: 2018/08/25 07:33:32 [T] AppPath: /usr/bin/gitea
Aug 25 07:33:32 neth-test.familybrown.org esmith::event[2873]: 2018/08/25 07:33:32 [T] AppWorkPath: /var/lib/nethserver/gitea
Aug 25 07:33:32 neth-test.familybrown.org esmith::event[2873]: 2018/08/25 07:33:32 [T] Custom path: /var/lib/nethserver/gitea/custom
Aug 25 07:33:32 neth-test.familybrown.org esmith::event[2873]: 2018/08/25 07:33:32 [T] Log path: /var/log/gitea/
Aug 25 07:33:34 neth-test.familybrown.org su[2901]: pam_unix(su:session): session closed for user gitea
Aug 25 07:33:34 neth-test.familybrown.org esmith::event[2873]: 
Aug 25 07:33:36 neth-test.familybrown.org esmith::event[2873]: Session terminated, killing shell... ...killed.
Aug 25 07:33:36 neth-test.familybrown.org su[2916]: (to gitea) root on none
Aug 25 07:33:36 neth-test.familybrown.org su[2916]: pam_unix(su:session): session opened for user gitea by root(uid=0)
Aug 25 07:33:36 neth-test.familybrown.org esmith::event[2873]: 2018/08/25 07:33:36 [I] XORM Log Mode: File(Warn)
Aug 25 07:33:36 neth-test.familybrown.org esmith::event[2873]: CreateUser: Error 1146: Table 'gitea.email_address' doesn't exist
Aug 25 07:33:36 neth-test.familybrown.org su[2916]: pam_unix(su:session): session closed for user gitea
Aug 25 07:33:36 neth-test.familybrown.org esmith::event[2873]: ~
Aug 25 07:33:36 neth-test.familybrown.org esmith::event[2873]: ERROR 1146 (42S02) at line 1: Table 'gitea.login_source' doesn't exist
Aug 25 07:33:36 neth-test.familybrown.org esmith::event[2873]: Action: /etc/e-smith/events/nethserver-gitea-update/S30nethserver-gitea-conf FAILED: 1 [4.39021]

mark_nl · August 25, 2018, 12:00pm

... esmith::event[2873]: 2018/08/25 07:33:32 [T] Log path: /var/log/gitea/
... su[2901]: pam_unix(su:session): session closed for user gitea

vs my log

... esmith::event[2858]: 2018/08/25 09:19:43 [T] Log path: /var/log/gitea/
... esmith::event[2858]: 2018/08/25 09:19:44 Serving [::]:3000 with pid 2887
... su[2886]: pam_unix(su:session): session closed for user gitea

gitea does not come in a full running state Serving [::]:3000 with pid 2887

question is why?

you slould be able to “save” your install with

systemctl stop gitea
pushd /var/lib/nethserver/gitea

su gitea -c \
"GITEA_WORK_DIR='/var/lib/nethserver/gitea' /usr/bin/gitea admin create-user --name gitea --password 'gitea' --email gitea@example.com --admin --config /etc/gitea/app.ini"

popd

export epoch=$(date +%s)

/usr/bin/mysql --defaults-file=/root/.my.cnf -e \
"INSERT INTO gitea . login_source (id, type, name, is_actived, is_sync_enabled, cfg, created_unix, updated_unix) 
VALUES ('1', '4', 'Nethserver', '1', '0', '{\"ServiceName\":\"password-auth\"}', '$epoch', '$epoch');"

systemctl start gitea

But this is unacceptable!

Maybe ease the time gitea is running to build up the database (2 second now) could be an answer although i cant believe a RPI is faster than what-ever-other-environment.

danb35 · August 25, 2018, 12:06pm

Yes, that fixes it:

[root@neth-test ~]# systemctl stop gitea
[root@neth-test ~]# pushd /var/lib/nethserver/gitea
/var/lib/nethserver/gitea ~
[root@neth-test gitea]# su gitea -c \
> "GITEA_WORK_DIR='/var/lib/nethserver/gitea' /usr/bin/gitea admin create-user --name gitea --password 'gitea' --email gitea@example.com --admin --config /etc/gitea/app.ini"
2018/08/25 08:02:13 [I] XORM Log Mode: File(Warn)
New user 'gitea' has been successfully created!
[root@neth-test gitea]# popd
~
[root@neth-test ~]# export epoch=$(date +%s)
[root@neth-test ~]# /usr/bin/mysql --defaults-file=/root/.my.cnf -e \
> "INSERT INTO gitea . login_source (id, type, name, is_actived, is_sync_enabled, cfg, created_unix, updated_unix) 
> VALUES ('1', '4', 'Nethserver', '1', '0', '{\"ServiceName\":\"password-auth\"}', '$epoch', '$epoch');"
[root@neth-test ~]# systemctl start gitea
[root@neth-test ~]#

I’m now able to log in both as gitea and also as a Neth user.

That would seem like the obvious culprit, but I’ve run into this on three separate installations on a decently-resourced VM. If two seconds is adequate on a Pi, it would seem like at least one of these ought to work.

mark_nl · August 25, 2018, 12:08pm

if you are whiling and have the time you could try to rebuild nethserver-gitea after changing the timeout time to something ridiculously long as 10s

timeout 10s su gitea -c \

/root/etc/e-smith/events/actions/nethserver-gitea-conf

github.com

markVnl/nethserver-gitea/blob/master/root/etc/e-smith/events/actions/nethserver-gitea-conf#L52


/usr/bin/mysql --defaults-file=/root/.my.cnf -e \
"CREATE USER gitea@localhost IDENTIFIED BY '$password';"
/usr/bin/mysql --defaults-file=/root/.my.cnf -e \
"GRANT ALL PRIVILEGES ON gitea .* TO gitea@localhost;"


# populate database with tables on the fly by running gitea for 2s
# and add a local admin user
expand-template /etc/gitea/app.ini


pushd /var/lib/nethserver/gitea
timeout 2s su gitea -c \
"GITEA_WORK_DIR='/var/lib/nethserver/gitea' /usr/bin/gitea web --config /etc/gitea/app.ini"
su gitea -c \
"GITEA_WORK_DIR='/var/lib/nethserver/gitea' /usr/bin/gitea admin create-user --name gitea --password 'gitea' --email gitea@example.com --admin --config /etc/gitea/app.ini"
popd


# add PAM login source
epoch=$(date +%s)
/usr/bin/mysql --defaults-file=/root/.my.cnf -e \
"INSERT INTO gitea . login_source (id, type, name, is_actived, is_sync_enabled, cfg, created_unix, updated_unix) 
VALUES ('1', '4', 'Nethserver', '1', '0', '{\"ServiceName\":\"password-auth\"}', '$epoch', '$epoch');"

danb35 · August 25, 2018, 12:18pm

With that change, it works. I built 0.0.4-2 with (only) that change, rolled back the VM to a clean state, installed gitea and nethserver-gitea, and I’m able to log in to the installation both as gitea and as a Neth user.

mark_nl · August 25, 2018, 12:20pm

thanx for helping me out here!

danb35 · August 25, 2018, 12:25pm

Now back to checking the backup/restore scenarios…

Edit: clone and push via HTTPS and SSH continue to work.

mark_nl · August 25, 2018, 2:13pm

nice!

BTW until a better solution is found, an if loop until the tables are available comes in to mind
i’ve set the time out to even 20s better save than sorry; it’s only done once on new installs…

danb35 · August 25, 2018, 7:58pm

Is there a configuration to make Gitea use its own credentials? Like my Dokuwiki install, I wouldn’t really expect that other users with Neth accounts on my system (my family, in my case) would have be editing or adding content, but other folks could. It’d be helpful if it could do its own authentication.

mark_nl · August 25, 2018, 8:18pm

Yes there are a lot of possibilities, including AD/Ldap bind using group permisions.
Still need to figure out the right group filter syntax to accomplish this.

https://docs.gitea.io/en-us/authentication/

you can simply deactivate the PAM authentication.
as admin user (gitea) in drop down menu Site Administration > Authentication Sources > Edit (PAM) > uncheck “is active”

this is a partial solution because everybody can register…

In my local wip branch this configuration option in the app.ini at [service] section is already moved to the esmith-db. DISABLE_REGISTRATION: false

https://docs.gitea.io/en-us/config-cheat-sheet/

EDIT: if you want to bind to ad/ldap i can share my finding so far
EDIT2 or make a pam unit for a group