102 Create LEEF docker Host

This document is in a draft version and needs to be tidied up

Pipeline Server

Before starting, make sure that you have enough space for the LEEF data upload and processing. This can be done either by having a large enough system drive or have a second drive mounted. For this document, we will assume that the space for the LEEF data is at /mnt/leef_data.

basic stuff

sudo timedatectl set-timezone Europe/Zurich
sudo apt-get update
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y install mc zile keyutils
sudo apt-get -y install make mailutils

sudo reboot now

Install and setup samba client

install cifs samba client and smbclient

sudo apt-get -y install cifs-utils smbclient

Prepare for the mounting of the volume

 sudo mkdir /mnt/leef_data
 sudo chown -R ubuntu:ubuntu /mnt/leef_data
 sudo touch /mnt/leef_data/NOT_MOUNTED
 ###
 # depends on local setup!
 sudo mount /dev/vdb /mnt/leef_data/
 ###
 sudo chown -R ubuntu:ubuntu /mnt/leef_data
 # sudo umount /mnt/leef_data/

install docker and setup of user

install docker

sudo apt-get -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt update

# apt-cache policy docker-ce

sudo apt -y install docker-ce

Check if running

sudo systemctl status docker

Add user ubuntu to docker group

sudo usermod -aG docker ${USER}

Now logout and in again to activate the new memberships.

Samba Server

add mounting to fstab

sudo cp /etc/fstab /etc/fstab.old
sudo lsblk -o NAME,FSTYPE,UUID,SIZE,LABEL
sudo zile /etc/fstab

Add the following line, but adapt the UUIB to the actual one

# UUID=a10379cf-40c6-490a-9233-1dee0552b953 /mnt/leef_data ext4 noauto,x-systemd.automount,x-systemd.idle-timeout=300 0 0
sudo mount -a

install and configure samba

sudo apt-get -y install samba

configure samba

sudo cp -pf /etc/samba/smb.conf /etc/samba/smb.conf.bak
sudo addgroup smbgrp
sudo useradd leef -G smbgrp
sudo smbpasswd -a leef
sudo chmod -R 0770 /mnt/leef_data/LEEF /mnt/leef_data/Diverse /mnt/leef_data/sample\ one\ day
sudo chown -R ubuntu:smbgrp /mnt/leef_data/LEEF /mnt/leef_data/Diverse /mnt/leef_data/sample\ one\ day

sudo zile /etc/samba/smb.conf

The file should be as followed:

#======================= Global Settings =======================

[global]
   workgroup = WORKGROUP
   dns proxy = no

#### Networking ####

#### Debugging/Accounting ####

   log file = /var/log/samba/log.%m

   syslog = 0

   panic action = /usr/share/samba/panic-action %d


####### Authentication #######

   passdb backend = tdbsam

   obey pam restrictions = yes

   unix password sync = yes

   passwd program = /usr/bin/passwd %u
   passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .

   pam password change = yes

   map to guest = bad user

    ########## Domains ###########

    ############ Misc ############

   usershare allow guests = yes

    #======================= Share Definitions =======================

    [leef_data]
    comment = LEEF Data
       path = /mnt/leef_data
       browsable = yes
       writable = yes
   read only = no
   guest ok = no

Now restart samba

sudo service smbd restart

Install tools for SWIFT object store

See S3IT for details

sudo apt-get update
sudo apt install python-pip

Pull LEEF-UZH/docker repo and the leef docker image.

cd
git clone https://github.com/LEEF-UZH/docker.git
cd docker
make pull

In order to setup your environment, please download Openstack RC file from ScienceCloud GUI by: Access & Security > API Access > Download OpenStack RC file.

Then import this variables into your shell environment by:

source /path/to/downloaded/rc_file.sh

This needs to be done before each using of the commands.

Better: Service account.

Follow S3IT Info to set this up.

=======================================================

=======================================================

Old stuff which is not needed anymore

Install syncthing and configure for leef_data volume

This is not needed at the moment, although it was a nice idea. Still here to remember how it was done.

echo “deb https://apt.syncthing.net/ syncthing stable” | sudo tee /etc/apt/sources.list.d/syncthing.list curl -s https://syncthing.net/release-key.txt | sudo apt-key add - printf “Package: *: origin apt.syncthing.net-Priority: 990” | sudo tee /etc/apt/preferences.d/syncthing sudo apt-get update sudo apt-get install syncthing

Edit ~/.config/syncthing/config.xml

Change

    <gui enabled="true" tls="false" debugging="false">
        <address>127.0.0.1:8384</address>

to

    <gui enabled="true" tls="false" debugging="false">
        <address>0.0.0.0:8384</address>

to enable global access ### Start syncthing

sudo systemctl enable [email protected]
sudo systemctl start [email protected]

Install cifs to be able to mount smb shares

This is not needed at the moment, as all transfers are done remotely using rsync.

add mounting to fstab

some tries which did not work

Add The SAMBA Share

sudo cp /etc/fstab /etc/fstab.old
sudo zile /etc/fstab

Add the following lines

# THIS DOES NOT WORK# leef_§data/leef_data /mnt/leef_data cifs -o user,credentials=/usr/local/etc/whisper.credentials,uid=1000,umask=000,noauto,x-systemd.automount,x-systemd.idle-timeout=300 0 0
# //leef_data/leef_data   /mnt/leef_data cifs     rw,credentials=/usr/local/etc/whisper.credential,uid=ubuntu,gid=ubuntu,file_mode=0770,dir_mode=0770,x-systemd.mount-timeout=300 0 0
# /mnt/leef_data/LEEF /home/ubuntu/LEEF        none    bind

finally

sudo mount -a

Configure everything for kerberos / the D.UZH.CH domain (as root):

sudo zile /etc/krb5.conf

relevant new lines (add them if missing):

[libdefaults]
       default_realm = D.UZH.CH

[realms]
       D.UZH.CH = {
               kdc = d.uzh.ch
               admin_server = d.uzh.ch
               default_domain = uzh.ch
       }

[domain_realm]
       d.uzh.ch = D.UZH.CH