Info | ||
---|---|---|
| ||
This tutorial is still under development, but nearly finalized. Please try out and post any necessary comment! |
Metadata list | ||
---|---|---|
| ||
|| workpackage | WP6 | || task | 2 | || document number | 004 | || document version | 1.6 | || document title | EPN-TAP Server Installation for VESPA Data Provider Tutorial | || document type | TD | |
EPN2020-RI
EUROPLANET2020 Research Infrastructure
H2020-INFRAIA-2014-2015
Grant agreement no: 654208
Document: VESPA-
- Metadata from 1 workpackage
- Metadata from 1 task
- Metadata from 1 document number
-v Metadata from 1 document type Metadata from 1 document version
Metadata from | ||
---|---|---|
|
Date:
Start date of project: 01 September 2015
Duration: 48 Months
Responsible WP Leader: OBSPARIS / Stéphane Erard
Project co-funded by the European Union's Horizon 2020 research and innovation programme | ||
Dissemination level | ||
PU | Public | |
PP | Restricted to other programme participants (including the Commission Service) | |
RE | Restricted to a group specified by the consortium (including the Commission Services) | |
CO | Confidential, only for members of the consortium (excluding the Commission Services) |
Project Number | 654208 |
Project Title | EPN2020 - RI |
Project Duration | 48 months: 01 September 2015 – 30 August 2019 |
Pagebreak |
---|
Document Number | VESPA-
| |||||||||||||||||||||
Delivery date | 2015.11.21 | |||||||||||||||||||||
Title of Document |
| |||||||||||||||||||||
Contributing Work package (s) |
| |||||||||||||||||||||
Dissemination level | PU | |||||||||||||||||||||
Author (s) | Baptiste Cecconi, Pierre Le Sidaner, Stéphane Erard, Renaud Savalle, Markus Deimletner, Paul Sladen, Mikhail Minin |
Abstract: |
Document history (to be deleted before submission to Commission) | ||||||||||
Date | Version | Editor | Change | Status | ||||||
| 0.1 | First issue (imported from internal MS-Word document) |
| |||||||
| 0.2 | Updated AWStats and Apache Proxy configuration |
| |||||||
| 0.3 | Fixed several errors |
| |||||||
| 0.4 | Simplified the server configuration (no more proxy) |
| |||||||
0.5 | Update of first installation step. |
| ||||||||
0.6 | Changed Awstats configuration |
| ||||||||
0.7 | Added DaCHS web configuration (title and logos) Fixed /etc/gavo.rc configuration |
| ||||||||
0.8 | Added screenshots and some adding testing steps |
| ||||||||
0.9 | Modification of the ordering of Apache and DaCHS configuration, and various small changes for better clarity. |
| ||||||||
0.10 | Added PgAdmin section. Added notes on Virtual Box network configuration. Small modifications |
| ||||||||
0.11 | Small modifications |
| ||||||||
1.0 | Public Release |
| ||||||||
1.1 | Changed PgAdmin configuration. Changed document name. |
| ||||||||
1.2 | Fixed /etc/gavo.rc file |
| ||||||||
1.3 | Going back to draft status for updating tutorial. Including "test" and "prod" set up. |
| ||||||||
1.4 | Modified /etc/apt/sources.list input. |
| ||||||||
| 1.5 | replay installation tutorial, update the documentation |
| |||||||
| 1.5 | typos, fixes, etc... |
| |||||||
1.6 | add logFormat: combined to gavo.rc to use awstats in curent DaCHS version |
|
Table of Contents1.
Table of Contents |
---|
Pagebreak |
---|
1. Introduction
This document presents the installation procedure to set up data distribution server for VESPA. We propose here to set up three service interfaces on the same physical server (and the same public IP address): a standard web server (Apache) to serve data files, a tool to have access statistics (using AWStats), and a DaCHS server to provide the Virtual Observatory (VO) interfaces (including a TAP service). Two server names have to be set up: one for the data access and the statistics, and the other for the VO access:
- http://my_servername.my_domain VO interface
- http://my_servername.my_domain:8080 Data repository and usage statistic service
Section 2 describes the Debian distribution installation. Section 3 describes the installation of DaCHS and its configuration. Section 4 describes the installation and configuration of AWStatAWStats. Section 5 describes the configuration of the Apache server and of DNS.
Further configuration of the VO interface is described in the second part of the tutorial.
Info |
---|
In all this document, you will have to replace all text between |
In order to prepare the installation, here is a table to be filled with the parameters of you installation:
Parameter Name | Your Value | Example Value for tutorial |
---|---|---|
my_servername | .................................................................................. | voparis-test-bc |
my_domain | .................................................................................. | obspm.fr |
maintainer_email (1) | .................................................................................. | vo.paris@obspm.fr |
authority-ivo-name (2) | .................................................................................. | vopdc.obspm |
server_title | .................................................................................. | My new test DaCHS server |
my_user | .................................................................................. | user |
(1) This should be a generic address, rather than a specific person address.
(2) Read the Service Identifier page when filling these lines. Note that this specific item do not contain "ivo://
" at the beginning.
2. Debian Stable (Jessie) Installation
DaCHS (Data Center Helper Suite) must be installed on a fresh Debian Stable distribution .
2.1 Install virtualization software
Test | Production | |||||||
---|---|---|---|---|---|---|---|---|
The installation demonstration is done on a Virtual Box guest computer, so you first need to install VirtualBox on your computer. On Debian based systems:
On other systems, download VirtualBox and follow the installation instructions.
| For a real installation, a physical server (or a bare metal virtualization framework) should be preferred. |
2.2 Download Debian
The current installation image for this distribution is available here: https://www.debian.org/distrib/netinst#smallcd (under Small CDs or USB sticks). In this example, we used the following ISO image: debian-8.6.0-amd64-netinst.iso
.
2.3 Virtual machine installation
Test | Production | |||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Click on the New icon, then fill:
Click create on the last step, then click on the Configuration icon. Go to Storage, on IDE controller, click on Add optical drive, Choose a disk, and select your downloaded iso. | Nothing to do. |
2.4 Network configuration
Test | Production | |||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Always in Configuration window, go to Network, keep "NAT" for the default network access mode, then click on Port forwarding and add 4 rules as the following:
| For operation server, you must have a public address. The DNS configuration must be completed. Ports to be open:
|
2.5 Debian installation
Step | Screenshot | Configuration | ||||||
---|---|---|---|---|---|---|---|---|
Installation selection | Select "Install " | |||||||
Language used for the installation process, and for the system, once installed | Select " English " | |||||||
Location of the installation | Select your country | |||||||
Locale used for the installation | Select "en-US.UTF-8 " | |||||||
Keyboard map | Select your keyboard map | |||||||
Hostname for this computer | Type the hostname (here "
| |||||||
Domain name for your institute | Type the domain name (here "
| |||||||
Root password | Enter root password (pick a password, which is not easy to guess and don't forget it!) | |||||||
Main user configuration | Enter Full Name, user name and password In the following, | |||||||
Disk configuration | Select "Use entire disk " | |||||||
Select Disk to partition | Select the main disk (there should be only one). | |||||||
Partition Scheme | Select "All files in one partition " | |||||||
Confirmation of partition set up | Select "Finish partioning and write changes to disk " | |||||||
write partition to disk | select <Yes> | |||||||
Configuration of package manager | Select a mirror close to your place | |||||||
Proxy configuration | If needed enter the proxy information to go on internet (if your internet access is via a proxy, otherwise leave it blank). | |||||||
Configuring popularity-contest | This is up to you (here "No " is selected) | |||||||
Software selection | Select a minimal set of package, namely "
| |||||||
Installation of GRUB boot loader, part 1 | Select "yes " | |||||||
Installation of GRUB boot loader, part 2 | Select the newly created partition. | |||||||
Installation completion | Select "continue " to finish installation and reboot. | |||||||
Reboot | Go to next section |
From now, it is recommended to access to your server through ssh:
Test (from a terminal, or use PuTTY on Windows) | Production |
---|---|
ssh -p 2222 <<my_user>>@localhost | ssh <<my_user>>@<<my_servername>>.<<my_domain>> |
And you don't need to use the guest interface anymore.
Now you will be able (and you should) to use copy-paste along the tutorial.
Info | |||||
---|---|---|---|---|---|
To change the keyboard map on your guest, you have to edit a file:
|
3. Install and configure sudo
On the server, login with the <<my_user>>
user created during installation (either on the computer itself, or remotely with ssh).
The first thing is to install and set up the sudo command, which allows a user to issue commands with root rights. Use the su command to log as root, and type the root password.
Code Block | ||
---|---|---|
| ||
su |
Install the sudo package:
Code Block | ||
---|---|---|
| ||
apt-get install sudo |
Set up rights for your regular user to use the sudo command:
Code Block | ||
---|---|---|
| ||
adduser "<<my_user>>" sudo |
You have now the log out from your root shell:
Code Block | ||
---|---|---|
| ||
exit |
and your regular user shell:
Code Block | ||
---|---|---|
| ||
exit |
(this will disconnect your ssh connection, so you have to connect again)
Now you will be able to use the sudo command.
4. Apache Configuration
Connect to the server with ssh
. Apache should already be installed during initial configuration, otherwise :
Code Block | ||
---|---|---|
| ||
sudo apt-get install apache2 |
Enable ModCGI:
Code Block |
---|
sudo a2enmod cgid |
Update the default Apache port to 8080 instead of 80 by changing the /etc/apache2/ports.conf
file, using your preferred command line editor, for instance, you can use the pico editor (useful short cuts: ^O to save, ^X to quit):
(sometimes you have to check the line NameVirtualHost *:8080 in ports.conf and <VirtualHost *:8080> in /etc/apache2/sites-available/default)
Code Block |
---|
sudo pico /etc/apache2/ports.conf |
Modify the first line containing the Listen
command and replace the port number:
Code Block |
---|
Listen 8080 |
Then restart Apache2:
Code Block | ||
---|---|---|
| ||
sudo service apache2 restart |
You can check that your regular web server is running connecting to the URL: http://<<my_servername>>.<<my_domain>>:8080/
. You should see a standard splash page:
/var/www/html/
. Pagebreak
5. DaCHS Installation and Configuration
There are several ways to install DaCHS as presented by his author in [1]. For DaCHs installation you need an Debian Stable distribution, at this time Jessie (see Section 2).
Install DaCHS framework
The repository source list must be updated to include DaCHS packages. Edit the file /etc/apt/sources.list
:
Code Block | ||
---|---|---|
| ||
sudo pico /etc/apt/sources.list |
Add the following line at the end of file (see Fig. 3.1):
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
deb http://vo.ari.uni-heidelberg.de/debian beta main |
/etc/apt/sources.list
Code Block | ||
---|---|---|
| ||
wget -qO - http://docs.g-vo.org/archive-key.asc | sudo apt-key add - |
Code Block | ||
---|---|---|
| ||
sudo apt-get update sudo apt-get upgrade |
After these first steps, start the installation:
Code Block | ||
---|---|---|
| ||
sudo apt-get install gavodachs-server |
In order to safely install the various components of the server, we now turn off the Apache web server (we will configure it later):
Code Block |
---|
sudo apachectl stop |
Initial DaCHS Configuration
We now configure DaCHS so that the server can run. The registry and data services configuration will be done in the second part of the tutorial.
As we are using DaCHS in a non secure configuration, we need to disable some features:
Code Block | ||
---|---|---|
| ||
sudo cp /var/gavo/etc/trustedquery /var/gavo/etc/untrustedquery |
Create a file /etc/gavo.rc
using your preferred editor (for instance pico):
Code Block | ||
---|---|---|
| ||
sudo pico /etc/gavo.rc |
Insert the following content in that file :
Test | Production | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
In this file, replace http://<<my_servername>>.<<my_domain>>
by your actual server name, <<server_title>>
by server title and <<authority-ivo-name>>
by your own authority name that you will create during the registration of your service (e.g., at Observatoire de Paris, our authority name is vopdc.obspm
). If this is not ready at this time, you may modify it later when your have registered your service with IVOA. The << maintainer_email>>
field is the contact of the technical maintainer of the service, use a generic address if possible.
If you want to add your institute logo on the DaCHS server instead of the default GAVO logo, you have to prepare three PNG files, with width of 200, 120 and 50 pixels, and the following respective names logo_big.png
, logo_medium.png
and logo_tiny.png
. These files should be placed in the /var/gavo/web/nv_static/img/
directory (you will have to create the img/
directory if necessary).
Code Block | ||
---|---|---|
| ||
sudo mkdir /var/gavo/web/nv_static/img |
The image transfer can be done with the following command, on the server you have prepared your images:
Test | Production | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
|
|
This command has copied to your images on the home account of the DaCHS server. Then you have to copy them in the right directory, on your DaCHS server:
Code Block |
---|
cd ~ sudo mkdir /var/gavo/web/nv_static/img sudo chown dachsroot:gavo /var/gavo/web/nv_static/img sudo cp logo_big.png logo_medium.png logo_tiny.png /var/gavo/web/nv_static/img/. |
Warning |
---|
This logo will be displayed in the DaCHS web interface, but will also be used by VESPA clients to display your logo next to your data products, so that the origin of the data is clearly identified. |
Finally start Apache and DaCHS servers:
Code Block | ||
---|---|---|
| ||
sudo apachectl start sudo gavo serve start |
You can check that your server is running by connecting to the server with a web browser, using the server URL:
Test | Production |
---|---|
http://127.0.0.1:8000 | http://<<my_servername>>.<<my_domain>> |
You should see the following welcome page:
Pagebreak |
---|
6.
AWStatAWStats Installation and Configuration including Apache
Install AWStats:
Code Block | ||
---|---|---|
| ||
sudo apt-get install awstats |
Apache Configuration for AWStats
AWStats is a script, which must be enabled in the Apache default configuration file.
Edit the file /etc/apache2/sites-enabled/000-default.conf
:
Code Block | ||
---|---|---|
| ||
sudo pico /etc/apache2/sites-enabled/000-default.conf |
Replace the entire file content by the following lines ( ctrl-K to cut/delete a line on Pico):
Code Block | ||||
---|---|---|---|---|
| ||||
<VirtualHost *:8080> ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ <Directory "/usr/lib/cgi-bin"> Options FollowSymLinks AddHandler cgi-script .pl AllowOverride None Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch Order allow,deny Allow from all </Directory> ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined </VirtualHost> |
Configuration of AWStats
We configure here AWStats to use the DaCHS and Apache log files and eliminate queries coming from monitoring server at ObsParis.
Apache statistics
Go into the /etc/awstats
directory and open a new file for edition:
Code Block | ||
---|---|---|
| ||
cd /etc/awstats sudo pico awstats.conf |
In this file, edit the following lines (search by keyword - ctrl-W on Pico - : LogFile
, LogFormat
, etc) and replace by the proposed content:
Keyword to search | Line to input |
---|---|
LogFile | LogFile="/usr/share/awstats/tools/logresolvemerge.pl /var/log/apache2/access.log* |" |
LogFormat | LogFormat=1 |
SiteDomain | SiteDomain="<<my_servername>>.<<my_domain>>" |
HostAliases | HostAliases="localhost 127.0.0.1 <<my_domain>>" |
DNSLookup | DNSLookup=2 |
AllowFullYearView | AllowFullYearView=3 |
SkipHosts | SkipHosts="145.238.187.13 145.238.187.29 145.238.193.18" |
The VESPA technical team plan to monitor the status of all VESPA servers using a Nagios server. Its hits should be removed from the statistics. This is the purpose of the line starting with SkipHosts
.
The Apache2 log files must also be readable by AWStats scripts. First we modify the permissions of Apache2 log files. Those permissions are defined in the /etc/logrotate.d/apache2
file. We have to modify the line starting with the create
command by:
Code Block |
---|
create 644 root adm |
If you need to fix permissions of already existing log files, the following commands will fix the Apache2 log permissions:
Code Block | ||
---|---|---|
| ||
sudo chmod 644 /var/log/apache2/access.log* sudo chmod 755 /var/log/apache2 |
DaCHS statistics
Staying in the /etc/awstats
directory, copy the file awstats.conf
to awstats.dachs.conf
:
Code Block | ||
---|---|---|
| ||
sudo cp awstats.conf awstats.dachs.conf sudo pico awstats.dachs.conf |
In this file, edit the following lines (search by keyword - ctrl-W on Pico - : LogFile
, LogFormat
, etc):
Keyword to search | Line to input |
---|---|
LogFile | LogFile="/usr/share/awstats/tools/logresolvemerge.pl /var/gavo/logs/web.log* |" |
LogFormat | LogFormat="%other %other %other %host %other %other %time1 %methodurl %code %bytesd %refererquot %uaquot" |
URLWithQuery | URLWithQuery=1 |
Setup a crontab to update statistics every day
First create a file containing the calling sequence to AWStats:
Code Block | ||
---|---|---|
| ||
cd /usr/local/bin sudo pico run_awstats |
In this file, input the three following lines:
Code Block | ||||
---|---|---|---|---|
| ||||
#!/bin/bash /usr/bin/perl /usr/lib/cgi-bin/awstats.pl -config=dachs -update /usr/bin/perl /usr/lib/cgi-bin/awstats.pl -config=apache -update |
Finally, change the access control configuration for this file:
Code Block | ||
---|---|---|
| ||
sudo chmod 777 run_awstats |
A task can then be added to the crontab of your system using :
Code Block | ||
---|---|---|
| ||
sudo crontab -e |
Add those lines at the end of the file:
Code Block | ||||
---|---|---|---|---|
| ||||
0 0 * * * www-data /usr/local/bin/run_awstats 2>/dev/null |
Info | ||
---|---|---|
If you use the
If you are not very familiar with |
Run AWStats
Code Block | ||
---|---|---|
| ||
sudo /usr/local/bin/run_awstats |
Access AWStats
The AWStats reports are available at:
- http://my_servername.my_domain:8080/cgi-bin/awstats.pl?config=dachs
- http://my_servername.my_domain:8080/cgi-bin/awstats.pl?config=apache
Figure 5.1 shows a screenshot of the AWStats monitoring page.
Fig 5.1: View of the AWStats monitoring page.
7. Allowing up external access to database
In order to connect to the PostgreSQL (PgSQL) database that is used by DaCHS, you can either set up an SSH tunnel or configure DaCHS server to allow external access to the database:
Method A: By opening an SSH tunnel
From your host:
Code Block | ||
---|---|---|
| ||
ssh <<my_user>>@127.0.0.1 -p 2222 -L 5432:127.0.0.1:5432 |
Then leave the console open during the tutorial.
Method B: By configuring DaCHS server
On the DaCHS server, you have to edit 2 files (pg_hba.conf
and postgresql.conf
). The first file is used to set up which range of IP have access, with which user and what role. Open the file:
Code Block |
---|
sudo pico /etc/postgresql/9.4/main/pg_hba.conf |
and at the end of the file add the following line:
Code Block |
---|
host all gavoadmin 0.0.0.0/0 md5 |
Warning |
---|
That configuration will allow all IP address to connect. That's good for a tutorial, but not for real operations. You will have to restrict the range of IPs that are allowed to connect. Check with your local system administrator if needed. |
The second file is used to configure which IP range the database is listening to for queries. Open the file:
Code Block |
---|
sudo pico /etc/postgresql/9.4/main/postgresql.conf |
and modify the line containing the "listen_addresses" directive:
Code Block |
---|
listen_addresses = '*' |
The PgSQL database as well as DaCHS must now be restarted:
Code Block |
---|
sudo /etc/init.d/postgresql restart sudo gavo serve restart |
8. Setting up pgAdmin (optional)
Installation
pgAdmin aims to provide a graphical interface to view and manage a database. We will use this tool on section Setting up an EPN-TAP service. If you don't want to install pgAdmin on you computer, you can however use psql commands instead.
First, install pgAdmin on your host machine. If you are on Debian based distribution:
Code Block | ||
---|---|---|
| ||
sudo apt-get install pgadmin3 |
Compiled sources are also available for MacOS and Windows.
Info |
---|
You need pgAdmin v. 1.20 or upper in order to deal with postgreSQL 9.4. Once installed, check pgdmin version (Help menu, About). |
Configuration
Check for the gavo
user password:
Code Block | ||
---|---|---|
| ||
cat /var/gavo/etc/feed |
Now start pgAdmin, then click on Add a connection to a server (power socket icon) and fill the fields as the following:
Test | Production | Comment | |||
---|---|---|---|---|---|
Name | DaCHS-test | DaCHS | As you want | ||
Host | 127.0.0.1 | <<my_server>>.<<my_domain>> | |||
TCP port | 5433 | ||||
Service | leave empty | ||||
Base maintenance | postgres | ||||
Use name | gavoadmin | ||||
password | ******* | Get it with the command above | |||
save password | yes | As you want |
Checking installation
Now you should be able to see your database: