Work in Progress
This tutorial is still under development, but nearly finalized. Please try out and post any necessary comment!
EPN2020-RI
EUROPLANET2020 Research Infrastructure
H2020-INFRAIA-2014-2015
Grant agreement no: 654208
Document: VESPA- WP6-2-004-TD-v0.6
EPN-TAP Installation for VESPA Data Provider Tutorial — Part 1
Date: $action.dateFormatter.formatGivenString("yyyy-MM-dd",$content.getLastModificationDate())
Start date of project: 01 September 2015
Duration: 48 Months
Responsible WP Leader: OBSPARIS / Stéphane Erard
Project co-funded by the European Union's Horizon 2020 research and innovation programme | ||
Dissemination level | ||
PU | Public | |
PP | Restricted to other programme participants (including the Commission Service) | |
RE | Restricted to a group specified by the consortium (including the Commission Services) | |
CO | Confidential, only for members of the consortium (excluding the Commission Services) |
Project Number | 654208 |
Project Title | EPN2020 - RI |
Project Duration | 48 months: 01 September 2015 – 30 August 2019 |
Document Number | VESPA-WP6-2-004-TD-v0.6(86) |
Delivery date | 2015.11.21 |
Title of Document | EPN-TAP Installation for VESPA Data Provider Tutorial — Part 1 |
Contributing Work package (s) | WP6 |
Dissemination level | PU |
Author (s) | Baptiste Cecconi, Pierre Le Sidaner, Stéphane Erard, Renaud Savalle, Markus Deimletner, Paul Sladen, Mikhail Minin |
Abstract: |
Document history (to be deleted before submission to Commission) | ||||
Date | Version | Editor | Change | Status |
| 0.1 | First issue (imported from internal MS-Word document) | DRAFT | |
| 0.2 | Updated AWStats and Apache Proxy configuration | DRAFT | |
| 0.3 | Fixed several errors | DRAFT | |
| 0.4 | Simplified the server configuration (no more proxy) | DRAFT | |
0.5 | Baptiste Cecconi, Mikhail Minin | Update of first installation step. | DRAFT | |
0.6 | Pierre Le Sidaner, Baptiste Cecconi | Changed Awstats configuration | DRAFT |
Table of Contents
1. Introduction
This document presents the installation procedure to set up data distribution server for VESPA. We propose here to set up three service interfaces on the same physical server (and the same public IP address): a standard web server (Apache) to serve data files, a tool to have access statistics (using AWStats), and a DaCHS server to provide the Virtual Observatory (VO) interfaces (including a TAP service). Two server names have to be set up: one for the data access and the statistics, and the other for the VO access:
- http://my_servername.my_domain VO interface
- http://my_servername.my_domain:8080 Data repository and usage statistic service
Section 2 describes the Debian distribution installation. Section 3 describes the installation of DaCHS and its configuration. Section 4 describes the installation and configuration of AWStat. Section 5 describes the configuration of the Apache server and of DNS.
Further configuration of the VO interface is described in the second part of the tutorial.
In all this document, you will have to replace all text between <<...>>
with the value that you have selected for your installation.
In order to prepare the installation, here is a table to be filled with the parameters of you installation:
Parameter Name | Your Value | Example Value for tutorial |
---|---|---|
my_servername | .................................................................................. | voparis-test-bc |
my_domain | .................................................................................. | obspm.fr |
maintainer_email (1) | .................................................................................. | vo.paris@obspm.fr |
my_datacenter-ivoid (2) | .................................................................................. | ivo://vopdc.obspm |
(1) This should be a generic address, rather than a specific person address.
(2) Read the Service Identifier page when filling these lines.
(3) May be different from the contact_name
. Contact Name is the Data Center contact person, whereas the Creator Name is the contact for this specific server.
(4) The logo must have the following maximum dimensions: width = 120px and height = 60px.
(5) This date must be formatted in ISO-8901, see the example.
Your server must have a public address. The DNS configuration must be completed so that the tests can be done.
2. Debian Stable (Jessie) Installation
DaCHS (Data Center Helper Suite) must be installed on a fresh Debian Stable distribution. The current installation image for this distribution is available here: https://www.debian.org/distrib/netinst#smallcd.
In this example, we used the following ISO image: debian-8.2.0-amd64-netinst.iso
. The installation demonstration is done on a Virtual Box guest computer. This virtual machine is emulating an AMD 64bits processor based system. For a real installation, a physical server (or a bare metal virtualization framework) should be preferred.
If you use a virtualization software, the Debian installation ISO image must be mounted on boot (see Virtual Box documentation for more details), so that the system can boot from it. If you use a physical server, the installation disk (extracted from the ISO image) must be available to the system on boot.
Step | Screenshot | Configuration |
---|---|---|
Installation selection | Select "Install " | |
Language used for the installation process, and for the system, once installed | Select " English " | |
Location of the installation | Select your country | |
Locale used for the installation | Select "en-US.UTF-8 " | |
Keyboard map | Select your keyboard map | |
Hostname for this computer | Type the hostname (here " replace by << | |
Domain name for your institute | Type the domain name (here " replace by << | |
Root password | Enter root password (pick a password, which is not easy to guess and don't forget it!) | |
Main user configuration | Enter Full Name, user name and password | |
Disk configuration | Select "Use entire disk " | |
Select Disk to partition | Select the main disk (there should be only one). | |
Partition Scheme | Select "All files in one partition " | |
Confirmation of partition set up | Select "Finish partioning and write changes to disk " | |
Configuration of package manager | Select a mirror close to your place | |
Proxy configuration | If needed enter the proxy information to go on internet | |
Configuring popularity-contest | This is up to you (here "No " is selected) | |
Software selection | Select a minimal set of package, namely " For package selection/unselection, use the "space" key In basic configuration, we do not select any desktop environment, but it is possible to do so, it makes you more confident. If you don't load any desktop environment, you must make sure that your server is reachable through | |
Installation of GRUB boot loader, part 1 | Select "yes " | |
Installation of GRUB boot loader, part 2 | Select the newly created partition. | |
Installation completion | Select "continue " to finish installation and reboot. | |
Reboot | Go to next section |
3. DaCHS Installation and Configuration
There are several ways to install DaCHS as presented by his author in [1]. For DaCHs installation you need an Debian Stable distribution, at this time Jessie (see Section 2).
Install DaCHS framework
On the server, log with the regular user created during installation (either on the computer itself, or remotely with ssh). The first thing is to install and set up the sudo command, which allows a user to issue commands with root rights. Use the su command to log as root, and type the root password.
su
Install the sudo package:
apt-get install sudo
Set up rights for your regular user to use the sudo command:
adduser "user" sudo
You have now the log out from your root shell and your regular user shell. Once you log back in with the regular user, you will be able to use the sudo command.
The repository source list must be updated to include DaCHS packages.Edit the file /etc/apt/sources.list with your preferred command line editor, for instance, you can use the pico editor (useful short cuts: ^O to save, ^X to quit):
sudo pico /etc/apt/sources.list
Add the following lines at the end of file (see Fig. 3.1):
# DaCHS repository deb http://vo.ari.uni-heidelberg.de/debian stable main deb-src http://vo.ari.uni-heidelberg.de/debian stable main
wget -qO - http://docs.g-vo.org/archive-key.asc | sudo apt-key add -
sudo apt-get update sudo apt-get upgrade
After these first steps, start the installation:
sudo apt-get install gavodachs-server
Initial DaCHS Configuration
We configure DaCHS so that the server can run. The registry and data services configuration will be done in the second part of the tutorial.
As we are using DaCHS in a non reduce configuration, we need to disable some features:
sudo cp /var/gavo/etc/trustedquery /var/gavo/etc/untrustedquery
Create a file /etc/gavo.rc
using your preferred editor (for instance pico):
sudo pico /etc/gavo.rc
Insert the following content in that file:
[general] rootDir: /var/gavo maintainerAddress: <<maintainer_email>> [web] bindAddress: serverPort: 80 serverURL: http://<<my_servername>>.<<my_domain>> preloadRDs: rr/q [ivoa] authority: <<my_datacenter-ivoid>>
In this file, replace
http://my_servername.my_domain
by your actual server name and my_datacenter-ivoid
by your own authority ID that you will create during the registration of your service (e.g., at Observatoire de Paris, our authority ID is ivo://vopdc.obspm
). If this is not ready at this time, you may modify it later when your have registered your service with IVOA. The maintainer.address@my_domain
field is the contact of the technical maintainer of the service, use a generic address if possible.
Finally restart DaCHS server:
sudo gavo serve restart
4. Apache Configuration
Apache should already be installed during initial configuration, otherwise :
sudo apt-get install apache2
Enable ModCGI:
sudo a2enmod cgi
Update the default Apache port to 8080 instead of 80 by changing the /etc/apache2/ports.conf
file:
sudo pico /etc/apache2/ports.conf
Modify the line containing the Listen
command and replace the port number:
Listen 8080
Then restart Apache2 as well as DaCHS:
sudo service apache2 restart sudo gavo serve restart
5. AWStat Installation and Configuration including Apache
Install AWStats:
sudo apt-get install awstats
Apache Configuration for AWStats
AWStats is a script, which must be enabled in default config the script execution in you Apache default config file.
Edit the file /etc/apache2/sites-enabled/000-default.conf
and check for the following lines:
<VirtualHost *:8080> ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ <Directory "/usr/lib/cgi-bin"> Options FollowSymLinks AddHandler cgi-script .pl AllowOverride None Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch Order allow,deny Allow from all </Directory> ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined </VirtualHost>
Configuration of AWStats
We configure here AWStats to use the DaCHS and Apache log files and eliminate queries coming from monitoring server at ObsParis.
DaCHS statistics
Go into the /etc/awstats
directory and open a new file for edition:
cd /etc/awstats sudo mv awstats.conf awstats.conf.orig sudo cp awstats.conf.orig awstats.dachs.conf sudo pico awstats.dachs.conf
In this file, edit the following lines (search by keyword: LogFile
, LogFormat
, etc):
LogFile="/usr/share/awstats/tools/logresolvemerge.pl /var/gavo/logs/web.log* |" ... LogFormat="%other %other %other %host %other %other %time1 %methodurl %code %bytesd %refererquot %uaquot" ... SiteDomain="<<my_servername>>.<<my_domain>>" ... HostAliases="localhost 127.0.0.1 <<my_domain>>" ... DNSLookup=2 ... AllowFullYearView=3 ... SkipHosts="145.238.187.13 145.238.187.29" ... URLWithQuery=1
The VESPA technical team plan to monitor the status of all VESPA servers using a Nagios server. Its hits should be removed from the statistics. This is the purpose of the line starting with SkipHosts
.
Apache statistics
Go into the /etc/awstats
directory and open a new file for edition:
cd /etc/awstats sudo cp awstats.conf.orig awstats.apache.conf sudo pico awstats.apache.conf
In this file, edit the following lines (search by keyword: LogFile
, LogFormat
, etc):
LogFile="/usr/share/awstats/tools/logresolvemerge.pl /var/log/apache2/access.log* |" ... LogFormat=1 ... SiteDomain="<<my_servername>>.<<my_domain>>" ... HostAliases="localhost 127.0.0.1 <<my_domain>>" ... DNSLookup=2 ... AllowFullYearView=3 ... SkipHosts="145.238.187.13 145.238.187.29"
The VESPA technical team plan to monitor the status of all VESPA servers using a Nagios server. Its hits should be removed from the statistics. This is the purpose of the line starting with SkipHosts
.
For this monitoring purpose, please send the public IP address (or <<my_servername>>.<<my_domain>>
address) to support.vespa@obspm.fr, so that we can configure our servers.
Setup a crontab to update statistics every day
First create a file containing the calling sequence to AWStats:
cd /usr/local/bin sudo pico run_awstats
In this file, input the two following lines:
#!/bin/bash /usr/bin/perl /usr/lib/cgi-bin/awstats.pl -config=dachs -update /usr/bin/perl /usr/lib/cgi-bin/awstats.pl -config=apache -update
Finally, change the access control configuration for this file:
sudo chmod 777 run_awstats
A task can then be added to the crontab of your system using :
sudo crontab -e
Add those lines at the end of the file:
0 0 * * * www-data /usr/local/bin/run_awstats 2>/dev/null
If you use the crontab -e
command for the first time, you may have to select en editor. The following lines should then appear in your terminal:
Select an editor. To change later, run 'select-editor'. 1. /bin/nano <---- easiest 2. /usr/bin/vim.tiny Choose 1-2 [1]:
If you are not very familiar with vim
, prefer the nano
option.
Run AWStats
sudo /usr/local/bin/run_awstats
Access AWStats
The AWStats reports are available at:
- http://my_servername.my_domain:8080/cgi-bin/awstats.pl?config=dachs
- http://my_servername.my_domain:8080/cgi-bin/awstats.pl?config=apache
Figure 5.1 shows a screenshot of the AWStats monitoring page.
Fig 5.1: View of the AWStats monitoring page.