Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Info
titleWork in Progress

This tutorial is still under development, but nearly finalized. Please try out and post any necessary comment! (smile)



Metadata list
hiddentrue
|| workpackage | WP6 |
|| task | 2 |
|| document number | 004 |
|| document version | 1.6 |
|| document title | EPN-TAP Server Installation for VESPA Data Provider Tutorial |
|| document type | TD |




 

EPN2020-RI

 


EUROPLANET2020 Research Infrastructure 

H2020-INFRAIA-2014-2015 

Grant agreement no: 654208 


Document: VESPA-

Metadata from
1workpackage
-
Metadata from
1task
-
Metadata from
1document number
-
Metadata from
1document type
-v
Metadata from
1document version

 

 



Metadata from
1document title

 

 

 




Date:  

 


Start date of project: 01 September  2015

 Duration: 48 Months

Responsible WP Leader: OBSPARIS / Stéphane Erard 


Project co-funded by the European Union's Horizon 2020 research and innovation programme

Dissemination level

PU

Public

  •   

PP

Restricted to other programme participants (including the Commission Service)

  •   

RE

Restricted to a group specified by the consortium (including the Commission Services)

  •   

CO

Confidential, only for members of the consortium (excluding the Commission Services)

  •   


Project Number

654208

Project Title

EPN2020 - RI

Project Duration

48 months: 01 September 2015 – 30 August 2019

Pagebreak

Document Number

VESPA-

Metadata from
1workpackage
-
Metadata from
1task
-
Metadata from
1document number
-
Metadata from
1document type
-v
Metadata from
1document version
(
Current page version
)

Delivery date

2015.11.21

Title of Document

Metadata from
1document title

Contributing Work package (s)

Metadata from
1workpackage

Dissemination level

PU

Author (s)


Abstract:

 

 



Document history (to be deleted before submission to Commission)

Date

Version

Editor

Change

Status

 

0.1

First issue (imported from internal MS-Word document)

Status
colourYellow
titledraft

 

0.2

Updated AWStats and Apache Proxy configuration

Status
colourYellow
titledraft

 

0.3

Fixed several errors

Status
colourYellow
titledraft

 

0.4

Simplified the server configuration (no more proxy)

Status
colourYellow
titledraft

 
0.5

Update of first installation step.
Added comments and updated Apache2 configuration. 

Status
colourYellow
titledraft
 
0.6Changed Awstats configuration
Status
colourYellow
titledraft
 
0.7Added DaCHS web configuration (title and logos)
Fixed /etc/gavo.rc configuration 
Status
colourYellow
titledraft
 
0.8Added screenshots and some adding testing steps
Status
colourYellow
titledraft
 
0.9Modification of the ordering of Apache and DaCHS configuration, and various small changes for better clarity.
Status
colourYellow
titledraft
0.10

Added PgAdmin section. Added notes on Virtual Box network configuration. Small modifications

Status
colourYellow
titledraft
0.11Small modifications
Status
colourYellow
titledraft
 
1.0Public Release

Status
colourGreen
titleissued

 
1.1Changed PgAdmin configuration.
Changed document name.
Status
colourGreen
titleissued
 
1.2Fixed /etc/gavo.rc file
Status
colourGreen
titleissued
 
1.3Going back to draft status for updating tutorial.
Including "test" and "prod" set up.
Status
colourYellow
titledraft
 
1.4

Modified /etc/apt/sources.list input.
Added note about server logo. 
Fixed AWSTATS configuration. 

Status
colourYellow
titledraft

 

 1.5

replay installation tutorial, update the documentation

Status
colourYellow
titledraft

 

1.5

typos, fixes, etc...

Status
colourYellow
titledraft


1.6

add logFormat: combined to gavo.rc to use awstats in curent DaCHS version

Status
colourYellow
titledraft

 


Table of Contents1.

Table of Contents

 


Pagebreak
 


1. Introduction

This document presents the installation procedure to set up data distribution server for VESPA. We propose here to set up three service interfaces on the same physical server (and the same public IP address): a standard web server (Apache) to serve data files, a tool to have access statistics (using AWStats), and a DaCHS server to provide the Virtual Observatory (VO) interfaces (including a TAP service). Two server names have to be set up: one for the data access and the statistics, and the other for the VO access:

Section 2 describes the Debian distribution installation. Section 3 describes the installation of DaCHS and its configuration. Section 4 describes the installation and configuration of AWStatAWStats. Section 5 describes the configuration of the Apache server and of DNS.

Further configuration of the VO interface is described in the second part of the tutorial.

Info

In all this document, you will have to replace all text between <<...>> with the value that you have selected for your installation.

In order to prepare the installation, here is a table to be filled with the parameters of you installation:

Parameter NameYour ValueExample Value for tutorial
my_servername..................................................................................voparis-test-bc
my_domain..................................................................................obspm.fr
maintainer_email (1)..................................................................................vo.paris@obspm.fr
authority-ivo-name (2)..................................................................................vopdc.obspm
server_title ..................................................................................My new test DaCHS server
my_user..................................................................................user

(1) This should be a generic address, rather than a specific person address. 
(2) Read the Service Identifier page when filling these lines. Note that this specific item do not contain "ivo://" at the beginning.

2. Debian Stable (Jessie) Installation

DaCHS (Data Center Helper Suite) must be installed on a fresh Debian Stable distribution .

2.1 Install virtualization software

TestProduction

 The installation demonstration is done on a Virtual Box guest computer, so you first need to install VirtualBox on your computer.

On Debian based systems:

Code Block
languagebash
sudo apt-get install virtualbox

On other systems, download VirtualBox and follow the installation instructions.

Warning

The Debian installation ISO image must be mounted on boot (see Virtual Box documentation for more details), so that the system can boot from it. If you use a physical server, the installation disk (extracted from the ISO image) must be available to the system on boot. 


For a real installation, a physical server (or a bare metal virtualization framework) should be preferred.

2.2 Download Debian

 The current installation image for this distribution is available here: https://www.debian.org/distrib/netinst#smallcd (under Small CDs or USB sticks). In this example, we used the following ISO image: debian-8.6.0-amd64-netinst.iso

2.3 Virtual machine installation

TestProduction

Click on the New icon, then fill:

LabelValueComment

Name

Debian8.3_VESPAAs you want

Type

Linux 
VersionDebian (64bits) 
RAM1GoNo more than 50% of your actual RAM.
Virtual HDDCreate a virtual drive now 
Drive file typeVDI 
Storagedynamically allocated 
Location 
The default option is usually good.
Size8Go 

Click create on the last step, then click on the Configuration icon.

Go to Storage, on IDE controller, click on Add optical drive, Choose a disk, and select your downloaded iso.

Nothing to do.

2.4 Network configuration

TestProduction

Always in Configuration window, go to Network, keep "NAT" for the default network access mode, then click on Port forwarding and add 4 rules as the following:

NameProtocolhost IPhost portguest IPguest port
sshTCP 
2222 
22
web-apacheTCP 
8080 
8080
web-dachsTCP 
8000 
8000
psqlTCP 
5433 
5432


Info

Note that as part of this tutorial, the DaCHS port is set to 8000 instead of 80 because the host can't access to a guest through ports numbers lower than 1024 for security reasons.


Info

For advanced test purposes, you can configure the network interface of the VM to "Bridged", and then use the "Production" commands.

In this case, the VM will have its own IP address, as provided by the DHCP server (you may need to provide the MAC address of your VM to your network admin to get a proper IP address).


For operation server, you must have a public address. The DNS configuration must be completed.

Ports to be open:

  • 22: Secure shell login for administration and configuration.
  • 80 (open to all): VO interface (DaCHS server)
  • 8080 (open to all): WWW interface (Apache2 server)

2.5 Debian installation

StepScreenshotConfiguration
Installation selection
Select "Install"
Language used for the installation process, and for the system, once installed
Select " English "
Location of the installation
Select your country
Locale used for the installation
Select "en-US.UTF-8"
Keyboard map 
Select your keyboard map
Hostname for this computer

Type the hostname (here "voparis-test-bc")

Info

replace by <<my_servername>>


Domain name for your institute

Type the domain name (here "obspm.fr")

Info

replace by <<my_domain>>


Root password
 Enter root password (pick a password, which is not easy to guess and don't forget it!)
Main user configuration

Enter Full Name, user name and password

In the following, <<my_user>> will refer to the login of this user, you will need a user account.

Disk configuration
Select "Use entire disk"
Select Disk to partition
Select the main disk (there should be only one).
Partition Scheme
Select "All files in one partition"
Confirmation of partition set up
Select "Finish partioning and write changes to disk"
write partition to disk
select <Yes>
Configuration of package manager
Select a mirror close to your place
Proxy configuration
If needed enter the proxy information to go on internet (if your internet access is via a proxy, otherwise leave it blank).
Configuring popularity-contest
This is up to you (here "No" is selected)
Software selection

Select a minimal set of package, namely "web server", "ssh server", "standard system utilities". Note that no desktop environment is selected here. As a server, only shell access is necessary.

Info

For package selection/unselection, use the "space" key.


Info

In basic configuration, we do not select any desktop environment, but it is possible to do so, it makes you more confident.


Warning

If you don't load any desktop environment, you must make sure that your server is reachable through ssh. This may imply configuration on the virtualization software and on the DNS server of your institution.


Installation of GRUB boot loader, part 1
Select "yes"
Installation of GRUB boot loader, part 2
Select the newly created partition.
Installation completion
Select "continue" to finish installation and reboot.
Reboot
Go to next section

From now, it is recommended to access to your server through ssh:

Test (from a terminal, or use PuTTY on Windows)
Production
ssh -p 2222 <<my_user>>@localhost
ssh <<my_user>>@<<my_servername>>.<<my_domain>>

And you don't need to use the guest interface anymore.

Now you will be able (and you should) to use copy-paste along the tutorial.

Info

To change the keyboard map on your guest, you have to edit a file:

Code Block
languagebash
sudo nano /etc/default/keyboard



3. Install and configure sudo 

On the server, login with the <<my_user>> user created during installation (either on the computer itself, or remotely with ssh). 

The first thing is to install and set up the sudo command, which allows a user to issue commands with root rights. Use the su command to log as root, and type the root password. 

Code Block
languagebash
su

Install the sudo package: 

Code Block
languagebash
apt-get install sudo 

Set up rights for your regular user to use the sudo command: 

Code Block
languagebash
adduser "<<my_user>>" sudo

You have now the log out from your root shell:

Code Block
languagebash
exit

and your regular user shell:

Code Block
languagebash
exit

(this will disconnect your ssh connection, so you have to connect again)

Now you will be able to use the sudo command. 

4. Apache Configuration

Connect to the server with ssh. Apache should already be installed during initial configuration, otherwise :

Code Block
languagebash
sudo apt-get install apache2

Enable ModCGI:

Code Block
sudo a2enmod cgid

Update the default Apache port to 8080 instead of 80 by changing the /etc/apache2/ports.conf  file, using your preferred command line editor, for instance, you can use the pico editor (useful short cuts: ^O to save, ^X to quit):

(sometimes you have to check the line NameVirtualHost *:8080 in ports.conf and <VirtualHost *:8080> in /etc/apache2/sites-available/default)

Code Block
sudo pico /etc/apache2/ports.conf

Modify the first line containing the  Listen  command and replace the port number:

Code Block
Listen 8080

Then restart Apache2:

Code Block
languagebash
sudo service apache2 restart

You can check that your regular web server is running connecting to the URL:  http://<<my_servername>>.<<my_domain>>:8080/. You should see a standard splash page:

Fig. 4.1: View of the Apache server web default welcome page.

You can create a regular website that presents your project and database putting HTML code into /var/www/html/

Pagebreak

5. DaCHS Installation and Configuration

There are several ways to install DaCHS as presented by his author in [1]. For DaCHs installation you need an Debian Stable distribution, at this time Jessie (see Section 2).

Install DaCHS framework

The repository source list must be updated to include DaCHS packages. Edit the file /etc/apt/sources.list

Code Block
languagebash
sudo pico /etc/apt/sources.list 

Add the following line at the end of file (see Fig. 3.1):

Code Block
languagebash
firstline1
linenumberstrue
deb http://vo.ari.uni-heidelberg.de/debian beta main

Fig. 3.1: Adding GAVO/DaCHS repository into  /etc/apt/sources.list 

  Add the key to use the repository:

Code Block
languagebash
wget -qO - http://docs.g-vo.org/archive-key.asc | sudo apt-key add -

  Get the latest debian packages:

Code Block
languagebash
sudo apt-get update
sudo apt-get upgrade 

 After these first steps, start the installation:

Code Block
languagebash
sudo apt-get install gavodachs-server

In order to safely install the various components of the server, we now turn off the Apache web server (we will configure it later):

Code Block
sudo apachectl stop

Initial DaCHS Configuration

We now configure DaCHS so that the server can run. The registry and data services configuration will be done in the second part of the tutorial.

As we are using DaCHS in a non secure configuration, we need to disable some features: 

Code Block
languagebash
sudo cp /var/gavo/etc/trustedquery /var/gavo/etc/untrustedquery 

Create a file /etc/gavo.rc using your preferred editor (for instance pico): 

Code Block
languagebash
sudo pico /etc/gavo.rc 

Insert the following content in that file : 

TestProduction


Code Block
languagetext
firstline1
linenumberstrue
[general]
rootDir: /var/gavo 
maintainerAddress: <<maintainer_email>>
[web]
bindAddress:
serverPort: 8000
serverURL: http://127.0.0.1:8000
sitename: <<server_title>>
corsOriginPat: http.*
logFormat: combined
[ivoa]
authority: <<authority-ivo-name>>



Code Block
languagetext
linenumberstrue
[general]
rootDir: /var/gavo 
maintainerAddress: <<maintainer_email>>
[web]
bindAddress:
serverPort: 80
serverURL: http://<<my_servername>>.<<my_domain>>
sitename: <<server_title>>
corsOriginPat: http.*
logFormat: combined
[ivoa]
authority: <<authority-ivo-name>>


In this file, replace http://<<my_servername>>.<<my_domain>>  by your actual server name,  <<server_title>> by server title and <<authority-ivo-name>> by your own authority name that you will create during the registration of your service (e.g., at Observatoire de Paris, our authority name is vopdc.obspm). If this is not ready at this time, you may modify it later when your have registered your service with IVOA. The << maintainer_email>>  field is the contact of the technical maintainer of the service, use a generic address if possible.

If you want to add your institute logo on the DaCHS server instead of the default GAVO logo, you have to prepare three PNG files, with width of 200, 120 and 50 pixels, and the following respective names logo_big.png, logo_medium.png and logo_tiny.png. These files should be placed in the /var/gavo/web/nv_static/img/ directory (you will have to create the img/ directory if necessary).

Code Block
languagebash
sudo mkdir /var/gavo/web/nv_static/img

The image transfer can be done with the following command, on the server you have prepared your images:

TestProduction


Code Block
languagebash
cd /your/local/path/to/img
scp -P 2222 logo_big.png logo_medium.png logo_tiny.png <<my_user>>@127.0.0.1:~ 



Code Block
languagebash
cd /your/local/path/to/img
scp logo_big.png logo_medium.png logo_tiny.png <<my_user>>@<<my_servername>>.<<my_domain>>:~ 


This command has copied to your images on the home account of the DaCHS server. Then you have to copy them in the right directory, on your DaCHS server:

Code Block
cd ~
sudo mkdir /var/gavo/web/nv_static/img
sudo chown dachsroot:gavo /var/gavo/web/nv_static/img
sudo cp logo_big.png logo_medium.png logo_tiny.png /var/gavo/web/nv_static/img/.


Warning

 This logo will be displayed in the DaCHS web interface, but will also be used by VESPA clients to display your logo next to your data products, so that the origin of the data is clearly identified.

Finally start Apache and DaCHS servers:

Code Block
languagebash
sudo apachectl start
sudo gavo serve start

You can check that your server is running by connecting to the server with a web browser, using the server URL:

TestProduction
http://127.0.0.1:8000 http://<<my_servername>>.<<my_domain>>

You should see the following welcome page:

Fig. 3.2: View of the welcome DaCHS server.

Pagebreak

6.

AWStat

AWStats Installation and Configuration including Apache

Install AWStats:

Code Block
languagebash
sudo apt-get install awstats 

Apache Configuration for AWStats

AWStats is a script, which must be enabled in the Apache default configuration file.

Edit the file /etc/apache2/sites-enabled/000-default.conf:

Code Block
languagebash
sudo pico /etc/apache2/sites-enabled/000-default.conf

Replace the entire file content by the following lines ( ctrl-K to cut/delete a line on Pico):

Code Block
languagetext
linenumberstrue
<VirtualHost *:8080>


       ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
       <Directory "/usr/lib/cgi-bin">
               Options FollowSymLinks
               AddHandler cgi-script  .pl
               AllowOverride None
               Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
               Order allow,deny
               Allow from all
       </Directory>

       ErrorLog ${APACHE_LOG_DIR}/error.log
       CustomLog ${APACHE_LOG_DIR}/access.log combined


</VirtualHost>

Configuration of AWStats

We configure here AWStats to use the DaCHS and Apache log files and eliminate queries coming from monitoring server at ObsParis.

Apache statistics

Go into the /etc/awstats directory and open a new file for edition:

Code Block
languagebash
cd /etc/awstats
sudo pico awstats.conf

In this file, edit the following lines (search by keyword - ctrl-W on Pico - : LogFileLogFormat, etc) and replace by the proposed content:

Keyword to searchLine to input
LogFile
LogFile="/usr/share/awstats/tools/logresolvemerge.pl /var/log/apache2/access.log* |"
LogFormat
LogFormat=1
SiteDomain
SiteDomain="<<my_servername>>.<<my_domain>>"
HostAliases
HostAliases="localhost 127.0.0.1 <<my_domain>>"
DNSLookup
DNSLookup=2
AllowFullYearView
AllowFullYearView=3
SkipHosts
SkipHosts="145.238.187.13 145.238.187.29 145.238.193.18"

The VESPA technical team plan to monitor the status of all VESPA servers using a Nagios server. Its hits should be removed from the statistics. This is the purpose of the line starting with SkipHosts.

The Apache2 log files must also be readable by AWStats scripts. First we modify the permissions of Apache2 log files. Those permissions are defined in the /etc/logrotate.d/apache2 file. We have to modify the line starting with the create command by:

Code Block
	create 644 root adm

If you need to fix permissions of already existing log files, the following commands will fix the Apache2 log permissions:

Code Block
languagebash
sudo chmod 644 /var/log/apache2/access.log*
sudo chmod 755 /var/log/apache2

DaCHS statistics

Staying in the /etc/awstats directory, copy the file awstats.conf to awstats.dachs.conf:

Code Block
languagebash
sudo cp awstats.conf awstats.dachs.conf
sudo pico awstats.dachs.conf

In this file, edit the following lines (search by keyword - ctrl-W on Pico - : LogFileLogFormat, etc):


 

Keyword to searchLine to input
LogFile
LogFile="/usr/share/awstats/tools/logresolvemerge.pl /var/gavo/logs/web.log* |"
LogFormat
LogFormat="%other %other %other %host %other %other %time1 %methodurl %code %bytesd %refererquot %uaquot"
URLWithQuery
URLWithQuery=1
 


Setup a crontab to update statistics every day

First create a file containing the calling sequence to AWStats: 

Code Block
languagebash
cd /usr/local/bin
sudo pico run_awstats

In this file, input the three following lines: 

Code Block
languagebash
linenumberstrue
#!/bin/bash
/usr/bin/perl /usr/lib/cgi-bin/awstats.pl -config=dachs -update
/usr/bin/perl /usr/lib/cgi-bin/awstats.pl -config=apache -update

Finally, change the access control configuration for this file:

Code Block
languagebash
sudo chmod 777 run_awstats

A task can then be added to the crontab of your system using :

Code Block
languagebash
sudo crontab -e

Add those lines at the end of the file:

Code Block
languagetext
linenumberstrue
0 0 * * * www-data /usr/local/bin/run_awstats 2>/dev/null


Info

If you use the crontab -e command for the first time, you may have to select en editor. The following lines should then appear in your terminal:

Code Block
Select an editor.  To change later, run 'select-editor'.
  1. /bin/nano        <---- easiest
  2. /usr/bin/vim.tiny

Choose 1-2 [1]: 

If you are not very familiar with vim, prefer the nano option.

Run AWStats

Code Block
languagebash
sudo /usr/local/bin/run_awstats

Access AWStats

The AWStats reports are available at:

Figure 5.1 shows a screenshot of the AWStats monitoring page.

Fig 5.1: View of the AWStats monitoring page.

7. Allowing up external access to database

In order to connect to the PostgreSQL (PgSQL) database that is used by DaCHS, you can either set up an SSH tunnel or configure DaCHS server to allow external access to the database:

Method A: By opening an SSH tunnel

From your host:

Code Block
languagebash
ssh <<my_user>>@127.0.0.1 -p 2222 -L 5432:127.0.0.1:5432

Then leave the console open during the tutorial.

Method B: By configuring DaCHS server

On the DaCHS server, you have to edit 2 files (pg_hba.conf and postgresql.conf). The first file is used to set up which range of IP have access, with which user and what role. Open the file:

Code Block
sudo pico /etc/postgresql/9.4/main/pg_hba.conf

and at the end of the file add the following line:

Code Block
host	all		gavoadmin	0.0.0.0/0	md5


Warning

That configuration will allow all IP address to connect. That's good for a tutorial, but not for real operations. You will have to restrict the range of IPs that are allowed to connect. Check with your local system administrator if needed.

The second file is used to configure which IP range the database is listening to for queries. Open the file:

Code Block
sudo pico /etc/postgresql/9.4/main/postgresql.conf

and modify the line containing the "listen_addresses" directive:

Code Block
listen_addresses = '*'

The PgSQL database as well as DaCHS must now be restarted:

Code Block
sudo /etc/init.d/postgresql restart
sudo gavo serve restart

8. Setting up pgAdmin (optional)

Installation

pgAdmin aims to provide a graphical interface to view and manage a database. We will use this tool on section Setting up an EPN-TAP service. If you don't want to install pgAdmin on you computer, you can however use psql commands instead.

First, install pgAdmin on your host machine. If you are on Debian based distribution:

Code Block
languagebash
sudo apt-get install pgadmin3

Compiled sources are also available for MacOS and Windows.

Info

You need pgAdmin v. 1.20 or upper in order to deal with postgreSQL 9.4. Once installed, check pgdmin version (Help menu, About).

Configuration

Check for the gavo user password:

Code Block
languagebash
cat /var/gavo/etc/feed

Now start pgAdmin, then click on Add a connection to a server (power socket icon) and fill the fields as the following:

 
 TestProductionComment
Name DaCHS-testDaCHS As you want
Host127.0.0.1<<my_server>>.<<my_domain>> 
TCP port 5433 
Service   leave empty
Base maintenancepostgres 
Use namegavoadmin 
password******* Get it with the command above
save password  yes As you want
 


Checking installation

Now you should be able to see your database: