Sites installation and setup tips: gLite Configuration

• Issues specific to SL4 installations
• The site-info directory tree
• Tuning and caveats, before running yaim

Assuming a basic installation of gLite3.2 has already been performed (in particular, packages glite-yaim-core-4.0.10-2 and ig-yaim-4.0.10-2) on one node, copy the whole tree below /opt/glite/yaim/examples/siteinfo to a safe place and start modifying it. For example:

rsync -av /opt/glite/yaim/examples/siteinfo /root/

Note that we assume that the site will be configured 'a la INFN-Grid'. The files which will need to be modified are:

• ig-site-info.def file: this contains most of the configuration for your site: if you are installing a CREAM-CE use this file as a substitute for the siteinfo.def mentioned in the CREAM-CE specific instructions.
• files like services/ig-< service >: these contain service-specific configurations

You can download siteinfo_20110210.tgz and use it as an example of what you will need to do for your site customization: original files have been saved with extension .orig. Names of servers offering core services have been taken from this page: other servers could/should be used for EuMed .

You are free to define new variables and use them as you would in a regular bash script. For example you may want to define MY_DOMAIN to contain the value of your domain (ex.: MY_DOMAIN = mydomain.org), or MY_INT_DOMAIN to contain the value of your internal network domain (ex.: MY_INT_DOMAIN = research.cluster).

Issues specific to SL4 installations

During the VOMS server installation, I found some dependency problems. It looks like such problems exist since March 2010 and have not (yet) been fixed, so here is my workaround:

• after OS installation is complete, make sure you don't have jdk installed

     yum remove jdk
     

• download this RPM and install it
• follow instructions as usual:

     yum install glite-VOMS_mysql
     ...
     

The site-info directory tree

The file is rather long, make sure you read through all of it. A non-exhaustive list of variables which need to be set/modified according to your configuration, follows:

WN_LIST

A file containing the list of your worker nodes. When configuring a worker-node, the list should already include its name: this can be done by the kickstart, appending the hostname to the base file.

USERS_CONF, GROUPS_CONF

Typically modified only when supporting non-standard VOs.

USER_HOME_PREFIX

This is the path under which the generic GRID accounts will be created: defaults to /home. It is advised you change this to some other value (=/home/GridUsers/ is used in the example siteinfo tarfile) for at least two reasons: keeping generic accounts (there will be a lot of them) separate from local accounts, and easily allowing sharing of home directories throughout the site, since you'd just need to NFS-export this path to all CE/WN/UI. This is particularly useful when configuring the site for MPI support.

MYSQL_PASSWORD

Obvious meaning

SITE_NAME, SITE_EMAIL, SITE_LAT, SITE_LONG

These could/should be in the BDII file.

MY_DOMAIN

This is not defined anywhere in the file, yet it is used: set it to the domain part of your IP hosts

PRIVATE_NETWORK

if your nodes belong to a private network. In this case you should also set INT_NET, CE_INT_HOST, INT_HOST_SW_DIR, MY_INT_DOMAIN.

CE_PHYSCPU, CE_LOGCPU, CE_SMPSIZE, CE_CAPABILITY, CE_OTHERDESCR, CE_SI00, CE_SF00

Set these according to your CE type, and the average CPU you have. Note that CE_PHYSCPU is the number of physical chips at your site (please don't ask me why a user would be interested in the number of silicon chips), CE_LOGCPU is the total number of cores at your site, Cores (to be defined in CE_OTHERDESCR) is the number of cores/chip (same as above, please don't ask me), CE_SMPSIZE is the number of cores per machine. Also note that CE_PHYSCPU times Cores must be equal to CE_LOGCPU: not sure floats are allowed, nor exactly in what other places are these variables used. To avoid problems I suggest to start from LOGCPU, set Cores to the maximum (e.g., 8 if you have 4core CPUs with HyperThreading enabled) and derive PHYSCPU: round numbers to integers! Note the difference between Cores and CE_SMPSIZE: the former is rather useless in my opinion, the latter is the number of cores per WN and I suggest to set it to the highest available at your site (e.g., 16 if you have dual-4core machines with HyperThreading ON). Refer to this document to set reasonable numbers for the published average HepSpec06 for your nodes, and for the scaling factor between HepSpec00 and HepSpec06 (to make a long story short, you can use SI00 = SF00 = HS06*250): in case you did not find your CPU listed at the link above, you may be interested in knowing that I have measured HepSpec06 = 11.93 for a dual-processor 64bit-capable Intel Xeon 3 GHz with HyperThreading off. Note that CE_OTHERDESCR accpets floats, so you can set it, for example, to "Cores=1,Benchmark=5.96-HEP-SPEC06" (hence, in this case you would set SI00 = 1490)

BASE_SW_DIR, HOST_SW_DIR

Path for the NFS-exported software area, and NFS server name

BATCH_BIN_DIR

this needs to be set only if running ig_yaim. Unlike what is written in the example, however, it MUST be set also for a CREAM CE using PBS/Torque ( BATCH_BIN_DIR = /usr/bin)

BATCH*, JOBMANAGER

Set these according to your batch system type: note in particular that BATCH_LOG_DIR should point to the directory below which subdirectory server_logs/ can be found, hence for Torque it will be /var/spool/pbs

WMS_HOST, LB_HOST, DGAS_HLR_RESOURCE, BDII_HOST

Note that LB_HOST is used in the example, but not defined.

QUEUES

List of queues defined on your CE. You should also set the mapping between VO and batch queue by properly setting variable _GROUP_ENABLE=_queue-name_

VO_SW_DIR

Base path where VO software areas will appear. For VO=eumed you will also be asked to set variable VO_EUMED_SW_DIR=$VO_SW_DIR/eumed

Note: part on DGAS, MON, SE still missing...

The services/ig-bdii_site or services/glite-bdii_site file

First of all, please make sure ONLY ONE of the above files exists, or yaim will get confused (and you never know which files it uses to source the variable definitions).

A non-exhaustive list of variables which need to be set/modified according to your configuration, follows. Note that the value for all variables mentioned below has to be enclosed between double-quotes.

SITE_NAME

your site name. Note that this variable must be declared in both the siteinfo and the BDII-specific configuration file, otherwise an incomplete LDAP description of your site will result (see also this Savannah bug). There is a convention to name sites like this nationCode-cardinal-siteDesc (also check the Gstat page) where:

• nationCode is the two-letter code for your state (e.g., DZ for Algeria, or JO for Jordan)
• cardinal is a two-digit number, starting from 01
• siteDesc is a short string describing your site (e.g., ARN for Algerian Cerist site, JUNET for Jordan University site)

SITE_DESC

A long format name for your site

SITE_SUPPORT_EMAIL

Contact email for support, possibly a helpdesk system

SITE_SECURITY_EMAIL

Contact email for security

SITE_LOC

City, Country. Please stick to this form or you may get errors in the GSTAT page.

SITE_WEB

Site website

SITE_OTHER_GRID

An or'ed combination of GRIDs, which has to include EUMED. For example you may want to set it to "WLCG|EGEE|EUMED"

SITE_OTHER_WLCG_PARENT

Set this only if your site is part of WLCG

BDII_REGIONS

List of symbols for your CE(s) and SE(s). You can call symbols as you please: for each symbol you will then have to define a variable =BDII__URL". Assuming you have just one CE and one SE, you may end up with something like:

    BDII_REGIONS="CE01 SE01"
    BDII_CE01_URL="ldap://<yourCE>:2170/mds-vo-name=resource,o=grid"
    BDII_SE01_URL="ldap://<yourSE>:2170/mds-vo-name=resource,o=grid"
   

The services/glite-creamce file

A non-exhaustive list of variables which need to be set/modified according to your configuration, follows.

CEMON_HOST

not sure this is really needed/used, though

CREAM_DB_USER

mandatory setting

CREAM_DB_PASSWORD

set it to some string

BLPARSER_HOST

most probably set this to $CE_HOST

CREAM_CE_STATE

set it to reflect the state of your CE, whether Production or Special (for testing purposes only), read file for more info.

Tuning configuration and caveats, before running yaim

Please follow these random advices before you try to configure your CREAM-CE (and possibly, other services):

• make sure package xml-commons-apis is installed: there must be some missing dependency, it was not installed on my machine.
• only for Torque server installation, if your worker-nodes are on a private network: follow these instructions as the automatic procedure makes some mess with the private and the public name for the CE
  • edit /var/spool/pbs/server_priv/acl_svr/acl_hosts and make sure it contains both the public and the private fully-qualified CE names
  • edit /var/spool/pbs/server_priv/acl_svr/managers and /var/spool/pbs/server_priv/acl_svr/operators and make sure they contain (at least!) both root@<_private_CE_name_> and root@<_public_CE_name_>
  • edit /opt/glite/yaim/functions/local/config_torque_server and use the public CE name ${CE_HOST} in the line defining /var/spool/pbs/server_name
  • edit /opt/glite/yaim/functions/local/config_torque_server and change the qmgr commands to read

          set server managers += root@`hostname -f`
          set server managers += root@`echo $CE_INT_HOST`
          set server operators += root@`hostname -f` 
          set server operators += root@`echo $CE_INT_HOST`
          

    so that both the public and private names of the CE are present
  • edit /opt/glite/yaim/functions/local/config_maui_cfg and use the public CE name ${CE_HOST} in the line defining SERVERHOST
• configuring NFS, if your worker nodes are on a private network: edit /opt/glite/yaim/functions/local/config_nfs_sw_dir_server, look for the line containing no_root_squash and substitute MY_DOMAIN with MY_INT_DOMAIN

-- FulvioGaleazzi - 2010-11-17