Sites installation and setup tips: gLite Configuration

Assuming a basic installation of gLite3.2 has already been performed (in particular, packages glite-yaim-core-4.0.10-2 and ig-yaim-4.0.10-2) on one node, copy the whole tree below /opt/glite/yaim/examples/siteinfo to a safe place and start modifying it. For example:

rsync -av /opt/glite/yaim/examples/siteinfo /root/

Note that we assume that the site will be configured 'a la INFN-Grid'. The files which will need to be modified are:

  • ig-site-info.def file: this contains most of the configuration for your site: if you are installing a CREAM-CE use this file as a substitute for the siteinfo.def mentioned in the CREAM-CE specific instructions.
  • files like services/ig-< service >: these contain service-specific configurations

You can download siteinfo_20110210.tgz and use it as an example of what you will need to do for your site customization: original files have been saved with extension .orig. Names of servers offering core services have been taken from this page: other servers could/should be used for EuMed .

You are free to define new variables and use them as you would in a regular bash script. For example you may want to define MY_DOMAIN to contain the value of your domain (ex.: MY_DOMAIN =, or MY_INT_DOMAIN to contain the value of your internal network domain (ex.: MY_INT_DOMAIN = research.cluster).

Issues specific to SL4 installations

During the VOMS server installation, I found some dependency problems. It looks like such problems exist since March 2010 and have not (yet) been fixed, so here is my workaround:

  • after OS installation is complete, make sure you don't have jdk installed
       yum remove jdk
  • download this RPM and install it
  • follow instructions as usual:
       yum install glite-VOMS_mysql

The site-info directory tree

The file is rather long, make sure you read through all of it. A non-exhaustive list of variables which need to be set/modified according to your configuration, follows:

A file containing the list of your worker nodes. When configuring a worker-node, the list should already include its name: this can be done by the kickstart, appending the hostname to the base file.
Typically modified only when supporting non-standard VOs.
This is the path under which the generic GRID accounts will be created: defaults to /home. It is advised you change this to some other value (=/home/GridUsers/ is used in the example siteinfo tarfile) for at least two reasons: keeping generic accounts (there will be a lot of them) separate from local accounts, and easily allowing sharing of home directories throughout the site, since you'd just need to NFS-export this path to all CE/WN/UI. This is particularly useful when configuring the site for MPI support.
Obvious meaning
These could/should be in the BDII file.
This is not defined anywhere in the file, yet it is used: set it to the domain part of your IP hosts
if your nodes belong to a private network. In this case you should also set INT_NET, CE_INT_HOST, INT_HOST_SW_DIR, MY_INT_DOMAIN.
Set these according to your CE type, and the average CPU you have. Note that CE_PHYSCPU is the number of physical chips at your site (please don't ask me why a user would be interested in the number of silicon chips), CE_LOGCPU is the total number of cores at your site, Cores (to be defined in CE_OTHERDESCR) is the number of cores/chip (same as above, please don't ask me), CE_SMPSIZE is the number of cores per machine. Also note that CE_PHYSCPU times Cores must be equal to CE_LOGCPU: not sure floats are allowed, nor exactly in what other places are these variables used. To avoid problems I suggest to start from LOGCPU, set Cores to the maximum (e.g., 8 if you have 4core CPUs with HyperThreading enabled) and derive PHYSCPU: round numbers to integers! Note the difference between Cores and CE_SMPSIZE: the former is rather useless in my opinion, the latter is the number of cores per WN and I suggest to set it to the highest available at your site (e.g., 16 if you have dual-4core machines with HyperThreading ON). Refer to this document to set reasonable numbers for the published average HepSpec06 for your nodes, and for the scaling factor between HepSpec00 and HepSpec06 (to make a long story short, you can use SI00 = SF00 = HS06*250): in case you did not find your CPU listed at the link above, you may be interested in knowing that I have measured HepSpec06 = 11.93 for a dual-processor 64bit-capable Intel Xeon 3 GHz with HyperThreading off. Note that CE_OTHERDESCR accpets floats, so you can set it, for example, to "Cores=1,Benchmark=5.96-HEP-SPEC06" (hence, in this case you would set SI00 = 1490)
Path for the NFS-exported software area, and NFS server name
this needs to be set only if running ig_yaim. Unlike what is written in the example, however, it MUST be set also for a CREAM CE using PBS/Torque ( BATCH_BIN_DIR = /usr/bin)
Set these according to your batch system type: note in particular that BATCH_LOG_DIR should point to the directory below which subdirectory server_logs/ can be found, hence for Torque it will be /var/spool/pbs
Note that LB_HOST is used in the example, but not defined.
List of queues defined on your CE. You should also set the mapping between VO and batch queue by properly setting variable _GROUP_ENABLE=_queue-name_
Base path where VO software areas will appear. For VO=eumed you will also be asked to set variable VO_EUMED_SW_DIR=$VO_SW_DIR/eumed

Note: part on DGAS, MON, SE still missing...

The services/ig-bdii_site or services/glite-bdii_site file

First of all, please make sure ONLY ONE of the above files exists, or yaim will get confused (and you never know which files it uses to source the variable definitions).

A non-exhaustive list of variables which need to be set/modified according to your configuration, follows. Note that the value for all variables mentioned below has to be enclosed between double-quotes.

your site name. Note that this variable must be declared in both the siteinfo and the BDII-specific configuration file, otherwise an incomplete LDAP description of your site will result (see also this Savannah bug). There is a convention to name sites like this nationCode-cardinal-siteDesc (also check the Gstat page) where:
  • nationCode is the two-letter code for your state (e.g., DZ for Algeria, or JO for Jordan)
  • cardinal is a two-digit number, starting from 01
  • siteDesc is a short string describing your site (e.g., ARN for Algerian Cerist site, JUNET for Jordan University site)
A long format name for your site
Contact email for support, possibly a helpdesk system
Contact email for security
City, Country. Please stick to this form or you may get errors in the GSTAT page.
Site website
An or'ed combination of GRIDs, which has to include EUMED. For example you may want to set it to "WLCG|EGEE|EUMED"
Set this only if your site is part of WLCG
List of symbols for your CE(s) and SE(s). You can call symbols as you please: for each symbol you will then have to define a variable =BDII__URL". Assuming you have just one CE and one SE, you may end up with something like:

The services/glite-creamce file

A non-exhaustive list of variables which need to be set/modified according to your configuration, follows.

not sure this is really needed/used, though
mandatory setting
set it to some string
most probably set this to $CE_HOST
set it to reflect the state of your CE, whether Production or Special (for testing purposes only), read file for more info.

Tuning configuration and caveats, before running yaim

Please follow these random advices before you try to configure your CREAM-CE (and possibly, other services):

  • make sure package xml-commons-apis is installed: there must be some missing dependency, it was not installed on my machine.
  • only for Torque server installation, if your worker-nodes are on a private network: follow these instructions as the automatic procedure makes some mess with the private and the public name for the CE
    • edit /var/spool/pbs/server_priv/acl_svr/acl_hosts and make sure it contains both the public and the private fully-qualified CE names
    • edit /var/spool/pbs/server_priv/acl_svr/managers and /var/spool/pbs/server_priv/acl_svr/operators and make sure they contain (at least!) both root@<_private_CE_name_> and root@<_public_CE_name_>
    • edit /opt/glite/yaim/functions/local/config_torque_server and use the public CE name ${CE_HOST} in the line defining /var/spool/pbs/server_name
    • edit /opt/glite/yaim/functions/local/config_torque_server and change the qmgr commands to read
            set server managers += root@`hostname -f`
            set server managers += root@`echo $CE_INT_HOST`
            set server operators += root@`hostname -f` 
            set server operators += root@`echo $CE_INT_HOST`
      so that both the public and private names of the CE are present
    • edit /opt/glite/yaim/functions/local/config_maui_cfg and use the public CE name ${CE_HOST} in the line defining SERVERHOST
  • configuring NFS, if your worker nodes are on a private network: edit /opt/glite/yaim/functions/local/config_nfs_sw_dir_server, look for the line containing no_root_squash and substitute MY_DOMAIN with MY_INT_DOMAIN

-- FulvioGaleazzi - 2010-11-17

Topic revision: r6 - 2011-07-04 - 13:10:52 - FulvioGaleazzi
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback