Tutorial 2: Reaching the Grid

In this tutorial is presented the way to run a workflow on a different environment and especially on a glite computational grid.

Prerequisites

Running OpenMOLE workflows on a grid doesn't imply to be aware of grid mechanisms and protocols, but the user should get valid credential through the validation process of it's certification authority. Getting Grid access certificates is for instance compulsory. For instance for the accessing the virtual organization "vo.iscpif.fr", a good start is to visit the simple steps to access the grid (Steps 1 and 2)

Introduction

One of the most important valuable feature of OpenMOLE is to make an easy the access to grid computing. The idea is to declaratively specify environments (a virtual organization definition for instance) where each task will to be run. In the future, we also intend to implements an automatic environment scheduler, which is able to determine where it is optimal to run each task (locally, on a grid or on a cluster...).

Configure preferences for the glite environment (need to be done only once)

• In OpenMOLE's console:

init org.openmole.plugin.environment.glite.GliteEnvironment

• You will be asked a few question to configure necessary preferences for accessing glite VO.

CertificateType type (default=pem; old=pem) [pem,p12]:

Choose the appropriate type rather you use pem or p12 certificate type.

Note: if you chose p12 and you use sun Java Virtual Machine (JVM) you should disable legal restriction on cryptography in your JVM by downloading and installing the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 6. (http://www.oracle.com/technetwork/java/javase/downloads/index.html). With OpenJDK it should work ok.

• Then you will be asked for your certificate location and decryption key.

• Last you will be asked for the runtime location. The runtime is a file named "org.openmole.runtime-0.3-SNAPSHOT.tar.bz2" distributed with OpenMOLE. You should indicate the full path to this file on your machine.

Define an execution environment

The first step consists in defining the environment, that is to say defining:

• the VO name
• the VOMS url
• the BDII url

Example of the definition of glite environments for several virtual organizations:

• vo.iscpif.fr:

  import org.openmole.plugin.environment.glite.*
  
  env = new GliteEnvironment("vo.iscpif.fr",
          "voms://grid12.lal.in2p3.fr:20013/O=GRID-FR/C=FR/O=CNRS/OU=LAL/CN=grid12.lal.in2p3.fr",
          "ldap://topbdii.grif.fr:2170")

• vo.complex-systems.eu:

  env = new GliteEnvironment("vo.complex-systems.eu",
          "vomss://voms.grid.auth.gr:15160/C=GR/O=HellasGrid/OU=auth.gr/CN=voms.grid.auth.gr",
          "ldap://topbdii.grif.fr:2170")

• biomed

  env = new GliteEnvironment("biomed",
          "voms://cclcgvomsli01.in2p3.fr:15000/O=GRID-FR/C=FR/O=CNRS/OU=CC-IN2P3/CN=cclcgvomsli01.in2p3.fr",
          "ldap://topbdii.grif.fr:2170")

Using myProxy

For using renewable proxies with OpenMOLE you should specify it in your environment, by adding your myProxy server, your DN and a password:

env = new GliteEnvironment("biomed",
        "voms://cclcgvomsli01.in2p3.fr:15000/O=GRID-FR/C=FR/O=CNRS/OU=CC-IN2P3/CN=cclcgvomsli01.in2p3.fr",
        "ldap://topbdii.grif.fr:2170",
        "myproxy.cern.ch", 
        "/O=GRID-FR/C=FR/O=CNRS/OU=LPSC/CN=Romain Reuillon",
        "password")

Specifying requirements

When declaring an environment you can specify requirements. For instance for declaring an environment using renewable proxy, allowing jobs of more than 10 hours and using 2GB of RAM on Scientific Linux 5 and above, I should specify it like so:

import static org.openmole.plugin.environment.glite.GliteAttributes.*
import static org.openmole.plugin.environment.jsaga.JSAGAAttributes.*

env = new GliteEnvironment("biomed",
        "voms://cclcgvomsli01.in2p3.fr:15000/O=GRID-FR/C=FR/O=CNRS/OU=CC-IN2P3/CN=cclcgvomsli01.in2p3.fr",
        "ldap://topbdii.grif.fr:2170","myproxy.cern.ch", 
        "/O=GRID-FR/C=FR/O=CNRS/OU=LPSC/CN=Romain Reuillon",
        "password",
        [(CPU_TIME):"PT10H", (MEMORY):"2000", (REQUIREMENTS):"other.GlueHostOperatingSystemRelease >= 5.0"])

Delegating execution of a task to glite

Once an environment has been defined, jobs for a given capsule could be set to be executed on that environment.

The syntax is very simple and only descriptive:

import org.openmole.core.implementation.mole.*
import org.openmole.plugin.task.groovy.*
import org.openmole.core.implementation.capsule.*
import org.openmole.core.implementation.mole.*

//Let us define some task and task capsule
exampleTask = new GroovyTask("exampleTask")
exampleCapsule = new Capsule(exampleTask)

//This capsule has to be run on the 'env' environment
strategy = new FixedEnvironmentSelection()
strategy.select(exampleCapsule,env)

Applying a grouping strategy

Sometimes, when running short jobs (lasting only a few minutes or seconds) within an exploration, it is longer to carry out the submission on the grid than to run the program itself. That is the reason why functionnalities have been developed to group jobs executions on an execution node. The most two famous grouping strategies are:

• by variable (all the executions having the same value for a particular variable are run on the same execution node),
• by fixing the number of jobs per execution node (if set it to 10 for instance, each execution node will run 10 jobs).

Both strategies are useful. Grouping per variable can be for instance a nice strategy when dealing with large input files (grouping on value of this files means all the jobs in the group use the same file as input, and so fewer download on the execution node). Grouping on a fix number of jobs enables to build homogeneous execution durations on each node (if the execution time of jobs is homogeneous).

An example of grouping according to a fixed number of jobs per node

import org.openmole.plugin.groupingstrategy.batch.*
import org.openmole.core.implementation.mole.*

jobGrouping = new MoleJobGrouping()
jobGrouping.set(exampleCapsule,new NumberOfMoleJobsGroupingStrategy(4))

Running jobs on the grid

Running jobs on a particular environment is quite similar to run it locally. All that is required is to apply the environment and the optional grouping strategy to the execution mole.

ex = new MoleExecution(new Mole(exampleCapsule),
                       strategy,
                       jobGrouping)
ex.start()