Andy: Update from Wiki

2006-12-18T13:27:31Z

Update from Wiki

New page

'''This Page Under Construction'''

= Overview =

Programmer's Week Project Material: [[Media:2005_UCSD_ProgrammingWeek-final.ppt|Project Summary Slide]]

 

= Compute Resources =

A 32 node Compute Cluster has been made available to the NAMIC community during the Programmer's Week. Under Condor this allows for 64 available processors. We have set up a 'submit node' where you can log in and run jobs from. It is also possible to access data from the Birn Data Grid (SRB) for computational use, utilizing the proxy delegation method (i.e. you can delegate a user proxy to condor which will then delegate the proxy to the selected computational resources which will then be able to interact directly with the Data Grid). To obtain an account, please contact either Jason Gerk or Jeff Grethe during the Programmer's Week. Once you have a standard Unix account, you will SSH to the 'submit node' '''namic-srb.nbirn.net''' .

= Remote Access to Data Grid via GSI Proxies =

Once you login to the machine '''namic-srb.nbirn.net''' run the command '''birn-proxy-retrieve.pl''' .

[jgerk@namic-srb jgerk]$ birn-proxy-retrieve.pl

You are about to retrieve your BIRN proxy for authentication.

Please enter your portal username.
jgerk
Please enter your portal passphrase to retrieve your proxy.

You've successfully retrieved your BIRN proxy.

Displayed below is your proxy information:
-----------------------------------------------
subject : /C=US/O=BIRN/OU=UCSD - BIRN Coordinating Center/CN=Jason Gerk/USERID=jgerk/CN=proxy
issuer : /C=US/O=BIRN/OU=UCSD - BIRN Coordinating Center/CN=Jason Gerk/USERID=jgerk
identity : /C=US/O=BIRN/OU=UCSD - BIRN Coordinating Center/CN=Jason Gerk/USERID=jgerk
type : full legacy globus proxy
strength : 512 bits
path : /tmp/x509up_ujgerk
timeleft : 11:59:59
0

This will retrieve your proxy and set up your environment in that shell and allow you to add Scommands into your shell script.

= A Condor/SRB Example =

Here is a working example. In my home directory I have a file called '''job_submit_file''' which contains the executable program (small shell script) the I want to be run when I launch the Condor job. This file is called '''getInfoSRB''', but you can name it anything.

[jgerk@namic-srb birn_condor]$ more job_submit_file

Executable = getInfoSRB
Universe = vanilla
Output = output.stdout
Error = output.stderr
Log = output.log
getenv = TRUE
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = /tmp/x509up_u$ENV(LOGNAME)

 Below is an example of what my shell script looks like. This will be run in the Compute Cluster environment, calling some Scommands, then pulling out the '''hello1''' program to a standard Linux environment where the permissions are changed and the file is run.

[jgerk@namic-srb birn_condor]$ more getInfoSRB
#!/bin/sh

echo " "
echo "This is an example of a shell script that uses Scommands."
echo " "

# Run a Sls
echo "Do and 'Sls' and show NAMIC related files:"
Sls | grep -i namic

#Get hello from SRB and download it locally on condor execute dir
echo " "
echo "Get a file from SRB: 'Sget hello' "
Sget hello hello1

#Run hello on condor execution machine
echo " "
echo "Set the permissions, and execute the program: chmod +x hello1 then ./hello1 "
echo "(output below) "
chmod +x hello1
./hello1

Next, we are ready to launch the job. The '''job_submit_file''' calls the shell script which calls other programs within the Compute Cluster environment. Here we go.

[jgerk@namic-srb birn_condor]$ birn_condor_submit job_submit_file
Submitting job(s).
Logging submit event(s).
1 job(s) submitted to cluster 68.

Once complete, a sucessful job should write to output.stdout (or whatever you defing in your submit file).

[jgerk@namic-srb birn_condor]$ more output.stdout

This is an example of a shell script that uses Scommands.

Do and 'Sls' and show NAMIC related files:
C-/home/jgerk.ucsd-bcc/namic
C-/home/jgerk.ucsd-bcc/namic_backup

Get a file from SRB: 'Sget hello'

Set the permissions, and execute the program: chmod +x hello1 then ./hello1
(output below)
hello, Condor

= Submitting A Condor Job =

 Submit template file focusing Condor with GSI and SRB. Replace YOUR_EXECUTALBE with the name of your executable, which will usually be a shell script with SRB commands and the executables you really want to run. add other input files to the end of transfer_input_files as needed for example /tmp/x509up_u$(LOGNAME), your_input1, your_input2, etc. Note that if you are using a shell script as your condor executable, you will have to either copy over your scientific code as an input file or store it in SRB and Sget it.

== Template File ==

Executable = YOUR_EXECUTABLE
Universe = vanilla
Output = YOUR_EXECUTABLE.stdout
Error = YOUR_EXECUTABLE.stderr
Log = YOUR_EXECUTABLE.log
getenv = TRUE
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = /tmp/x509up_u$ENV(LOGNAME)

Queue

= A NAMIC Example =

Sample data and an associated executable from the NA-MIC svn Sandbox can be found [[Engineering:ITK_Grid_Pipeline|here]]

In this example the sample data has been uploaded to a directory: /home/jegrethe.ucsd-bcc/TestData/NAMIC Results are written to: /home/jegrethe.ucsd-bcc/TestData/NAMIC_results

The job submit file is as follows:

<nowiki>########################################################
# Submit template file focusing Condor with GSI and SRB.
# Replace YOUR_EXECUTALBE with the name of your executable,
# which will usually be a shell script with SRB commands and
# the executables you really want to run. add other input files to
# the end of transfer_input_files as needed for example /tmp/x509up_u$(USER,
# your_input1, your_input2, etc. Note that if you are using
# a shell script as your condor executable, you will have to
# either copy over your scientific code as an input file or
# store it in SRB and Sget it.
#######################################################

Executable = condorSH.sh
Universe = vanilla
Output = condorSH.stdout
Error = condorSH.stderr
Log = condorSH.log
getenv = TRUE
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = /tmp/x509up_ujegrethe

</nowiki>

The shell script executed on the condor compute node is as follows:

<nowiki>#!/bin/sh

/usr/local/bin/Sget -r /home/jegrethe.ucsd-bcc/TestData/NAMIC

cd NAMIC
chmod a+x ./MultiModalityRigidRegistration
./MultiModalityRigidRegistration reg.nhdr mrt.nhdr xformIN xformOUT 2 1 > reg.out

cd ..
mv NAMIC NAMIC_results
/usr/local/bin/Sput -r NAMIC_results /home/jegrethe.ucsd-bcc/TestData

/usr/local/bin/Sput condorSH.stderr /home/jegrethe.ucsd-bcc/TestData/NAMIC_results
/usr/local/bin/Sput condorSH.stdout /home/jegrethe.ucsd-bcc/TestData/NAMIC_results
</nowiki>

The condor log file for the job is as follows (Note: the log doesn't have any information on the SRB data movement, Condor only manages the movement of the proxy in this case):

000 (069.000.000) 06/17 01:40:08 Job submitted from host: <198.202.95.80:9646>
...
001 (069.000.000) 06/17 01:40:13 Job executing on host: <198.202.95.110:9607>
...
006 (069.000.000) 06/17 01:40:21 Image size of job updated: 35248
...
006 (069.000.000) 06/17 02:00:21 Image size of job updated: 93844
...
005 (069.000.000) 06/17 03:06:38 Job terminated.
(1) Normal termination (return value 0)
Usr 0 01:26:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 01:26:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
3489 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
3489 - Total Bytes Received By Job
...

= Condor Quick Help and Useful Commands =

To see what resources are available on the cluster: '''condor_status''' 

[jgerk@namic-srb jgerk]$ condor_status

Name OpSys Arch State Activity LoadAv Mem ActvtyTime

vm1@birn-clus LINUX INTEL Claimed Idle 0.000 1979 0+00:00:03
vm2@birn-clus LINUX INTEL Claimed Idle 0.000 1979 0+00:00:03
vm1@birn-clus LINUX INTEL Claimed Idle 0.000 1979 0+00:00:03
vm2@birn-clus LINUX INTEL Claimed Idle 0.000 1979 0+00:00:04
vm1@birn-clus LINUX INTEL Claimed Idle 0.000 1979 0+00:00:03
vm2@birn-clus LINUX INTEL Unclaimed Idle 0.000 1979 0+00:00:05
vm1@birn-clus LINUX INTEL Claimed Busy 0.000 1979 0+00:00:04
vm2@birn-clus LINUX INTEL Claimed Idle 0.000 1979 0+00:00:06
vm1@birn-clus LINUX INTEL Claimed Idle 0.000 1979 0+00:00:03
vm2@birn-clus LINUX INTEL Claimed Idle 0.000 1979 0+00:00:03

 

Machines Owner Claimed Unclaimed Matched Preempting

INTEL/LINUX 62 0 61 1 0 0

Total 62 0 61 1 0 0

 

 To launch a job using a submit file: '''birn_condor_submit <job_submit_file>''' 

[jgerk@namic-srb birn_condor]$ birn_condor_submit job_submit_file
Submitting job(s).
Logging submit event(s).
1 job(s) submitted to cluster 68.

To see what is queued to be run: '''condor_q''' 

[jgerk@namic-srb birn_condor]$ condor_q

-- Submitter: namic-srb.public : <132.239.132.232:9663> : namic-srb.public
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
67.0 jgerk 6/27 04:02 0+00:03:53 R 0 0.0 getInfoSRB

1 jobs; 0 idle, 1 running, 0 held

To remove a job from the queue: '''condor_rm <job_id>''' 

[jgerk@namic-srb birn_condor]$ condor_rm 67.0
Job 67.0 marked for removal

 

= Syntax Strawman =

namic-submit [--save-submit-file <submitfile>] [--job-directory <jobdir>] <cmd> [<args>...]

where:

* ''submitfile'' is a condor job submit file; if not specified a temp file is generated
* ''jobdir'' is a working state directory for a set of submissions that you want to later observe with namic-wait or manually inspect
* ''cmd'' is a shell script or executable that can be run on the remote environment
* ''args'' are arguments to the cmd

functionality: namic-submit takes the given command and args and runs the job on a condor-determined external compute node. By default stdin, stdout, and stderr of the ''cmd'' are mapped to the corresponding files of namic-submit.

If srb proxy is not set up, namic-submit prompts you for srb user id and password and sets up proxy.

namic-wait [--job-directory <jobdir>] [<groupid>]

where:

* ''jobdir'' is a directory that corresponds to the state for jobs submitted by namic-submit

functionality: namic-wait blocks until all jobs submitted with the given groupid have completed running

Engineering:Project:Condor Job Submission - Revision history

Andy: Update from Wiki