[GE users] Delegated filestaging problem

sumyee sum-yee.lai at hp.com
Thu Mar 18 18:21:00 GMT 2010


Hi Reuti,

Thanks for the clarification!

Sum Yee

-----Original Message-----
From: reuti [mailto:reuti at staff.uni-marburg.de] 
Sent: Wednesday, March 17, 2010 3:55 PM
To: users at gridengine.sunsource.net
Subject: Re: [GE users] Delegated filestaging problem

Hi,

Am 17.03.2010 um 20:10 schrieb sumyee:

> Hi Reuti,
>
> $fs_stdin_file_staging is one of the prolog arguments.  As part of  
> the prolog script, I write out $fs_stdin_file_staging and the other  
> $fs_ variables.  They are all '0' or "", even when I use the python  
> script below to submit the job using drmaa.

only valid path names for jt.drmaa_input_pat and output/error will  
allow the flag to be set. AFAICS you are setting a directory for the  
input file, but directories are only valid for output/error  
specification. The input file must be a single file.

-- Reuti


> Thanks,
>
> Sum Yee
> ---------------------------------------
>
> qconf -sq default.q
> qname                 default.q
> hostlist              @allhosts
> seq_no                10
> load_thresholds       NONE
> suspend_thresholds    NONE
> nsuspend              1
> suspend_interval      00:05:00
> priority              0
> min_cpu_interval      00:05:00
> processors            UNDEFINED
> qtype                 BATCH INTERACTIVE
> ckpt_list             reloc
> pe_list               make
> rerun                 FALSE
> slots                 4
> tmpdir                /tmp/sge_tmp
> shell                 /bin/bash
> prolog                /usr/local/GridEngine/custom/bin/prolog.py \
>                      $fs_stdin_file_staging $fs_stdin_host  
> $fs_stdin_path
> epilog                /usr/local/GridEngine/custom/bin/epilog.py \
>                      $fs_stdout_file_staging $fs_stdout_host  
> $fs_stdout_path
> shell_start_mode      unix_behavior
> starter_method        /usr/local/GridEngine/custom/bin/starter.py
> suspend_method        NONE
> resume_method         NONE
> terminate_method      NONE
> notify                00:05:00
> owner_list            NONE
> user_lists            NONE
> xuser_lists           NONE
> subordinate_list      slots=4(low.q:0:lr)
>
> -----Original Message-----
> From: reuti [mailto:reuti at staff.uni-marburg.de]
> Sent: Wednesday, March 17, 2010 10:12 AM
> To: users at gridengine.sunsource.net
> Subject: Re: [GE users] Delegated filestaging problem
>
> Hi,
>
> Am 17.03.2010 um 17:54 schrieb sumyee:
>
>> Hi,
>>
>> I am trying to use delegated filestaging to transfer files.  Even
>> though I have delegated filestaging turned on and I am submitting
>> jobs via drmaa, $fs_stdin_file_staging is always '0'.  Is there
>> something else I need to do to run delegated filestaging?
>
> when is $fs_stdin_file_staging always 0? It's not clear where you
> check it. It's a pseudo variable which can be in the definition of
> the prolog/epilog. It's not to be used in the job script.
>
> -- Reuti
>
>
>>
>> Thanks!
>>
>> Sum
>>
>>
>> Script to submit job:
>> #!/usr/bin/env python
>>
>> import os, sys, drmaa
>>
>> def main():
>> 	s = drmaa.Session()
>> 	s.initialize()
>>
>> 	jt = s.createJobTemplate()
>> 	jt.remoteCommand = os.getcwd() + '/simple.sh'
>> 	path = '%s:%s' % (os.uname()[1], os.getcwd())
>> 	jt.drmaa_input_path=path
>> 	jt.drmaa_transfer_files='io'
>> 	jobid = s.runJob(jt)
>> 	s.deleteJobTemplate(jt)
>> 	s.exit()
>>
>> if __name__ == '__main__':
>> 	main()
>>
>> ----------------------------------------------------------------------
>> --------------------
>> qconf -sconf
>> #global:
>> execd_spool_dir              /usr/local/GridEngine/default/spool
>> mailer                       /bin/mail
>> xterm                        /usr/bin/X11/xterm
>> load_sensor                  /usr/local/GridEngine/custom/bin/load.py
>> prolog                       none
>> epilog                       none
>> shell_start_mode             unix_behavior
>> login_shells                 sh,ksh,csh,tcsh,bash
>> min_uid                      0
>> min_gid                      0
>> user_lists                   none
>> xuser_lists                  none
>> projects                     none
>> xprojects                    none
>> enforce_project              false
>> enforce_user                 auto
>> load_report_time             00:00:40
>> max_unheard                  00:05:00
>> reschedule_unknown           00:00:00
>> loglevel                     log_warning
>> administrator_mail           none
>> set_token_cmd                none
>> pag_cmd                      none
>> token_extend_time            none
>> shepherd_cmd                 none
>> qmaster_params               MAX_DYN_EC=10000,ENABLE_FORCED_QDEL=true
>> reporting_params             accounting=true reporting=true \
>>                             flush_time=00:00:15 joblog=false
>> sharelog=00:00:00
>> finished_jobs                100
>> gid_range                    20000-30000
>> qlogin_command               builtin
>> qlogin_daemon                builtin
>> rlogin_command               /usr/bin/ssh -t -Y
>> rlogin_daemon                /usr/sbin/sshd -i
>> rsh_command                  /usr/bin/ssh -t -Y
>> rsh_daemon                   /usr/sbin/sshd -i
>> max_aj_instances             2000
>> max_aj_tasks                 75000
>> max_u_jobs                   0
>> max_jobs                     0
>> max_advance_reservations     15
>> auto_user_oticket            0
>> auto_user_fshare             0
>> auto_user_default_project    none
>> auto_user_delete_time        86400
>> delegated_file_staging       true
>> reprioritize                 false
>> jsv_url                      none
>> jsv_allowed_mod              ac,h,i,e,o,j,M,N,p,w,l
>> execd_params                 INHERIT_ENV=false SET_LIB_PATH=true
>>
>> ------------------------------------------------------
>> http://gridengine.sunsource.net/ds/viewMessage.do?
>> dsForumId=38&dsMessageId=249193
>>
>> To unsubscribe from this discussion, e-mail: [users-
>> unsubscribe at gridengine.sunsource.net].
>>
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=249196
>
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net 
> ].
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=249210
>
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net 
> ].
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=249248

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=249467

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list