[GE users] DRMAA output redirection oddness

reuti reuti at staff.uni-marburg.de
Tue Dec 7 15:40:59 GMT 2010


Please don't CC me. My answers may go to the wrong address.

Am 07.12.2010 um 14:54 schrieb Scott Wilson:

> I get:
> 
> stdout_path_list:           NONE:host123:./SGE-test.log

This is interesting. I was always wondering about the two NONE in front of the output file name. The first one is set as I found if you specify a hostname before the output filename resp. output directory. I never saw the submit host popping up there.

Can it be, that it's coincidence because you prefix the output path by a ":" in your job.setOutputPath (which I wondered about already)?

-- Reuti


> where host123 is the name (but not FQDN) of the host the job was submitted on. Could this be the issue? My home folder is on an NFS mount, so it should be accessible "locally" from all nodes. Could GE be trying to do file transfers? I haven't consciously set any up.
> 
> Scott
> 
> Scott Wilson
> Epistemy
> 
> On 7 Dec 2010, at 13:40, reuti wrote:
> 
>> Am 07.12.2010 um 14:25 schrieb templedf:
>> 
>>> Well, then.  Last suggestion: don't use anything.  The default 
>>> assumption is that your starting point for all paths is the user's home 
>>> directory.  If you set the workingDirectory to ./SGE-test, it should 
>>> resolve to ~/SGE-test.
>> 
>> In libs/japi/drmaa.c also $TASK_ID is referrenced. Hence the idea is not to use the environment variables of any shell, but the pseudo variables which can be used for -e/-o. Question is, why they are not expanded lateron.
>> 
>> What is `qstat -j <job_id>` showing for such a job of yours? For a `qsub -o ...`:
>> 
>> $ qstat -j 123
>> ...
>> stdout_path_list:           NONE:NONE:$HOME/test
>> 
>> should be there.
>> 
>> -- Reuti
>> 
>> 
>>> 
>>> Daniel
>>> 
>>> On 12/7/10 8:21 AM, Scott Wilson wrote:
>>>> No bother. If only I got a penny every time *I* missed something. ;-)
>>>> 
>>>> Setting the working directory to "~/SGEtest" gets me:
>>>> 
>>>> 12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory
>>>> 
>>>> Is expansion of '~' not a shell feature, too?
>>>> 
>>>> Scott
>>>> 
>>>> Scott Wilson
>>>> Epistemy
>>>> 
>>>> On 7 Dec 2010, at 13:07, Daniel Templeton wrote:
>>>> 
>>>>> Sorry, missed that.  I'm answering emails from a hotel room while listening to a conference call. ;)
>>>>> 
>>>>> In that case, yes that should concern you.  For now, try just using ~ instead of the placeholder.  I'll try to reproduce it and file an issue if needed.
>>>>> 
>>>>> Daniel
>>>>> 
>>>>> On 12/7/10 8:01 AM, Scott Wilson wrote:
>>>>>> I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
>>>>>> 
>>>>>> Scott
>>>>>> 
>>>>>> Scott Wilson
>>>>>> Epistemy
>>>>>> 
>>>>>> On 7 Dec 2010, at 12:54, Daniel Templeton wrote:
>>>>>> 
>>>>>>> When you're using DRMAA, there's no shell to expand $HOME for you.  It's being taken literally, e.g. \$HOME.  There are two ways to reference the user's home directory from the workingDirectory, inputPath, and outputPath properties.  The first is just to use standard UNIX conventions, e.g. ~.  The second is to use the JobTemplate.HOME_DIRECTORY placeholder:
>>>>>>> 
>>>>>>> http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
>>>>>>> 
>>>>>>> like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
>>>>>>> 
>>>>>>> Daniel
>>>>>>> 
>>>>>>> On 12/7/10 6:51 AM, epistemyscott wrote:
>>>>>>>> Dear all,
>>>>>>>> I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
>>>>>>>> 
>>>>>>>> For instance, when using the code below results in the following message in the qmaster messages file:
>>>>>>>> 
>>>>>>>> 12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
>>>>>>>> 
>>>>>>>> Touching the output files beforehand makes no difference.
>>>>>>>> 
>>>>>>>> If I comment out the setOutputPath and setErrorPath lines, I get the following:
>>>>>>>> 
>>>>>>>> 12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
>>>>>>>> 
>>>>>>>> If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
>>>>>>>> 
>>>>>>>> Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
>>>>>>>> 
>>>>>>>> Your help would be most appreciated.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Scott
>>>>>>>> 
>>>>>>>> public static void submitJob(){
>>>>>>>> 	File runFolder = new File(System.getProperty("user.home"), "SGE-test");
>>>>>>>> 	runFolder.mkdir();
>>>>>>>> 	
>>>>>>>> 	System.out.println("Folder created");
>>>>>>>> 	
>>>>>>>> 	SessionFactory factory = SessionFactory.getFactory();
>>>>>>>> 	Session session = factory.getSession();
>>>>>>>> 	
>>>>>>>> 	try{
>>>>>>>> 		session.init("");
>>>>>>>> 		JobTemplate job = session.createJobTemplate();
>>>>>>>> 		
>>>>>>>> 		job.setRemoteCommand("pwd");
>>>>>>>> 		
>>>>>>>> 		job.setJobName("SGE-test");
>>>>>>>> 		job.setBlockEmail(true);
>>>>>>>> 		job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
>>>>>>>> 		System.out.println("Working directory is: " + job.getWorkingDirectory());
>>>>>>>> 		job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
>>>>>>>> 		job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
>>>>>>>> 		
>>>>>>>> 		String identifier = session.runJob(job);
>>>>>>>> 		
>>>>>>>> 		System.out.println("Job submitted with identifier '" + identifier + "'.");
>>>>>>>> 		
>>>>>>>> 		session.deleteJobTemplate(job);
>>>>>>>> 	}
>>>>>>>> 	catch(DrmaaException e){
>>>>>>>> 		System.out.println(e);
>>>>>>>> 		e.printStackTrace();
>>>>>>>> 		throw e;
>>>>>>>> 	}
>>>>>>>> 	finally{
>>>>>>>> 		try {
>>>>>>>> 			session.exit();
>>>>>>>> 		}
>>>>>>>> 		catch(DrmaaException e) {
>>>>>>>> 			System.out.println("Closing DRMAA session failed. " + e);
>>>>>>>> 			e.printStackTrace();
>>>>>>>> 		}
>>>>>>>> 	}
>>>>>>>> }
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Scott Wilson
>>>>>>>> Developer
>>>>>>>> Epistemy
>>>>>>>> Energy Academy
>>>>>>>> Riccarton
>>>>>>>> EH14 4AS
>>>>>>>> 
>>>>>>>> tel: +44 131 564 0232
>>>>>>>> Scott.Wilson at Epistemy.com
>>>>>>>> http://www.Epistemy.com/
>>>>>>>> Epistemy Limited is a company registered in Scotland, number SC365481.
>>>>>>>> Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
>>>>>>>> 
>>>>>>>> ------------------------------------------------------
>>>>>>>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
>>>>>>>> 
>>>>>>>> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
>>> 
>>> ------------------------------------------------------
>>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302773
>>> 
>>> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
>> 
>> ------------------------------------------------------
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302778
>> 
>> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
> 
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302807

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list