[GE users] Emailing users their usage statistics

Kirk Patton kpatton at transmeta.com
Mon Nov 15 14:45:53 GMT 2004


On Mon, Nov 15, 2004 at 12:08:15PM +0000, Aaron Turner wrote:
> Dear All,
> 
> We reccomend that users specify job resource requirements rather than 
> queues (a sensible policy)

Using resources rather that queues is always a good idea.  :-)

> but often users are unsure what those requirements are. What might be 
> helpful is for them to be
> informed by the usage that their last job of a similar type required, at 
> least as a rule-of-thumb.
> We'd like to  support this for command line users. The easiest way of 
> doing this would be probably
> emailing users, on completion of their job, with the CPU and memory usage.

The way I handled the issue of re-using job requirements was to implement a
framework of job classes using a submission script.  The idea is to use a 
centralized file that contains job definitions.  These definitions contain
the necessary magic that the users would normally issue as part of the command.

Instead of issuing a long submission command, the user instead specifies the 
job definition to use along with the command to submit.  The definition from 
the central file is parsed and the command is constructed on the fly.

Ex.

#Central file

%%big_job%%
description	'Run on linux servers with lots of memory'
gridware
-l 		'-l mem_free=3G,os=Linux'
-N		'my_Big_honkin_job'
-P		'important_project'

This would be specified by running

bwrap --def big_job  some_command

And would be translated by the submission script into

qsub -l -l mem_free=3G,os=Linux -N my_Big_honkin_job -P important_project some_command

User are also free to implement their own job definitions by creating a definition file in their
home directory.  These definitions override those specified in the central config.  The users can
also specify individual option overrides at submission time.

The wrapper script is on the file exchange.
http://gridengine.sunsource.net/servlets/ProjectDownloadList?JServSessionIdservlets=x0ba2x33q1

Kirk

> 
> I wondered if anyone had already done this. If not we will look at 
> implementing this and can report
> back on how users find it. It should be possible to implement this 
> functionality via the various
> internal SGE scripts, but we'll probably take the simplest approach and 
> simply create a wrapper
> around qsub.
> 
> The interesting part won't really be the technical implementation, but 
> more whether it helps users
> make decsions about where their jobs might run. It isn't a perfect 
> solution, of course, as two jobs
> with almost identical parameters might have wildly different resource 
> requirements, but it might
> be a useful tool to give users some idea of their requirements.
> 
> A more sophisticated approach might be to compile some sort of database 
> of past usage with
> respect to user jobs, but given that users may use a variety of names 
> for scripts executing similar
> jobs it would probably be hard to do this in practice.
> 
> Sincerely,
> 
> Aaron Turner
> White Rose Grid (York)
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
> 

-- 
Kirk Patton
Unix Administrator
Transmeta Inc.
Tel. 408 919-3055

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list