[GE users] semaphore leftovers

Reuti reuti at staff.uni-marburg.de
Thu Feb 10 01:40:39 GMT 2005


    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Hi David,

sorry for mentioning the "One solution would be to compile MPICH without shared 
memory support" between the lines. I saw semaphores up now only when the MPICH 
was compiled *with* shared memory support (--with-comm=shared). If you compile 
it without, there shoudn't be any (unless of course the programs use it on 
their own). Since the default libs of MPICH are static, the applications would 
have to be linked again.

And if it's compiled into MPICH to use shared memory, it will allocate 
semaphores, although you won't use it, if you have a machinefile like:

node00
node00
node01
node01

you have two shared memory areas per node and make 3 times rsh. I think it will 
not use it.

When the machinefile is like:

node00:2
node01:2

with only one time rsh and one shared memory area per node. One important thing 
to mention: whether batch or interactive, on the headnode of the MPICH job the 
cleanipcs is called by default in mpirun.ch_p4, unless you set 
MPICH_CLEAN_IPCS="no". This may already now hit other jobs of the same user on 
the head node of the job. But it's not builtin to run on the slave nodes.

When your users stay in their session after ctrl-C, the "rsh cleanipcs" to the 
slave nodes in any stop_proc_args will not work immediately. But in this case, 
you could supply a script looping over the nodes on the machines file. When the 
PE is configured to guarantee that only one job of a user is on each node (e.g. 
allocation rule = 2 like), it's safe to call it from the command line after 
they pressed ctrl-C.

Also when your users would use a qdel, for now it's not in SGE to remove any 
semaphores on the nodes.


But anyway: are the applications linked dynamically? I could look into such a 
wrapper. It would be most useful in batch of course, and not in interactive 
mode. But I can only supply Linux versions for x86 and Opteron.

Cheers - Reuti


Quoting David Farrell <d-farrell2 at northwestern.edu>:

> 
> On Feb 9, 2005, at 4:33 PM, Reuti wrote:
> 
> > Hi,
> >
> > are you using MPICH as you mentioned cleanipcs? One solution would be 
> > to
> > compile MPICH without shared memory support. cleanipcs will simply 
> > remove all
> > ipcs stuff from one user on a node. If there are two jobs from the 
> > same user,
> > the second might be killed also by removing the "wrong" semaphores by 
> > accident.
> > But instead of a cron job, the cleanipcs could be put in the 
> > stop_proc_args or
> > queue_epilog.
> Yes this is MPICH, I will give this bit a try. The issue here is that 
> the users tend to use ctrl-C sorts to kill a job when running in 
> interactive mode, rather than using a more elegant technique. It seems 
> that in a way, the users are bypassing SGE by doing this and this may 
> be part of the problem. I think that having an epilog script to clean 
> up after they have ended the job may work well, but I wonder if there 
> are other ways (just so I know). It appears they are using the cluster 
> to debug their code on, and in so doing they end up using it like a 
> series of workstations through interactive shells. Perhaps there is a 
> better way to go about them using the nodes in this manner?
> 
> >
> > Another solution for dynamically linked application:
> >
> > A wrapper lib which will trap semget(), shmget(), msgget() and so on 
> > which is
> > loaded before the job by using LD_PRELOAD for this wrapper lib. This 
> > wrapper
> > will call the real semget() and remember the assigned ids on the 
> > return to the
> > application. When you shutdown the application by qdel, you would know 
> > all the
> > ids of the semaphores you have to remove. It's just an idea, but it 
> > would be a
> > cool addition to SGE.
> That does sound interesting, but I am not sure my skills are up to the 
> task
> 
> Thanks again,
> 
> Dave
> 
> >
> > Cheers - Reuti
> >
> >
> > Quoting David Farrell <d-farrell2 at northwestern.edu>:
> >
> >> I am running into the issue where semaphores build up and create a
> >> situation in which uses can no longer start jobs, and on occasion, 
> >> jobs
> >> die prematurely. The errors point to the semaphore issue and cleaning
> >> it manually has become a bother, as users wish to use the machine for
> >> testing(so abnormal exits are common). Is there any good solution to
> >> this problem? I have heard that making a cleanup script into a cron 
> >> job
> >> sometimes results in jobs being killed, so I would be interested in
> >> other possibilities. In addition, I sometimes see a situation in which
> >> running the cleanipcs script as root does not clean out some of the
> >> semaphores. Is there any solution?
> >>
> >> Thanks in advance,
> >>
> >> Dave
> >>
> >>
> >>
> >> David E. Farrell
> >> Graduate Student
> >> Mechanical Engineering
> >> Northwestern University
> >> email: d-farrell2 at northwestern.edu
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> > For additional commands, e-mail: users-help at gridengine.sunsource.net
> >
> >
> David E. Farrell
> Graduate Student
> Mechanical Engineering
> Northwestern University
> email: d-farrell2 at northwestern.edu
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list