[GE users] Parallel 'make'

David Olbersen dolbersen at nextwave.com
Wed Jun 4 17:36:20 BST 2008


Rayson,

Thanks for the pointers but I was only able to find an incomplete email
thread and no patches.
Am I blind or has there been some nasty bit rot?

-- 
David Olbersen
 

-----Original Message-----
From: Rayson Ho [mailto:rayrayson at gmail.com] 
Sent: Monday, June 02, 2008 3:27 PM
To: users at gridengine.sunsource.net
Subject: Re: [GE users] Parallel 'make'

If it works with gmake 3.81, then may be it's because the makefiles are
using features that are not available in 3.78. In that case, you may
consider compiling a newer qmake using the 3.8.1 source:

http://gridengine.sunsource.net/issues/show_bug.cgi?id=2544

Rayson



On 6/2/08, David Olbersen <dolbersen at nextwave.com> wrote:
> Reuti,
>
> Yes, it seems to work with the trivial case.
> Using the simple Makefile below the two calls to gcc happened 
> sequentially, on different hosts.
>
> Modifying that Makefile to make test.3, test.4, ..., test.6 worked as 
> well, with it running sequentially.
>
> Modifying that Makefile further, to make test.1, test.2, ..., test.6 
> their own targets showed the biggest improvement, as all 6 
> compilations happened at once
>
> So now that I can see that this works, I suppose I get to dig through 
> my users Makefile to figure out what it's trying to do? Or does 
> anybody have some other suggestions?
>
> --
> David Olbersen
>
>
> -----Original Message-----
> From: Reuti [mailto:reuti at staff.uni-marburg.de]
> Sent: Monday, June 02, 2008 3:01 PM
> To: users at gridengine.sunsource.net
> Subject: Re: [GE users] Parallel 'make'
>
> Am 02.06.2008 um 23:26 schrieb David Olbersen:
>
> > Sorry, I misunderstood:
> >
> >   [dolbersen at jotan ~]$ qrsh -q ncsim hostname
> >   sge-alfrodull-065
> >   [dolbersen at jotan ~]$
>
> No prob. Difference is, that in the first case (w/o command) a /usr/ 
> sbin/in.rlogind is used, while with a command /usr/sge/utilbin/lx24- 
> x86/rshd (on the nodes).
>
> Is your makefile conforming to the rules laid out in `man qmake`? Can 
> you try with a simple makefile whether it's working in principle?
>
> all:
>        gcc -o test.1 test.c
>        gcc -o test.2 test.c
>
> -- Reuti
>
> >
> > --
> > David Olbersen
> >
> >
> > -----Original Message-----
> > From: Reuti [mailto:reuti at staff.uni-marburg.de]
> > Sent: Monday, June 02, 2008 2:17 PM
> > To: users at gridengine.sunsource.net
> > Subject: Re: [GE users] Parallel 'make'
> >
> > Am 02.06.2008 um 22:53 schrieb David Olbersen:
> >
> >> Rayson,
> >>
> >> Yes, interactive jobs work well.
> >> qrsh works as well:
> >>
> >>   [dolbersen at jotan ~]$ qrsh -q rfic
> >>   Last login: Mon Jun  2 10:13:00 2008 from jotan.eng.atg.nw.net
> >>   [dolbersen at sge-alfrodull-069 ~]$
> >
> > qrsh with command is also working? - Reuti
> >
> >>
> >> --
> >> David Olbersen
> >>
> >>
> >> -----Original Message-----
> >> From: Rayson Ho [mailto:rayrayson at gmail.com]
> >> Sent: Monday, June 02, 2008 1:52 PM
> >> To: users at gridengine.sunsource.net
> >> Subject: Re: [GE users] Parallel 'make'
> >>
> >> David,
> >>
> >> Does interactive job work?? Can you run "qrsh hostname"??
> >>
> >> Rayson
> >>
> >>
> >>
> >> On 6/2/08, David Olbersen <dolbersen at nextwave.com> wrote:
> >>> Reuti,
> >>>
> >>> No firewalls, no other parallel integration at all.
> >>> Most of our jobs are numerous runs of single-threaded 
> >>> applications, with different input for each run.
> >>> Now one of my users has asked if they can run "make" on the 
> >>> cluster to
> >>
> >>> get some speed benefit.
> >>> This is to be the first parallel application.
> >>>
> >>> --
> >>> David Olbersen
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Reuti [mailto:reuti at staff.uni-marburg.de]
> >>> Sent: Monday, June 02, 2008 1:30 PM
> >>> To: users at gridengine.sunsource.net
> >>> Subject: Re: [GE users] Parallel 'make'
> >>>
> >>> Hi,
> >>>
> >>> Am 02.06.2008 um 19:34 schrieb David Olbersen:
> >>>
> >>>> I must be missing something, but I thought it would be pretty 
> >>>> straightforward to run `make` via `qmake` to get some parallel 
> >>>> advantage. It looks like this was a bad assumption. All the 
> >>>> documentation I could find mentions using LAM, MPI, or some other

> >>>> extra application: is that really required to get "make" running 
> >>>> on
>
> >>>> a few machines in my cluster?
> >>>>
> >>>> Here's what I've tried so far:
> >>>>
> >>>> % qmake -cwd -V -pe test1 1-8 -q rfic -l arch=lx24-x86 -- 
> >>>> PLATFORM=nw2000e
> >>>> qmake: *** Warning: File `builds/nw2000e-debug-opt-rvct-3.0b617-
> >>>> threadx42/depends.mak' has modification time in the future
> >>>> (2008-06-02 10:20:40 > 2008-06-02 10:20:30) 
> >>>> sge-alfrodull-069.eng.atg.nw.net
> >>>> error: executing task of job 435644 failed:
> >>>> qmake: *** remote_status: No child processes.  Stop.
> >>>> qmake: *** Waiting for unfinished jobs....
> >>>> qmake: *** remote_status: No child processes.  Stop.
> >>>>
> >>>> ... which exits very quickly and doesn't really do any builds.
> >>>>
> >>>> % qmake -cwd -V -q rfic -l arch=lx24-x86 -- PLATFORM=nw2000e
> >>>> qmake: *** Warning: File `builds/nw2000e-debug-opt-rvct-3.0b617-
> >>>> threadx42/depends.mak' has modification time in the future
> >>>> (2008-06-02 10:16:40 > 2008-06-02 10:16:30) dynamic mode Build
> >>>> info:
> >>>> nw2000e-debug-opt-rvct-3.0b617-threadx42 Mon Jun 2 10:16:30 PDT
> >>>> 2008
> >>>> dolbersen at sge-alfrodull-071 dynamic mode dynamic mode dynamic 
> >>>> mode dynamic mode dynamic mode dynamic mode dynamic mode dynamic 
> >>>> mode dynamic mode dynamic mode dynamic mode dynamic mode
> >>>> error: error waiting on socket for client to connect: Interrupted

> >>>> system call
> >>>
> >>> other parallel apps are running using a Tight Integration? Any
> >> firewall?
> >>> - Reuti
> >>>
> >>>>
> >>>> ... which does each job one at a time, as though I'd just run 
> >>>> "make"
> >>
> >>>> -- no advantage at all.
> >>>>
> >>>> Can anybody help? Pointing me to documentation should be more 
> >>>> than enough -- I've not been able to find what I'm looking for.
> >>>>
> >>>> David Olbersen
> >>>>
> >>>>
> >>>
> >>>
> >>> ------------------------------------------------------------------
> >>> --
> >>> -
> >>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> >>> For additional commands, e-mail: 
> >>> users-help at gridengine.sunsource.net
> >>>
> >>>
> >>>
> >>> ------------------------------------------------------------------
> >>> --
> >>> -
> >>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> >>> For additional commands, e-mail: 
> >>> users-help at gridengine.sunsource.net
> >>>
> >>>
> >>
> >> -------------------------------------------------------------------
> >> -- To unsubscribe, e-mail: 
> >> users-unsubscribe at gridengine.sunsource.net
> >> For additional commands, e-mail: 
> >> users-help at gridengine.sunsource.net
> >>
> >>
> >>
> >> -------------------------------------------------------------------
> >> -- To unsubscribe, e-mail: 
> >> users-unsubscribe at gridengine.sunsource.net
> >> For additional commands, e-mail: 
> >> users-help at gridengine.sunsource.net
> >>
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> > For additional commands, e-mail: users-help at gridengine.sunsource.net
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> > For additional commands, e-mail: users-help at gridengine.sunsource.net
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list