qmake 8.1.0 puts "\n" in multiline commands, causing the commands to fail.

[sbridgett]

When there are multi-line commands (with the "\" continuation character at end of each line) in a "Makefile", qmake 8.0 and latest 8.1.0 replace the "\newline" with "\n", causing the command to fail. (where "newline" means an actual newline character, and "n" means the character "n").

Whereas the earlier Grid engine GE 6.1u6 replaced the "\newline" with a space so the command succeeded.

For example:

The "Makefile" contains:

all:
[Tab]echo Hello \
        how \
        are \
        you?; \
        ls \
        -al  

where "[Tab]" is a Horizontal Tab character, and the lines below it start with spaces rather than tabs.

The "runqmake.sh" contains:

#!/bin/bash

# Print the version of Grid engine:
qsub -help | grep GE

qmake -cwd -v PATH -- -j 2

Then submitting with qsub using:

  qsub -cwd -v PATH -sync yes runqmake.sh

In the resulting "runqmake.sh.o281786" the earlier grid engine GE 6.1 outputs:

GE 6.1u6
echo Hello \
        how \
        are \
        you?; \
        ls \
        -al
dynamic mode
Hello how are you?
total 64
drwxrwxr-x  2 sbridget biology  4096 Jun 14 17:24 .
drwxrwxr-x 33 root     biology 16384 Jun 14 17:05 ..
-rw-rw-r--  1 sbridget biology    89 Jun 14 17:23 Makefile
-rwxrwxr-x  1 sbridget biology    98 Jun 14 17:19 runqmake.sh

Whereas the latest SGE 8.1.0 outputs:

SGE 8.1.0
echo Hello \
        how \
        are \
        you?; \
        ls \
        -al
dynamic mode
Hello \n how \n are \n you?

NOTE: The "\n" embedded within the "Hello \n how \ are \n you?"

And the "ls -al" command fails, outputting a file "runqmake.sh.e9188" containing:

/bin/sh: \n: command not found
qmake: *** [all] Error 127

Which I think is because the ls command is being output as "ls \n -al"
rather than the "ls -al", so the error message "\n: command not found"
is given.

I've looked into the qmake source code, including the read.c and misc.c
files, but am not just sure why this is occurring.

Perhaps there is something I have set incorrectly in the sge configuration?

We are running SGE 8.1.0 on Centos 6 (whereas the GE 6.1u is running in Centos 5)

Thank you for help with this.

If I just use "make" on the Centos 6 cluster (where SGE 8.1.0 is installed), I get the
correct output:

make

echo Hello \
        how \
        are \
        you?; \
        ls \
        -al
Hello  how are you?
total 259K
drwxrwxr-x  2 sbridget biology  545 Jun 14 17:23 .
drwxrwxr-x 25 sbridget biology 2414 Jun 14 11:37 ..
-rw-rw-r--  1 sbridget biology   91 Jun 14 17:23 Makefile
-rw-rw-r--  1 sbridget biology   98 Jun 14 17:20 runqmake.sh

Several of the perl scripts that we use to run processes have long multiline commands with many command-line options, so these programs fail on SGE 8.1.0.

[sbridgett]

Makefile with multiline command.

[sbridgett]

runqmake.sh script submitted by qsub

[sbridgett]

Alternatively if I run "qmake" directly (ie. instead of the "qsub ... runqmake.sh" used above), with the parameters: "-cwd -v PATH --" then the same error occurs as shown below:

$ qmake -cwd -v PATH --
echo Hello \
        how \
        are \
        you?; \
        ls \
        -al
dynamic mode
Hello \n how \n are \n you?
/bin/sh: \n: command not found
qmake: *** [all] Error 127

If I only run "qmake", then the correct output is given (but this just runs make locally, and doesn't submit job to grid engine):

$ qmake
echo Hello \
        how \
        are \
        you?; \
        ls \
        -al
Hello  how are you?
total 259K
drwxrwxr-x  2 sbridget biology  545 Jun 14 17:23 .
drwxrwxr-x 25 sbridget biology 2414 Jun 14 11:37 ..
-rw-rw-r--  1 sbridget biology   91 Jun 14 17:23 Makefile
-rw-rw-r--  1 sbridget biology   98 Jun 14 17:20 runqmake.sh

I'm using the Linux AMD 64, ("lx-amd64") SGE 8.10 version, with source code compiled locally, rather than the rpm.

[dlove]

SGE <sge-bugs@…> writes:

When there are multi-line commands (with the "\" continuation character at
end of each line) in a "Makefile", qmake 8.0 and latest 8.1.0 replace the
"\newline" with "\n", causing the command to fail. (where "newline" means
an actual newline character, and "n" means the character "n").

Whereas the earlier Grid engine GE 6.1u6 replaced the "\newline" with a
space so the command succeeded.

Hmm... GNU Make's previous behaviour was POSIXly wrong -- the backslash
and newline should be passed to the shell. The fix is obviously
infelicitous for qrsh, though. Output from qmake -verbose looks right,
so something seems wrong with qrsh argument handling when /bin/sh is
invoked. (Simple make commands that don't need to invoke the shell are
OK, if you can use them.)

Pending investigation, you might be able to kludge it with a
QRSH_WRAPPER (see qsub(1)) to filter the newlines or modify the argument
for the "qrsh" command in qmake/remote-sge.c/start_remote_job if you're
up for hacking it.

[Dave Love <d.love@…>]

In [4265/sge]:

Fix #1420: convert \n in QRSH_COMMAND in qrsh_starter