[GE users] restriction on stdout

Mag Gam magawake at gmail.com
Sun Nov 2 02:35:02 GMT 2008


    [ The following text is in the "UTF-8" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some characters may be displayed incorrectly. ]

we are running on a 10G network and the nodes are very underutilized.
I managed to get it to work and get consistent results by splitting
the file earlier in a seperate process.

BTW, I actually have a awk statement instead o cat. The awk is used to
eliminate the header.




On Sat, Nov 1, 2008 at 7:06 PM, Reuti <reuti at staff.uni-marburg.de> wrote:
> Am 01.11.2008 um 19:49 schrieb Mag Gam:
>
>> I have noticed some unusual behavior when I pipe data to stdout and it
>> gets chopped off in an array job. For example:
>>
>> $ wc -l file.txt
>> 199971 file.txt
>>
>> $ cat file.txt | split --lines =50137 -a 4 -d - splitted1
>>
>> When I run this normally I get the correct portions.
>>
>> However, when I run it using qsub, in a script one of the files is
>> completly different. Its missing close to 40% of the contents.
>
> I would assume it's an NFS problem. Is it run on an NFS client? If possible,
> such task should be directed to the file server (if it's possible to use it
> for such special types of jobs - less network traffic, faster execution).
>
> On the other hand: why are you using cat with a pipe? You can save a process
> with:
>
> $ split --lines 50137 -a 4 -d file.txt splitted1
>
> -- Reuti
>
>
>> Also, the " - " means stdout in the split command. Has anyone seen this
>> before?
>>
>> TIA
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list