[GE users] Lockfiles and not doing the process on the same file.

reuti reuti at staff.uni-marburg.de
Mon May 17 19:26:11 BST 2010


Hi,

Am 17.05.2010 um 16:13 schrieb mnuzaihan:

> Thanks for the reply, now i had realised on the race condition issue  
> when implementing a lock file.
>
> The script i modified creates a lockfile on the NFS, shared by the  
> cluster.
>
> I've did something like (if output file [resulting encoded file]  
> exists or lockfile exists), it skips the file and then do a loop to  
> do other files.
>
> So the setting that i did involves putting the directory path of  
> where it searches for the raw files to encode and then executes the  
> process.
>
> In fact, the original script was intended to be done on a single  
> local machine but i added the lockfile stuff in the "if ( ! -e  
> encoded_file || ! -e lockfile ) then encodes, else skip. But  
> executing like you had mentioned, causes a race condition of  
> machines in the cluster to check on the lock file which my idea  
> doesn't work well.

then I would suggest make two scripts out of the one you have:

The first part is a loop checking for new files (which is an endless  
loop I assume). When it finds a new file, it won't convert it, but  
submit a job which will do the actual conversion (this second script  
is a sub-part of the original one).

As movies are large files (which will put some heavy load on the NFS  
server), maybe you can improve performance when you first copy the  
file to a local node (into the $TMPDIR which is maintained by SGE),  
and then the result back.

-- Reuti


> I'm sure i had about some movie houses had used gridengine but on  
> how they did it, i'm not really sure. But if someone on this list  
> had done it and it would be nice to share their experiences on this  
> topic.
>
> But i know this might not be limited to just encoding files though.
>
> Thanks!,
> Muhammad Nuzaihan
>
> On 17-May-2010, at 5:36 PM, reuti wrote:
>
>> Hi,
>>
>> Am 15.05.2010 um 21:13 schrieb mnuzaihan:
>>
>>> I am having a problem. We do encoding of large and many videos  
>>> over the gridengine cluster. However, the problem is no matter how  
>>> much i tried to create a lockfile in the script so the other  
>>> machines would know there's a lockfile (if the encoding is started  
>>> on one machine) and try to encode the next file. It doesn't seem  
>>> to work.
>>
>> how do you create the lockfile - and where?
>>
>> But: there is nothing inside SGE which would prevent a race- 
>> condition, where two nodes would start with the same with the same  
>> movie. The lockfile-creation will never be atomic when you do it  
>> inside the script.
>>
>> Can't you just give the filename to the script, and each submitted  
>> job will handle exactly this movie? So there wouldn't be a need for  
>> a lockfile.
>>
>> -- Reuti
>>
>>
>>> Is there someone who had done this before? Any workarounds to this  
>>> problem?
>>>
>>> Thanks in advance!
>>>
>>> Muhammad Nuzaihan
>>>
>>> ------------------------------------------------------
>>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=257423
>>>
>>> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net 
>>> ].
>>
>> ------------------------------------------------------
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=257578
>>
>> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net 
>> ].
>
> Best Regards,
> Muhammad Nuzaihan Kamal
> Network Consultant
> Mobile: +65 97473874
>
> Asfa Systems Pte Ltd
> 91, Alps Avenue. #03-10. Singapore 498787
>
> Tel:  +65 62538211
> Fax: +65 62504814
> www.asfasystems.com.sg
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=257629

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list