[GE users] Lockfiles and not doing the process on the same file.

mnuzaihan muhammad at asfasystems.com
Mon May 17 15:13:39 BST 2010


Thanks for the reply, now i had realised on the race condition issue when implementing a lock file.

The script i modified creates a lockfile on the NFS, shared by the cluster.

I've did something like (if output file [resulting encoded file] exists or lockfile exists), it skips the file and then do a loop to do other files.

So the setting that i did involves putting the directory path of where it searches for the raw files to encode and then executes the process.

In fact, the original script was intended to be done on a single local machine but i added the lockfile stuff in the "if ( ! -e encoded_file || ! -e lockfile ) then encodes, else skip. But executing like you had mentioned, causes a race condition of machines in the cluster to check on the lock file which my idea doesn't work well.

I'm sure i had about some movie houses had used gridengine but on how they did it, i'm not really sure. But if someone on this list had done it and it would be nice to share their experiences on this topic.

But i know this might not be limited to just encoding files though.

Muhammad Nuzaihan

On 17-May-2010, at 5:36 PM, reuti wrote:


Am 15.05.2010 um 21:13 schrieb mnuzaihan:

I am having a problem. We do encoding of large and many videos over the gridengine cluster. However, the problem is no matter how much i tried to create a lockfile in the script so the other machines would know there's a lockfile (if the encoding is started on one machine) and try to encode the next file. It doesn't seem to work.

how do you create the lockfile - and where?

But: there is nothing inside SGE which would prevent a race-condition, where two nodes would start with the same with the same movie. The lockfile-creation will never be atomic when you do it inside the script.

Can't you just give the filename to the script, and each submitted job will handle exactly this movie? So there wouldn't be a need for a lockfile.

-- Reuti

Is there someone who had done this before? Any workarounds to this problem?

Thanks in advance!

Muhammad Nuzaihan


To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].


To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

Best Regards,
Muhammad Nuzaihan Kamal
Network Consultant
Mobile: +65 97473874

Asfa Systems Pte Ltd
91, Alps Avenue. #03-10. Singapore 498787

Tel:  +65 62538211
Fax: +65 62504814

More information about the gridengine-users mailing list