Massey Render Farm
- fractal.design
- Posts: 51
- Joined: Wed Sep 08, 2010 4:38 pm
Massey Render Farm
Hi,
I am helping get the render farm at Massey in Wellington under way and we are having a few issues with the Network Manager.
So far we are running Indigo as "nice" slaves on the MacPro's all the time, co-ordinating the slaves with the masters using a network manager on one of the servers.
It seems a bit flakey...
Can the license manager be Windows based for the Mac slaves?
Would the licenses need to be converted?
Also, is there anymore documentation on the networking side of indigo than that I found here: http://www.indigorenderer.com/forum/vie ... 19&t=10188
and that in the technical manuals?
Specifically, how does the process work? Do the slaves get locked? (If we kill a master, how do the slaves know to be free?)
Maybe one of you guys can come up and give us a hand at some stage?
Cheers
I am helping get the render farm at Massey in Wellington under way and we are having a few issues with the Network Manager.
So far we are running Indigo as "nice" slaves on the MacPro's all the time, co-ordinating the slaves with the masters using a network manager on one of the servers.
It seems a bit flakey...
Can the license manager be Windows based for the Mac slaves?
Would the licenses need to be converted?
Also, is there anymore documentation on the networking side of indigo than that I found here: http://www.indigorenderer.com/forum/vie ... 19&t=10188
and that in the technical manuals?
Specifically, how does the process work? Do the slaves get locked? (If we kill a master, how do the slaves know to be free?)
Maybe one of you guys can come up and give us a hand at some stage?
Cheers
Re: Massey Render Farm
Hi fractal.design,
Cheers!
Yes, a license manager run on windows can give out licenses to mac slaves (and the other way around).fractal.design wrote:Can the license manager be Windows based for the Mac slaves?
No, the licenses are not OS specific.fractal.design wrote:Would the licenses need to be converted?
Afaik that is all documentation we have, sorryfractal.design wrote:Also, is there anymore documentation on the networking side of indigo than that I found here: http://www.indigorenderer.com/forum/vie ... 19&t=10188
and that in the technical manuals?
The slaves should never just lock. When you kill a master, all slaves connections to it will be dropped and they become free.fractal.design wrote:Specifically, how does the process work? Do the slaves get locked? (If we kill a master, how do the slaves know to be free?)
Sure, just let us know what time suits you the best and we will come aroundfractal.design wrote:Maybe one of you guys can come up and give us a hand at some stage?
Cheers
Cheers!
- fractal.design
- Posts: 51
- Joined: Wed Sep 08, 2010 4:38 pm
Re: Massey Render Farm
Hey,
Ok great, have linked the Massey admin to this post.
We are going to test 2.4.13 (when the Mac version comes out)
I think maybe the hostnames were not setup correctly, so we will try some more ideas,
and if we run into any trouble will let you guys know.
Cheers for the (speedy) help,
Oli.
Ok great, have linked the Massey admin to this post.
We are going to test 2.4.13 (when the Mac version comes out)
I think maybe the hostnames were not setup correctly, so we will try some more ideas,
and if we run into any trouble will let you guys know.
Cheers for the (speedy) help,
Oli.
Re: Massey Render Farm
OK, so using Indigo v.2.4.13 I have setup a LaunchDaemon that runs this command on startup:
/Applications/Indigo/Indigo.app/Contents/MacOS/IndigoConsole -n s -h test-server:7200
Trying to connect to a test Network Master machine, running the same release of Indigo..
These are the error messages from the network slave:
Indigo Renderer v2.4.13, OS X 64-bit build.
Licence not verified, running in free mode.
Attempting to connect to master at test-server:7200...
Connected to master.
Connection Exception: Connection Closed.
Connection to master terminated.
Attempting to connect to master at test-server:7200...
Connected to master.
Connection Exception: Connection Closed.
Connection to master terminated.
Attempting to connect to master at test-server:7200...
Connected to master.
Connection Exception: Connection Closed.
Connection to master terminated.
And the Server:
10/09/10 12:38:05 PM [0x0-0x1f01f].com.apple.Indigo[248] *** process 248 exceeded 500 log message per second limit - remaining messages this second discarded ***
10/09/10 12:38:06 PM [0x0-0x1f01f].com.apple.Indigo[248] Error while talking with 1xx.1xx.1xx.1xx:55964: Invalid hello, received 539
10/09/10 12:38:06 PM [0x0-0x1f01f].com.apple.Indigo[248] Error while talking with 1xx.1xx.1xx.1xx:55965: Invalid hello, received 539
However, if I set the Indigo Network Options on the slave machine, then launch the Indigo Network Client GUI, or run the following command as admin, or any user (other than root) works:
/Applications/Indigo/Indigo.app/Contents/MacOS/IndigoConsole -n s
Indigo Renderer v2.4.13, OS X 64-bit build.
Licence not verified, running in free mode.
Connected to Network Manager at test-server:7200 successfully.
Running that command as root user fires up the slave:
Indigo Renderer v2.4.13, OS X 64-bit build.
Licence not verified, running in free mode.
ClientDiscoveryPacketSender thread started.
Listening on port 7100
But obviously isn't reading the settings.xml file that the Indigo Client creates, and doesn't connect to any server.
So I'm interested to know if it's possible to get the slave running as root, to enable it to run while no user is logged in (which would be must useful in a computer lab scenario).
/Applications/Indigo/Indigo.app/Contents/MacOS/IndigoConsole -n s -h test-server:7200
Trying to connect to a test Network Master machine, running the same release of Indigo..
These are the error messages from the network slave:
Indigo Renderer v2.4.13, OS X 64-bit build.
Licence not verified, running in free mode.
Attempting to connect to master at test-server:7200...
Connected to master.
Connection Exception: Connection Closed.
Connection to master terminated.
Attempting to connect to master at test-server:7200...
Connected to master.
Connection Exception: Connection Closed.
Connection to master terminated.
Attempting to connect to master at test-server:7200...
Connected to master.
Connection Exception: Connection Closed.
Connection to master terminated.
And the Server:
10/09/10 12:38:05 PM [0x0-0x1f01f].com.apple.Indigo[248] *** process 248 exceeded 500 log message per second limit - remaining messages this second discarded ***
10/09/10 12:38:06 PM [0x0-0x1f01f].com.apple.Indigo[248] Error while talking with 1xx.1xx.1xx.1xx:55964: Invalid hello, received 539
10/09/10 12:38:06 PM [0x0-0x1f01f].com.apple.Indigo[248] Error while talking with 1xx.1xx.1xx.1xx:55965: Invalid hello, received 539
However, if I set the Indigo Network Options on the slave machine, then launch the Indigo Network Client GUI, or run the following command as admin, or any user (other than root) works:
/Applications/Indigo/Indigo.app/Contents/MacOS/IndigoConsole -n s
Indigo Renderer v2.4.13, OS X 64-bit build.
Licence not verified, running in free mode.
Connected to Network Manager at test-server:7200 successfully.
Running that command as root user fires up the slave:
Indigo Renderer v2.4.13, OS X 64-bit build.
Licence not verified, running in free mode.
ClientDiscoveryPacketSender thread started.
Listening on port 7100
But obviously isn't reading the settings.xml file that the Indigo Client creates, and doesn't connect to any server.
So I'm interested to know if it's possible to get the slave running as root, to enable it to run while no user is logged in (which would be must useful in a computer lab scenario).
- fractal.design
- Posts: 51
- Joined: Wed Sep 08, 2010 4:38 pm
Re: Massey Render Farm
Hey,
Under OSX u can add commands to the "rc.local" file on the machine so the process will load on boot as root.
Under OSX u can add commands to the "rc.local" file on the machine so the process will load on boot as root.
Re: Massey Render Farm
Hey Oli,
Launchd essentially replaces the rc.local method when using the option 'RunAtLoad'.
http://developer.apple.com/macosx/launchd.html
Here's what the command looks like as a Launchd plist:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Disabled</key>
<false/>
<key>KeepAlive</key>
<true/>
<key>Label</key>
<string>IndigoNetworkSlave</string>
<key>ProgramArguments</key>
<array>
<string>/Applications/Indigo/Indigo.app/Contents/MacOS/IndigoConsole</string>
<string>-n</string>
<string>s</string>
<string>-h</string>
<string>test-server:7200</string>
</array>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>
Launchd essentially replaces the rc.local method when using the option 'RunAtLoad'.
http://developer.apple.com/macosx/launchd.html
Here's what the command looks like as a Launchd plist:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Disabled</key>
<false/>
<key>KeepAlive</key>
<true/>
<key>Label</key>
<string>IndigoNetworkSlave</string>
<key>ProgramArguments</key>
<array>
<string>/Applications/Indigo/Indigo.app/Contents/MacOS/IndigoConsole</string>
<string>-n</string>
<string>s</string>
<string>-h</string>
<string>test-server:7200</string>
</array>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>
Re: Massey Render Farm
So very close to having network rendering working seamlessly...
Setting up the settings.xml file at: /Applications/Indigo/Indigo.app/Contents/Resources/ with our Network Manager hostname has made the simple LaunchDaemon (in the last post) work a treat for running slaves!
Now we've run into an issue when rendering images above 1000x700.. no slaves are used (anything below that res, & slaves are used)
The Indigo app on the server is licensed, but do we need a separate license for Network Rendering to work above the 'free mode' resolution?
In the Network Manager, we have the following error:
Floating License Status:
Cannot validate licence: Could not find '</licence_data>'
Any help regarding our setup would be muchly appreciated
Thanks,
Hamish
Setting up the settings.xml file at: /Applications/Indigo/Indigo.app/Contents/Resources/ with our Network Manager hostname has made the simple LaunchDaemon (in the last post) work a treat for running slaves!
Now we've run into an issue when rendering images above 1000x700.. no slaves are used (anything below that res, & slaves are used)
The Indigo app on the server is licensed, but do we need a separate license for Network Rendering to work above the 'free mode' resolution?
In the Network Manager, we have the following error:
Floating License Status:
Cannot validate licence: Could not find '</licence_data>'
Any help regarding our setup would be muchly appreciated
Thanks,
Hamish
Re: Massey Render Farm
Hi Hamish,
The slave computers need to have a valid Indigo licence to help with renderings above 0.7 megapixels.
Does the normal Indigo GUI read the licence when run on the slave computers? If so, and if the slave process runs as a different user, it may be an issue with the slave processes reading from a different user directory when reading the licence file.
The slave computers need to have a valid Indigo licence to help with renderings above 0.7 megapixels.
Does the normal Indigo GUI read the licence when run on the slave computers? If so, and if the slave process runs as a different user, it may be an issue with the slave processes reading from a different user directory when reading the licence file.
Re: Massey Render Farm
Thanks for the quick response!
The Indigo GUI reads the license fine, but the slave process is running as root.
Ahh the joy of running as root user...
Is there any way to let the root user 'see' the license file?
I know the standard procedure would be to run the slave as a 'normal' user...
The Indigo GUI reads the license fine, but the slave process is running as root.
Ahh the joy of running as root user...
Is there any way to let the root user 'see' the license file?
I know the standard procedure would be to run the slave as a 'normal' user...
Re: Massey Render Farm
OK, resurrecting an old thread I know, but just picking up where we left off and wondering if the sort of functionality we've created could be worked into Indigo as standard? (much like the Windows version).
The new setup is a LaunchAgent which runs at Login Window:
This then runs a one liner shell script:
Which runs IndigoConsole as Admin (Settings.xml file has been deployed to all Admin accounts so IndigoConsole runs with a network floating license, connects to our network manager & does a master search broadcast).
This seems to run OK, however we seem to have some machines connecting multiple times to the master, via different ports... Maybe we should specify a port?
The next part of the equation is running the IndigoConsole when users are logged in, after a period of inactivity. For this we are using AppStartSaver http://www.themcdonalds.net/richard/ind ... StartSaver
This is setup with the following ByHost preferences enforced for each user:
com.ve3nvm.AppStartSaver.plist
com.apple.screensaver.plist
It would be really neat to finally get this running smoothly, and available for all our students to cut down their render times
The new setup is a LaunchAgent which runs at Login Window:
Code: Select all
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>nz.ac.massey.IndigoLoginWindow</string>
<key>LimitLoadToSessionType</key>
<string>LoginWindow</string>
<key>KeepAlive</key>
<true/>
<key>ProgramArguments</key>
<array>
<string>/Library/Indigo/IndigoLoginWindow.sh</string>
</array>
</dict>
</plist>
Code: Select all
sudo -u admin /Applications/Indigo\ Renderer/Indigo.app/Contents/MacOS/IndigoConsole -n s >>/tmp/Indigo.log
This seems to run OK, however we seem to have some machines connecting multiple times to the master, via different ports... Maybe we should specify a port?
The next part of the equation is running the IndigoConsole when users are logged in, after a period of inactivity. For this we are using AppStartSaver http://www.themcdonalds.net/richard/ind ... StartSaver
This is setup with the following ByHost preferences enforced for each user:
com.ve3nvm.AppStartSaver.plist
Code: Select all
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>bringFront</key>
<true/>
<key>commandOptions</key>
<string>-n s</string>
<key>displayStatus</key>
<true/>
<key>displayTime</key>
<false/>
<key>docsList</key>
<array/>
<key>hideOnLaunch</key>
<false/>
<key>pathName</key>
<string>/Applications/Indigo Renderer/Indigo.app/Contents/MacOS/IndigoConsole</string>
<key>quitOnDisplayDim</key>
<false/>
<key>quitType</key>
<integer>0</integer>
<key>restartProgram</key>
<true/>
<key>sendAppQuit</key>
<true/>
<key>updateSeconds</key>
<integer>5</integer>
<key>windowAlpha</key>
<real>1</real>
</dict>
</plist>
Code: Select all
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CleanExit</key>
<string>YES</string>
<key>PrefsVersion</key>
<integer>100</integer>
<key>idleTime</key>
<integer>300</integer>
<key>moduleDict</key>
<dict>
<key>moduleName</key>
<string>AppStartSaver</string>
<key>path</key>
<string>/Library/Screen Savers/AppStartSaver.saver</string>
<key>type</key>
<integer>0</integer>
</dict>
<key>showClock</key>
<true/>
</dict>
</plist>
Who is online
Users browsing this forum: No registered users and 15 guests