Request: chain .igs files

[Wedge]

ZomB: I had forgot the console build...Now that you reminded me (thanks!), I should try to see if it crashes now. I was talking about the gui version back around 0.7 I think. New tests I should make, see if I can find a new memory limit.

Big Fan: I know the exporter I worked on allows you to export single size meshes around 900,000 faces/verts. (My system has 2gb ram.) An actual total scene size limit I do not remember, probably limited to ram size, but I suppose I should check. I guess now I need to make some new tests with the console build of Indigo to see if it will crash when I add a few of these massive 900k meshes to make a scene sized greater than a million.

It sounds like a good feature, just need to check which object is in which layer, like the animation function checks for which layout is in which frame, and then export with numbers at the end of the file linked to layers the meshes were in.

[Big Fan]

yeah xp will only run out about 1.65 gb before it refuses despite the apparent 2gb for an application.
I tried the 3gb switch user limiting to 2900 and it processes a lot more faces but it still seems to need to have 500 mb ** real ram available to actually save it right at the end or it balks.
perhaps if I had a full 2-3gb physical ram it might work...but for my 1.5 gb 1m faces seems the practical limit for a single export regardless of the switch.
I guess I need to test some large 'include' files to see how much indigo will handle Id like to hope it will run 3m

Ideally I would like the separate materials file to include the lights, camera. the various options/parameters, and of course the 'include' paths.
This would make it quite easy to adjust things in one small file rather than mess with multiple 100mb+ mesh files

**EDIT actually on reflection I think it is probably more than 2x the blend size that is required to be free to actually save successfully

[Wedge]

Please let me know what you find out for scene limits if you finish it before I do. I will also try but I have yet to start. I am having trouble setting aside the time I guess.

[Big Fan]

ok well some more experiments today
I may as well post here for posterity
I cranked up the export resolution from SW of the same scene in pursuit of 6m faces
actually too far to begin with cos SW ran out of memory at about 3gb commit file while I went grocery shopping and despite using the switch
I backed off in a rerun and I ended up with 400 mb of wrl files for the 16000 parts
Using a new strategy I managed to import these much easier this time
- open the win task manager to keep an eye on resources and processor activity
- use the blender console to monitor the script progress
- disable global undo
- close the outliner window
- set the 3d view to wire frame
- group the wrl files into parcels - for this torture test I used 4000 each but it would be prudent to aim for a number giving 1m faces each layer to save work later not the 3m I ended up with..
- import first parcel then change the 3d view to another empty layer in preparation and save and reopen the blend
- repeat for other parcels - necessary cos the commit climbs to well over 2gb and chokes if you try to keep adding...(actually I got in the first 2 4000 mesh parcels consecutively in this case but as you add more it definitely requires reloads or at least if you see the commit getting very big)
- be very patient to see the CPU activity has finished completely before doing something else
- downside discovered SW seems to export 90 deg rotated - different 'up' I suppose - this needs a rotate -not actually a biggy in wire frame , but clear apply rotation and see a very busy computer

so finally I have 12m faces in with a 1.6gb mem use indicated in blender
saved this is 550mb blend
so you can get in in there if you try and don't rush
now the mission is to export in <1m face parcels using the layers again
will test this next to see if it copes with a big blend and also exporting from it if needs be I'll delete meshes until I find the sweet spot

[Big Fan]

some more..
I managed to get a 1m face export out of a 5m face blend having only 1 visible layer out of the 5 to limit the commit. this is about the limit for 32 bit with 1.5gb ram
the blend was 245mb and the resulting .igs was 115mb
so all in all it looks like you are best just to import 1m faces into a series of blends for exporting again.
...I guess 64 bit might be best for large models...but it looks like 3-4mb might be needed
let you know how include goes soon

[Wedge]

I have a question for Ono:

If I failed to export a scene from Blender due to massive size, if I exported chunks of it in multiple files and used include feature, would Indigo also run out of memory putting these together or how does it load? What might happen?

I am guessing the only way to see for sure is to code it and find out but I have not been in the mood for programming lately.

Just to rephrase, if a scene is too big in one file, will breaking it apart and using include allow me to load it or will it just break again?

[Big Fan]

I can answer you that

I just tried including 5 x 1m face .igs and it runs very nicely in less than 1.1 gb commit
the igs are about 100mb each
each time it reads another it peaks at about the same commit so its very friendly/graceful
at a guess it will run 2m face includes before it peaks somewhere at the 1.65gb allowed by windows
it doesnt seem to matter how many there are they don't accumulate..
during the render itself the commit dropped to 650mb or so

the only issue is that is won't quit until it gets right to the warm up phase
so ono needs to include a poll to see if there is a close request before starting the next .igs include

the way to deal with this issue I think is to have an option to write a <scenedata> tag instead of <scene> in the exporters and that doesn't save the camera,system options etc with it either - then do an export of a layer with just the camera and the sun say, that picks up the system options as usual but optionally will write a user set number of 'empty' <include> tag sets for easy editing.
this way you don't need to touch the large mesh sets at all
and the small controlling igs is very easy to edit as required

this is the method I used in my trial ( manually -I haven't altered my exporter yet)

attach is a pic of the 5m faces rendering in indigo
there are some parts missing cos its from the full 12m face model I cut back until I could use it as one blend
I haven't tried using all 12 as includes but I think it will be fine on the basis of what I observed using 5 provided individual igs dont go over 2m faces (guess) [assuming you could even export that many out of blender]

[Big Fan]

just to help people in the future here is the controlling igs I made referencing my 5 includes

the include igs were edited to remove the first bits you see here from each - and also the <scene> tags top and bottom were changed to <scenedata> (dont forget the / in the bottom one)
--------------------------------------------------
<?xml version="1.0" encoding="utf-8"?>

<scene>

<renderer_settings>
<width>480</width>
<height>270</height>
<metropolis>true</metropolis>
<max_num_consec_rejections>100</max_num_consec_rejections>
<bidirectional>false</bidirectional>
<max_depth>1000</max_depth>
<logging>false</logging>
<save_untonemapped_exr>false</save_untonemapped_exr>
<save_tonemapped_exr>false</save_tonemapped_exr>
<save_igi>false</save_igi>
<display_period>120.0</display_period>
<halt_time>600</halt_time>
<frame_upload_period>60</frame_upload_period>
<image_save_period>360</image_save_period>
<super_sample_factor>3</super_sample_factor>
<hybrid>false</hybrid>
<bih_tri_threshold>500000</bih_tri_threshold>
<cache_trees>0</cache_trees>
<ray_origin_nudge_distance>0.0010000000475</ray_origin_nudge_distance>
<halt_samples_per_pixel>-1</halt_samples_per_pixel>
</renderer_settings>

<tonemapping>
<reinhard>
<pre_scale>2</pre_scale>
<post_scale>3</post_scale>
<burn>6</burn>
</reinhard>
</tonemapping>

<skylight>
<sundir>0.309478 -0.111062 0.944398</sundir>
<turbidity>2.200000</turbidity>
<extra_atmospheric>false</extra_atmospheric>
</skylight>

<camera>
<pos>7.481132 -6.507640 5.343665</pos>
<up>-0.317370 0.312469 0.895343</up>
<forwards>-0.654862 0.610666 -0.445245</forwards>
<aperture_radius>0.000895</aperture_radius>
<focus_distance>11.227551</focus_distance>
<aspect_ratio>1.77778</aspect_ratio>
<sensor_width>0.036000</sensor_width>
<lens_sensor_dist>0.039375</lens_sensor_dist>
<white_balance>E</white_balance>
<autofocus/>
</camera>

<include>
<pathname>layer1.igs</pathname>
</include>

<include>
<pathname>layer2.igs</pathname>
</include>

<include>
<pathname>layer3.igs</pathname>
</include>

<include>
<pathname>layer4.igs</pathname>
</include>

<include>
<pathname>layer5.igs</pathname>
</include>

<material>
<name>Default</name>
<diffuse>
<albedo_spectrum>
<rgb>
<rgb>1 1 1</rgb>
<gamma>2.2</gamma>
</rgb>
</albedo_spectrum>
</diffuse>
</material>

<material>
<name>null</name>
<null_material/>
</material>

</scene>

[Wedge]

Hey Big Fan, I tried to code something to export meshes out in certain layers but it appears harder than I originally thought.

A few problems come up, but it has been a few days, I erased my code, and do not remember the problems exactly.

Here are problems I had:

* Objects can be in more than one layer - but which layer will it export to? If object in layer 1,2,3 export to layer 1 or 2 or 3?

* If object is in layer 1 export it to main render settings file or a layer 1 file?

* Support all 20 layers? - this is where I started to copy / paste code to make support for all 20 layers in Blender, this is a lot of code to copy and paste but I can't think how to open / write / close 20 different files randomly any better than copy / paste the same code from layer 1 and change to 2....3.....4....

Basically:
check object layers, if object in layer 1 open scene layer 1 file and write mesh, add include layer 1 to main file, if object in any more layers ignore

I will try to write something again soon.

EDIT: Thanks for your testing Big Fan!

[Big Fan]

yes I know it isnt easy I have been trying too
perhaps its easier to have a button matrix on another UI page to set up things like ( bit messy hope it makes sense)


X####X#### save these layers as: include1.igs
#########X use this include X

##X####X## save these layers as: include2.igs
###X###### use this include X

#########X save these layers as: include3.igs
X######### use this include #

make includes set # | make a controller X


possibly you only need say 5 or maybe 10 includes max for practical purposes?
this way some layers could be grouped together for convenience or you just might have 1 per include
it would be clever of course to only have one set of layer buttons referenced by a further set of 10 include buttons to switch through to set up the appropriate data and name for each

I think having all these includes set up and saved together will cause memory use problems if there are a large number of faces ie. I dont think you will be able to export all the includes in one hit..
hmm still with 64 bit or perhaps some guidance notes for 32 bit limitations in the manual/wiki it could be ok
also it occurs to me that making layers visible/active is what sucks up the memory among other things and ruins exporting of large no of faces so it might be advantageous to actually export from hidden layers in the worst case... if its possible?

I think you might still want to have the option of separating the materials from the meshes available though perhaps that makes things too complex...
and it would be convenient to use the above matrix and names to make the controller set of include tags or remake it as required
beware though you want to disable autorun for making the includes
the controller could be saved and autorun an existing include set or parts of it though

dont let me dictate though I am sure there are plenty of ways to do this
Indigo readers here please speak up with your own ideas..

EDIT: add pic -something like this perhaps

[deltaepsylon]

it wont quit until after the warmup phase

doesnt that normally happen? usually i kill indigo via process explorer when i realize i messed something up, cuz the close wont actually close it until its done with warmup, right?

[Big Fan]

usually it doesnt matter too much but if you chain 10 together you wait quite a while until the warm up...
of course you can kill it ruthlessly any time but if ono actually puts in a polite quit it would be good to cover this situation as well IMHO

[Wedge]

I have a working version as of tonight. It appears to export meshes automatically to files with layer extensions. I will post an updated exporter sometime Saturday my time. (-5 gmt)

Finally made the code efficient, (I hope anyways) so it appears early tests show no noticeable speed loss in writing multiple files.

So all you do is push the button and it will export your scene out in layers that you designed it in. A little more testing tomorrow after some sleep and it should be ready to go.

[deltaepsylon]

yay! nice, wedge. I have a 3 billion poly scene, cant wait to test your exporter

[Wedge]

Hi Big Fan, just wanted to say thanks for all of your help with this request. Your testing was very helpful and I'm sure you know, testing is half of the battle when adding new features. Fixed the last bug with layers (I hope) that came up in certain situations. It was a little bit tricky to code this feature, since objects can be in multiple layers and you can render multiple layers. The bug caused cube located in layer 3 and 5, to export to layer 3 when exporting of only layer 5 was selected. It is fixed now though.

Thanks again for your help. Soon I hope to do a big scene test with this feature to see how it works. (all of my tests were less than 250,000 faces/verts)