Problem with large projects
On Sun, Jun 7, 2020 at 12:55 PM nop head nop.head@gmail.com wrote:
I have been designing a 3D printer on and off for a few years now. When I
came back to it recently, haing updated my library and OpenSCAD in the
meantime, I find it very slow to update the preview when I make a small
change. This is mainly the "Compiling design (CSG Tree generation)...",
i.e. when the code is instantiated and echo modules come out, before it
actually does any geometry. The geometry takes no time when nothing has
changed because it is all cached.
Considering that this is all CPU bound, have you tried getting a perf based
CPU flamegraph of the rendering process? My current OpenSCAD setup is
living in a docker container and I have never attempted to perf such, but
it would be an interesting thing to figure out.
Regards, Jan
I remember it being about 12 seconds, which wasn't too bad but now it is a
minute.
It is by far the biggest project I have done in OpenSCAD, and is broken up
into 21 files, not including the library. In contrast most of other
projects are just a single file or two or three and will do that part of
the compilation in less than a second. For example something like this uses
the same library and will do F5 with no changes in 1 second.
[image: image.png]
It only takes 53 seconds to draw the whole thing when first loaded. My
large project only takes 4 minutes to render from scratch, so whereas the
small project has a 53:1 difference between the first F5 and a second one.
The large project only has a 4:1 difference.
It is hard to see where the time is taken, it seems to be fairly slow all
over just due to splitting it into multiple files. I think the main problem
is a small file of global constants that gets included everywhere. Putting
an echo function in that file shows the constants are instantiated about
1/4 million times! In the smaller project it is instantiated 4700 times.
Still huge but it seems to explode when you have more files.
This is because whenever you call a module or function in a different file
it instantiates all the constants. Is this really necessary or can it be
fixed? I think it would make a huge difference to a project like this.
OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
--
Jan Wieck
Principal Database Engineer
I have been designing a 3D printer on and off for a few years now. When I
came back to it recently, haing updated my library and OpenSCAD in the
meantime, I find it very slow to update the preview when I make a small
change. This is mainly the "Compiling design (CSG Tree generation)...",
i.e. when the code is instantiated and echo modules come out, before it
actually does any geometry. The geometry takes no time when nothing has
changed because it is all cached.
I remember it being about 12 seconds, which wasn't too bad but now it is a
minute.
It is by far the biggest project I have done in OpenSCAD, and is broken up
into 21 files, not including the library. In contrast most of other
projects are just a single file or two or three and will do that part of
the compilation in less than a second. For example something like this uses
the same library and will do F5 with no changes in 1 second.
[image: image.png]
It only takes 53 seconds to draw the whole thing when first loaded. My
large project only takes 4 minutes to render from scratch, so whereas the
small project has a 53:1 difference between the first F5 and a second one.
The large project only has a 4:1 difference.
It is hard to see where the time is taken, it seems to be fairly slow all
over just due to splitting it into multiple files. I think the main problem
is a small file of global constants that gets included everywhere. Putting
an echo function in that file shows the constants are instantiated about
1/4 million times! In the smaller project it is instantiated 4700 times.
Still huge but it seems to explode when you have more files.
This is because whenever you call a module or function in a different file
it instantiates all the constants. Is this really necessary or can it be
fixed? I think it would make a huge difference to a project like this.
Yeah, when I posted about this a few months ago people seemed to think it
didn't matter. The issue has been open for years and seems catastrophic to
me.
Our solution is to never use "use" and only use "include" which avoids the
problem. I think there are other cases where "use" is slow as well, though
not as serious.
--
Sent from: http://forum.openscad.org/
Yes I remember that post. I seem to get away with it on all my projects
except this one. It also has a global file with constants at the project
level and they are positions of things with several function calls each to
other modules, so they are more costly. Each of the function calls likely
needs all the constants instantiated again and it explodes.
I can get the time down to 37 seconds by including sub assemblies instead
of using them but I had to add a bodge to stop them instantiating
themselves as I like to load a sub assembly when working on it.
I am on Windows so to profile OpenSCAD I think I would need to build it
with Visual Studio, which I have done before, but I am pretty sure the
problem is evaluating constants over and over again. Every single screw,
nut, washer, etc are calls to the library and the library includes its
global constants in every module as again, they all work standalone.
It is counter intuitive that making constants in a file module is slower
than working everything out inside functions. You would expect the constant
to be evaluated once when the file is first used, not every time a function
or module is called in it.
It might be faster to make all constants into functions but then you would
have a big tree of function calls that explode as nothing can be worked out
once and stored for later use.
On Sun, 7 Jun 2020 at 18:33, adrianv avm4@cornell.edu wrote:
Yeah, when I posted about this a few months ago people seemed to think it
didn't matter. The issue has been open for years and seems catastrophic to
me.
Our solution is to never use "use" and only use "include" which avoids the
problem. I think there are other cases where "use" is slow as well, though
not as serious.
--
Sent from: http://forum.openscad.org/
OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Yes this has really become a problem for me. I have a list of
transformations that position screws in a shelf bracket that is reasonably
complex to calculate and since it generates the bracket and also positions
inserts and screws into it plus it needs to drill holes in 6 panels that it
mates with I need to reference it many times.
function shelf_screw_positions() = [
translate([z_axis_x + spine_width() / 2, box_depth / 2 - spine_depth()
-
spine_clearance + shelf_boss_r + eps]) * rotate([-90, 0,
90]),
translate([z_axis_x + spine_width() / 2, box_depth / 2 - shelf_boss_r -
eps]) * rotate([-90, 0,
90]),translate([-box_width / 4, box_depth / 2])
* rotate([-90, 0,
0]),translate([box_width / 2, box_depth / 2 -
box_intrusion(box_type) - shelf_boss_r - box_bezel_clearance(box_type) -
eps]) * rotate([-90, 0, -90]),
translate([box_width / 2, -box_depth / 2 +
box_intrusion(box_type) + shelf_boss_r + box_bezel_clearance(box_type) +
eps]) * rotate([-90, 0, -90]),translate([ 10, -box_depth / 2]) * rotate([-90, 0, 180]),
translate([-10, -box_depth / 2]) * rotate([-90, 0, 180]),translate([-box_width / 2, box_depth / 2 -
box_intrusion(box_type) - shelf_boss_r - box_bezel_clearance(box_type) -
eps]) * rotate([-90, 0, 90]),
translate([-box_width / 2, -box_depth / 2 +
box_intrusion(box_type) + shelf_boss_r + box_bezel_clearance(box_type) +
eps]) * rotate([-90, 0, 90]),translate([z_axis_x - spine_width() / 2, box_depth / 2 - spine_depth()
-
spine_clearance + shelf_boss_r + eps]) * rotate([-90, 0,
-90]),
translate([z_axis_x - spine_width() / 2, box_depth / 2 - shelf_boss_r -
eps]) * rotate([-90, 0,
-90]),
];
[image: image.png]
If I store it in a variable it gets calculated every time I call a function
or module in a file that includes its definition. If I wrap it in a
function it gets computed every time I use it. It is bonkers because it is
a constant, so only needs to be calculated once.
Adding this shelf to my project doubles the time it takes to do "Compiling
design (CSG Tree generation)...", from about 30 seconds to 1 minute simply
because this self touches six panels, which are all separate files in the
machine. Compared to the rest of the complexity of the machine it is
trivial but because it interacts with most of it it has a huge impact on
the speed and the time needed to see every change.
Is it true that the only way to have a constant calculated once is to make
my project into one massive file?
On Sun, 7 Jun 2020 at 21:03, nop head nop.head@gmail.com wrote:
Yes I remember that post. I seem to get away with it on all my projects
except this one. It also has a global file with constants at the project
level and they are positions of things with several function calls each to
other modules, so they are more costly. Each of the function calls likely
needs all the constants instantiated again and it explodes.
I can get the time down to 37 seconds by including sub assemblies instead
of using them but I had to add a bodge to stop them instantiating
themselves as I like to load a sub assembly when working on it.
I am on Windows so to profile OpenSCAD I think I would need to build it
with Visual Studio, which I have done before, but I am pretty sure the
problem is evaluating constants over and over again. Every single screw,
nut, washer, etc are calls to the library and the library includes its
global constants in every module as again, they all work standalone.
It is counter intuitive that making constants in a file module is slower
than working everything out inside functions. You would expect the constant
to be evaluated once when the file is first used, not every time a function
or module is called in it.
It might be faster to make all constants into functions but then you would
have a big tree of function calls that explode as nothing can be worked out
once and stored for later use.
On Sun, 7 Jun 2020 at 18:33, adrianv avm4@cornell.edu wrote:
Yeah, when I posted about this a few months ago people seemed to think it
didn't matter. The issue has been open for years and seems catastrophic
to
me.
Our solution is to never use "use" and only use "include" which avoids the
problem. I think there are other cases where "use" is slow as well,
though
not as serious.
--
Sent from: http://forum.openscad.org/
OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
I have my own "constructive" library (kind of a strong dialect of
openscad entirely written in openscad) i have been using for many years
now for complex projects.
in that library all transformations are just functiontions producing
matrices, and a sequence of transformations is easily accumulated into a
single multmatrix by the library, so essentialy all your code below will
break down into a single multmatrix operation in the CSG tree. all the
calculation itself is not passed CSG, it is just matrix multiplications,
so essentially every combination of input variabled becomes one matrix
value and i think this will be chached right?
will this help the case? i am not 100% sure i understand your case
completely.
i planned to publish it anyway, but since the reaction to my work in
this group so far was never very enthusiastic, it seems it will probably
not pay of the effort of cleaning up and documenting the code.
but in case it might help you with your problem i am happy to share it
with you, it has many other useful things like automatic
backwards(reverse) transformations, ability to mark parts of the code as
relevant to one design part or other, and then, render only certain
parts, or boolean them, or having "difference()" work over different
transformation origins (the components of the difference operation do
not have to have the same parent,like with normal difference() where the
subtracted componets are always ancestors of the difference()
element),selective hull(),...
On 15.06.20 01:11, nop head wrote:
Yes this has really become a problem for me. I have a list of
transformations that position screws in a shelf bracket that is
reasonably complex to calculate and since it generates the bracket and
also positions inserts and screws into it plus it needs to drill holes
in 6 panels that it mates with I need to reference it many times.
function shelf_screw_positions() = [
translate([z_axis_x + spine_width() / 2, box_depth / 2 -
spine_depth() + spine_clearance + shelf_boss_r + eps]) *
rotate([-90, 0, 90]),
translate([z_axis_x + spine_width() / 2, box_depth / 2 -
shelf_boss_r - eps]) *
rotate([-90, 0, 90]),
translate([-box_width / 4, box_depth / 2])
*
rotate([-90, 0, 0]),
translate([box_width / 2, box_depth / 2 -
box_intrusion(box_type) - shelf_boss_r - box_bezel_clearance(box_type)
- eps]) * rotate([-90, 0, -90]),
translate([box_width / 2, -box_depth / 2 +
box_intrusion(box_type) + shelf_boss_r + box_bezel_clearance(box_type)
- eps]) * rotate([-90, 0, -90]),
translate([ 10, -box_depth / 2]) * rotate([-90, 0, 180]),
translate([-10, -box_depth / 2]) * rotate([-90, 0, 180]),
translate([-box_width / 2, box_depth / 2 -
box_intrusion(box_type) - shelf_boss_r - box_bezel_clearance(box_type)
- eps]) * rotate([-90, 0, 90]),
translate([-box_width / 2, -box_depth / 2 +
box_intrusion(box_type) + shelf_boss_r + box_bezel_clearance(box_type)
- eps]) * rotate([-90, 0, 90]),
translate([z_axis_x - spine_width() / 2, box_depth / 2 -
spine_depth() + spine_clearance + shelf_boss_r + eps]) *
rotate([-90, 0, -90]),
translate([z_axis_x - spine_width() / 2, box_depth / 2 -
shelf_boss_r - eps]) *
rotate([-90, 0, -90]),
];
image.png
If I store it in a variable it gets calculated every time I call a
function or module in a file that includes its definition. If I wrap
it in a function it gets computed every time I use it. It is bonkers
because it is a constant, so only needs to be calculated once.
Adding this shelf to my project doubles the time it takes to do
"Compiling design (CSG Tree generation)...", from about 30 seconds to
1 minute simply because this self touches six panels, which are all
separate files in the machine. Compared to the rest of the complexity
of the machine it is trivial but because it interacts with most of it
it has a huge impact on the speed and the time needed to see every change.
Is it true that the only way to have a constant calculated once is to
make my project into one massive file?
On Sun, 7 Jun 2020 at 21:03, nop head <nop.head@gmail.com
mailto:nop.head@gmail.com> wrote:
Yes I remember that post. I seem to get away with it on all my
projects except this one. It also has a global file with constants
at the project level and they are positions of things with several
function calls each to other modules, so they are more costly.
Each of the function calls likely needs all the constants
instantiated again and it explodes.
I can get the time down to 37 seconds by including sub assemblies
instead of using them but I had to add a bodge to stop them
instantiating themselves as I like to load a sub assembly when
working on it.
I am on Windows so to profile OpenSCAD I think I would need to
build it with Visual Studio, which I have done before, but I am
pretty sure the problem is evaluating constants over and over
again. Every single screw, nut, washer, etc are calls to the
library and the library includes its global constants in every
module as again, they all work standalone.
It is counter intuitive that making constants in a file module is
slower than working everything out inside functions. You would
expect the constant to be evaluated once when the file is first
used, not every time a function or module is called in it.
It might be faster to make all constants into functions but then
you would have a big tree of function calls that explode as
nothing can be worked out once and stored for later use.
On Sun, 7 Jun 2020 at 18:33, adrianv <avm4@cornell.edu
<mailto:avm4@cornell.edu>> wrote:
Yeah, when I posted about this a few months ago people seemed
to think it
didn't matter. The issue has been open for years and seems
catastrophic to
me.
Our solution is to never use "use" and only use "include"
which avoids the
problem. I think there are other cases where "use" is slow as
well, though
not as serious.
--
Sent from: http://forum.openscad.org/
_______________________________________________
OpenSCAD mailing list
Discuss@lists.openscad.org <mailto:Discuss@lists.openscad.org>
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
in that library all transformations are just functiontions producing
matrices, and a sequence of transformations is easily accumulated into a
single multmatrix by the library, so essentialy all your code below will
break down into a single multmatrix operation in the CSG tree. all the
calculation itself is not passed CSG, it is just matrix multiplications, so
essentially every combination of input variabled becomes one matrix value
and i think this will be chached right?
As far as I know, they will not be cached. Only solids, models, objects are
cached. Variables are not.
Yes that is the problem. If you have some complex code that generates a
solid it is all run on every update, the parameters it generates are then
compared with the cached solids to see if the geometry needs updating. In
my project that takes no time at all but running all the code to generate
thousands of parts starts to take a long time. I think this is mainly
because every constant that isn't in your top level file gets evaluated
over and over again.
For isolation I have individual files that I can load on their own, for
example the z axis assembly, which is quite complex. At the start of the
file it has a lot of constants to work out positions, etc. It also includes
definitions of all the vitamins it uses, which are hundreds of lists. It
also needs to export a few functions and modules to the rest of the
machine, which just export constants, like the total height.
However calling any of these means all the hundreds of constants it uses
are all evaluated over and over again. So even on a PC it starts to take
significant time. It really shouldn't take that long to evaluate a script
but the constant evaluation explosion does as soon as you separate a
project into lots of files.
On Mon, 15 Jun 2020 at 01:16, Ronaldo Persiano rcmpersiano@gmail.com
wrote:
in that library all transformations are just functiontions producing
matrices, and a sequence of transformations is easily accumulated into a
single multmatrix by the library, so essentialy all your code below will
break down into a single multmatrix operation in the CSG tree. all the
calculation itself is not passed CSG, it is just matrix multiplications, so
essentially every combination of input variabled becomes one matrix value
and i think this will be chached right?
As far as I know, they will not be cached. Only solids, models, objects
are cached. Variables are not.
OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
i think one possible approach would be to do big Part of the caching
yourself, that is use lookup tables.
so if all transformation sequences are accumulated into single matrices,
i would create a vector of matrices of all transformations you will
need, and write it out as long string into lookup.scad file
this lookup.scad containing only single assignment of the lookup vector
of matrices: lookup=[[1,2,2,2,...}[5,34,887,87,..], yadda yadda];
then just include the lookup.scad and use the
multmatrix(lookup[indexofNeededVal])
in the main code
creating lookup tables is never the most elegant approach,but still,
flexible enough, and will probably save a lot of rendering time.
On 15.06.20 10:17, nop head wrote:
Yes that is the problem. If you have some complex code that generates
a solid it is all run on every update, the parameters it generates are
then compared with the cached solids to see if the geometry needs
updating. In my project that takes no time at all but running all the
code to generate thousands of parts starts to take a long time. I
think this is mainly because every constant that isn't in your top
level file gets evaluated over and over again.
For isolation I have individual files that I can load on their own,
for example the z axis assembly, which is quite complex. At the start
of the file it has a lot of constants to work out positions, etc. It
also includes definitions of all the vitamins it uses, which are
hundreds of lists. It also needs to export a few functions and
modules to the rest of the machine, which just export constants, like
the total height. However calling any of these means all the hundreds
of constants it uses are all evaluated over and over again. So even on
a PC it starts to take significant time. It really shouldn't take that
long to evaluate a script but the constant evaluation explosion does
as soon as you separate a project into lots of files.
On Mon, 15 Jun 2020 at 01:16, Ronaldo Persiano <rcmpersiano@gmail.com
mailto:rcmpersiano@gmail.com> wrote:
in that library all transformations are just functiontions
producing matrices, and a sequence of transformations is
easily accumulated into a single multmatrix by the library, so
essentialy all your code below will break down into a single
multmatrix operation in the CSG tree. all the calculation
itself is not passed CSG, it is just matrix multiplications,
so essentially every combination of input variabled becomes
one matrix value and i think this will be chached right?
As far as I know, they will not be cached. Only solids, models,
objects are cached. Variables are not.
_______________________________________________
OpenSCAD mailing list
Discuss@lists.openscad.org <mailto:Discuss@lists.openscad.org>
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
But it is then no longer parametric, the main reason for using OpenSCAD.
It isn't particularly slow to multiply matrices. It is only when it is done
over and over again. I sped it up considerably by caching functions calls
like this.
function shelf_screw_positions() = let(sw = spine_width(), sd =
spine_depth(), bc = box_intrusion(box_type) +
box_bezel_clearance(box_type)) [
translate([z_axis_x + sw / 2, box_depth / 2 - sd + spine_clearance +
shelf_boss_r + eps]) * rotate([-90, 0, 90]),
translate([z_axis_x + sw / 2, box_depth / 2 - shelf_boss_r - eps])
* rotate([-90, 0, 90]),
translate([-box_width / 4, box_depth / 2])
* rotate([-90, 0, 0]),
translate([box_width / 2, box_depth / 2 - bc -
shelf_boss_r - eps]) * rotate([-90, 0, -90]),
translate([box_width / 2, -box_depth / 2 + bc +
shelf_boss_r + eps]) * rotate([-90, 0, -90]),
translate([ 10, -box_depth / 2])
* rotate([-90, 0, 180]),
translate([-10, -box_depth / 2])
* rotate([-90, 0, 180]),
translate([-box_width / 2, box_depth / 2 - bc -
shelf_boss_r - eps]) * rotate([-90, 0, 90]),
translate([-box_width / 2, -box_depth / 2 + bc +
shelf_boss_r + eps]) * rotate([-90, 0, 90]),
translate([z_axis_x - sw / 2, box_depth / 2 - sd + spine_clearance +
shelf_boss_r + eps]) * rotate([-90, 0, -90]),
translate([z_axis_x - sw / 2, box_depth / 2 - shelf_boss_r - eps])
* rotate([-90, 0, -90]),
];
The three functions I calculate once in the let() and then reuse two or
three times are trivial functions but they are slow because they are in a
used file that has lots of constants defined in it. So each time they are
called lots of constants are evaluated again. Also this function should be
a constant, not a function, but then it gets evaluated whenever something
else is used from the file that defines it.
I got a big speed up making it a function until I added more code that
called it, so I can't win. There is no way to calculate a constant once and
store it except to do it in the main file, but then it can't be accessed
from any other file unless they are all included. Then they can't be used
standalone and that breaks my build system because it locates stls and
assemblies in the file that defines them and calls them from the command
line. So each file has to be self contained and include or use all its
dependencies.
On Mon, 15 Jun 2020 at 12:37, pproj@posteo.de pproj@posteo.de wrote:
i think one possible approach would be to do big Part of the caching
yourself, that is use lookup tables.
so if all transformation sequences are accumulated into single matrices,
i would create a vector of matrices of all transformations you will need,
and write it out as long string into lookup.scad file
this lookup.scad containing only single assignment of the lookup vector
of matrices: lookup=[[1,2,2,2,...}[5,34,887,87,..], yadda yadda];
then just include the lookup.scad and use the
multmatrix(lookup[indexofNeededVal])
in the main code
creating lookup tables is never the most elegant approach,but still,
flexible enough, and will probably save a lot of rendering time.
On 15.06.20 10:17, nop head wrote:
Yes that is the problem. If you have some complex code that generates a
solid it is all run on every update, the parameters it generates are then
compared with the cached solids to see if the geometry needs updating. In
my project that takes no time at all but running all the code to generate
thousands of parts starts to take a long time. I think this is mainly
because every constant that isn't in your top level file gets evaluated
over and over again.
For isolation I have individual files that I can load on their own, for
example the z axis assembly, which is quite complex. At the start of the
file it has a lot of constants to work out positions, etc. It also includes
definitions of all the vitamins it uses, which are hundreds of lists. It
also needs to export a few functions and modules to the rest of the
machine, which just export constants, like the total height.
However calling any of these means all the hundreds of constants it uses
are all evaluated over and over again. So even on a PC it starts to take
significant time. It really shouldn't take that long to evaluate a script
but the constant evaluation explosion does as soon as you separate a
project into lots of files.
On Mon, 15 Jun 2020 at 01:16, Ronaldo Persiano rcmpersiano@gmail.com
wrote:
in that library all transformations are just functiontions producing
matrices, and a sequence of transformations is easily accumulated into a
single multmatrix by the library, so essentialy all your code below will
break down into a single multmatrix operation in the CSG tree. all the
calculation itself is not passed CSG, it is just matrix multiplications, so
essentially every combination of input variabled becomes one matrix value
and i think this will be chached right?
As far as I know, they will not be cached. Only solids, models, objects
are cached. Variables are not.
OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
OpenSCAD mailing listDiscuss@lists.openscad.orghttp://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org