Yes, that is a huge problem. I don't intend to teach math to a calculus
teacher. I am just seeking a sensible way to get rid of the bug by using
both use<> and include<>. Isn't it possible in your case to put all those
small functions accessed everywhere in an used<> file and the global
constants in an included<> file?

Ronaldo wrote

Yes, if you get 100k evaluations then it starts to add up to detectable
amounts of time.

I wonder would using functions instead of constants enable you to work
around the use bug?  This would prevent recalculation.

--
Sent from: http://forum.openscad.org/

That would not work for calculated constants.

I have played around moving things between include and use and between
constants and functions. I think what l currently have is reasonably
optimal. The problem is what you gain on the swings you lose on the
roundabouts in both cases.

I do have all the global constants in includes but they have to be included
everywhere, so they are parsed and evaluated lots of times. I have moved
some expensive constants to functions to avoid them being re-evaluated but
then they get evaluated every time you use them, so it only gains if they
are not used very many times. And lots of constants depend on other
constants so wrapping them in functions starts to create a tree of function
calls and that soon explodes in time.

Similarly including files instead of using them avoids the constants being
evaluated lots of times due to calling into a used file but because they
are needed in more than one file you get them parsed and evaluated multiple
times. When they are used they are only parsed once and then cached.

The re-evaluation of constants actually placed a fundamental limit on
project complexity that doesn't exist in any other language I know.
Normally factoring sub expressions into constants makes the code faster and
shorter but in OpenSCAD it can make it much slower.

For example I made springs with sweep and since I wanted multiple springs
the same for battery contacts I made a function that returned the
polyhedron of the spring and stored it in a variable in the battery module
for reuse.  That caused a slowdown because it got evaluated every time a
function in the battery module was called instead of just once and reused
for each battery. It is much faster to do the sweep again for each battery.
Totally counter intuitive.

On Thu, 30 Jul 2020, 12:40 adrianv, avm4@cornell.edu wrote:

nophead, I know you have posted before about this performance issue for
large projects.  I believe you that it's a major problem for your use case,
but it's difficult for me to reproduce.
Could you open an issue with a clear list of steps to reproduce?
Normally a self contained single file test case is preferable, but due to
the nature of this problem it would be fine if the steps to reproduce are
something like:

1. Download some public repo
2. Preview some top level file which exemplifies the poor performance you
   encounter

It's possible there are multiple issues at play here, so it would be useful
to do some profiling on such a case and verify where time is being spent.

On Thu, Jul 30, 2020 at 7:29 AM nop head nop.head@gmail.com wrote:

When I noticed this issue last year I posted an example here
http://forum.openscad.org/variable-definition-in-used-file-cause-huge-run-time-hit-td26001.html

This problem is very simple.  It doesn't require fancy profiling or a big
setup.  It's trivial to reproduce and doesn't require any thing complicated.
The problem is simply that if you "use" a file and then call a function
whose definition is in the used file, then all variable definitions at the
top level in the used file get (re)executed.  And they get executed every
time you run any function in the file.  No big deal if it's a=3 and you
call the function a couple times, but if it's a complex calculation, or you
call functions hundreds or thousands of times it gets to be a big drain on
performance. My understanding is that in nophead's projects, file are used a
lot and he has lots of variable definitions, so it ends up being a big
problem.  He also gave the example of wanting to build a big polyhedron once
and save the result in a variable.  Can't do it because the polyhedron gets
recalculated every time any function in the file is run.

See https://github.com/openscad/openscad/issues/782. Linde, who opened the
issue in 2014 says: "Fixing the issue is easy, but I've not attached a patch
as there is a remote chance that the behavior is intended, and because there
is also a remote chance of this breaking / changing the behavior of some
existing code that could be relying on this."

Here's an even simpler example than my original one:

file.scad:

use<test.scad>
output = [for(i=[1:100]) test(i)];
echo(output);

test.scad:

a = [for(i=[1:1e5]) sin(i)];
b = echo("evaluating b") 3;
function test(x) = x;

If I run this it takes 9s to run and the "evaluating b" message displays 100
times.  If I either change "use" to include it runs in 0s and "evaluating b"
displays only once.  Or if I use "use" but comment out the time consuming
definition of a then it also runs in 0s.  In my original example I had the
run time go from 5s up to
9 minutes in code that was derived from a real case, where the definitions
in the lib file were only slightly expensive but functions in the file were
called thousands of times.

thehans wrote

--
Sent from: http://forum.openscad.org/

The only project large enough for the effect to be noticeable is my half
finished 3D printer, which I don't particularly want to publish in its
current state.

The effect can be seen with the tiny example project bundled with my
library. If you add function echo to one of the global constant
definitions, you can see it gets defined 951 times despite the project only
using a handful of files in the library that include the global
definitions. In the small example it isn't noticeable because as
adrianv said each definition only takes a few us. The problem is every
function call or module call into the library causes all the global
constants to be reevaluated. So even for this tiny project that adds up to
nearly 1000 times.

As project complexity grows the number of calls into the library grows,
especially if there are some loops, for example making a 2D grill with
poly_circles.  Because poly_circle() is in a different module to grill() it
means for every single hole, all the global constants are evaluated again.

Even for moderate size projects it still isn't a big overhead, because it
is very fast but on my large project that is split into lots of separate
files it completely explodes because not only does every library call
evaluate every constant, every call between projects files evaluates every
constant in those as well. That includes the thousands of object definition
lists in the library and also much more expensive constant expressions that
work out positions of things depending on other things in the project. It
was ending up taking a minute to evaluate all the expressions until I added
some hacks such as storing an expensive expression in a $variable, setting
it at the start of main.scad and then including all the other files that
see if it is set and use it or calculate it themselves if they have been
opened standalone. They should simply of been able to use the file that
makes the list once, stores it in a variable, and exports it with a
function.

1. Download https://github.com/nophead/NopSCADlib
2. Edit this line
   https://github.com/nophead/NopSCADlib/blob/master/global_defs.scad#L48 and
   change it to

inf = echo("inf") 1e10;      // very big

3. Preview
   https://github.com/nophead/NopSCADlib/blob/master/examples/MainsBreakOutBox/scad/bob_main.scad
   and
   count how many times you see ECHO: "inf"

The complication is that constant espressions can include $variables and
then they need reevaluating for each value of the $variable.

On Thu, 30 Jul 2020 at 18:18, Hans L thehans@gmail.com wrote:

Ronaldo wrote

Yes it would, as long as recalculating the constants as needed was (much)
faster than recalculating every time a function in the library runs.  If
we're talking about hundreds of thousands of re-evaluations due the the
"use" then it's entirely possible that full recalculation as needed would be
faster.

--
Sent from: http://forum.openscad.org/

+1

One of the really good things about OpenSCAD is its simplicity.

Beginners can get started with uncomplicated code generating straight forward objects.

But there is also much sophistication that can be used once experienced enough.

Adding strong checks then requires complexity, Nopheads ' if(is_list(type) && len(type) > 3 &&
type[3])' example shows this.

See how other languages draw a cube: https://rosettacode.org/wiki/Draw_a_cuboid

There are plenty of fully formed languages with strong typing and all the extra complexity
(https://en.wikipedia.org/wiki/Syntactic_sugar#Syntactic_salt)

We have been able to add more features like list comprehensions, assert, advanced echo, which are
useful for advanced programmers, but which the beginner can ignore.

The change to len() nophead mentions was the start of forcing additional complexity.

If you want a language which forces the programmer to write code that enforces ridged safe
programming, there are plenty of them to choose.

From: Discuss [mailto:discuss-bounces@lists.openscad.org] On Behalf Of nop head
Sent: Thu, 30 Jul 2020 04:57
To: OpenSCAD general discussion
Subject: Re: [OpenSCAD] What happened to booleans?

The issue is that OpenSCAD has been used for many years and has always returned undef for an out of
bounds index so I use it as a shortcut. Instead of if(len(type) > 3 && type[3]) is just
if(type[3]), which is a lot less ugly. It saves adding empty values to the end of my object
descriptors and makes the lack of an entry the same as an empty entry, which makes semantic sense
to me. And len should go back to silently returning undef when applied to a scalar. Otherwise it
becomes if(is_list(type) && len(type) > 3 && type[3]), which is even more ugly. In practice the
list element is wrapped in an accessor function, so the intent becomes obvious. E.g.

    conn = camera_connector_size(type);
    if(conn) {

             use conn to draw the connector

    }

I doubt there are many cases where people do if(list[n]) where it is a mistake, using list[n] to
compute something or compare it to anything other than undef is definitely an error if the list
isn't that long and my proposal would catch all other errors due to bad indexing.

If an undef value goes down an execution path that doesn't use it then it isn't a bug as far as I
am concerned. undef exists as a value, can be assigned to variables and passed as parameters and
silently tested as false with if, so I don't think that should change. But if it always stores how
it became undef then debugging is easy because if you ever use it in an expression you will get a
warning showing where it became undef and where it was finally used. Exactly what you need to
debug.

Whether operators get overloaded to work on strings in the future doesn't matter. At the moment
they don't so you get undef.

On Wed, 29 Jul 2020 at 19:09, Jordan Brown openscad@jordan.maileater.net wrote:

[ Please forgive my tendency to write a lot about this.  It's actually a primary interest in my
professional life - how to write programs to make it difficult to have bugs, how to detect the bugs
you can't prevent, and how to design languages, libraries, and other infrastructure so as to
prevent and detect bugs. A related topic is "how to report errors so that the error reports are
useful". ]

On 7/29/2020 12:10 AM, Rogier Wolff wrote:

Some "failures" are not failures,  [...]

So... when talking abstractly: "You should throw errors as close as possible to the line causing
the problem." I'm all for it.... But when you think about it, it is not as clear cut as you might
think.

Yes.  Distinguishing a likely bug from an "expected" error is important.

Nop head's "when you need a concrete answer" test is certainly a good start.  But I don't think
it's quite enough.  The next question is "what is a likely bug, and what is merely missing data?".

A missing parameter (in OpenSCAD, where missing parameters are expected) is clearly not an error.
(But maybe OpenSCAD should have a syntax that says "this parameter is required".)

An attempt to divide by zero is clearly an error.  In the unlikely event that you really want
something infinity-like as the result, require the programmer to explicitly check first.  (atan2()
comes to mind.)

I think it's clear that if a function requires a particular parameter, then not supplying it is an
error.  (But not everybody agrees.  In C, I often see people checking required parameters for NULL
and returning an error if they're NULL.  No!  The parameter is required; if the caller supplied a
NULL then the caller unquestionably has a bug!)

But an array index that's out of bounds?  Hard to say.  You might say it's a bug, or you might say
that returning undef allows the program to probe for the dimensions of the array.  I would come
down on the side of calling it an error.  If you want to probe for the dimensions of the array,
you can use len() and then the intent is obvious.  That's a key thing:  having the program convey
the programmer's intent.  If you have out-of-bounds references simply return undef, neither the
interpreter nor a reviewer can tell what the intent is, and so you can't detect the bug.  If, on
the other hand, you consider it an error, then you have two distinct idioms:  if your intent is to
retrieve a value (and you don't expect your index to be out of bounds) then you do a simple
reference; if your intent is to check for the presence of the value then you use len().  Yes, if
"undef" was a reasonable default value you have to go a bit out of your way and say i < len(x) ?
x[i] : undef, but that doesn't seem like an onerous price to pay for protecting all of the other
cases.

Mixed-type expressions are particularly tricky.  There are reasonable definitions for many
combinations.  I'd draw the line at the point where you have to "stretch" to say that the
definition is reasonable.  String concatenation through '+' is reasonable.  String replication
through multiplication seems reasonable.  Saying that boolean*string is equivalent to (boolean ?
string : "") seems past the limit.  Automatic to-string when you mix a string and something else
with "+"... I'm not sure.  It's reasonably obvious and awfully convenient, but is implicated in any
number of cases of "null" appearing in user interfaces.

Hans's "undef with a reason" does help a lot.  I'm not sure it's enough.  Because undef is
considered false for "if" purposes (a definition that seems unlikely to change) an undef that came
from a bug can lead to silently going down an execution path that doesn't happen to ever use the
value in a "concrete" way.

OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

--
This email has been checked for viruses by AVG.
https://www.avg.com

I've now implemented the capability for undef types to carry a reason
string.
Warnings are now emitted for basically any binary operator (and unary -)
that returns undef.
Holding a reason string does not necessarily trigger a warning unless the
higher level code explicitly checks for undef and prints the warning.

These changes (still in a Pull Request, not merged into master yet) can be
reviewed here https://github.com/openscad/openscad/pull/3383

Changing behavior of operator[] and len() are out of the scope of this pull
request and I've left them as they are.

On Thu, Jul 30, 2020 at 10:32 PM OzAtMichael oz.at.michael@gmail.com
wrote: