I am attempting to figure out how to do operations with "large" sets (>10,000 elements), which presents a couple problems in OpenSCAD that I believe there is a solution to, but I can't quite figure it out. I posted earlier about issues with for() being limited to an iteration range of 10,000, looking back this seems like a reasonable limitation since it points out that you're probably doing something inefficient that should otherwise be done with list comprehensions or something that is not stack-intensive. Example code: function generate_random_polar(n=5001, r=1)=[for (i=[0:1:n-1]) [r, rands(0, 360,1)[0]]]; function polar_to_xy(r_theta=[1,0]) = [r_theta[0]*cos(r_theta[1]),r_theta[0 ]*sin(r_theta[1])]; list1 = concat(generate_random_polar(),generate_random_polar()); list2 = [for (x = list1) polar_to_xy(x)]; Problem 1: Algorithmic generation of a list with greater than 10,000 elements The only way I've figured out how to do this is simple concatenation of two more or more smaller lists that are generated by a function call with <10,000 iterations in the for() loop. For lists of random points, no big deal. If I was trying to generate a set of ordered points with certain spaces in a certain canonical order, this could get more complicated to have to "batch" the function calls. Is there a better way? Problem 2: Performing actions that require access to the inside of a list element (e.g., a coordinate pair) Say I have a list of coordinates (x,y), and that list is greater than 10,000 elements. How would I loop over the entire list and perform an action such as plotting a cube at that point?, e.g.: cube_size = 0.01; for(i = [0:1:len(list2)-1]) {translate([list2[i][0],list2[i][1],0]) cube (cube_size);} List comprehensions in OpenSCAD are, from what I understand, limited to taking a list as input, performing an operation on the individual elements (and even doing the operation may be conditional depending on the contents of the element), and returning a list. "Actions" can't be performed such as translate(); and cube();, but I can perform math on the elements and return a list of modified elements (e.g., convert from polar to cartesian coordinates). List comprehensions seem to be implemented in such a way that they are not limited to 10,000 element lists (I don't see any errors when I input lists bigger than that in this example code), so it must have been written in the back-end to not create a huge stack I assume. I don't mind switching to this syntax for operations on lists - it is no doubt more efficient - but on the end of a) creating, and b) doing something with lists that large after the math is done, there doesn't seem to be a way to do that, since both would require "action" syntax that I can only get to work in a normal for() loop. Is there a way around this with a better use of OpenSCAD's language features that I just don't understand well enough yet, or am I stuck with batching the for() loop calls to sets with less than 10,000 elements? Thanks, David

The for loop limit is not really anything to do with stack use, that would be a recursion limit. I think it is just an arbitrary limit to catch infinite loops or massive trees of objects that will cause CGAL to explode in time and or memory use. A list on the other hand is a much smaller data structure than a CGAL object. If you have a practical use for such massive trees of objects then perhaps the limit can be increased. How many do you need and are you going to need to F6 the result? F5 might be OK but F6 with 10000 objects doesn't seem practical. On 8 March 2017 at 19:47, David Coneff <david.coneff@gmail.com> wrote: > I am attempting to figure out how to do operations with "large" sets > (>10,000 elements), which presents a couple problems in OpenSCAD that I > believe there is a solution to, but I can't quite figure it out. I posted > earlier about issues with for() being limited to an iteration range of > 10,000, looking back this seems like a reasonable limitation since it > points out that you're probably doing something inefficient that should > otherwise be done with list comprehensions or something that is not > stack-intensive. > > Example code: > > function generate_random_polar(n=5001, r=1)=[for (i=[0:1:n-1]) [r, rands(0 > ,360,1)[0]]]; > function polar_to_xy(r_theta=[1,0]) = [r_theta[0]*cos(r_theta[1]),r_theta[ > 0]*sin(r_theta[1])]; > list1 = concat(generate_random_polar(),generate_random_polar()); > list2 = [for (x = list1) polar_to_xy(x)]; > > Problem 1: Algorithmic generation of a list with greater than 10,000 > elements > > The only way I've figured out how to do this is simple concatenation of > two more or more smaller lists that are generated by a function call with > <10,000 iterations in the for() loop. For lists of random points, no big > deal. If I was trying to generate a set of ordered points with certain > spaces in a certain canonical order, this could get more complicated to > have to "batch" the function calls. Is there a better way? > > Problem 2: Performing actions that require access to the inside of a list > element (e.g., a coordinate pair) > > Say I have a list of coordinates (x,y), and that list is greater than > 10,000 elements. How would I loop over the entire list and perform an > action such as plotting a cube at that point?, e.g.: > > cube_size = 0.01; > for(i = [0:1:len(list2)-1]) {translate([list2[i][0],list2[i][1],0]) cube > (cube_size);} > > > List comprehensions in OpenSCAD are, from what I understand, limited to > taking a list as input, performing an operation on the individual elements > (and even doing the operation may be conditional depending on the contents > of the element), and returning a list. "Actions" can't be performed such as > translate(); and cube();, but I can perform math on the elements and return > a list of modified elements (e.g., convert from polar to cartesian > coordinates). > > List comprehensions seem to be implemented in such a way that they are not > limited to 10,000 element lists (I don't see any errors when I input lists > bigger than that in this example code), so it must have been written in the > back-end to not create a huge stack I assume. I don't mind switching to > this syntax for operations on lists - it is no doubt more efficient - but > on the end of a) creating, and b) doing something with lists that large > after the math is done, there doesn't seem to be a way to do that, since > both would require "action" syntax that I can only get to work in a normal > for() loop. Is there a way around this with a better use of OpenSCAD's > language features that I just don't understand well enough yet, or am I > stuck with batching the for() loop calls to sets with less than 10,000 > elements? > > Thanks, > David > > > > > > > > > _______________________________________________ > OpenSCAD mailing list > Discuss@lists.openscad.org > http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org > >

Why not make the limit configurable, with a default of 10,000? Jon On 3/8/2017 4:50 PM, nop head wrote: > The for loop limit is not really anything to do with stack use, that > would be a recursion limit. I think it is just an arbitrary limit to > catch infinite loops or massive trees of objects that will cause CGAL > to explode in time and or memory use. A list on the other hand is a > much smaller data structure than a CGAL object. > > If you have a practical use for such massive trees of objects then > perhaps the limit can be increased. How many do you need and are you > going to need to F6 the result? F5 might be OK but F6 with 10000 > objects doesn't seem practical.

Only any point if there is a practical use for 10000+ objects. On 8 March 2017 at 22:00, jon <jon@jonbondy.com> wrote: > Why not make the limit configurable, with a default of 10,000? > > Jon > > On 3/8/2017 4:50 PM, nop head wrote: > >> The for loop limit is not really anything to do with stack use, that >> would be a recursion limit. I think it is just an arbitrary limit to catch >> infinite loops or massive trees of objects that will cause CGAL to explode >> in time and or memory use. A list on the other hand is a much smaller data >> structure than a CGAL object. >> >> If you have a practical use for such massive trees of objects then >> perhaps the limit can be increased. How many do you need and are you going >> to need to F6 the result? F5 might be OK but F6 with 10000 objects doesn't >> seem practical. >> > > > _______________________________________________ > OpenSCAD mailing list > Discuss@lists.openscad.org > http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >

For cases where you have a lot of objects overlapping each other (say you generate 1000s of cubes lying on the same unit circle), CGAL will sieze up trying to render maybe the first 1000 objects. In the case of, say, 10000 cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much of a problem although it does take a while to render. The main issue where I see things start to lag is when you try to render objects that overlap in a lot of areas and it has to compute the end result inner surface or combined outer surface of many objects. So if you are careful about how much intersection happens (e.g., by creating a wire-frame grid where there are regular intersections only at the ends of the wires), CGAL can potentially render a lot more than 10000 objects in one call. On Wed, Mar 8, 2017 at 2:50 PM, nop head <nop.head@gmail.com> wrote: > The for loop limit is not really anything to do with stack use, that would > be a recursion limit. I think it is just an arbitrary limit to catch > infinite loops or massive trees of objects that will cause CGAL to explode > in time and or memory use. A list on the other hand is a much smaller data > structure than a CGAL object. > > If you have a practical use for such massive trees of objects then perhaps > the limit can be increased. How many do you need and are you going to need > to F6 the result? F5 might be OK but F6 with 10000 objects doesn't seem > practical. > > > > On 8 March 2017 at 19:47, David Coneff <david.coneff@gmail.com> wrote: > >> I am attempting to figure out how to do operations with "large" sets >> (>10,000 elements), which presents a couple problems in OpenSCAD that I >> believe there is a solution to, but I can't quite figure it out. I posted >> earlier about issues with for() being limited to an iteration range of >> 10,000, looking back this seems like a reasonable limitation since it >> points out that you're probably doing something inefficient that should >> otherwise be done with list comprehensions or something that is not >> stack-intensive. >> >> Example code: >> >> function generate_random_polar(n=5001, r=1)=[for (i=[0:1:n-1]) [r, rands( >> 0,360,1)[0]]]; >> function polar_to_xy(r_theta=[1,0]) = [r_theta[0]*cos(r_theta[1 >> ]),r_theta[0]*sin(r_theta[1])]; >> list1 = concat(generate_random_polar(),generate_random_polar()); >> list2 = [for (x = list1) polar_to_xy(x)]; >> >> Problem 1: Algorithmic generation of a list with greater than 10,000 >> elements >> >> The only way I've figured out how to do this is simple concatenation of >> two more or more smaller lists that are generated by a function call with >> <10,000 iterations in the for() loop. For lists of random points, no big >> deal. If I was trying to generate a set of ordered points with certain >> spaces in a certain canonical order, this could get more complicated to >> have to "batch" the function calls. Is there a better way? >> >> Problem 2: Performing actions that require access to the inside of a list >> element (e.g., a coordinate pair) >> >> Say I have a list of coordinates (x,y), and that list is greater than >> 10,000 elements. How would I loop over the entire list and perform an >> action such as plotting a cube at that point?, e.g.: >> >> cube_size = 0.01; >> for(i = [0:1:len(list2)-1]) {translate([list2[i][0],list2[i][1],0]) cube >> (cube_size);} >> >> >> List comprehensions in OpenSCAD are, from what I understand, limited to >> taking a list as input, performing an operation on the individual elements >> (and even doing the operation may be conditional depending on the contents >> of the element), and returning a list. "Actions" can't be performed such as >> translate(); and cube();, but I can perform math on the elements and return >> a list of modified elements (e.g., convert from polar to cartesian >> coordinates). >> >> List comprehensions seem to be implemented in such a way that they are >> not limited to 10,000 element lists (I don't see any errors when I input >> lists bigger than that in this example code), so it must have been written >> in the back-end to not create a huge stack I assume. I don't mind switching >> to this syntax for operations on lists - it is no doubt more efficient - >> but on the end of a) creating, and b) doing something with lists that large >> after the math is done, there doesn't seem to be a way to do that, since >> both would require "action" syntax that I can only get to work in a normal >> for() loop. Is there a way around this with a better use of OpenSCAD's >> language features that I just don't understand well enough yet, or am I >> stuck with batching the for() loop calls to sets with less than 10,000 >> elements? >> >> Thanks, >> David >> >> >> >> >> >> >> >> >> _______________________________________________ >> OpenSCAD mailing list >> Discuss@lists.openscad.org >> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >> >> > > _______________________________________________ > OpenSCAD mailing list > Discuss@lists.openscad.org > http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org > >

The trouble is OpenSCAD doesn't know which scenerio it is. 10000 might be far too many overlapping objects, or not enough when they don't overlap. It seems like too blunt a tool, so perhaps the limit should simply go and the user gets what they deserve. On 8 March 2017 at 22:05, David Coneff <david.coneff@gmail.com> wrote: > For cases where you have a lot of objects overlapping each other (say you > generate 1000s of cubes lying on the same unit circle), CGAL will sieze up > trying to render maybe the first 1000 objects. In the case of, say, 10000 > cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much > of a problem although it does take a while to render. The main issue where > I see things start to lag is when you try to render objects that overlap in > a lot of areas and it has to compute the end result inner surface or > combined outer surface of many objects. So if you are careful about how > much intersection happens (e.g., by creating a wire-frame grid where there > are regular intersections only at the ends of the wires), CGAL can > potentially render a lot more than 10000 objects in one call. > > On Wed, Mar 8, 2017 at 2:50 PM, nop head <nop.head@gmail.com> wrote: > >> The for loop limit is not really anything to do with stack use, that >> would be a recursion limit. I think it is just an arbitrary limit to catch >> infinite loops or massive trees of objects that will cause CGAL to explode >> in time and or memory use. A list on the other hand is a much smaller data >> structure than a CGAL object. >> >> If you have a practical use for such massive trees of objects then >> perhaps the limit can be increased. How many do you need and are you going >> to need to F6 the result? F5 might be OK but F6 with 10000 objects doesn't >> seem practical. >> >> >> >> On 8 March 2017 at 19:47, David Coneff <david.coneff@gmail.com> wrote: >> >>> I am attempting to figure out how to do operations with "large" sets >>> (>10,000 elements), which presents a couple problems in OpenSCAD that I >>> believe there is a solution to, but I can't quite figure it out. I posted >>> earlier about issues with for() being limited to an iteration range of >>> 10,000, looking back this seems like a reasonable limitation since it >>> points out that you're probably doing something inefficient that should >>> otherwise be done with list comprehensions or something that is not >>> stack-intensive. >>> >>> Example code: >>> >>> function generate_random_polar(n=5001, r=1)=[for (i=[0:1:n-1]) [r, rands >>> (0,360,1)[0]]]; >>> function polar_to_xy(r_theta=[1,0]) = [r_theta[0]*cos(r_theta[1 >>> ]),r_theta[0]*sin(r_theta[1])]; >>> list1 = concat(generate_random_polar(),generate_random_polar()); >>> list2 = [for (x = list1) polar_to_xy(x)]; >>> >>> Problem 1: Algorithmic generation of a list with greater than 10,000 >>> elements >>> >>> The only way I've figured out how to do this is simple concatenation of >>> two more or more smaller lists that are generated by a function call with >>> <10,000 iterations in the for() loop. For lists of random points, no big >>> deal. If I was trying to generate a set of ordered points with certain >>> spaces in a certain canonical order, this could get more complicated to >>> have to "batch" the function calls. Is there a better way? >>> >>> Problem 2: Performing actions that require access to the inside of a >>> list element (e.g., a coordinate pair) >>> >>> Say I have a list of coordinates (x,y), and that list is greater than >>> 10,000 elements. How would I loop over the entire list and perform an >>> action such as plotting a cube at that point?, e.g.: >>> >>> cube_size = 0.01; >>> for(i = [0:1:len(list2)-1]) {translate([list2[i][0],list2[i][1],0]) cube >>> (cube_size);} >>> >>> >>> List comprehensions in OpenSCAD are, from what I understand, limited to >>> taking a list as input, performing an operation on the individual elements >>> (and even doing the operation may be conditional depending on the contents >>> of the element), and returning a list. "Actions" can't be performed such as >>> translate(); and cube();, but I can perform math on the elements and return >>> a list of modified elements (e.g., convert from polar to cartesian >>> coordinates). >>> >>> List comprehensions seem to be implemented in such a way that they are >>> not limited to 10,000 element lists (I don't see any errors when I input >>> lists bigger than that in this example code), so it must have been written >>> in the back-end to not create a huge stack I assume. I don't mind switching >>> to this syntax for operations on lists - it is no doubt more efficient - >>> but on the end of a) creating, and b) doing something with lists that large >>> after the math is done, there doesn't seem to be a way to do that, since >>> both would require "action" syntax that I can only get to work in a normal >>> for() loop. Is there a way around this with a better use of OpenSCAD's >>> language features that I just don't understand well enough yet, or am I >>> stuck with batching the for() loop calls to sets with less than 10,000 >>> elements? >>> >>> Thanks, >>> David >>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> OpenSCAD mailing list >>> Discuss@lists.openscad.org >>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >>> >>> >> >> _______________________________________________ >> OpenSCAD mailing list >> Discuss@lists.openscad.org >> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >> >> > > _______________________________________________ > OpenSCAD mailing list > Discuss@lists.openscad.org > http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org > >

David, The main aspect to understand is that there is a for limit just for ranges. If you don't use range in a for, there is no limit. p = rands(0,1,1000000); // p is a list of 1000000 elements for(pi=p){ // there is no range here if(pi<1e-6) echo(pi=pi); } Or if all your ranges have less than 10000 elements, you will not get an error message. // a very long for with ranges for(i=[0:999], j=[0:999], k=1000*i+j){ if(k<10 || k> 999990) echo(k=k); } The same happens with for in list comprehension: // short lists filtered from a long lists q = [for(pi=p) if(pi<1e-6) pi]; s = [for(i=[0:999], j=[0:999], k=1000*i+j) if(k<len(p)-1 && p[k]<1e-6 ) p[k] ]; echo(q=q); echo(s=s);

davidconeff wrote > In the case of, say, 10000 > cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much > of a problem although it does take a while to render. Are you sure of that? As far as I know neither OpenSCAD nor CGAL use bounding boxes. -- View this message in context: http://forum.openscad.org/for-loop-iteration-limitation-list-comprehensions-tp20806p20813.html Sent from the OpenSCAD mailing list archive at Nabble.com.

for(x = [0:99], y = [0:99]) translate([x * 10, y * 10, 0]) sphere(1); Takes 1 hour 18 minutes on my system and uses 2.6GB of memory. So perhaps 10K is a bit low, but not much before it would bring my system to its knees as Windows doesn't handle running out of memory very well. On 8 March 2017 at 23:02, Ronaldo <rcmpersiano@gmail.com> wrote: > davidconeff wrote > > In the case of, say, 10000 > > cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as > much > > of a problem although it does take a while to render. > > Are you sure of that? As far as I know neither OpenSCAD nor CGAL use > bounding boxes. > > > > -- > View this message in context: http://forum.openscad.org/for- > loop-iteration-limitation-list-comprehensions-tp20806p20813.html > Sent from the OpenSCAD mailing list archive at Nabble.com. > > _______________________________________________ > OpenSCAD mailing list > Discuss@lists.openscad.org > http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >

Ronaldo, p = rands(0,1,1000000); // p is a list of 1000000 elements for(pi=p){ // there is no range here if(pi<1e-6) echo(pi=pi); } Thanks for that example, it's exactly what I was looking for. I hadn't seen that syntax case used in the for() argument before, it looks like that just knows to go over the entire list in canonical order without having to specify a range and iteration interval. The more common (0:1:len(list)-1) is what I see in most other people's code, but this is a really cumbersome way to do it, and introduces that range limitation on a for() loop unnecessarily. As for the unit sphere example, I have some code that can produce that type of list. From what I have observed in test-case code I've written, it has nothing to do with bounding box calculations - the CSG rendering does a quick and dirty render that doesn't complain about intersecting surfaces because it is also not trying to be precise. CGAL seems to have awareness of when primitives intersect and tries to produce one solid geometry that is a simplification of all the intersecting volumes, so if you create something like the thread of a screw using progressively rotated and translated rectangular cubes (someone else's library that I played with a little), then the result is a very slow render because it has to figure out how many of the vertices of the intersecting volumes are inside the end-result shape in order to discard them. I found that creating a thread by first starting with a 2D primitive and going through a linear_extrusion(rotation) operation was much more efficient both in CSG and CGAL since it is generating the end-result vertices in the first place rather than having to do loads of extra 3D math. I would assume that if you also just limit the number of objects that intersect in a given region to a small number, the math is probably much simpler for CGAL as well. I won't know until I have a fully fleshed out test-framework, I'm fairly close to being there. On Wed, Mar 8, 2017 at 4:02 PM, Ronaldo <rcmpersiano@gmail.com> wrote: > davidconeff wrote > > In the case of, say, 10000 > > cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as > much > > of a problem although it does take a while to render. > > Are you sure of that? As far as I know neither OpenSCAD nor CGAL use > bounding boxes. > > > > -- > View this message in context: http://forum.openscad.org/for- > loop-iteration-limitation-list-comprehensions-tp20806p20813.html > Sent from the OpenSCAD mailing list archive at Nabble.com. > > _______________________________________________ > OpenSCAD mailing list > Discuss@lists.openscad.org > http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >