M
MichaelAtOz
Sun, Dec 20, 2015 1:22 AM
As jp said the full list produces an error, seems to be some fixed parser
limit:
ERROR: Parser error in line 20017: memory exhausted
ERROR: Compilation failed!
I first split it in 1/2, that compiled, but rendering ran out of memory.
I then wrapped every 100 (of all 37K) in a union(), that cleared the parser
memory limit & compiled:
http://forum.openscad.org/file/n15231/td15226_F5.jpg
mmm...chips...
Rendering now, will take a while to find out if it hits my memory limit.
Newly minted Admin - PM me if you need anything, or if I've done something stupid...
Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work. Obviously inclusion of works of previous authors is not included in the above.
The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
View this message in context: http://forum.openscad.org/Test-compile-big-openSCAD-model-tp15226p15231.html
Sent from the OpenSCAD mailing list archive at Nabble.com.
As jp said the full list produces an error, seems to be some fixed parser
limit:
> ERROR: Parser error in line 20017: memory exhausted
> ERROR: Compilation failed!
I first split it in 1/2, that compiled, but rendering ran out of memory.
I then wrapped every 100 (of all 37K) in a union(), that cleared the parser
memory limit & compiled:
<http://forum.openscad.org/file/n15231/td15226_F5.jpg>
mmm...chips...
Rendering now, will take a while to find out if it hits my memory limit.
-----
Newly minted Admin - PM me if you need anything, or if I've done something stupid...
Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work. Obviously inclusion of works of previous authors is not included in the above.
The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
--
View this message in context: http://forum.openscad.org/Test-compile-big-openSCAD-model-tp15226p15231.html
Sent from the OpenSCAD mailing list archive at Nabble.com.
M
MichaelAtOz
Sun, Dec 20, 2015 9:22 PM
Rendering now, will take a while to find out if it hits my memory limit.
That hit my limit of 8GB (4GB+4GB swap).
It (the 100 union'ed version) has been running for 1/2 a day on my bigger
box. (8 core 3.5GHz 16BG+swap raid SSD)
Just checked it this AM, unfortunately it went to sleep at some time...
It is about to hit commit of 7GB, at this stage the peak working set is
5.3GB and current WS <2GB.
(OpenSCAD progress meter showing 0/1000)
Newly minted Admin - PM me if you need anything, or if I've done something stupid...
Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work. Obviously inclusion of works of previous authors is not included in the above.
The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
View this message in context: http://forum.openscad.org/Test-compile-big-openSCAD-model-tp15226p15253.html
Sent from the OpenSCAD mailing list archive at Nabble.com.
MichaelAtOz wrote
> Rendering now, will take a while to find out if it hits my memory limit.
That hit my limit of 8GB (4GB+4GB swap).
It (the 100 union'ed version) has been running for 1/2 a day on my bigger
box. (8 core 3.5GHz 16BG+swap raid SSD)
Just checked it this AM, unfortunately it went to sleep at some time...
It is about to hit commit of 7GB, at this stage the peak working set is
5.3GB and current WS <2GB.
(OpenSCAD progress meter showing 0/1000)
-----
Newly minted Admin - PM me if you need anything, or if I've done something stupid...
Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work. Obviously inclusion of works of previous authors is not included in the above.
The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
--
View this message in context: http://forum.openscad.org/Test-compile-big-openSCAD-model-tp15226p15253.html
Sent from the OpenSCAD mailing list archive at Nabble.com.
DV
david vanhorn
Sun, Dec 20, 2015 9:36 PM
What is eating so much time? The cpu is practically at idle. I run em
sims that peg the cpu and graphics card for hours to days and make a very
effective room heater. This isnt even twitching the needle.
What is eating so much time? The cpu is practically at idle. I run em
sims that peg the cpu and graphics card for hours to days and make a very
effective room heater. This isnt even twitching the needle.
M
MichaelAtOz
Sun, Dec 20, 2015 10:38 PM
What is eating so much time? The cpu is practically at idle. I run em
sims that peg the cpu and graphics card for hours to days and make a very
effective room heater. This isnt even twitching the needle.
OpenSCAD mailing list
OpenSCAD is single threaded and does not use GPU either.
So best is the highest GHz CPU and plenty of memory.
Newly minted Admin - PM me if you need anything, or if I've done something stupid...
Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work. Obviously inclusion of works of previous authors is not included in the above.
The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
View this message in context: http://forum.openscad.org/Test-compile-big-openSCAD-model-tp15226p15255.html
Sent from the OpenSCAD mailing list archive at Nabble.com.
dbvanhorn wrote
> What is eating so much time? The cpu is practically at idle. I run em
> sims that peg the cpu and graphics card for hours to days and make a very
> effective room heater. This isnt even twitching the needle.
>
> _______________________________________________
> OpenSCAD mailing list
> Discuss@.openscad
> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
OpenSCAD is single threaded and does not use GPU either.
So best is the highest GHz CPU and plenty of memory.
-----
Newly minted Admin - PM me if you need anything, or if I've done something stupid...
Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work. Obviously inclusion of works of previous authors is not included in the above.
The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
--
View this message in context: http://forum.openscad.org/Test-compile-big-openSCAD-model-tp15226p15255.html
Sent from the OpenSCAD mailing list archive at Nabble.com.
DV
david vanhorn
Sun, Dec 20, 2015 10:41 PM
I know that but its not even fully tasking one processor.
I know that but its not even fully tasking one processor.
DM
doug moen
Mon, Dec 21, 2015 12:37 AM
If the arguments to a CSG operation like union are so large they don't fit
in primary cache, then I'd expect cache thrashing, and a big slowdown as
the CPU will be waiting on main memory.
On Sunday, 20 December 2015, david vanhorn kc6ete@gmail.com wrote:
I know that but its not even fully tasking one processor.
If the arguments to a CSG operation like union are so large they don't fit
in primary cache, then I'd expect cache thrashing, and a big slowdown as
the CPU will be waiting on main memory.
On Sunday, 20 December 2015, david vanhorn <kc6ete@gmail.com> wrote:
> I know that but its not even fully tasking one processor.
>
DV
david vanhorn
Mon, Dec 21, 2015 2:12 AM
Done.
ECHO: "processed 37100"
ECHO: "processed 37200"
Rendering Polygon Mesh using CGAL...
Geometries in cache: 74403
Geometry cache size in bytes: 54164656
CGAL Polyhedrons in cache: 0
CGAL cache size in bytes: 0
Total rendering time: 7 hours, 20 minutes, 9 seconds
Top level object is a 3D object:
Simple: no
Vertices: 1244484
Halfedges: 3943810
Edges: 1971905
Halffacets: 1384044
Facets: 692022
Volumes: 2341
WARNING: Object may not be a valid 2-manifold and may need repair!
Rendering finished.
Memory is sitting at 20G for OpenSCAD 37% used, and CPU at 4%.
I would expect that my EM simulations with 10G datasets would also be far
too big to fit in cache, but when they are running the processor(s)
assigned to the task are 100% loaded, and the system acts accordingly.
On Sun, Dec 20, 2015 at 5:37 PM, doug moen doug@moens.org wrote:
If the arguments to a CSG operation like union are so large they don't fit
in primary cache, then I'd expect cache thrashing, and a big slowdown as
the CPU will be waiting on main memory.
On Sunday, 20 December 2015, david vanhorn kc6ete@gmail.com wrote:
I know that but its not even fully tasking one processor.
--
K1FZY (WA4TPW) SK 9/29/37-4/13/15
Done.
ECHO: "processed 37100"
ECHO: "processed 37200"
Rendering Polygon Mesh using CGAL...
Geometries in cache: 74403
Geometry cache size in bytes: 54164656
CGAL Polyhedrons in cache: 0
CGAL cache size in bytes: 0
Total rendering time: 7 hours, 20 minutes, 9 seconds
Top level object is a 3D object:
Simple: no
Vertices: 1244484
Halfedges: 3943810
Edges: 1971905
Halffacets: 1384044
Facets: 692022
Volumes: 2341
WARNING: Object may not be a valid 2-manifold and may need repair!
Rendering finished.
Memory is sitting at 20G for OpenSCAD 37% used, and CPU at 4%.
I would expect that my EM simulations with 10G datasets would also be far
too big to fit in cache, but when they are running the processor(s)
assigned to the task are 100% loaded, and the system acts accordingly.
On Sun, Dec 20, 2015 at 5:37 PM, doug moen <doug@moens.org> wrote:
> If the arguments to a CSG operation like union are so large they don't fit
> in primary cache, then I'd expect cache thrashing, and a big slowdown as
> the CPU will be waiting on main memory.
>
>
> On Sunday, 20 December 2015, david vanhorn <kc6ete@gmail.com> wrote:
>
>> I know that but its not even fully tasking one processor.
>>
>
> _______________________________________________
> OpenSCAD mailing list
> Discuss@lists.openscad.org
> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>
>
--
K1FZY (WA4TPW) SK 9/29/37-4/13/15
AC
Alan Cox
Mon, Dec 21, 2015 10:24 AM
What is eating so much time? The cpu is practically at idle. I run em
sims that peg the cpu and graphics card for hours to days and make a very
effective room heater. This isnt even twitching the needle.
The CPU is idle because it's continually waiting for the disk because its
thrashing and needs more memory. OpenSCAD is built upon libraries that are
open and very powerful but unfortunately neither stunningly fast nor
memory efficient, especially as OpenSCAD uses them. The algorithms used
are also memory and performance dependant upon the number of vertices
involved.
Your test compile triggers those cases quite spectacularly, it rapidly
needs lots of memory and the moment it exceeds your RAM your are waiting
for disk all the time, which is very very slow relative to CPU
performance.
If you had a real world need for such a complex object you'd want to use
different tools whose algorithms are instead strongly time dependant upon
the detail level of the final render. ImplicitCAD should for example
handle it fairly well.
It's not a normal use case for 3D printing, so while OpenSCAD has some
speed annoyances they usually don't bite you for real world work.
Alan
On Sun, 20 Dec 2015 14:36:21 -0700
david vanhorn <kc6ete@gmail.com> wrote:
> What is eating so much time? The cpu is practically at idle. I run em
> sims that peg the cpu and graphics card for hours to days and make a very
> effective room heater. This isnt even twitching the needle.
The CPU is idle because it's continually waiting for the disk because its
thrashing and needs more memory. OpenSCAD is built upon libraries that are
open and very powerful but unfortunately neither stunningly fast nor
memory efficient, especially as OpenSCAD uses them. The algorithms used
are also memory and performance dependant upon the number of vertices
involved.
Your test compile triggers those cases quite spectacularly, it rapidly
needs lots of memory and the moment it exceeds your RAM your are waiting
for disk all the time, which is very very slow relative to CPU
performance.
If you had a real world need for such a complex object you'd want to use
different tools whose algorithms are instead strongly time dependant upon
the detail level of the final render. ImplicitCAD should for example
handle it fairly well.
It's not a normal use case for 3D printing, so while OpenSCAD has some
speed annoyances they usually don't bite you for real world work.
Alan
DM
doug moen
Mon, Dec 21, 2015 4:59 PM
I checked stack exchange about how %cpu usage is calculated. I don't think
my guess about CPU cache thrashing is correct.
Alan notes that the CPU won't be at 100% if the swap is thrashing. That
can't explain David's result where OpenScad is consuming 20Gb of a total
64Gb RAM.
Running a profiling tool on openscad might reveal the cause.
On Sunday, 20 December 2015, david vanhorn kc6ete@gmail.com wrote:
Done.
ECHO: "processed 37100"
ECHO: "processed 37200"
Rendering Polygon Mesh using CGAL...
Geometries in cache: 74403
Geometry cache size in bytes: 54164656
CGAL Polyhedrons in cache: 0
CGAL cache size in bytes: 0
Total rendering time: 7 hours, 20 minutes, 9 seconds
Top level object is a 3D object:
Simple: no
Vertices: 1244484
Halfedges: 3943810
Edges: 1971905
Halffacets: 1384044
Facets: 692022
Volumes: 2341
WARNING: Object may not be a valid 2-manifold and may need repair!
Rendering finished.
Memory is sitting at 20G for OpenSCAD 37% used, and CPU at 4%.
I would expect that my EM simulations with 10G datasets would also be far
too big to fit in cache, but when they are running the processor(s)
assigned to the task are 100% loaded, and the system acts accordingly.
On Sun, Dec 20, 2015 at 5:37 PM, doug moen <doug@moens.org
javascript:_e(%7B%7D,'cvml','doug@moens.org');> wrote:
I know that but its not even fully tasking one processor.
--
K1FZY (WA4TPW) SK 9/29/37-4/13/15
I checked stack exchange about how %cpu usage is calculated. I don't think
my guess about CPU cache thrashing is correct.
Alan notes that the CPU won't be at 100% if the swap is thrashing. That
can't explain David's result where OpenScad is consuming 20Gb of a total
64Gb RAM.
Running a profiling tool on openscad might reveal the cause.
On Sunday, 20 December 2015, david vanhorn <kc6ete@gmail.com> wrote:
> Done.
>
> ECHO: "processed 37100"
>
> ECHO: "processed 37200"
>
> Rendering Polygon Mesh using CGAL...
>
> Geometries in cache: 74403
>
> Geometry cache size in bytes: 54164656
>
> CGAL Polyhedrons in cache: 0
>
> CGAL cache size in bytes: 0
>
> Total rendering time: 7 hours, 20 minutes, 9 seconds
>
> Top level object is a 3D object:
>
> Simple: no
>
> Vertices: 1244484
>
> Halfedges: 3943810
>
> Edges: 1971905
>
> Halffacets: 1384044
>
> Facets: 692022
>
> Volumes: 2341
>
> WARNING: Object may not be a valid 2-manifold and may need repair!
>
> Rendering finished.
>
>
> Memory is sitting at 20G for OpenSCAD 37% used, and CPU at 4%.
>
>
>
> I would expect that my EM simulations with 10G datasets would also be far
> too big to fit in cache, but when they are running the processor(s)
> assigned to the task are 100% loaded, and the system acts accordingly.
>
>
> On Sun, Dec 20, 2015 at 5:37 PM, doug moen <doug@moens.org
> <javascript:_e(%7B%7D,'cvml','doug@moens.org');>> wrote:
>
>> If the arguments to a CSG operation like union are so large they don't
>> fit in primary cache, then I'd expect cache thrashing, and a big slowdown
>> as the CPU will be waiting on main memory.
>>
>>
>> On Sunday, 20 December 2015, david vanhorn <kc6ete@gmail.com
>> <javascript:_e(%7B%7D,'cvml','kc6ete@gmail.com');>> wrote:
>>
>>> I know that but its not even fully tasking one processor.
>>>
>>
>> _______________________________________________
>> OpenSCAD mailing list
>> Discuss@lists.openscad.org
>> <javascript:_e(%7B%7D,'cvml','Discuss@lists.openscad.org');>
>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>>
>>
>
>
> --
> K1FZY (WA4TPW) SK 9/29/37-4/13/15
>
M
Mekko
Mon, Dec 21, 2015 5:10 PM
If its performance is bound by disk seek-latency, wouldn't an SSD (a.k.a.
"flash drive") improve performance considerably? They're getting cheap these
days...
--
View this message in context: http://forum.openscad.org/Test-compile-big-openSCAD-model-tp15226p15264.html
Sent from the OpenSCAD mailing list archive at Nabble.com.
If its performance is bound by disk seek-latency, wouldn't an SSD (a.k.a.
"flash drive") improve performance considerably? They're getting cheap these
days...
--
View this message in context: http://forum.openscad.org/Test-compile-big-openSCAD-model-tp15226p15264.html
Sent from the OpenSCAD mailing list archive at Nabble.com.
