discuss@lists.openscad.org

OpenSCAD general discussion Mailing-list

View all threads

Auto Reload with lots of files causes massive slowdown again

NH
nop head
Sat, Feb 16, 2019 6:56 PM

I have fixed this problem twice but it is back again. My ability to debug
it is severely limited now because I can no longer build at home. I trashed
my MSYS environment by trying to add the double-conversion package.

Has anybody changed something recently that might have broken it? As far as
I can tell the StatCache is working and only has one entry for each file
but does everything still go through it?

I have fixed this problem twice but it is back again. My ability to debug it is severely limited now because I can no longer build at home. I trashed my MSYS environment by trying to add the double-conversion package. Has anybody changed something recently that might have broken it? As far as I can tell the StatCache is working and only has one entry for each file but does everything still go through it?
MK
Marius Kintel
Sat, Feb 16, 2019 8:12 PM

On Feb 16, 2019, at 13:56, nop head nop.head@gmail.com wrote:

I have fixed this problem twice but it is back again. My ability to debug it is severely limited now because I can no longer build at home. I trashed my MSYS environment by trying to add the double-conversion package.

Has anybody changed something recently that might have broken it? As far as I can tell the StatCache is working and only has one entry for each file but does everything still go through it?

I cannot think of any recent breakage, but a lot of PRs came through recently, so we may have introduced a bug somewhere..

I still think that the correct solution to this is to use a filesystem monitor, which most filesystems, including NTFS, should support.

It would be good if we could get a local MSYS build up and running again. Just a severe lack of Windows developers around here : /
Looks like my XP VM finally died, so I'm out of luck at the moment as well.

A native (MSVC) build env would probably make life easier for anyone using Windows proper, but that's still quite a bit of work to pull off, mostly getting binaries to build for all dependencies.

-Marius

> On Feb 16, 2019, at 13:56, nop head <nop.head@gmail.com> wrote: > > I have fixed this problem twice but it is back again. My ability to debug it is severely limited now because I can no longer build at home. I trashed my MSYS environment by trying to add the double-conversion package. > > Has anybody changed something recently that might have broken it? As far as I can tell the StatCache is working and only has one entry for each file but does everything still go through it? I cannot think of any recent breakage, but a lot of PRs came through recently, so we may have introduced a bug somewhere.. I still think that the correct solution to this is to use a filesystem monitor, which most filesystems, including NTFS, should support. It would be good if we could get a local MSYS build up and running again. Just a severe lack of Windows developers around here : / Looks like my XP VM finally died, so I'm out of luck at the moment as well. A native (MSVC) build env would probably make life easier for anyone using Windows proper, but that's still quite a bit of work to pull off, mostly getting binaries to build for all dependencies. -Marius
NH
nop head
Sun, Feb 17, 2019 12:31 AM

Yes a filesystem monitor would be more efficient but multi-platform and on
networked filesystems, etc?

Basically I have about 100 unique files and checking their dates doesn't
take the five seconds I am seeing now. It should be a small fraction of a
second. However lots of files are included many times so the original
problem was they were being checked thousands of times.

Somehow I think the problem is back and and what ever has broken it would
probably still break a filesytem monitor solution. To take 5 seconds
something must be accessing the files over and over again.

I only noticed the problem when I downloaded Hans' polygon speed up fork
but I don't think that is the cause. It must have been another change to
master.

The obvious solution would be git bisect but my code won't work on old
versions any more because the language keeps changing. It won't even work
on the release candidate, which is disappointing because I was expecting to
be able to release code that would work on the upcoming release,

Perhaps I can do git bisect using Mendel90 code as the test. However that
would require getting MSYS2 to work again, or building a Linux box. I don't
have long before I go overseas again with only a laptop.

On Sat, 16 Feb 2019 at 20:13, Marius Kintel marius@kintel.net wrote:

On Feb 16, 2019, at 13:56, nop head nop.head@gmail.com wrote:

I have fixed this problem twice but it is back again. My ability to

debug it is severely limited now because I can no longer build at home. I
trashed my MSYS environment by trying to add the double-conversion package.

Has anybody changed something recently that might have broken it? As far

as I can tell the StatCache is working and only has one entry for each file
but does everything still go through it?

I cannot think of any recent breakage, but a lot of PRs came through
recently, so we may have introduced a bug somewhere..

I still think that the correct solution to this is to use a filesystem
monitor, which most filesystems, including NTFS, should support.

It would be good if we could get a local MSYS build up and running again.
Just a severe lack of Windows developers around here : /
Looks like my XP VM finally died, so I'm out of luck at the moment as well.

A native (MSVC) build env would probably make life easier for anyone using
Windows proper, but that's still quite a bit of work to pull off, mostly
getting binaries to build for all dependencies.

-Marius


OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

Yes a filesystem monitor would be more efficient but multi-platform and on networked filesystems, etc? Basically I have about 100 unique files and checking their dates doesn't take the five seconds I am seeing now. It should be a small fraction of a second. However lots of files are included many times so the original problem was they were being checked thousands of times. Somehow I think the problem is back and and what ever has broken it would probably still break a filesytem monitor solution. To take 5 seconds something must be accessing the files over and over again. I only noticed the problem when I downloaded Hans' polygon speed up fork but I don't think that is the cause. It must have been another change to master. The obvious solution would be git bisect but my code won't work on old versions any more because the language keeps changing. It won't even work on the release candidate, which is disappointing because I was expecting to be able to release code that would work on the upcoming release, Perhaps I can do git bisect using Mendel90 code as the test. However that would require getting MSYS2 to work again, or building a Linux box. I don't have long before I go overseas again with only a laptop. On Sat, 16 Feb 2019 at 20:13, Marius Kintel <marius@kintel.net> wrote: > > On Feb 16, 2019, at 13:56, nop head <nop.head@gmail.com> wrote: > > > > I have fixed this problem twice but it is back again. My ability to > debug it is severely limited now because I can no longer build at home. I > trashed my MSYS environment by trying to add the double-conversion package. > > > > Has anybody changed something recently that might have broken it? As far > as I can tell the StatCache is working and only has one entry for each file > but does everything still go through it? > > I cannot think of any recent breakage, but a lot of PRs came through > recently, so we may have introduced a bug somewhere.. > > I still think that the correct solution to this is to use a filesystem > monitor, which most filesystems, including NTFS, should support. > > It would be good if we could get a local MSYS build up and running again. > Just a severe lack of Windows developers around here : / > Looks like my XP VM finally died, so I'm out of luck at the moment as well. > > A native (MSVC) build env would probably make life easier for anyone using > Windows proper, but that's still quite a bit of work to pull off, mostly > getting binaries to build for all dependencies. > > -Marius > > > _______________________________________________ > OpenSCAD mailing list > Discuss@lists.openscad.org > http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >
MK
Marius Kintel
Sun, Feb 17, 2019 3:19 AM

On Feb 16, 2019, at 19:31, nop head nop.head@gmail.com wrote:

The obvious solution would be git bisect but my code won't work on old versions any more because the language keeps changing. It won't even work on the release candidate, which is disappointing because I was expecting to be able to release code that would work on the upcoming release,

What did we change/not include that caused this to not work?

-Marius

> On Feb 16, 2019, at 19:31, nop head <nop.head@gmail.com> wrote: > > The obvious solution would be git bisect but my code won't work on old versions any more because the language keeps changing. It won't even work on the release candidate, which is disappointing because I was expecting to be able to release code that would work on the upcoming release, > What did we change/not include that caused this to not work? -Marius
NH
nop head
Sun, Feb 17, 2019 7:46 AM

The warnings about len and undef and the introduction of is_indef and
is_list as work arounds.

On Sun, 17 Feb 2019 at 03:20, Marius Kintel marius@kintel.net wrote:

On Feb 16, 2019, at 19:31, nop head nop.head@gmail.com wrote:

The obvious solution would be git bisect but my code won't work on old

versions any more because the language keeps changing. It won't even work
on the release candidate, which is disappointing because I was expecting to
be able to release code that would work on the upcoming release,

What did we change/not include that caused this to not work?

-Marius


OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

The warnings about len and undef and the introduction of is_indef and is_list as work arounds. On Sun, 17 Feb 2019 at 03:20, Marius Kintel <marius@kintel.net> wrote: > > On Feb 16, 2019, at 19:31, nop head <nop.head@gmail.com> wrote: > > > > The obvious solution would be git bisect but my code won't work on old > versions any more because the language keeps changing. It won't even work > on the release candidate, which is disappointing because I was expecting to > be able to release code that would work on the upcoming release, > > > What did we change/not include that caused this to not work? > > -Marius > > > _______________________________________________ > OpenSCAD mailing list > Discuss@lists.openscad.org > http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >
NH
nop head
Sun, Feb 17, 2019 8:03 AM

If I load Mendel90 into the latest version I get 38000 warnings.

On Sun, 17 Feb 2019 at 07:46, nop head nop.head@gmail.com wrote:

The warnings about len and undef and the introduction of is_indef and
is_list as work arounds.

On Sun, 17 Feb 2019 at 03:20, Marius Kintel marius@kintel.net wrote:

On Feb 16, 2019, at 19:31, nop head nop.head@gmail.com wrote:

The obvious solution would be git bisect but my code won't work on old

versions any more because the language keeps changing. It won't even work
on the release candidate, which is disappointing because I was expecting to
be able to release code that would work on the upcoming release,

What did we change/not include that caused this to not work?

-Marius


OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

If I load Mendel90 into the latest version I get 38000 warnings. On Sun, 17 Feb 2019 at 07:46, nop head <nop.head@gmail.com> wrote: > The warnings about len and undef and the introduction of is_indef and > is_list as work arounds. > > On Sun, 17 Feb 2019 at 03:20, Marius Kintel <marius@kintel.net> wrote: > >> > On Feb 16, 2019, at 19:31, nop head <nop.head@gmail.com> wrote: >> > >> > The obvious solution would be git bisect but my code won't work on old >> versions any more because the language keeps changing. It won't even work >> on the release candidate, which is disappointing because I was expecting to >> be able to release code that would work on the upcoming release, >> > >> What did we change/not include that caused this to not work? >> >> -Marius >> >> >> _______________________________________________ >> OpenSCAD mailing list >> Discuss@lists.openscad.org >> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >> >
NH
nop head
Sun, Feb 17, 2019 8:05 AM

On the other hand it previews in about 5 minutes, whereas it used to take
about 12 IIRC. Not many programs get faster!

On Sun, 17 Feb 2019 at 08:03, nop head nop.head@gmail.com wrote:

If I load Mendel90 into the latest version I get 38000 warnings.

On Sun, 17 Feb 2019 at 07:46, nop head nop.head@gmail.com wrote:

The warnings about len and undef and the introduction of is_indef and
is_list as work arounds.

On Sun, 17 Feb 2019 at 03:20, Marius Kintel marius@kintel.net wrote:

On Feb 16, 2019, at 19:31, nop head nop.head@gmail.com wrote:

The obvious solution would be git bisect but my code won't work on old

versions any more because the language keeps changing. It won't even work
on the release candidate, which is disappointing because I was expecting to
be able to release code that would work on the upcoming release,

What did we change/not include that caused this to not work?

-Marius


OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

On the other hand it previews in about 5 minutes, whereas it used to take about 12 IIRC. Not many programs get faster! On Sun, 17 Feb 2019 at 08:03, nop head <nop.head@gmail.com> wrote: > If I load Mendel90 into the latest version I get 38000 warnings. > > On Sun, 17 Feb 2019 at 07:46, nop head <nop.head@gmail.com> wrote: > >> The warnings about len and undef and the introduction of is_indef and >> is_list as work arounds. >> >> On Sun, 17 Feb 2019 at 03:20, Marius Kintel <marius@kintel.net> wrote: >> >>> > On Feb 16, 2019, at 19:31, nop head <nop.head@gmail.com> wrote: >>> > >>> > The obvious solution would be git bisect but my code won't work on old >>> versions any more because the language keeps changing. It won't even work >>> on the release candidate, which is disappointing because I was expecting to >>> be able to release code that would work on the upcoming release, >>> > >>> What did we change/not include that caused this to not work? >>> >>> -Marius >>> >>> >>> _______________________________________________ >>> OpenSCAD mailing list >>> Discuss@lists.openscad.org >>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >>> >>
NH
nop head
Sun, Feb 17, 2019 8:23 AM

It also works fine with Auto Reload so it actually seems to be something to
do with my new machine design. I did change the way I include my library.
Ironically instead of including every file everywhere I included just the
files required. So I would expect it to be faster not many orders of
magnitude slower.

On thing that stands out is the paths in the cache are a mixture of forward
/ and \ because it depends whether the file is used via OPENSCADPATH or via
a relative path. The StatCache doesn't have multiple entries, so I don't
know why this causes a problem, but I have noticed a similar problem in
the past. When I was inconsistent with the case of my library path it
caused a big slowdown. I assumed that was because it doubled the size of
Cache but maybe something else goes wrong.

For consistency I think all paths should use / inside OpenSCAD. I think
is only needed on the command line in Windows, so I never use it C code.

On Sun, 17 Feb 2019 at 08:05, nop head nop.head@gmail.com wrote:

On the other hand it previews in about 5 minutes, whereas it used to take
about 12 IIRC. Not many programs get faster!

On Sun, 17 Feb 2019 at 08:03, nop head nop.head@gmail.com wrote:

If I load Mendel90 into the latest version I get 38000 warnings.

On Sun, 17 Feb 2019 at 07:46, nop head nop.head@gmail.com wrote:

The warnings about len and undef and the introduction of is_indef and
is_list as work arounds.

On Sun, 17 Feb 2019 at 03:20, Marius Kintel marius@kintel.net wrote:

On Feb 16, 2019, at 19:31, nop head nop.head@gmail.com wrote:

The obvious solution would be git bisect but my code won't work on

old versions any more because the language keeps changing. It won't even
work on the release candidate, which is disappointing because I was
expecting to be able to release code that would work on the upcoming
release,

What did we change/not include that caused this to not work?

-Marius


OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

It also works fine with Auto Reload so it actually seems to be something to do with my new machine design. I did change the way I include my library. Ironically instead of including every file everywhere I included just the files required. So I would expect it to be faster not many orders of magnitude slower. On thing that stands out is the paths in the cache are a mixture of forward / and \ because it depends whether the file is used via OPENSCADPATH or via a relative path. The StatCache doesn't have multiple entries, so I don't know why this causes a problem, but I have noticed a similar problem in the past. When I was inconsistent with the case of my library path it caused a big slowdown. I assumed that was because it doubled the size of Cache but maybe something else goes wrong. For consistency I think all paths should use / inside OpenSCAD. I think \ is only needed on the command line in Windows, so I never use it C code. On Sun, 17 Feb 2019 at 08:05, nop head <nop.head@gmail.com> wrote: > On the other hand it previews in about 5 minutes, whereas it used to take > about 12 IIRC. Not many programs get faster! > > On Sun, 17 Feb 2019 at 08:03, nop head <nop.head@gmail.com> wrote: > >> If I load Mendel90 into the latest version I get 38000 warnings. >> >> On Sun, 17 Feb 2019 at 07:46, nop head <nop.head@gmail.com> wrote: >> >>> The warnings about len and undef and the introduction of is_indef and >>> is_list as work arounds. >>> >>> On Sun, 17 Feb 2019 at 03:20, Marius Kintel <marius@kintel.net> wrote: >>> >>>> > On Feb 16, 2019, at 19:31, nop head <nop.head@gmail.com> wrote: >>>> > >>>> > The obvious solution would be git bisect but my code won't work on >>>> old versions any more because the language keeps changing. It won't even >>>> work on the release candidate, which is disappointing because I was >>>> expecting to be able to release code that would work on the upcoming >>>> release, >>>> > >>>> What did we change/not include that caused this to not work? >>>> >>>> -Marius >>>> >>>> >>>> _______________________________________________ >>>> OpenSCAD mailing list >>>> Discuss@lists.openscad.org >>>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >>>> >>>
NH
nop head
Sun, Feb 17, 2019 8:49 AM

This looks like the culprit.
https://github.com/openscad/openscad/blob/c6a485651fa29f1a878ea454558192492f7467ec/src/parsersettings.cc#L31

It is calling file system functions bypassing the cache. This will be tens
of thousand of times as it is files squared with a library that is included
everywhere.

On Sun, 17 Feb 2019 at 08:23, nop head nop.head@gmail.com wrote:

It also works fine with Auto Reload so it actually seems to be something
to do with my new machine design. I did change the way I include my
library. Ironically instead of including every file everywhere I included
just the files required. So I would expect it to be faster not many orders
of magnitude slower.

On thing that stands out is the paths in the cache are a mixture of
forward / and \ because it depends whether the file is used via
OPENSCADPATH or via a relative path. The StatCache doesn't have multiple
entries, so I don't know why this causes a problem, but I have noticed a
similar problem in the past. When I was inconsistent with the case of my
library path it caused a big slowdown. I assumed that was because it
doubled the size of Cache but maybe something else goes wrong.

For consistency I think all paths should use / inside OpenSCAD. I think
is only needed on the command line in Windows, so I never use it C code.

On Sun, 17 Feb 2019 at 08:05, nop head nop.head@gmail.com wrote:

On the other hand it previews in about 5 minutes, whereas it used to take
about 12 IIRC. Not many programs get faster!

On Sun, 17 Feb 2019 at 08:03, nop head nop.head@gmail.com wrote:

If I load Mendel90 into the latest version I get 38000 warnings.

On Sun, 17 Feb 2019 at 07:46, nop head nop.head@gmail.com wrote:

The warnings about len and undef and the introduction of is_indef and
is_list as work arounds.

On Sun, 17 Feb 2019 at 03:20, Marius Kintel marius@kintel.net wrote:

On Feb 16, 2019, at 19:31, nop head nop.head@gmail.com wrote:

The obvious solution would be git bisect but my code won't work on

old versions any more because the language keeps changing. It won't even
work on the release candidate, which is disappointing because I was
expecting to be able to release code that would work on the upcoming
release,

What did we change/not include that caused this to not work?

-Marius


OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

This looks like the culprit. https://github.com/openscad/openscad/blob/c6a485651fa29f1a878ea454558192492f7467ec/src/parsersettings.cc#L31 It is calling file system functions bypassing the cache. This will be tens of thousand of times as it is files squared with a library that is included everywhere. On Sun, 17 Feb 2019 at 08:23, nop head <nop.head@gmail.com> wrote: > It also works fine with Auto Reload so it actually seems to be something > to do with my new machine design. I did change the way I include my > library. Ironically instead of including every file everywhere I included > just the files required. So I would expect it to be faster not many orders > of magnitude slower. > > On thing that stands out is the paths in the cache are a mixture of > forward / and \ because it depends whether the file is used via > OPENSCADPATH or via a relative path. The StatCache doesn't have multiple > entries, so I don't know why this causes a problem, but I have noticed a > similar problem in the past. When I was inconsistent with the case of my > library path it caused a big slowdown. I assumed that was because it > doubled the size of Cache but maybe something else goes wrong. > > For consistency I think all paths should use / inside OpenSCAD. I think \ > is only needed on the command line in Windows, so I never use it C code. > > On Sun, 17 Feb 2019 at 08:05, nop head <nop.head@gmail.com> wrote: > >> On the other hand it previews in about 5 minutes, whereas it used to take >> about 12 IIRC. Not many programs get faster! >> >> On Sun, 17 Feb 2019 at 08:03, nop head <nop.head@gmail.com> wrote: >> >>> If I load Mendel90 into the latest version I get 38000 warnings. >>> >>> On Sun, 17 Feb 2019 at 07:46, nop head <nop.head@gmail.com> wrote: >>> >>>> The warnings about len and undef and the introduction of is_indef and >>>> is_list as work arounds. >>>> >>>> On Sun, 17 Feb 2019 at 03:20, Marius Kintel <marius@kintel.net> wrote: >>>> >>>>> > On Feb 16, 2019, at 19:31, nop head <nop.head@gmail.com> wrote: >>>>> > >>>>> > The obvious solution would be git bisect but my code won't work on >>>>> old versions any more because the language keeps changing. It won't even >>>>> work on the release candidate, which is disappointing because I was >>>>> expecting to be able to release code that would work on the upcoming >>>>> release, >>>>> > >>>>> What did we change/not include that caused this to not work? >>>>> >>>>> -Marius >>>>> >>>>> >>>>> _______________________________________________ >>>>> OpenSCAD mailing list >>>>> Discuss@lists.openscad.org >>>>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >>>>> >>>>