Empathy List Archives

R

runsun

Wed, Jul 8, 2015 1:10 AM

Just finished a script that allows users to download the entire documentation
from the wiki for offline reading.

All the non-OpenSCAD-related stuff (like wiki menu, wiki warning, etc) are
stripped off.

Here are 2 screenshots:

http://forum.openscad.org/file/n13096/openscad_offliner_screenshot1.png
http://forum.openscad.org/file/n13096/openscad_offliner_screenshot2.png

It has only one file. You can find it in the repo:
https://github.com/runsun/openscad_offliner
https://github.com/runsun/openscad_offliner

Require: python 2.7, BeautifulSoup 4.0 above

Usage:

	1) Save this file in folder x.
	2) In folder x, type: 
		
		  python openscad_offliner.py 

	   or to save log to a file:  

		  python openscad_offliner.py > openscad_offliner.log

	All web pages will be saved in x/openscad_docs, 
	and all images in x/openscad_docs/imgs

It will generate 94 pages (1.3 MB) + 102 images (1.4 MB).

$ Runsun Pan, PhD

$ -- OpenScad_DocTest: doc and unit test ( Github , Thingiverse )

$ -- hash parameter model: here , here

$ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly

--
View this message in context: http://forum.openscad.org/Use-openscad-offliner-to-download-online-doc-tp13096.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

Just finished a script that allows users to download the entire documentation from the wiki for offline reading. All the non-OpenSCAD-related stuff (like wiki menu, wiki warning, etc) are stripped off. Here are 2 screenshots: <http://forum.openscad.org/file/n13096/openscad_offliner_screenshot1.png> <http://forum.openscad.org/file/n13096/openscad_offliner_screenshot2.png> It has only one file. You can find it in the repo: https://github.com/runsun/openscad_offliner <https://github.com/runsun/openscad_offliner> Require: python 2.7, BeautifulSoup 4.0 above Usage: > 1) Save this file in folder x. > 2) In folder x, type: > > python openscad_offliner.py > > or to save log to a file: > > python openscad_offliner.py > openscad_offliner.log > > All web pages will be saved in x/openscad_docs, > and all images in x/openscad_docs/imgs It will generate *94 pages (1.3 MB) + 102 images (1.4 MB).* ----- $ Runsun Pan, PhD $ -- OpenScad_DocTest: doc and unit test ( Github , Thingiverse ) $ -- hash parameter model: here , here $ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly -- View this message in context: http://forum.openscad.org/Use-openscad-offliner-to-download-online-doc-tp13096.html Sent from the OpenSCAD mailing list archive at Nabble.com.

DM

doug moen

Wed, Jul 8, 2015 2:55 AM

Looks pretty good.

On 7 July 2015 at 21:10, runsun runsun@gmail.com wrote:

Just finished a script that allows users to download the entire
documentation from the wiki for offline reading.

All the non-OpenSCAD-related stuff (like wiki menu, wiki warning, etc) are
stripped off.

Here are 2 screenshots:

It has only one file. You can find it in the repo:
https://github.com/runsun/openscad_offliner

Require: python 2.7, BeautifulSoup 4.0 above

Usage:

Save this file in folder x.
In folder x, type:

python openscad_offliner.py

or to save log to a file:

python openscad_offliner.py > openscad_offliner.log

All web pages will be saved in x/openscad_docs,
and all images in x/openscad_docs/imgs

It will generate 94 pages (1.3 MB) + 102 images (1.4 MB).

$ http://forum.openscad.org/mailing_list/MailingListOptions.jtp?forum=1 Runsun
Pan, PhD
$ -- OpenScad_DocTest: doc and unit test ( Github
https://github.com/runsun/openscad_doctest, Thingiverse
https://www.thingiverse.com/thing:410831 )
$ -- hash parameter model: here
http://forum.openscad.org/parameterized-models-td8303.html#a8306, here
http://forum.openscad.org/Can-I-get-some-code-review-up-in-here-tp12341p12355.html
$ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/
2015.04.01.nightly

View this message in context: Use openscad_offliner to download online doc
http://forum.openscad.org/Use-openscad-offliner-to-download-online-doc-tp13096.html
Sent from the OpenSCAD mailing list archive http://forum.openscad.org/
at Nabble.com.

OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

Looks pretty good. On 7 July 2015 at 21:10, runsun <runsun@gmail.com> wrote: > Just finished a script that allows users to download the entire > documentation from the wiki for offline reading. > > All the non-OpenSCAD-related stuff (like wiki menu, wiki warning, etc) are > stripped off. > > Here are 2 screenshots: > > > It has only one file. You can find it in the repo: > https://github.com/runsun/openscad_offliner > > Require: python 2.7, BeautifulSoup 4.0 above > > Usage: > > 1) Save this file in folder x. > 2) In folder x, type: > > python openscad_offliner.py > > or to save log to a file: > > python openscad_offliner.py > openscad_offliner.log > > All web pages will be saved in x/openscad_docs, > and all images in x/openscad_docs/imgs > > > It will generate *94 pages (1.3 MB) + 102 images (1.4 MB).* > > $ <http://forum.openscad.org/mailing_list/MailingListOptions.jtp?forum=1> *Runsun > Pan, PhD* > $ -- *OpenScad_DocTest*: doc and unit test ( Github > <https://github.com/runsun/openscad_doctest>, Thingiverse > <https://www.thingiverse.com/thing:410831> ) > $ -- *hash parameter model*: here > <http://forum.openscad.org/parameterized-models-td8303.html#a8306>, here > <http://forum.openscad.org/Can-I-get-some-code-review-up-in-here-tp12341p12355.html> > $ -- *Linux Mint 17.1 Rebecca x64* + *OpenSCAD 2015.03.15*/ > *2015.04.01.nightly* > > ------------------------------ > View this message in context: Use openscad_offliner to download online doc > <http://forum.openscad.org/Use-openscad-offliner-to-download-online-doc-tp13096.html> > Sent from the OpenSCAD mailing list archive <http://forum.openscad.org/> > at Nabble.com. > > _______________________________________________ > OpenSCAD mailing list > Discuss@lists.openscad.org > http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org > >

R

runsun

Wed, Jul 8, 2015 4:46 AM

It is possible to :

Write a shell script to zip the folder
Use a cron job to download and zip it nightly
Then put it in the main repo.

This way, the wiki page becomes the only source of doc we need to maintain.

$ Runsun Pan, PhD

$ -- OpenScad_DocTest: doc and unit test ( Github , Thingiverse )

$ -- hash parameter model: here , here

$ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly

--
View this message in context: http://forum.openscad.org/Use-openscad-offliner-to-download-online-doc-tp13096p13098.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

It is possible to : 1) Write a shell script to zip the folder 2) Use a cron job to download and zip it nightly 3) Then put it in the main repo. This way, the wiki page becomes the only source of doc we need to maintain. ----- $ Runsun Pan, PhD $ -- OpenScad_DocTest: doc and unit test ( Github , Thingiverse ) $ -- hash parameter model: here , here $ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly -- View this message in context: http://forum.openscad.org/Use-openscad-offliner-to-download-online-doc-tp13096p13098.html Sent from the OpenSCAD mailing list archive at Nabble.com.

JD

Jerry Davis

Sat, Jul 11, 2015 2:25 AM

If this works, and I will check if it does a little later.
If it does, and includes the cheat sheet, then maybe a future version of
openscad could have an optional Environment Variable to point to the
docroot (like file:///blahblah/), or if no env variable found, what it
points to now.

Jerry

--
Extra Ham Operator: K7AZJ
Registered Linux User: 275424
Raspberry Pi and Arduino developer

The most exciting phrase to hear in science - the one that heralds new
discoveries - is not "Eureka!" but "That's funny...".- Isaac. Asimov

I
*f you give someone a program, you will frustrate them for a day; if you
teach them how to program, you will frustrate them for a lifetime. *-
Anonymous

If writing good code requires very little comments, then writing really
excellent code requires no comments at all!- Ken Thompson

On Tue, Jul 7, 2015 at 11:46 PM, runsun runsun@gmail.com wrote:

It is possible to :

Write a shell script to zip the folder
Use a cron job to download and zip it nightly
Then put it in the main repo.

This way, the wiki page becomes the only source of doc we need to maintain.

$ Runsun Pan, PhD

$ -- OpenScad_DocTest: doc and unit test ( Github , Thingiverse )

$ -- hash parameter model: here , here

$ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly

--
View this message in context:
http://forum.openscad.org/Use-openscad-offliner-to-download-online-doc-tp13096p13098.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

OpenSCAD mailing list
Discuss@lists.openscad.org
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

If this works, and I will check if it does a little later. If it does, and includes the cheat sheet, then maybe a future version of openscad could have an optional Environment Variable to point to the docroot (like file:///blahblah/), or if no env variable found, what it points to now. Jerry -- Extra Ham Operator: K7AZJ Registered Linux User: 275424 Raspberry Pi and Arduino developer *The most exciting phrase to hear in science - the one that heralds new discoveries - is not "Eureka!" but "That's funny...".*- Isaac. Asimov *I* *f you give someone a program, you will frustrate them for a day; if you teach them how to program, you will frustrate them for a lifetime. *- Anonymous *If writing good code requires very little comments, then writing really excellent code requires no comments at all!*- Ken Thompson On Tue, Jul 7, 2015 at 11:46 PM, runsun <runsun@gmail.com> wrote: > It is possible to : > > 1) Write a shell script to zip the folder > 2) Use a cron job to download and zip it nightly > 3) Then put it in the main repo. > > This way, the wiki page becomes the only source of doc we need to maintain. > > > > > > > ----- > > $ Runsun Pan, PhD > > $ -- OpenScad_DocTest: doc and unit test ( Github , Thingiverse ) > > $ -- hash parameter model: here , here > > $ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly > > > > > -- > View this message in context: > http://forum.openscad.org/Use-openscad-offliner-to-download-online-doc-tp13096p13098.html > Sent from the OpenSCAD mailing list archive at Nabble.com. > > _______________________________________________ > OpenSCAD mailing list > Discuss@lists.openscad.org > http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org >

TP

Torsten Paul

Sat, Jul 11, 2015 11:37 PM

The scraped HTML would need quite some additional clean-up

it fires of a number of JavaScripts that access the Wikibooks site
all the CSS files are loaded online
most of the images don't work (at least in Firefox) as they have a
srcset attribute (which I did not even know until just now)
the index looks strange due to missing SVG files
the chapter links into other pages don't work
the edit links are displayed as []

I guess all that is fixable, but I do wonder how much of a moving
target this is chasing as it's trying to handle the output of
another generator which could change with every Wikimedia release.
It might still be a good idea to at least have a look at the
export option and render the original data.

But then if the script is maintained and updated to follow the
changes, that might not be a huge issue.

ciao,
Torsten.

The scraped HTML would need quite some additional clean-up * it fires of a number of JavaScripts that access the Wikibooks site * all the CSS files are loaded online * most of the images don't work (at least in Firefox) as they have a srcset attribute (which I did not even know until just now) * the index looks strange due to missing SVG files * the chapter links into other pages don't work * the edit links are displayed as [] I guess all that is fixable, but I do wonder how much of a moving target this is chasing as it's trying to handle the output of another generator which could change with every Wikimedia release. It might still be a good idea to at least have a look at the export option and render the original data. But then if the script is maintained and updated to follow the changes, that might not be a huge issue. ciao, Torsten.

R

runsun

Sun, Jul 12, 2015 12:21 AM

@Torsten,

Thx for pointing out those issues. I'll get them fixed.

tp3 wrote

the index looks strange due to missing SVG files
the chapter links into other pages don't work

Where are index and chapter links ? Maybe because thatI have
css files linked to the web(which should be fixed) so that I don't
see any problem ?

tp3 wrote

I guess all that is fixable, but I do wonder how much of a moving
target this is chasing as it's trying to handle the output of
another generator which could change with every Wikimedia release.
It might still be a good idea to at least have a look at the
export option and render the original data.

But then if the script is maintained and updated to follow the
changes, that might not be a huge issue.

ciao,
Torsten.

I think before we figure out a better way to solve the burden of
having to maintain several sources of doc, this might serve as
a temporary solution.

$ Runsun Pan, PhD

$ -- OpenScad_DocTest ( Thingiverse ), faces , Offliner

$ -- hash parameter model: here , here

$ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly

--
View this message in context: http://forum.openscad.org/Use-openscad-offliner-for-offline-documentation-tp13096p13118.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

@Torsten, Thx for pointing out those issues. I'll get them fixed. tp3 wrote > * the index looks strange due to missing SVG files > * the chapter links into other pages don't work Where are index and chapter links ? Maybe because thatI have css files linked to the web(which should be fixed) so that I don't see any problem ? tp3 wrote > I guess all that is fixable, but I do wonder how much of a moving > target this is chasing as it's trying to handle the output of > another generator which could change with every Wikimedia release. > It might still be a good idea to at least have a look at the > export option and render the original data. > > But then if the script is maintained and updated to follow the > changes, that might not be a huge issue. > > ciao, > Torsten. I think before we figure out a better way to solve the burden of having to maintain several sources of doc, this might serve as a temporary solution. ----- $ Runsun Pan, PhD $ -- OpenScad_DocTest ( Thingiverse ), faces , Offliner $ -- hash parameter model: here , here $ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly -- View this message in context: http://forum.openscad.org/Use-openscad-offliner-for-offline-documentation-tp13096p13118.html Sent from the OpenSCAD mailing list archive at Nabble.com.

R

runsun

Sun, Jul 12, 2015 12:24 AM

Btw, the [edit] => [] is intentional. I figure that there's no point of
editing an offline wiki doc ('cos it needs wiki's editing feature) so I
remove the word "edit".

tp3 wrote

the edit links are displayed as []

$ Runsun Pan, PhD

$ -- OpenScad_DocTest ( Thingiverse ), faces , Offliner

$ -- hash parameter model: here , here

$ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly

--
View this message in context: http://forum.openscad.org/Use-openscad-offliner-for-offline-documentation-tp13096p13119.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

Btw, the [edit] => [] is intentional. I figure that there's no point of editing an offline wiki doc ('cos it needs wiki's editing feature) so I remove the word "edit". tp3 wrote > * the edit links are displayed as [] ----- $ Runsun Pan, PhD $ -- OpenScad_DocTest ( Thingiverse ), faces , Offliner $ -- hash parameter model: here , here $ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly -- View this message in context: http://forum.openscad.org/Use-openscad-offliner-for-offline-documentation-tp13096p13119.html Sent from the OpenSCAD mailing list archive at Nabble.com.

TP

Torsten Paul

Sun, Jul 12, 2015 1:54 AM

On 07/12/2015 02:21 AM, runsun wrote:

Where are index and chapter links ? Maybe because thatI have
css files linked to the web(which should be fixed) so that I don't
see any problem ?

I mean the file://document#chapter links. The #chapter part seems
to be missing everywhere (maybe due to the link rewriting?)

Example:

file://.../openscad_docs/OpenSCAD_User_Manual

3.2 Primitive Solids - cube, sphere, cylinder & polyhedron

The "cylinder" link should go to Primitive_Solids#cylinder to
jump directly into that chapter.

Btw, the [edit] => [] is intentional. I figure that there's no point of
editing an offline wiki doc ('cos it needs wiki's editing feature) so I
remove the word "edit".

Ah, all the remaining [] looked a bit strange. If it's easy, it
would look better if those are removed completely. But that's
just minor cosmetics.

Talking cosmetics... :-) is it possible to add .html to the
files? It seems to work fine without, though.

I think before we figure out a better way to solve the burden of
having to maintain several sources of doc, this might serve as
a temporary solution.

I agree, it's much better than having different sources and it's
the only working solution to get a usable off-line documentation
right now.

Reading https://en.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_content
it looks like the footer should also be extended a bit to clarify
the license. I currently think it's best to use the CC-BY-SA, but
I'll have a look at the GNU-FDL too.
Using CC-BY-SA would mean to also add a license link and a short
info that the content was modified (maybe something like "page
modified for off-line reading" is sufficient?).

ciao,
Torsten.

On 07/12/2015 02:21 AM, runsun wrote: > Where are index and chapter links ? Maybe because thatI have > css files linked to the web(which should be fixed) so that I don't > see any problem ? > I mean the file://document#chapter links. The #chapter part seems to be missing everywhere (maybe due to the link rewriting?) Example: file://.../openscad_docs/OpenSCAD_User_Manual 3.2 Primitive Solids - cube, sphere, cylinder & polyhedron The "cylinder" link should go to Primitive_Solids#cylinder to jump directly into that chapter. > Btw, the [edit] => [] is intentional. I figure that there's no point of > editing an offline wiki doc ('cos it needs wiki's editing feature) so I > remove the word "edit". > Ah, all the remaining [] looked a bit strange. If it's easy, it would look better if those are removed completely. But that's just minor cosmetics. Talking cosmetics... :-) is it possible to add .html to the files? It seems to work fine without, though. > I think before we figure out a better way to solve the burden of > having to maintain several sources of doc, this might serve as > a temporary solution. > I agree, it's much better than having different sources and it's the only working solution to get a usable off-line documentation right now. Reading https://en.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_content it looks like the footer should also be extended a bit to clarify the license. I currently think it's best to use the CC-BY-SA, but I'll have a look at the GNU-FDL too. Using CC-BY-SA would mean to also add a license link and a short info that the content was modified (maybe something like "page modified for off-line reading" is sufficient?). ciao, Torsten.

R

runsun

Mon, Jul 13, 2015 4:39 AM

All issues are fixed (not update in repo yet), except the license part.
Please let me know what's the best description.

Will do more offline tests to make sure before I push it to the repo.

Also, if this comes to being usable, maybe consider if there's a better way
of using it. No matter how I'll write down clearly development instruction
in case someone wants to modify it.

$ Runsun Pan, PhD

$ -- OpenScad_DocTest ( Thingiverse ), faces , Offliner

$ -- hash parameter model: here , here

$ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly

--
View this message in context: http://forum.openscad.org/Use-openscad-offliner-for-offline-documentation-tp13096p13129.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

All issues are fixed (not update in repo yet), except the license part. Please let me know what's the best description. Will do more offline tests to make sure before I push it to the repo. Also, if this comes to being usable, maybe consider if there's a better way of using it. No matter how I'll write down clearly development instruction in case someone wants to modify it. ----- $ Runsun Pan, PhD $ -- OpenScad_DocTest ( Thingiverse ), faces , Offliner $ -- hash parameter model: here , here $ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly -- View this message in context: http://forum.openscad.org/Use-openscad-offliner-for-offline-documentation-tp13096p13129.html Sent from the OpenSCAD mailing list archive at Nabble.com.

R

runsun

Mon, Jul 13, 2015 5:33 AM

Just pushed it to the openscad_offliner repo. It seems working fine with
offline test on Chrome and Firefox.

Here are the fixes to tp3's findings :

it fires of a number of JavaScripts that access the Wikibooks site ==>
ALL javascripts are shutdown
all the CSS files are loaded online ==> All needed CSS files are
downloaded. There is still at least one page (Positioning_an_object)
trying to link to remote css, but it's not harmful ('cos all required css
are already downloaded) and hard to clean out (they seem to be retrieved
by javascript), so I leave them there.
most of the images don't work (at least in Firefox) as they have a
srcset attribute (which I did not even know until just now) ==> all
srcset attributes removed
the index looks strange due to missing SVG files ==> fixed
the chapter links into other pages don't work ==> fixed
the edit links are displayed as [] ==> all [edit] removed
is it possible to add .html to the files? It seems to work fine without,
though. ==> added

$ Runsun Pan, PhD

$ -- OpenScad_DocTest ( Thingiverse ), faces , Offliner

$ -- hash parameter model: here , here

$ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly

--
View this message in context: http://forum.openscad.org/Use-openscad-offliner-for-offline-documentation-tp13096p13131.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

Just pushed it to the openscad_offliner repo. It seems working fine with offline test on Chrome and Firefox. Here are the fixes to tp3's findings : > * it fires of a number of JavaScripts that access the Wikibooks site ==> > ALL javascripts are shutdown > > * all the CSS files are loaded online ==> All needed CSS files are > downloaded. There is still at least one page (Positioning_an_object) > trying to link to remote css, but it's not harmful ('cos all required css > are already downloaded) and hard to clean out (they seem to be retrieved > by javascript), so I leave them there. > > * most of the images don't work (at least in Firefox) as they have a > srcset attribute (which I did not even know until just now) ==> all > srcset attributes removed > > * the index looks strange due to missing SVG files ==> fixed > > * the chapter links into other pages don't work ==> fixed > > * the edit links are displayed as [] ==> all [edit] removed > > * is it possible to add .html to the files? It seems to work fine without, > though. ==> added ----- $ Runsun Pan, PhD $ -- OpenScad_DocTest ( Thingiverse ), faces , Offliner $ -- hash parameter model: here , here $ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly -- View this message in context: http://forum.openscad.org/Use-openscad-offliner-for-offline-documentation-tp13096p13131.html Sent from the OpenSCAD mailing list archive at Nabble.com.

discuss@lists.openscad.org

Use openscad_offliner to download online doc