Documentation and diagrams

older
Spectrum status update, April 2022

ville.ilvonen＠unikie.com

17 May 2022 17 May '22

5:52 a.m.

Hi, I asked thoughts on Spectrum documentation improvements some time back on #spectrum. I'd like to improve documentation in terms of architecture (views, description, diagrams) to support communication with people adopting Spectrum in our research/engineering project(s). Of course I can create presentations for such purposes but those are not supporting the project. There's nothing wrong with asciidoc but as of now it lacks support for diagrams. Another view to diagrams was how we want to create diagrams - writing dsl(diagram-specific-language)-to-diagrams or drawing. Based on our #spectrum discussion I was guided to asciidoc diagram extension - https://docs.asciidoctor.org/diagram-extension/latest/ - so I finally gave it a try last Friday with ditaa - an ascii-to-png-conversion. Long story short - I got it working by adding dependencies to jdk8 etc. but I'm not happy with the result for following reasons: Limitations: - ditaa (and other dsl-to-diagrams) have limited support in either diagram dsl comprehension or available editors. Namely, ascii-based box drawing programs exist (e.g. https://asciiflow.com/#/) but do not directly support ditaa. If ascii-drawings can be modified to ditaa, it's slow, error-prone and worst of all - importing version controlled ditaa-ascii-graph text files back to editors do not work as object imports so in worst case the objects need to be drawn again when one wants to modify the diagrams. - dsls have learning curves with different slopes. Some are easy, some take domain level understanding (like UML) in addition to dsl but it's there. - layout algorithms - many cases go quite ok with this but then there's points where the layout algorithms fail miserably with solutions beyond reasonable effort. - dsls are usually not readable in source format but beyond trivial diagrams always require validation visually Benefits: + Some asciidoc diagram-extension dsl:s (like plantuml, graphviz) work quite ok for both visualizing UML, graphs, dependencies, trees. + Focus on content, not presentation - kind of like LaTeX. Though this is limited by tools like ditaa which can get worst of both worlds. + dsls can be great for code level views. I've used tool where dsl has been combined into generating the diagrams from code or tree like structures. In fact, there's quite a few such tools to do this with nix-store dependency trees and have been useful in spectrum architecture comprehension. What I'd like to propose that in addition to aiming at text-to-diagrams we support diagram drawing tool(s) and agree on either png or svg with diagrams. With diagram drawing tools, we should store the diagrams in editable format in version control (not only png and svg). My 2 cents would go to https://github.com/jgraph/drawio but I'm fine with others as well - as long as they do not require commercial license. I'd like to hear thoughts/decisions on this before sending any patches to contribute to spectrum architecture documentation. Best regards, -Ville

Show replies by date

Mark Stosberg

17 May 17 May

11:42 a.m.

...

What I'd like to propose that in addition to aiming at text-to-diagrams we support diagram drawing tool(s) and agree on either png or svg with diagrams.

I’ve had good experiences with Mermaid: https://mermaid-js.github.io/mermaid/#/

Alyssa Ross

12:25 p.m.

Hi Ville, thanks for looking into this! On Tue, May 17, 2022 at 05:52:35AM -0000, ville.ilvonen@unikie.com wrote:

...

What I'd like to propose that in addition to aiming at text-to-diagrams we support diagram drawing tool(s) and agree on either png or svg with diagrams. With diagram drawing tools, we should store the diagrams in editable format in version control (not only png and svg). My 2 cents would go to https://github.com/jgraph/drawio but I'm fine with others as well - as long as they do not require commercial license.

I've heard very good things about draw.io. I agree with your assessment that having a graphical diagramming tool would be useful. draw.io being FOSS is a big plus for it as well — I think it's desirable that contributing to Spectrum does not depend on proprietary software. One question about draw.io: is there a tool we could use in the build process of our documentation to render the .drawio files to SVG, so we don't have to check in the SVG files?

Paco

7 p.m.

Hi all,

...

One question about draw.io: is there a tool we could use in the build process of our documentation to render the .drawio files to SVG, so we don't have to check in the SVG files?

drawio supports being called on the command-line like this: drawio in.drawio --export --output out.png # or svg but it's a huge dependency, as it's an Electron app, installing it requires ~400 MB…

Ville Ilvonen

18 May 18 May

12:44 p.m.

On Tue, May 17, 2022 at 10:00 PM Paco wrote:

...

Hi all,

...
One question about draw.io: is there a tool we could use in the build process of our documentation to render the .drawio files to SVG, so we don't have to check in the SVG files?

drawio supports being called on the command-line like this:

drawio in.drawio --export --output out.png # or svg

Yes, and no. Yes, command line but no in a headless environment. It was easy to test this before "jekyll build" with "drawio in.drawio --export --output test.svg" However, [vilvo@blop:~/dev/spectrum/Documentation]$ nix-build -I nixpkgs=../../nixpkgs-spectrum ... error: builder for '/nix/store/r8xf2kpba673ngi2fck4xsyvyc6pd3j1-spectrum-docs.drv' failed with exit code 133; last 10 log lines: > at new ElectronStore (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/node_modules/electron-store/index.js:48:32) > at Object.<anonymous> (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/electron.js:15:15) > at Module._compile (node:internal/modules/cjs/loader:1116:14) > at Object.Module._extensions..js (node:internal/modules/cjs/loader:1169:10) > at Module.load (node:internal/modules/cjs/loader:988:32) > at Module._load (node:internal/modules/cjs/loader:829:12) > at Function.c._load (node:electron/js2c/asar_bundle:5:13343) > at Object.<anonymous> (node:electron/js2c/browser_init:185:3104) > at Object../lib/browser/init.ts (node:electron/js2c/browser_init:185:3308) > /build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 1: 5 Trace/breakpoint trap (core dumped) drawio in.drawio --export --output test.svg Issue is the same with drawio versions: - version 15.7.3 from nixpkgs - version 17.4.2 from nixpkgs-spectrum - latest rpm available - 18.0.6 from updated nixpkgs-spectrum

...

but it's a huge dependency, as it's an Electron app, installing it requires ~400 MB…

This could be managed with caching but that's not the issue. People seem to have worked around the issue in CI with solutions like https://hub.docker.com/r/tomkludy/drawio-renderer I did not test it yet but would this be ok for a CI pipeline? Best, -Ville

Alyssa Ross

12:53 p.m.

On Wed, May 18, 2022 at 03:44:45PM +0300, Ville Ilvonen wrote:

...

On Tue, May 17, 2022 at 10:00 PM Paco wrote:

...
Hi all,

...
One question about draw.io: is there a tool we could use in the build process of our documentation to render the .drawio files to SVG, so we don't have to check in the SVG files?

drawio supports being called on the command-line like this:

drawio in.drawio --export --output out.png # or svg

Yes, and no. Yes, command line but no in a headless environment. It was easy to test this before "jekyll build" with "drawio in.drawio --export --output test.svg"

However,

[vilvo@blop:~/dev/spectrum/Documentation]$ nix-build -I nixpkgs=../../nixpkgs-spectrum ... error: builder for '/nix/store/r8xf2kpba673ngi2fck4xsyvyc6pd3j1-spectrum-docs.drv' failed with exit code 133; last 10 log lines: > at new ElectronStore (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/node_modules/electron-store/index.js:48:32) > at Object.<anonymous> (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/electron.js:15:15) > at Module._compile (node:internal/modules/cjs/loader:1116:14) > at Object.Module._extensions..js (node:internal/modules/cjs/loader:1169:10) > at Module.load (node:internal/modules/cjs/loader:988:32) > at Module._load (node:internal/modules/cjs/loader:829:12) > at Function.c._load (node:electron/js2c/asar_bundle:5:13343) > at Object.<anonymous> (node:electron/js2c/browser_init:185:3104) > at Object../lib/browser/init.ts (node:electron/js2c/browser_init:185:3308) > /build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 1: 5 Trace/breakpoint trap (core dumped) drawio in.drawio --export --output test.svg

Issue is the same with drawio versions: - version 15.7.3 from nixpkgs - version 17.4.2 from nixpkgs-spectrum - latest rpm available - 18.0.6 from updated nixpkgs-spectrum

Does it maybe require an X server / Wayland compositor to talk to? That would be a pain to do in the build but maybe not impossible. I know there are programs for dummy compositors out there but don't know how easy they are to use.

...

...
but it's a huge dependency, as it's an Electron app, installing it requires ~400 MB…

This could be managed with caching but that's not the issue.

People seem to have worked around the issue in CI with solutions like https://hub.docker.com/r/tomkludy/drawio-renderer

I did not test it yet but would this be ok for a CI pipeline?

If we can get it running in a Nix build, I don't think closure size will be an issue. If the size becomes an issue in future, we could try running the rendering code without Electron, by running the browser version in whatever the modern equivalent of PhantomJS is, or just trying to extract the renderer code and run it in Node directly. But if we can get the Electron version running, it should be fine to use that.

Michael Raskin

1:09 p.m.

...

Does it maybe require an X server / Wayland compositor to talk to? That would be a pain to do in the build but maybe not impossible. I know there are programs for dummy compositors out there but don't know how easy they are to use.

Dummy X is pretty easy. I use it. https://github.com/7c6f434c/lang-os/blob/master/wrapped-firefox-launcher.nix...

Alyssa Ross

1:11 p.m.

On Wed, May 18, 2022 at 12:53:53PM +0000, Alyssa Ross wrote:

...

On Wed, May 18, 2022 at 03:44:45PM +0300, Ville Ilvonen wrote:

...
On Tue, May 17, 2022 at 10:00 PM Paco wrote:

...
Hi all,

...
One question about draw.io: is there a tool we could use in the build process of our documentation to render the .drawio files to SVG, so we don't have to check in the SVG files?

drawio supports being called on the command-line like this:

drawio in.drawio --export --output out.png # or svg

Yes, and no. Yes, command line but no in a headless environment. It was easy to test this before "jekyll build" with "drawio in.drawio --export --output test.svg"

However,

[vilvo@blop:~/dev/spectrum/Documentation]$ nix-build -I nixpkgs=../../nixpkgs-spectrum ... error: builder for '/nix/store/r8xf2kpba673ngi2fck4xsyvyc6pd3j1-spectrum-docs.drv' failed with exit code 133; last 10 log lines: > at new ElectronStore (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/node_modules/electron-store/index.js:48:32) > at Object.<anonymous> (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/electron.js:15:15) > at Module._compile (node:internal/modules/cjs/loader:1116:14) > at Object.Module._extensions..js (node:internal/modules/cjs/loader:1169:10) > at Module.load (node:internal/modules/cjs/loader:988:32) > at Module._load (node:internal/modules/cjs/loader:829:12) > at Function.c._load (node:electron/js2c/asar_bundle:5:13343) > at Object.<anonymous> (node:electron/js2c/browser_init:185:3104) > at Object../lib/browser/init.ts (node:electron/js2c/browser_init:185:3308) > /build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 1: 5 Trace/breakpoint trap (core dumped) drawio in.drawio --export --output test.svg

Issue is the same with drawio versions: - version 15.7.3 from nixpkgs - version 17.4.2 from nixpkgs-spectrum - latest rpm available - 18.0.6 from updated nixpkgs-spectrum

Does it maybe require an X server / Wayland compositor to talk to? That would be a pain to do in the build but maybe not impossible. I know there are programs for dummy compositors out there but don't know how easy they are to use.

Actually, this looking more closely at the stack trace, this is more likely to be it not having a writeable XDG_CONFIG_HOME or HOME directory. So you could try setting those to temporary directories in the build. Something like export XDG_CONFIG_HOME="$(mktemp -d)"

Ville Ilvonen

19 May 19 May

7:25 a.m.

On Wed, May 18, 2022 at 4:11 PM Alyssa Ross wrote:

...

Actually, this looking more closely at the stack trace, this is more likely to be it not having a writeable XDG_CONFIG_HOME or HOME directory. So you could try setting those to temporary directories in the build. Something like

export XDG_CONFIG_HOME="$(mktemp -d)"

I think it was a good idea to test but unfortunately it did not work out: '' export XDG_CONFIG_HOME="$(mktemp -d)" echo $XDG_CONFIG_HOME drawio in.drawio --export --output test.svg jekyll build --disable-disk-cache -s $src -d $out '' $ nix-build this derivation will be built: /nix/store/k2z4xgs1dpp5p52sds98vl46x7myl924-spectrum-docs.drv building '/nix/store/k2z4xgs1dpp5p52sds98vl46x7myl924-spectrum-docs.drv'... /build/tmp.mOBtFDpZlO Fontconfig error: Cannot load default config file: No such file: (null) /build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 3: 4 Segmentation fault (core dumped) drawio in.drawio --export --output test.svg error: builder for '/nix/store/k2z4xgs1dpp5p52sds98vl46x7myl924-spectrum-docs.drv' failed with exit code 139; last 3 log lines: > /build/tmp.mOBtFDpZlO > Fontconfig error: Cannot load default config file: No such file: (null) > /build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 3: 4 Segmentation fault (core dumped) drawio in.drawio --export --output test.svg For full logs, run 'nix log /nix/store/k2z4xgs1dpp5p52sds98vl46x7myl924-spectrum-docs.drv'. Best, -Ville

Alyssa Ross

7:35 a.m.

On Thu, May 19, 2022 at 10:25:16AM +0300, Ville Ilvonen wrote:

...

On Wed, May 18, 2022 at 4:11 PM Alyssa Ross wrote:

...
Actually, this looking more closely at the stack trace, this is more likely to be it not having a writeable XDG_CONFIG_HOME or HOME directory. So you could try setting those to temporary directories in the build. Something like

export XDG_CONFIG_HOME="$(mktemp -d)"

I think it was a good idea to test but unfortunately it did not work out:

'' export XDG_CONFIG_HOME="$(mktemp -d)" echo $XDG_CONFIG_HOME drawio in.drawio --export --output test.svg jekyll build --disable-disk-cache -s $src -d $out ''

I was just looking at the draw.io packaging in Nixpkgs, and I discovered that there's already code in there for running headless draw.io! [1] But it's not exposed outside of this pandoc filter package. I'm going to see if it works, and if it does I'll see if we can pull it out to the top level, as a drawio-headless package or something. [1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/pandoc-drawio-f...

Ville Ilvonen

8:33 a.m.

On Thu, May 19, 2022 at 10:35 AM Alyssa Ross wrote:

...

On Thu, May 19, 2022 at 10:25:16AM +0300, Ville Ilvonen wrote:

...
On Wed, May 18, 2022 at 4:11 PM Alyssa Ross wrote:

...
Actually, this looking more closely at the stack trace, this is more likely to be it not having a writeable XDG_CONFIG_HOME or HOME directory. So you could try setting those to temporary directories in the build. Something like

export XDG_CONFIG_HOME="$(mktemp -d)"

I think it was a good idea to test but unfortunately it did not work out:

'' export XDG_CONFIG_HOME="$(mktemp -d)" echo $XDG_CONFIG_HOME drawio in.drawio --export --output test.svg jekyll build --disable-disk-cache -s $src -d $out ''

I was just looking at the draw.io packaging in Nixpkgs, and I discovered that there's already code in there for running headless draw.io! [1] But it's not exposed outside of this pandoc filter package. I'm going to see if it works, and if it does I'll see if we can pull it out to the top level, as a drawio-headless package or something.

[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/pandoc-drawio-f...

I was also starting to think that the X env is required. X virtual framebuffer is a great idea. Good catch with pandoc approach to generate drawio to pdf in xvfb.

Alyssa Ross

8:49 a.m.

On Thu, May 19, 2022 at 11:33:24AM +0300, Ville Ilvonen wrote:

...

On Thu, May 19, 2022 at 10:35 AM Alyssa Ross wrote:

...
On Thu, May 19, 2022 at 10:25:16AM +0300, Ville Ilvonen wrote:

...
On Wed, May 18, 2022 at 4:11 PM Alyssa Ross wrote:

...
Actually, this looking more closely at the stack trace, this is more likely to be it not having a writeable XDG_CONFIG_HOME or HOME directory. So you could try setting those to temporary directories in the build. Something like

export XDG_CONFIG_HOME="$(mktemp -d)"

I think it was a good idea to test but unfortunately it did not work out:

'' export XDG_CONFIG_HOME="$(mktemp -d)" echo $XDG_CONFIG_HOME drawio in.drawio --export --output test.svg jekyll build --disable-disk-cache -s $src -d $out ''

I was just looking at the draw.io packaging in Nixpkgs, and I discovered that there's already code in there for running headless draw.io! [1] But it's not exposed outside of this pandoc filter package. I'm going to see if it works, and if it does I'll see if we can pull it out to the top level, as a drawio-headless package or something.

[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/pandoc-drawio-f...

I was also starting to think that the X env is required. X virtual framebuffer is a great idea. Good catch with pandoc approach to generate drawio to pdf in xvfb.

https://github.com/NixOS/nixpkgs/pull/173589 :)

707

Age (days ago)

709

Last active (days ago)

List overview

Download

11 comments

6 participants

participants (6)

Alyssa Ross
Mark Stosberg
Michael Raskin
Paco
Ville Ilvonen
ville.ilvonen＠unikie.com

Documentation and diagrams

tags

participants (6)