W03: Bulma more

Week 3 was spent on putting finishing touches on the Bulma mockup and working on developing internal APIs for handling JS dependencies in HTML builds.

Bulma-based mockup

Here is an updated version of the Bulma mockup:

http://mortenpi.eu/gsoc2019-mockups/bulma/v2/

Overall, while some things need more tweaking, I would say it's ready to be implemented in HTMLWriter, i.e. make Documenter actually generate this new front end. I do not foresee any significant changes to the overall HTML structure.

Compared to last week's version, it features:

Most JS components are now included (e.g. KaTeX for maths rendering), the page is fully responsive and all components are styled now.
Uses bulma-dashboard for the overall layout (just two panels — the sidebar and the main body).
Is now based on a custom Sass style, which extends the default Bulma style.
Via developer tools you can also switch to a dark theme, which is a slightly extended version of the Darkly theme. As can be seen in the source SCSS file, existing Bulma themes can be re-used with relative ease, by just including the theme overrides in the appropriate places.

Relying on the standard Bulma styling was not sufficient, however. That is especially true for the navigation sidebar and top bar, but also for the styling of the docstring components.

However, where possible I have tried to re-use standard Bulma component styling. But instead of including them as classes in the HTML, I have opted to uses the Sass @extend rules (see e.g. how the docstring component is implemented).

Workarounds. I've also had to implement some workarounds. As an example, the .content class, which I apply on the main article tag to make sure that all the user-provided Markdown paragraphs, etc. get styled properly, can be problematic — you can't really use Bulma components in the .content, as the styling may conflict.

I had to override the styling of HTML tags in .message components (used for admonitions) if they are within the .content context. The main issue was that the .content styling took precedence over the text and background colors set by the .message class. As a result, in the dark theme, Bulma had light-colored text on light-colored backgrounds in admonitions.

Sidebar. I am not entirely happy with the sidebar and it needs more work. It presently replicates the old one, but it would be nice to (1) have some color there, and (2) make the levels a bit more distinct.

For (2): I have reached a design decision that the sidebar will only have two levels in general. A third level can also exist but will be collapsed by default^[1]. Any further levels get flattened into the third level. This will make it a bit easier to style the sidebar — I only have to figure out how to distinguish the two levels and the in-page headings.

JS dependency API

I also spent some time on prototyping an internal API for managing the JS dependencies.

In a nutshell, though, the goal is to generate the documenter.js file automatically. This is because we want to use RequireJS to manage JS dependencies. In each HTML file, we only have a single <script> tag that loads the main file

<script data-main="documenter" src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js"></script>

The documenter.js file, referred to in the data-main attribute, declares and loads all the dependent libraries and runs your actual code:

requirejs.config({
  paths: {
    'jquery': 'https://cdnjs.cloudflare.com/ajax/libs/jquery/3.1.1/jquery.min',
    'lunr': 'https://cdnjs.cloudflare.com/ajax/libs/lunr.js/2.3.5/lunr.min',
    'lodash': 'https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.11/lodash.min',
  }
});

require(["jquery", "lunr", "lodash"], function($, lunr, _) {
  // implementation
})

However, a problem arises when users want to include their own JS snippets, but use some libraries that are already available (e.g. JQuery). Currently, since documenter.js is immutable, this leads to a duplication, where the libraries have to be re-declared in another .js file which would then be loaded by RequireJS.

You can already see the duplication happening in Documenter if you compare the main documenter.js file and the search.js file, which gets loaded in addition to the main file on the search page.

So instead, the new idea is to construct documenter.js programmatically in Julia:

# Initialize an empty RequireJS file
r = RequireJS()
# Add a dependent library
push!(r, JSLibrary(
    "jquery",
    "https://cdnjs.cloudflare.com/ajax/libs/jquery/3.1.1/jquery.min.js",
    nothing, nothing
))
# Add a JS snippet that depends on the library
push!(r, JSSnippet(["jquery"], ["\$"],
    js"""
    $(document).ready(function() {
        ...
    })
    """
))
# Write out the RequrieJS file
writejs("documenter.js", r)

The writejs function will then generate the following file:

// Generated by Documenter.jl
requirejs.config({
  paths: {
    'jquery': 'https://cdnjs.cloudflare.com/ajax/libs/jquery/3.1.1/jquery.min',
  },
});
require(['jquery'], function($) {
// -- user-provided JS function body --
$(document).ready(function() {
    ...
})
// -- end of user-provided JS function body --
})

Of course, in general, it supports multiple libraries and snippets. Documenter will then combine them all together into a single file, with the libraries shared between the snippets, that can be loaded with RequireJS. A more substantial example can be found here..

In the end this will allow us to have ways to easily configure the JS code that gets shipped with Documenter builds. For example, it would allow the user to override MathJax's Config call, to globally define additional LaTeX commands:

function mathjax(config::Dict = MathJaxDefaultConfig)
    config_json = JSON.json(config, 2)
    js = """
    MathJax.Hub.Config($(config_json));
    """
    snippet = JSSnippet(["mathjax"], ["MathJax"], js)
    library = JSLibrary(
        "mathjax",
        "https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS_HTML",
        nothing, "MathJax"
    )
    ([library], [snippet])
end

# MathJaxDefaultConfig is not sufficient, so let's roll our own:
mathjax_config = Dict(
    :tex2jax => Dict(
        "inlineMath" => [["\$","\$"], ["\\(","\\)"]],
        "processEscapes" => true
    ),
    :TeX => Dict(
        :equationNumbers => Dict(:autoNumber => "AMS"),
        :Macros => Dict(
            "defd" => "≝",
            "diff" => ["\\mathrm{d}#1\\,", 1],
            "im" => "\\mathrm{i}",
        )
    )
)

makedocs(..., format = HTML(...,
    assets = [mathjax(mathjax_config)]
))

Next week

As the front-end overhaul is a pretty substantial change, it is probably better to wrap up #774 and tag 0.23.0 beforehand — the master branch has accumulated quite a few enhancements. The front end will then be included in 0.24.

So next week's targes involve quite a bit more Documenter coding (finally):

Finish off #774 — the doctesting PR.
Implement the Bulma mockup in Documenter, with a PR opened.
Implement basic theme building tools in DocumenterTools (should be able to build the default theme and a dark theme).

Wrapping up the JS assets API will come after the mockup implementation.

1Note to self: collapsible menus can be implemented without JS by using checkboxes.