W12: Require JS

At the start of the week I went ahead and merged the various open front end PRs: #1043, #1089, #1090. This was mainly to avoid a merge conflict hell which a lot of parallel PRs would create, as they all touch more or less the same part of the code base.

Since then the new front end has been gathering feedback (#1091, JuliaLang/julia#32865). Any fixes and improvements to the new front the can be done in follow-up PRs.

I also did a little bit of maintenance work on the test suite and CI setup (#1093, #1095, #1096).

JS APIs

The primary emphasis this week was on improving the handling of JS dependencies. The HTML front end loads most of the JS via a single documenter.js file that gets loaded with RequireJS.^[1] So far it has just been a single static JS file in the repository that we just copy verbatim when generating the output HTML site. That, however, makes it essentially impossible to have the user customize any parts of it without overriding the whole file.

A RequireJS configuration file consists of two parts. The first is a configuration shim, that tells RequireJS where it can find the dependencies (in our case, usually a URL to a CDN) and their internal dependency relation if need be. E.g. the one for highlight.js might look like this:

requirejs.config({
    paths: {
        'jquery': 'https://cdnjs.cloudflare.com/ajax/libs/jquery/3.1.1/jquery.min',
        'highlight': 'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min',
        'highlight-julia': 'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/languages/julia.min',
    },
    shim: {
        'highlight-julia': ['highlight'],
        'highlight-julia-repl': ['highlight'],
    }
});

The second part is a code snippet, wrapped in a require call, that uses the dependencies. In case of highlight.js it simply calls a single function to enable the highlighting code the code blocks:

require(['jquery', 'highlight', 'highlight-julia'], function($, hljs) {
    $(document).ready(function() {
        hljs.initHighlighting();
    })
})

Note that all the snippets will share the RequireJS configuration.

During the early weeks of GSoC I prototyped a new internal API which would generate documenter.js dynamically during the makedocs call (JS dependency API). This way we can also inject user-provided configuration into the file. Finally now, #1092 implements a small utility module called JSDependencies in Documenter which provides an internal API to build up the file dynamically.

The configuration bits are represented by RemoteLibrary objects, each specifying a single library and its dependencies.^[2] The Snippet objects represent individual require calls — they essentially just contain a string with the code that goes into the require call and declare which libraries this particular snippet depends on.

Those are then combined into a RequireJS object, representing the complete RequireJS configuration file, with simple push! calls.^[3] You then just have to call writejs on it and it will write out the full JS file which can be included on an HTML page with the appropriate <script> tag.

Highlight.js languages

Building on top of this new internal API, it was easy to implement support for additional highlighting languages. The main highlight.js script on cdnjs can only highlight a small subset of all the languages supported by highlight.js. But loading additional languages in highlight.js is easy — you just need to make sure that the necessary additional language-specific JS file is loaded.

E.g. to add support for LLVM IR, you just need to load the llvm.js language file from the CDN. With #1094, the user can just specify the names of the additional languages with the highlights keyword of Documenter.HTML.

Internally, Documenter handles this by having a highlightjs! function, which pushes all the necessary libraries and snippets into a RequireJS object. It also takes the list of extra languages as an argument and adds all the necessary libraries for those as well.

KaTeX & MathJax

We used to use MathJax as the LaTeX rendering engine in Documenter, but switched to KaTeX as the default with the new front end, as it provides better performance. However, MathJax still has some extra features that some more math-heavy docs might need, so we would like this to be something that the user can choose.

#1097 implements the option to choose the LaTeX rendering engine, by passing the mathengine keyword to Documenter.HTML. Moreover, you can now also customize the MathJax and KaTeX configuration options. For example, to use MathJax, but to also add in some custom global LaTeX commands, you can just pass

mathengine = Documenter.MathJax(Dict(:TeX => Dict(
    :equationNumbers => Dict(:autoNumber => "AMS"),
    :Macros => Dict(
        :ket => ["|#1\\rangle", 1],
        :bra => ["\\langle#1|", 1],
    ),
)))

Both MathJax and KaTeX use simple JS dictionaries to pass for the options. This means that it we can actually easily construct the configuration dictionaries in Julia and the use JSON.jl to convert that into JS. This allows us also to programmatically merge user-provided settings with the defaults.

Internally, it was again just a matter of implementing a simple mathengine! function which would push all the correct libraries and snippets required by the specified rendering engine. Also note that the user can remove any reference to either of the rendering engines in the output documenter.js file by passing mathengine = nothing.

Next week

The GSoC is slowly coming to an end. This week was the last week for major development work. The main focus for the next week will be writing up the GSoC submission.

In addition to that I also want to:

• Finalize and merge the open JS PRs. I would also like to add in some simple API to pass generic JS assets via the assets keyword.
• Finish up and add basic docs to DocumenterTools#27 (theme compilation API).
• Write up the NEWS for the 0.24 changes.
• Release 0.24 to make the new front end available to the world. Small fixes to the themes can still happen in patch releases of that version.