W02: Bulma & GFM

The second GSoC week was spent on prototyping the new Documenter front end in Bulma and doing some general feature and maintenance work on Documenter.

Bulma frontend

Compared to last week with the Bootstrap-based front end, I decided to take a slightly different approach here. I just started off by including the default Bulma CSS file and tried to see how far I will get with just playing around with the HTML by applying classes etc.

The current progress on the mockup is available here:

http://mortenpi.eu/gsoc2019-mockups/bulma/v1/

A few notes about the status of the mockup:

I have not included any of the JS libraries etc. at the moment.
The sidebar still needs quite a lot of work:
- Bulma does have a Menu component for vertical menus, but I am not a fan of its default styling.
- The menu component does not seem to allow you to disable certain menu items (i.e. having non-link menu items).
- We just probably have to extend the component in the Documenter stylesheet though.
- The centering of the logo is done with custom styling. I could not figure out how to make it centered without applying margin: auto; manually.
- Also the layout is sub-optimal. With Bulma natively, you can only create flexbox layouts. But I think I would prefer to stick to the static fixed-width version we have now.
  However, it was brought to my attention in #jsoc-standup that the bulma-dashboard Bulma extension might help me out there, instead of having to create the necessary CSS myself.
Docstrings are currently not styled. We probably need to create a custom component for these in the Documenter stylesheet.
I am also not a big fan of how much space the default footer class creates around the content (it's applied on the colophon container at the bottom of the page).
Admonitions are implemented by just applying the message class.
Blockquotes look nice out of the box!

Semantic UI. I also quickly looked at Semantic UI before I started working on the Bulma-based mockup. I primarily wanted to know if it also suffers from the same limitation that you always have to apply CSS classes in order to style components nicely. The answer turned out to be yes (e.g. tables), so I had to discard it.

In Bulma, the main <article> tag gets a content class and all the elements inside get all the necessary styling without having to apply any classes. This is what we want since we can't always control how the HTML tags will be emitted (by user code).

I also realized that Semantic UI uses LESS, instead of Sass, for compiling the stylesheets. This would mean that we'd have to rely on the Less.js Node library, instead of being able to use the libsass C/C++ library via the Sass.jl wrapper when compiling theme stylesheets in DocumenterTools in the future. It would be doable, but using a C/C++ library seems more straightforward and less error-prone.

So overall it seems that I will be sticking to Bulma as the framework of choice for Documenter.

READMEs, GFM, and Documenter

At the end of the week, I was working on a way to natively incorporate GitHub Flavored Markdown (GFM) into Documenter builds. GFM syntax slightly differs from the Julia Flavored Markdown implemented by the Markdown standard library, but you have to use GFM when writing documents that you want to be displayed on GitHub (e.g. the README of your repository).

Having a way to handle GFM in Documenter has been asked in several issues over the years (#12, #529, #551). The main blocker is that we don't have a parser that could handle the GFM syntax natively in Julia.

However, that was actually relatively easy to fix: I simply wrapped the cmark-gfm C/C++ library, which itself is an extension of the CommonMark reference parser library cmark, in Julia. The WIP Julia wrapper can be found at mortenpi/CMark.jl.

The wrapper provides the CMark.markdown function, which takes a string as an input and outputs the native Julia Markdown AST (i.e. a Markdown.MD object). Internally it calls to libcmark-gfm instead though and simply converts the library AST into Julia AST.

To handle Markdown features that do not have 1-to-1 correspondences between the two flavors, CMark.jl tries to approximate the GFM feature in the Julia AST in a reasonable way. For example, HTML code blocks, which GFM parses into separate leaves of the AST, are converted to Markdown.Code blocks with the language parameters set to @raw html, to mirror the standard way to include raw HTML strings in Documenter builds.

To be able to the CMark with Documenter I repurposed an old pull request, #552. That PR adds an external function that can be used to include pages that are external to the docs/src/ directory into Documenter builds.

I generalized it such that it also now take a keyword argument, which allows the parser to be overridden. So to call it with CMark, you'd have to pass a pages argument that looks something as follows to makedocs:

pages = [
    external("../README.md", parser=CMark.markdown)
]

The parser argument should be a function or a functor f(s::AbstractString) -> Markdown.MD, and its only job is to parse a string into valid Julia Markdown AST. The AST gets then passed on to Documenter and it gets processed as normal.

The API here is actually very general. It would, in principle, be possible to write a parser for reStructuredText and include pages written in that markup language in Documenter builds. Or, potentially, e.g. a wrapper function could be provided by Literate.jl to more easily incorporate literate Julia documents in Documenter builds:

import Literate, Markdown
function literateparser(s; kwargs...)
    (tmpfile, io) = mktemp()
    write(io, s); close(io)
    Literate.markdown(tmpfile, "$(tmpfile).md"; kwargs...)
    Markdown.parse_file("$(tmpfile).md")
end

pages = [
    external("literate-page.jl", parser = literateparser)
]

The WIP implementation of the updated external function lives on mp/external-pages branch. As a quick demo, it is being used on that branch to include README.md in the normal Documenter documentation builds (i.e. docs/make.jl). An example rendering is also available.

It is actually natural to introduce a way to specify a custom parser to the external function since it's main application will be to include README.md, LICENSE.md etc. files in documentation builds, which will generally use GFM.

pkg.julialang.org. One prospective application of this is to have better support for rendering READMEs for packages that do not provide full Documenter-based docs on the pkg.julialang.org site. Currently, it uses the commonmarker Ruby tool to parse the README and then simply include the resulting HTML in a Documenter build with an @raw html block. However, doing things this way has the downside that, for example, Documenter will not be able to render headings in the sidebar, etc.

Other work

I also did some other maintenance work on Documenter, which lead to the following pull requests:

The latter two grew out of the need to have a way to test the separate doctesting in #774. #1033 basically added proper tests to the current implementation of doctesting, where we now actually properly check whether doctests fail when they are supposed to etc.

Next week

I mainly now need to wrap up the front end mockup. So week 3 will focus on properly finishing up the Bulma mockup, and starting the design of new Documenter APIs for customizing the front end and the design of theming tools.