Yarner -- Introduction

Yarner is a language-independent Literate Programming command line tool for Markdown.

From Markdown documents written and structured for humans, Yarner extracts code blocks into compilable source code. It offers sufficient features and flexibility to be usable also for larger projects with numerous files and multiple languages.

GitHub project

Yarner is Open Source and available on GitHub.

We also provide examples there for the use of Yarner with different programming languages.

Contributing

For feature requests and issues, please use the issue tracker. Merge requests are welcome.

License

Yarner and all its sources are released under the MIT License.

Installation

There are multiple ways to install Yarner:

Binaries

Download the latest binaries for your platform
(Binaries are available for Linux, Windows and macOS)
Unzip somewhere
Optional: add the parent directory of the executable to your PATH environmental variable

From GitHub using `cargo`

In case you have Rust installed, you can install with cargo:

cargo install yarner

Clone and build

To build Yarner locally, e.g. to contribute to the project, you will have to clone the repository on your local machine:

git clone https://github.com/mlange-42/yarner

cd into yarner/ and run

cargo build

The resulting binary can be found in yarner/target/debug/ under the name yarner or yarner.exe.

Getting started

Yarner is very easy to use from the command line, requiring only two different commands, yarner init and yarner. This chapter demonstrates how to set up a project and extract sources from Markdown.

Create a project

To set up a new project, use the init sub-command. Run the following in your project's base directory:

> yarner init

This creates a file Yarner.toml with default settings, and a file README.md as starting point for Literate Programming.

The generated file already contains some content to get started with Yarner's basic features. These are explained in detail in the subsequent chapters.

Build a project

To build the project (i.e. extract code and create documentation), simply run:

> yarner

This creates two sub-directories, one containing the extracted code (a minimal but working Rust project), and another containing the final documentation.

Note that the contents of these directories can then be treated as usual, i.e. compiling the code with the normal compiler, or rendering Markdown to HTML or PDF.

What's next

The following chapters explain the use of Yarner in detail, feature by feature.

To get an impression how Yarner projects look like, or to find a template to get started with your preferred programming language, see also the examples in the GitHub repository.

Blocks and macros

Yarner's aim is to make it possible to create software from documents written and structured for humans. The most important features in that respect are (named) code blocks and macros. This chapter provides an introduction to their basic usage.

Code blocks

In the most basic scenario, Yarner extracts code blocks from a Markdown document and writes them to an equally-named file in the code output directory, with the file's md extension stripped off.

As an example, create a project like this:

> yarner init

This creates a file README.md in your project's directory, and a Yarner.toml file with the default settings.

File README.md has some template content, but we want to start with a minimal example:

# Simple example

The program's entry point:

```rust
fn main() {
    println!("Hello World!");
}
```

Running Yarner with

> yarner

creates a file README in sub-directory code, and a documentation file README.md in sub-directory docs. In this case, docs/README.md has the same content as the original file, and code/README contains the extracted code block:

fn main() {
    println!("Hello World!");
}

File extensions

A Rust file without the .rs extension is not very useful. We can rename the main file from README.md to main.rs.md. Further, in file Yarner.toml, we change option files in section [paths] to:

files = ["main.rs.md"]

Yarner now produces the files code/main.rs and docs/main.rs.md.

Please note that only the md extension is stripped for the code output, while any extension-like parts of the main file name are preserved.

Project structure

The project structure of the above example looks like this:

project
  │
  ├── code/              <code output>
  │     └── main.rs
  │
  ├── docs/              <doc output>
  │     └── main.rs.md
  │
  ├── main.rs.md         <source document>
  └── Yarner.toml        <config>

Project structure is flexible an can be configured in the project's Yarner.toml. See chapters Paths for configuration, and Project structure for other possible layouts.

Macros

To allow structuring for humans, Yarner uses macros. For that sake, code blocks can be given a name in their first line, by default prefixed with //-. Here is a code block named Say hello:

```rust
//- Say hello
fn say_hello() {
    println!("Hello World!");
}
```

During code extraction, code is drawn together by replacing macro invocations by the respective code block's content. By default, a macro invocation starts with // ==> and ends with ..

```rust
fn main() {
    say_hello();
}

// ==> Say hello.
```

As a complete example, the content of a file main.rs.md could look like this:

# Simple example

The program's entry point:

```rust
fn main() {
    say_hello();
}
// ==> Say hello.
```

Here is how we say hello:

```rust
//- Say hello
fn say_hello() {
    println!("Hello World!");
}
```

The resulting content of code/main.rs looks like this:

fn main() {
    say_hello();
}
fn say_hello() {
    println!("Hello World!");
}

Macro evaluation is recursive. Thus, code blocks that are referenced by macros can also contain macro invocations.

Named entrypoints

By default, unnamed code blocks are the entrypoints for code extraction. This can be changed in the config file Yarner.toml through the option entrypoint in section [paths]:

...
[paths]
entrypoint = "Main"
...

The code block named by //- Main in its first line would then be used as entrypoint.

Concatenated code blocks

Multiple code blocks with the same name are concatenated in their order of appearance in the source document. As an example, here is an alternative content for main.rs.md:

# Simple example

The program's entry point:

```rust
fn main() {
    say_hello();
}
```

Here is how we say hello:

```rust
fn say_hello() {
    println!("Hello World!");
}
```

The two code blocks are concatenated and result in this content of code/main.rs:

fn main() {
    say_hello();
}
fn say_hello() {
    println!("Hello World!");
}

Multiple code files

For most software projects, a single code file is not sufficient. Yarner provides several features for that purpose. The most basic, producing multiple code files from a single Markdown document, is described here.

Multiple files from a single document

It is possible to generate multiple code files from a single documentation source file through code blocks named with file paths, prefixed with file: (the default, configurable).

```rust
//- file:path/to/file.ext
...
```

As an example, a file main.rs.md with the following content creates file src/lib.rs, besides the main file main.rs:

# Multiple files example

File main.rs looks like this:

```rust
fn main () {

}
```

And here is the content of file `lib.rs`:

```rust
//- file:src/lib.rs
fn first_funtion() {}
fn second_funtion() {}

// ==> Further functions.
```

The remaining functions in `lib.rs`:

```rust
//- Further functions
fn third_funtion() {}
fn fourth_funtion() {}
```

Note that macro invocations are possible as usual, with no special syntax required.

Each code block named with the file: prefix is treated as separate entrypoint during the build process.

Further uses of the feature

This feature can also be used to avoid the somewhat uncommon file naming patterns that were used in this guide so far. We generated code files from source files of the same name, but with an additional md extension. With the file: prefix feature, it is possible to circumvent this restriction completely.

As an example, it may be desired that the primary documentation file is named README.md (because the project is hosted on GitHub or GitLab), but to create a file main.rs from it. A file README.md with the following content would achieve that:

# Simple example

The program's entry point:

```rust
//- file:main.rs
fn main() {
    println!("Hello World!");
}
```

Thus, file naming in documentation and code can be completely independent of each other.

Dead and hidden code

In some cases, it may be desirable to have code blocks that don't go into code output, or code that is hidden in the documentation.

Dead code

Code blocks like examples may be intended to be excluded from code output. This can be achieved by giving a block a name that is not used in any macro. In this example, the code block named Example will be absent from the generated code as it is not referenced from anywhere:

# Dead code example

A function to greet someone:

```rust
fn say_hello(name: &str) {
    println!("Hello {}", name);
}
```

Function `say_hello_to` can be used like that:

```rust
//- Example
fn say_hello("Paul");
```

Additionally, if the option entrypoint in section [paths] of the Yarner.toml is set, unnamed blocks are excluded from code output. This can be useful to ignore e.g. simple command line usage examples intended to instruct the reader rather than for code output.

Hidden code

Sometimes, it can be useful to exclude code that is of limited interest for the reader from documentation output. This can be achieved by prefixing block names with hidden: (the default, configurable):

```rust
//- hidden:A hidden function
fn hidden() {

}
```

For code output, hidden blocks are treated like regular code blocks.

Only named code blocks can be hidden.

The prefixes hidden: and file: can be combined, but only in that order:

```rust
//- hidden:file:secrets.rs
fn hidden() {

}
```

Also note the features for Links and transclusions and for Copying files. It is not necessary to have all code in the main document, nor to have it in Markdown code blocks at all.

Links and transclusions

For larger projects, not only multiple code files are desirable, but also multiple Markdown source and/or documentation files. This chapter explains two features serving that purpose.

Link following

By prefixing relative links with @ (by default, configurable), Yarner can be instructed to follow these links and include linked files in the build process. E.g. to include file linked.md, it can be linked from the main file like this:

The file @[linked.md](linked.md) is also part of this project.

The prefix is stripped from documentation output. The above content is modified to

The file [linked.md](linked.md) is also part of this project.

Transclusions

A transclusion means that the content of an entire file is drawn into another file. Transclusions are achieved by wrapping a file path or a relative link into @{{<path>}}. Here are two examples of valid transclusions:

@{{path/to/file.md}}
@{{[file.md](path/to/file.md)}}

In the documentation output, the transclusion is replaced by the content of the referenced file.

During transclusion, unnamed code blocks are renamed to produce code in the same output file as if the file was not transcluded, but "compiled" directly. E.g. an unnamed code block in file transcluded.rs.md is be renamed to file:transcluded.rs. Note the prefix file:. See chapter Multiple code files for details.

Transclusions are processed before macro evaluation. Thus, code blocks from the transcluded document can be used in the transcluding document, and vice versa.

A transclusion should be the only thing in a line.

Transclusions are recursive, so transcluded files can also transculde other files themselves.

Link correction

Relative links in transcluded files are corrected to still point to the correct target. E.g., when including a file path/file.md into README.md, a link to file2.md in that file would become path/file2.md in the documentation output of README.md.

Copying files

Sometimes it may be desired to include code that is not part of the Markdown sources. Further, it may be necessary to include files in the documentation that are not processed by Yarner, like images. For such cases, Yarner can automatically copy files into code and documentation output.

Copying files

Section [paths] of a project's Yarner.toml provides options to list files and patterns for copying files unchanged, code_files and doc_files. Both accept a list of file names or glob patterns.

As an example, with the following setting Yarner copies all .png and .jpg files from directory img to docs/img:

doc_files = ["img/*.png", "img/*.jpg"]

Equivalently, one could automatically copy all Rust files from src to code/src like this:

code_files = ["src/*.rs"]

Modifying paths

In some cases it may be inconvenient to be forces to equal structure in sources and outputs. Through options code_paths and doc_paths, paths can be modified to some extent during processing.

Currently, only replacement of path components as well as omission of components are supported.

As an example, some files may be required to end up at the top level directory of the code output, but should not be at the top level of sources. In the following example, all files and folders from directory additional-code are copied directly into the code output folder:

code_files = ["additional-code/**/*"]
code_paths = ["-"]

When present, options code_paths and doc_paths must have as many entries as code_files and doc_files, respectively. Each entry of e.g. code_paths is applied to the corresponding entry in code_files.

Possible modifications:

Use foo/bar to replace the first two path component by foo and bar.
Use - (minus) to omit, and _ (underscore) to preserve a component.
Use a single _ (underscore) if no path change is intended at all.

Reverse mode

Programming inside Markdown code blocks may be inconvenient due to missing syntax highlighting and the lack of IDE support in general. Yarner offers a "reverse mode" that lets you edit generated code files, e.g. in an IDE, and to play back changes into the Markdown sources.

Usage

To use the reverse mode, run the following after making changes to generated code files:

> yarner reverse

Reverse mode requires settings for the target language(s) to be defined. See the following section.

Language settings

To enable the reverse mode, Yarner needs to label code blocks in the generated sources to identify their origin.

The Yarner.toml file provides a section where settings for multiple languages can be specified. The language is determined from the extension of the output file. The following example provides settings for Rust that would be applied to all .rs files.

[language.rs]
clear_blank_lines = true
eof_newline = true

    [language.rs.block_labels]
    comment_start = "//"
    # comment_end = "*/"
    block_start = "<@"
    block_next = "<@>"
    block_end = "@>"

In most cases, only the option comment_start needs to be adapted to the line comment sequence of the target language. E.g., Python requires the following:

[language.py]
comment_start = "#"
...

Option comment_end is provided for languages that support only block comments and should be left out in all other cases.

For details on the available options, see chapter Languages.

Multiple languages can be defined by simply adding one section per language. It is, however, not necessary to provide language settings for every file extension present. Files with no language settings for their extension are simply ignored during reverse mode.

Code block labels

Code in output intended for the reverse mode is labelled to allow Yarner to identify its file and code block of origin. You can edit everything between labels, but do not modify or delete the labels themselves!

As an example, a simple Markdown source file main.rs.md could have the following content:

# Simple example

The program's entry point:

```rust
fn main() {
    // ==> Say hello.
}
```

Here is how we say hello:

```rust
//- Say hello
println!("Hello World!");
```

With language settings for Rust as given above, the generated code in main.rs looks like this:

// <@main.rs.md##0
fn main() {
    // <@main.rs.md#Say hello#0
    println!("Hello World!");
    // @>main.rs.md#Say hello#0
}
// @>main.rs.md##0

Copied files

If files were copied as explained in chapter Copying files, Yarner detects these in reverse mode and copies them back. I.e. code in copied files can be modified just like code extracted from code blocks, but without the need to care for block labels.

Lock file

When reverse mode for a project is enabled (by providing the required language settings), a file Yarner.lock is created in the project's root. The file is required to prevent accidental overwrites of user edits in Markdown sources as well as code output. E.g., after editing the code output, Yarner will refuse to do a forward build as your changes would then be lost. To build the project nonetheless, run with option --force:

> yarner --force
> yarner --force reverse

The file Yarner.lock should be ignored by Version Control Systems (i.e. add Yarner.lock to your .gitignore).

Clean code output

For clean code output without block labels, run Yarner with option --clean:

> yarner --clean

Of course, the reverse mode does not work with clean output.

Limitations

Block repetitions

When the same code block is use by multiple macro invocations, it is ambiguous which one to play back into the sources. Here is an example:

fn main() {
    // ==> Say hello.
    // ==> Say hello.
}

In such cases, Yarner emits a warning when called with subcommand reverse. If the occurrences differ, like in the following example of user-modified code output, it aborts with an error.

// <@main.rs.md#
fn main() {
    // <@main.rs.md#Say hello#0
    println!("Hello World!");
    // @>main.rs.md#Say hello#0
    // <@main.rs.md#Say hello#0
    println!("Hello Universe!");
    // @>main.rs.md#Say hello#0
}
// @>main.rs.md#

Watch command

Yarner has a subcommand watch to automatically build a project after files changed. This may be particularly convenient when switching back and forth between Markdown editing and code editing with reverse mode.

Warning: This feature is still experimental and modifies the original Markdown sources. Make a backup of the sources before using it!

Usage

To start watching, run subcommand watch:

> yarner watch

Yarner will do one forward build and then watch the detected source files, as well as the generated code files, for changes. When source files change, Yarner runs a forward build. When code files change, it runs a reverse build.

The config file Yarner.toml is watched as a source file, thus a forward run will be done on changes.

To stop watching, press Ctrl + C.

Reverse pass

In watch mode, a reverse build is immediately followed by a forward build, but exclusively for the documentation output. This way, code changes are immediately reflected in the documentation.

Error handling

When watching a project, some errors are handled less strict in order to not abort watching unnecessarily. Instead of aborting with an error, a warning is printed. In particular, this applies to plugin errors.

Plugins

Plugins allow further functionality to be provided for Yarner. A plugin is an external executable that is called by Yarner during the build process, and that receives and returns the parsed documents in JSON format.

The following sections explain how to use and how to write plugins.

Further, a list of known plugins is provided.

Using plugins

This chapter explains how to use plugins.

How it works

Plugins are external programs that can modify the documents parsed by Yarner. They communicate with Yarner using JSON through stdin/stdout.

Configuration

A plugin program must be on the PATH to be usable.

To use a plugin for a project, add the respective section to the Yarner.toml file. E.g. for a plugin yarner-block-links, add

[plugin.block-links]

Plugin options follow after the section:

[plugin.block-links]
join = " | "

Multiple plugins can be combined. They will process the document in the order they are given, each receiving the output of its precursor.

With these settings, run Yarner a usual.

Plugins have no effect in reverse mode.

Command

By default, the command of the plugin is derived from its name in the config file, prefixed with yarner-. E.g., the command derived from [plugin.block-links] is yarner-block-links.

Alternatively, each plugin section can have optional parameters command and arguments, e.g.

[plugin.xyz]
command = "path/to/binary"
arguments = ["--arg1", "value1", "--arg2", "value2"]

Known plugins

A list of plugins known to the authors of Yarner.

yarner-bib allows for citations using a BibTeX bibliography.

yarner-toc adds a Table of Content to each document.

yarner-block-links adds jump links between referenced and referencing code blocks.

yarner-fold-code puts all code blocks into collapsed <details> tags.

Writing plugins

This chapter explains how to write plugins for Yarner, in Rust or any other language.

Basic workflow

Each plugin is called by Yarner during the build process. It receives all documents in their parsed state, after transclusions are performed and code is extracted,but before documentation is printed out. Each plugin should report back with the changed documents, and potentially added documents.

Rust library

To use the Rust crate yarner-lib to write plugins, add it to the dependencies of your Cargo.toml:

[package]
...

[dependencies]
yarner-lib = "0.5"
...

Besides the structs that make up a document, the library offer some convenience functions for JSON conversion. Here is an example plugin that adds a simple text paragraph to the end of each document:

use yarner_lib::*;

fn main() {
    std::process::exit(match run() {
        Ok(()) => 0,
        Err(err) => {
            eprintln!("ERROR: {}", err);
            1
        }
    });
}

fn run() -> Result<(), Box<dyn std::error::Error>> {
    // Get documents from stdin JSON
    let mut data = yarner_lib::parse_input()?;

    // Get options from the plugin's section in the Yarner.toml
    let _foo = data.context.config.get("foo").unwrap();

    // Manipulate documents
    for (_path, doc) in data.documents.iter_mut() {
        doc.nodes.push(Node::Text(TextBlock {
            text: vec![
                String::new(),
                String::from("Edited by an example plugin."),
            ],
        }));
    }

    // Convert documents back to JSON and print them to stdout
    yarner_lib::write_output(&data)?;
    Ok(())
}

See the Known plugins for more complex code examples.

JSON schema

For plugins in languages other than Rust, a JSON schema is provided in the GitHub repository, folder schemas. yarner-data.json describes the data passed from Yarner to plugins (context with config, as well as documents), and back to Yarner.

Configuration

All aspects of Yarner's syntax and most of its behaviour can be configured in the project's Yarner.toml file. The file is structured into sections parser, paths, and potentially multiple language sections. These sections are explained individually in the following chapters.

Some options can be overwritten by command line arguments. See chapter Command line arguments for details.

Parser

Section parser of a project's Yarner.toml contains all configuration options for Yarner's syntax within Markdown sources.

Overview

The default parser section as generated by yarner init looks like this (comments removed):

[parser]

fence_sequence = "```"
fence_sequence_alt = "~~~"

block_name_prefix = "//-"

macro_start = "// ==>"
macro_end = "."

transclusion_start = "@{{"
transclusion_end = "}}"

link_prefix = "@"

file_prefix = "file:"
hidden_prefix = "hidden:"

Options

Option	Details
`fence_sequence`	The sequence used for normal code blocks
`fence_sequence_alt`	Alternative fence sequence to allow for code blocks inside code blocks. Use this for the outer block
`block_name_prefix`	Prefix sequence to indicate a block's name in the first line of a code block
`macro_start` `macro_end`	Start and end of a macro invocation
`transclusion_start` `transclusion_end`	Start and end of a transclusion. E.g. `@{{transclude.md}}`
`link_prefix`	Prefix for links to make Yarner include the linked file in the build process. E.g. `@[Linked file](linked.md)`
`file_prefix`	Prefix to treat block names as target file specifiers. E.g. `//- file:main.rs`
`hidden_prefix`	Prefix to hide a code block in documentation output. E.g. `//- hidden:Secret code block`

Paths

Section paths of a project's Yarner.toml contains configuration on which files to process and to copy.

Overview

The default paths section as generated by yarner init looks like this (comments removed):

[paths]
root = "."
code = "code/"
docs = "docs/"

files = ["README.md"]

# entrypoint = "Main"

# code_files = ["**/*.rs"]
# code_paths = ["_"]

# doc_files = ["**/*.png", "**/*.jpg"]
# doc_paths = ["_"]

Options

Option	Details
`root`	The root directory all paths are relative to
`code`	Path for code output
`docs`	Path for documentation output
`files`	List of files to process
`entrypoint`	Block name to be used as entrypoint. Optional. Uses unnamed entrypoints if not present
`code_files`	Files or glob patterns for files to copy to the code output. Optional. See Copying files
`code_paths`	Manipulation of code copy paths. Optional. See Copying files
`doc_files`	Files or glob patterns for files to copy to the documentation output. Optional. See Copying files
`doc_paths`	Manipulation of documentation copy paths. Optional. See Copying files

Languages

Sections language.<lang> of a project's Yarner.toml contain language-specific settings mainly used for the Reverse mode.

Overview

Language settings are optional. However, they are required for all languages/file extensions to be used in reverse mode.

Language settings are a section per language, identified from file extensions. Each section looks like this example for Rust (.rs files):

[language.rs]
clear_blank_lines = true
eof_newline = true

    [language.rs.block_labels]
    comment_start = "//"
    # comment_end = "*/"
    block_start = "<@"
    block_next = "<@>"
    block_end = "@>"

When language settings are requires for setting clear_blank_lines or eof_newline, but block labels in the target language are not wanted or not supported, leave out section [block_labels].

Options

Option	Details
`clear_blank_lines`	Replaces lines containing only whitespaces by blank lines, in code output. Defaults to `true` when no language settings are present
`eof_newline`	Enforces code files to always end with a blank line. Defaults to `true` when no language settings are present
`[language.<lang>.block_labels]`	Settings for block labels for reverse mode (see table below). Optional. When absent, reverse mode for the language is disabled

Reverse mode

Section [language.<lang>.block_labels] is disabled by default. To enable reverse mode for a language, that section with the following settings is required.

Option	Details
`comment_start`	Start of comments in the language. Used for code block labels for reverse mode. Can be start of line or block comments
`comment_end`	End of comments. Optional, only for languages that support only block comments
`block_start`	Start sequence of block labels
`block_next`	Start of next block with the same name
`block_end`	End of block labels

Command line arguments

Some configurations can be overwritten using command line arguments.

CLI help

CLI help

To get help on command line options, use yarner -h.

Literate Programming tool for Markdown
  https://github.com/mlange-42/yarner

The normal workflow is:
 1) Create a project with
    > yarner init
 2) Process the project by running
    > yarner

USAGE:
    yarner [FLAGS] [OPTIONS] [FILES]... [SUBCOMMAND]

FLAGS:
    -C, --clean      Produces clean code output, without block label comments.
    -F, --force      Forces building, although it would result in overwriting changed files.
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -o, --code <path>          Directory to output code files to. Optional. Defaults to 'path -> code' from config file.
    -c, --config <path>        Sets the config file path [default: Yarner.toml]
    -d, --docs <path>          Directory to output documentation files to. Optional. Defaults to 'path -> docs' from
                               config file.
    -e, --entrypoint <name>    The named entrypoint to use when tangling code. Optional. Defaults to 'path ->
                               entrypoint', or to the unnamed code block(s).
    -r, --root <path>          Root directory. Optional. Defaults to 'path -> root' from config file, or to the current
                               directory.

ARGS:
    <FILES>...    The input source file(s) as glob pattern(s). Optional. Defaults to 'path -> files' from config
                  file.

SUBCOMMANDS:
    help       Prints this message or the help of the given subcommand(s)
    init       Creates a yarner project in the current directory
    reverse    Reverse mode: play back code changes into source files
    watch      Watch files and build project on changes

Advanced topics

This part of the guide provides more advanced information for effective work with Yarner, and with Markdown in general.

Chapter Continuous integration demonstrates how to automate Yarner using different CIs (GitLab, GitHub Actions, GitHub Travis-CI), and how to publish the documentation output.

Chapter Project structure presents some typical project layouts for different Yarner use cases.

Chapter Effective Markdown provides a list of tools for Markdown, as well as some tips how to use Markdown effectively.

Chapter mdBook with Yarner shows how to use Yarner in conjunction with mdBook to write good-looking Literate Programming documents (the book you are currently reading was created with mdBook).

Continuous integration

This chapter shows some examples how to use Yarner with Continuous integration, and how to publish Literate Programming documents.

GitHub with Travis-CI

GitHub Actions

GitLab CI

In your GitLab project, create a file .gitlab-ci-yml with the following content:

image: ubuntu:latest

variables:
  YARNER_VERSION: 0.3.0

before_script:
  - apt-get update; apt-get -y install curl
  - curl -L -o- https://github.com/mlange-42/yarner/releases/download/${YARNER_VERSION}/yarner-${YARNER_VERSION}-linux-amd64.tar.gz | tar xz
  - export PATH="$PWD:$PATH"

build:
  script:
    - yarner --clean
  artifacts:
    paths:
      - docs/
      - code/

You should use the latest Yarner version for variable YARNER_VERSION.

Project structure

Yarner offers flexibility regarding the structure of projects. This chapter presents some project layouts recommended for different use cases.

Settings

For an explanation of all settings that serve to configure the project structure, see chapter Paths. The most important settings are:

[paths]
root = "."
code = "code/"
docs = "docs/"

files = ["README.md"]

root is the path all other paths are relative to. code is for code output, and docs for documentation output. files are the entrypoint files of the project (normally one, but can be more, or even glob patterns).

Default structure

The default setup created by init uses the setting listed above and produces this project structure:

project
  │
  ├── code/              <code output>
  ├── docs/              <doc output>
  │
  ├── README.md          <source document>
  └── Yarner.toml        <config>

The structure is well suited for small projects that use only a single source file and no transclusions or links to be followed (@). In this case, the documentation output looks exactly like the sources. The source file (e.g. README.md) can thus be used as the documentation directly, and directory docs can be ignored by Git (in file .gitignore):

/docs/

Three folders structure

Another possible structure is to have the Markdown sources in a separate sub-folder, e.g. lp:

project
  │
  ├── code/              <code output>
  ├── docs/              <doc output>
  ├── lp/                <source files>
  │     ├── README.md
  │     └── ...
  │
  └── Yarner.toml        <config>

The required settings look like this:

[paths]
root = "lp/"
code = "../code/"
docs = "../docs/"

files = ["README.md"]

This layout is suitable for larger projects with potentially many linked or transcluded source files.

Here, an additional file README.md that contains no Literate Programming code can be placed at the top level of the project. The compiled documentation for readers (after transclusion, and with clean links) can be found in directory docs.

Top-level documentation structure

A further useful layout is to place the documentation output at the top project level, while the sources are in a sub-folder (here, lp):

project
  │
  ├── code/              <code output>
  ├── lp/                <source files>
  │     ├── README.md
  │     └── ...
  │
  └── Yarner.toml        <config>

The required settings look like this:

[paths]
root = "lp/"
code = "../code/"
docs = "../"

files = ["README.md"]

Here, the "compiled" documentation output (e.g. README.md) is placed directly in the project directory and thus presented to the reader.

This structure is useful for larger projects that use link following and transclusions, but still want the Literate Programming document directly presented to the reader, e.g. as the repository's README.md.

Effective Markdown

This chapter gives advice on how to use Markdown effectively, with Yarner and in general. It lists tools like editors and explains how to create rendered documents. Further, it provides tips like how to render math formulas with Markdown.

Why Markdown?

Markdown is a lightweight markup language for creating formatted text documents. It is designed for writing in a plaintext editor, and to be simple and well readable for humans in its source code form. With that, Markdown is easy to learn and use, as well as future-proof. To the minimal Markdown syntax, Yarner adds some simple syntax elements for Literate Programming that do not break rendering of the source documents.

Using different tools, Markdown documents can be easily converted to HTML or PDF, or to other text editing formats like LaTeX or even MS Word.

When working with Git and a Git forge like GitHub or GitLab, Markdown offers the additional advantage that .md files are rendered on repository pages. Thus, Markdown sources and the "compiled" documentation output can be presented there directly, or automatically published using e.g. GitHub Pages or GitLab Pages.

Markdown editors

Notepad++ is a versatile Open Source text editor. Syntax highlighting for Markdown is available as plugin

Atom is a powerful and extensible Open Source text editor. Markdown syntax highlighting is built-in. For a rendered Markdown preview, multiple plugins are available, e.g. Markdown Preview Enhanced.

Most IDEs support Markdown syntax highlighting, and some even a rendered preview (potentially via a plugin).

MarkText is a pure Markdown editor with "real-time preview" (WYSIWYG). It is Open Source, and designed to be minimalistic and distraction-free. It supports all Markdown features necessary for using it in Yarner projects.

Typora is similar to MarkText, but only freeware, not Open Source.

Markdown conversion

Pandoc is a command line tool for conversion between different markup formats. Besides Markdown, it supports a vast range of other formats for conversion in both directions: HTML, LaTeX, WikiText, MS Word, OpenOffice, LibreOffice, ...

mdBook is a command line tool to create online books from Markdown files (e.g. the book you are currently reading). For details on how to use it with Yarner, see chapter mdBook with Yarner.

Dedicated Markdown editors like MarkText and Typora provide functionality to export Markdown documents as HTML and PDF.

Math formulas

Different renderers and platforms support different ways to write math formulas.

GitLab

GitLab supports inline math, enclosed in $`...`$, e.g. $`E = m c^2`$.

Math blocks are possible in fenced code blocks with language math:

```math
a^2+b^2=c^2
```

GitHub

GitHub does unfortunately not support any math syntax in Markdown. As a workaround, GitHub's math rendering service can be used to include formulas as images.

<img src="https://render.githubusercontent.com/render/math?math=<formula>">
<img src="https://render.githubusercontent.com/render/math?math=E = m c^2">

MarkText

MarkText supports inline math, enclosed in $...$ , e.g. $E = m c^2$ . This is similar to GitLab, except for the missing backticks.

For math blocks, fenced code blocks with language math are supported, like in GitLab:

```math
a^2+b^2=c^2
```

Further, TeX-like math blocks can be used:

$$
a^2+b^2=c^2
$$

Pandoc

Pandoc supports TeX-like inline math, surrounded by $...$ as well as math blocks:

$$
a^2+b^2=c^2
$$

mdBook

mdBook requires to explicitly enable math support in the book.toml config file through

[output.html]
mathjax-support = true

Inline equations must be enclosed in \$...\$.

Math blocks use \\[ and \\] as delimiters:

\\[
a^2+b^2=c^2
\\]

Diagrams

Some editors and platforms support Mermaid graphs through code blocks with language mermaid:

```mermaid
graph TD;
    A-->B;
    A-->C;
    B-->D;
    C-->D;
```

Mermaid is supported by GitLab, MarkText, Typora, and by mdBook through a pre-processor.

mdBook with Yarner

mdBook is a command line tool to create online books from Markdown files (e.g. the book you are currently reading). This chapter explains how to use it together with Yarner.

Initialization

To create a project for mdBook and Yarner, run both with their sub-command init:

> yarner init
> mdbook init

Delete the file README.md created by Yarner (or better, fill it with a readme for the project). Further, the correct paths need to be set in file Yarner.toml. See the next section.

Settings

Change section [paths] in file Yarner.toml to use these options:

[parser]
...

[paths]
root = "lp/"
code = "../code/"
docs = "../src/"

files = ["SUMMARY.md"]
...

Project structure

The recommended structure as resulting from the above initialization and settings looks like this (some directories are initially missing):

project
  │
  ├── book/        <───────┐       <rendered book>
  │                        │
  ├── code/                │       <code output>
  │     └── ...         <──┼──┐
  │                        │  │
  ├── src/                 │  │
  │     ├── SUMMARY.md  ───┘  │    <doc output/book sources>
  │     └── capter-1.md <─────┤
  │                           │
  ├── lp/                     │
  │     ├── SUMMARY.md   ─────┘    <yarner sources>
  │     └── capter-1.md
  │
  ├── book.toml
  └── Yarner.toml

Directory lp contains the Markdown source files. Write these files as you would normally write mdBook files in directory src. The only difference is that entries in SUMMARY.md that contain Literate Programming code are prefixed with @ (for link-following). As an example, lp/SUMMARY.md could look like this:

# Summary

* @[Chapter 1](./chapter-1.md)
* @[Chapter 2](./chapter-2.md)

From the Markdown sources in lp, Yarner creates files for mdBook in directory src, as well as extracted code in directory code:

> yarner

Finally, mdBook uses the files in src to create the HTML website in directory book:

> mdbook build

Yarner Documentation