Creating custom Julia Packages

Abstract

This post covers the following topics:

• Generating custom Julia package.
• Generating testing and documentation workflow for the package.
• Pushing the packlage to GitHub with GitHub Actions enabled for automatic deployment of documentation and unit testing.
• GitHub Actions Badges.

Introduction

This post will introduce (succinctly), how to create a package from scratch in Julia Programming Language.

Before starting, I’ll make a few assumptions:

• Julia Programming Language is installed. If not, please download and follow the instructions provided on the official website.
• You’re familiar with Julia REPL.
• You’re familiar with Git and GitHub.

First, generate a “starter pack” with Julia’s in-built package manager using the command generate <PKG_NAME>:

(@v1.8) pkg> generate jac
  Generating  project jac:
    jac/Project.toml
    jac/src/jac.jl

This will create a package jac with structure as follows:

jac/
├─ src/
│  └─ jac.jl
├─ Project.toml

Often you would like to keep this package as a remote repository so that others can use your code. For this, I’ll create a remote GitHub Reporsitory named after the package JAC. After cloning this GitHub Repository, I’ll move the contents of folder generated by Julia Package Manager into the local repository folder. Finally, the package structure will look like this:

jac/
├─ .git/
├─ src/
│  └─ jac.jl
├─ .gitignore
├─ LICENSE
├─ Project.toml
└─ README.md

Documentation

Let’s create documentation using Documenter and DocumenterTools packages. First, install these two packages:

(@v1.8) pkg> add Documenter
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.8/Project.toml`
  No Changes to `~/.julia/environments/v1.8/Manifest.toml`

(@v1.8) pkg> add DocumenterTools
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.8/Project.toml`
  No Changes to `~/.julia/environments/v1.8/Manifest.toml`

Now using these two packages, generate the documentation:

julia> using Documenter

julia> using DocumenterTools

julia> DocumenterTools.generate()
[ Info: name of package automatically determined to be `jac`.
[ Info: deploying documentation to `~/dev/jac/docs`
[ Info: Generating .gitignore at /Users/rvn/dev/jac/docs/.gitignore
[ Info: Generating make.jl at /Users/rvn/dev/jac/docs/make.jl
[ Info: Generating Project.toml at /Users/rvn/dev/jac/docs/Project.toml
[ Info: Generating src/index.md at /Users/rvn/dev/jac/docs/src/index.md

This will create a new folder named docs and the package structure will look like:

jac/
├─ .git/
├─ docs/
│  ├─ src/
│  │  └─ index.md
│  ├─ Project.toml
│  └─ make.jl
├─ src/
│  └─ jac.jl
├─ .gitignore
├─ LICENSE
├─ Project.toml
└─ README.md

Let’s now add our development package jac to the path so that Julia can find it:

(@v1.8) pkg> dev .
   Resolving package versions...
    Updating `~/.julia/environments/v1.8/Project.toml`
  [e91179df] + jac v0.1.0 `../../../dev/jac`
    Updating `~/.julia/environments/v1.8/Manifest.toml`
  [e91179df] + jac v0.1.0 `../../../dev/jac

What this does is let is us import jac as a library anywhere!

using jac

Now, let’s generate the documentation by executing make.jl file inside docs/src/ folder:

~/dev/jac/docs git:(master)
julia make.jl
[ Info: SetupBuildDirectory: setting up build directory.
[ Info: Doctest: running doctests.
[ Info: ExpandTemplates: expanding markdown templates.
[ Info: CrossReferences: building cross-references.
[ Info: CheckDocument: running document checks.
[ Info: Populate: populating indices.
[ Info: RenderDocument: rendering document.
[ Info: HTMLWriter: rendering HTML pages.

I’ve decided to document the development of this library and also provide information on how to use this library. For this, I’ve created two separate sections on the website:

• Development: It’s just a folder with same name inside docs/src/, and contains different articles explaining how the code was developed.
• Examples: It’s just a folder with same name inside docs/src/, and contains different articles explaining how code can be re-used.

To make this work I’ve also modified make.jl file:

using Documenter
using jac

makedocs(
    sitename = "jac",
    format = Documenter.HTML(),
    modules = [jac],
    pages = [
        "Home" => "index.md",
        "Development Blogs" => Any[
            "Development/Scalar_AD.md",
        ],
        "Examples" => Any[
            "Examples/Simple_Scalar_Arithmetics.md",
        ],
    ],
)

# Documenter can also automatically deploy documentation to gh-pages.
# See "Hosting Documentation" and deploydocs() in the Documenter manual
# for more information.
# deploydocs(
#     repo = #URL
# )

Automatic Testing

For this, I’ll use Test package which can be installed using Julia Package Manager. But first, activate jac (navigate into jac/ folder):

(@v1.8) pkg> activate .
  Activating project at `~/dev/jac`

Now, install Test package:

(jac) pkg> add Test
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `~/dev/jac/Project.toml`
  [8dfed614] + Test
    Updating `~/dev/jac/Manifest.toml`
  [2a0f44e3] + Base64
  [b77e0a4c] + InteractiveUtils
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [8dfed614] + Test

This will add Test as dependency in the Project.toml file.

To use automatic testing, I’ll have to create a folder test/ with runtests.jl file.

~/dev/jac git:(master)
mkdir test

~/dev/jac git:(master)
touch test/runtests.jl

Code for testing goes in runtests.jl file, where I import Test package and then use macro @test to run test statements:

using Test

@test true # Trivial true

Package Structure:

jac/
├─ .git/
├─ docs/
│  ├─ src/
│  │  ├─ Development
│  │  |     ├─
│  |  ├─ Examples
│  │  |     ├─
│  │  └─ index.md
│  ├─ Project.toml
│  └─ make.jl
├─ src/
│  └─ jac.jl
├─ test/
│  └─ runtests.jl
├─ .gitignore
├─ LICENSE
├─ Project.toml
└─ README.md

Now, I can automate testing using a single command test inside Julia Package Manager:

(jac) pkg> test
     Testing jac
      Status `/private/var/folders/46/gg5gxwf52hq4wqs0q04ll0wh0000gn/T/jl_P0oqm9/Project.toml`
  [e91179df] jac v0.1.0 `~/dev/jac`
  [8dfed614] Test `@stdlib/Test`
      Status `/private/var/folders/46/gg5gxwf52hq4wqs0q04ll0wh0000gn/T/jl_P0oqm9/Manifest.toml`
  [e91179df] jac v0.1.0 `~/dev/jac`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [8dfed614] Test `@stdlib/Test`
     Testing Running tests...
     Testing jac tests passed

GitHub Actions

GitHub Actions can automatically run tests and documenter everytime I push code to the GitHub Repository.

To do this, first create .github/ folder with Documenter.yml and Runtests.yml files (these are the config files for GitHub actions):

Documenter.yml

name: Documenter
on:
  push:
    branches:
      - master
    tags: '*'
  pull_request:
jobs:
  build:
    runs-on: ubuntu-latest # OS Support
    steps:
      - uses: actions/checkout@v2
      - uses: julia-actions/setup-julia@latest
        with:
          version: '1.8.3' # Julia Version
      - name: Install dependencies
        run: julia --project=docs/ -e 'using Pkg; Pkg.develop(PackageSpec(path=pwd())); Pkg.instantiate()'
      - name: Build and deploy
        env:
          GITHUB_TOKEN: $ # For authentication with GitHub Actions token
          DOCUMENTER_KEY: $ # For authentication with SSH deploy key
        run: julia --project=docs/ docs/make.jl

Runtests.yml

name: Runtests
on: [push, pull_request]
jobs:
  test:
    runs-on: $
    strategy:
      matrix:
        julia-version: ['1.8.3'] # Julia Version
        julia-arch: [x64] # 64 bit Architecture Support
        os: [macOS-latest, ubuntu-latest] # OS Support
    steps:
      - uses: actions/checkout@v2
      - uses: julia-actions/setup-julia@latest
        with:
          version: $
      - uses: julia-actions/julia-buildpkg@latest
      - uses: julia-actions/julia-runtest@latest

Package Structure:

jac/
├─ .git/
├─ .github/
│  └─ workflows/
│     ├─ Documenter.yml
│     └─ Runtests.yml
├─ docs/
│  ├─ src/
│  │  ├─ Development
│  │  |     ├─
│  |  ├─ Examples
│  │  |     ├─
│  │  └─ index.md
│  ├─ Project.toml
│  └─ make.jl
├─ src/
│  └─ jac.jl
├─ test/
│  └─ runtests.jl
├─ .gitignore
├─ LICENSE
├─ Project.toml
└─ README.md

Now, I have to deploy keys for GitHub Actions. First, generate the keys using DocumenterTools package:

julia> DocumenterTools.genkeys()
┌ Info: add the public key below to https://github.com/$USER/$REPO/settings/keys with read and write access
└ the title can be left empty as GitHub can infer it from the key comment

ssh-rsa WILL HAVE KEY HERE Documenter

[ Info: add a secure environment variable named 'DOCUMENTER_KEY' to https://travis-ci.com/$USER/$REPO/settings (if you deploy using Travis CI) or https://github.com/$USER/$REPO/settings/secrets (if you deploy using GitHub Actions) with value:

WILL HAVE KEY HERE

Follows the next steps:

• Copy the ssh key (starting with ssh-rsa) in Settings/Deploy keys on the GitHub repository’s website (figure below for details)

  

  Figure 1: Adding Documenter Key to GitHub.

• Copy the secret key in Settings/Secrets/Actions on the GitHub repository’s website (figure below for details)

  

  Figure 2: Adding Secret Key to GitHub.

One last thing before pushing the changes to remote repository, we have to modify docs/make.jl file to include the URL to the GitHub repository:

using Documenter
using jac

makedocs(
    sitename = "jac",
    format = Documenter.HTML(),
    modules = [jac]
)

# Documenter can also automatically deploy documentation to gh-pages.
# See "Hosting Documentation" and deploydocs() in the Documenter manual
# for more information.
deploydocs(
    repo = "github.com/tgautam03/jac.git" # URL to Repo
)

Now, push the changes to GitHub.

Final Touches

In the remote repository, I can now see the Tests and Documenter running inside Actions tab. I can also add Badges so that I don’t have to look inside Actions tab every time to check the status (this is done inside Actions/Documenter and Actions/Runtests using triple dots on the top right corner).

Once the documentation is deployed, I can check the URL on GitHub Repo itself (in my case it’s https://tgautam03.github.io/jac/dev/).

References

• Jaan’s blog post