Project Setup

Project Setup

Part of the functionality of DrWatson is creating and navigating through a project setup consistently. This works even if you move your project to a different location/computer or send it to a colleague with a different Julia installation. In addition, the navigation process is identical across any project that uses DrWatson.

This can "just work" (TM) because of the following principles:

  1. Your science project is also a Julia project defined by a Project.toml file. This way the project tracks the used packages (and their versions) and can be shared with any other Julia user.
  2. You first activate this project environment before running any code. This way you ensure that your project run on the specified package installation (instead of the global one)See Activating a Project for ways to do this.
  3. You use the functions scriptdir, datadir, etc. from DrWatson to navigate your project (see Navigating a Project).

Default Project Setup

DrWatson suggests a universal project structure for any scientific project, which is the following:

│projectdir          <- Project's main folder. It is initialized as a Git
│                       repository with a reasonable .gitignore file.
│
├── _research        <- WIP scripts, code, notes, comments,
│   |                   to-dos and anything in an alpha state.
│   └── tmp          <- Temporary data folder.
│
├── data             <- **Immutable and add-only!**
│   ├── sims         <- Data resulting directly from simulations.
│   ├── exp_pro      <- Data from processing experiments.
│   └── exp_raw      <- Raw experimental data.
│
├── plots            <- Self-explanatory.
├── notebooks        <- Jupyter, Weave or any other mixed media notebooks.
│
├── papers           <- Scientific papers resulting from the project.
│
├── scripts          <- Various scripts, e.g. simulations, plotting, analysis,
│   │                   The scripts use the `src` folder for their base code.
│   └── intro.jl     <- Simple file that uses DrWatson and uses its greeting.
│
├── src              <- Source code for use in this project. Contains functions,
│                       structures and modules that are used throughout
│                       the project and in multiple scripts.
│
├── test             <- Tests for code in src.
│   └── runtests.jl  <- Script to run all tests.
│
├── README.md        <- Optional top-level README for anyone using this project.
├── .gitignore       <- by default ignores _research, data, plots, videos,
│                       notebooks and latex-compilation related files.
│
├── Manifest.toml    <- Contains full list of exact package versions used currently.
└── Project.toml     <- Main project file, allows activation and installation.
                        Includes DrWatson by default.

src vs scripts

Seems like src and scripts folders have pretty similar functionality. However there is a distinction between these two. You can follow these mental rules to know where to put file.jl:

Initializing a Project

To initialize a project as described in the Default Project Setup section, we provide the following function:

DrWatson.initialize_project — Function.
initialize_project(path [, name]; kwargs...)

Initialize a scientific project expected by DrWatson inside the given path. If its name is not given, it is assumed to be the folder's name.

The new project remains activated for you to immidiately add packages.

Keywords

  • readme = true : adds a README.md file.
  • authors = nothing : if a string or container of strings, adds the authors in the Project.toml file.
  • force = false : If the path is not empty then throw an error. If however force is true then recursively delete everything in the path and create the project.
  • git = true : Make the project a Git repository.
source

Activating a Project

This part of DrWatson's functionality requires you to have your scientific project (and as a consequence, the Julia project) activated. This can be done in multiple ways:

  1. doing Pkg.activate("path/to/project") programmatically
  2. using the startup flag --project path when starting Julia
  3. by setting the JULIA_PROJECT environment variable
  4. using the functions quickactivate and findproject offered by DrWatson.

We recommend the fourth approach, although it does come with a caveat (see the docstring of quickactivate).

Here is how it works: the function quickactivate activates a project given some path by recursively searching the path and its parents for a valid Project.toml file. Typically you put this function in your script files like so:

using DrWatson # DONT USE OTHER PACKAGES HERE!
quickactivate(@__DIR__, "Best project in the WOLLRDD")
# Now you should start using other packages

where the second optional argument can assert if the activated project matches the name you provided. If not the function will throw an error.

DrWatson.quickactivate — Function.
quickactivate(path [, name::String])

Activate the project found by findproject of the path, which recursively searches the path and its parents for a valid Julia project file.

Optionally check if name is the same as the activated project's name. If it is not, throw an error.

This function is first activating the project and then checking if it matches the name.

Warning

Note that to access quickactivate you need to be using DrWatson. For this to be possible DrWatson must be already added in the existing global environment. The version of DrWatson loaded therefore will be the one of the global environment, and not of the activated project. To avoid unexpected behavior take care so that these two versions coincide.

In addition please be very careful to not write:

using DrWatson, Package1, Package2
quickactivate(@__DIR__)
# do stuff

but instead load packages after activating the project:

using DrWatson
quickactivate(@__DIR__)
using Package1, Package2
# do stuff

This ensures that the packages you use will all have the versions dictated by your activated project (besides DrWatson, since this is impossible to do using quickactivate).

source
DrWatson.findproject — Function.
findproject(path = pwd()) -> project_path

Recursively search path and its parents for a valid Julia project file (anything in Base.project_names). If it is found return its path, otherwise issue a warning and return nothing.

The function stops searching if it hits either the home directory or the root directory.

source

Notice that to get the current project's name you can use:

DrWatson.projectname — Function.
projectname()

Return the name of the currently active project.

source

Navigating a Project

To be able to navigate the project consistently, DrWatson provides the core function

DrWatson.projectdir — Function.
projectdir()

Return the directory of the currently active project. Ends with "/".

projectdir(folder::String) = joinpath(projectdir(), folder)*"/"

Return the directory of the folder in the active project.

source

Besides the above, the shortcut functions:

datadir()
srcdir()
plotsdir()
scriptdir()
papersdir()

immediately return the appropriate subdirectory. These are also defined due to the frequent use of these subdirectories.

In addition, the return value of all these functions ends with /. This means that you can directly chain them with a file name using just *. E.g. you could do

using DrWatson
file = makesimulation()
tagsave(datadir()*"sims/test.bson", file)

Reproducibility

The project setup approach that DrWatson suggests is designed to work flawlessly with Julia standards, to be easy to share and to be fully reproducible. There are three reasons that true reproducibility is possible:

  1. The project's used packages are embedded in the project because of Project.toml
  2. The navigation around the folders of the project uses local directories.
  3. The project is a Git repository, which means that it has a detailed (and re-traceable) history of all changes and additions.

If you send your entire project folder to a colleague, they only need to do:

julia> cd("path/to/project")
pkg> activate .
pkg> instantiate

to use your project. All required packages and dependencies will be installed and then any script that was running in your computer will also be running in their computer in the same way!

In addition, with DrWatson you have the possibility of "tagging" each simulation created with the commit id, see the discussion around current_commit and tag!. This way, any data result obtained at any moment can be truly reproduced simply by resetting the Git tree to the appropriate commit and running the code.