Part of the functionality of DrWatson is creating and navigating through a project setup consistently. This works even if you move your project to a different location/computer or send it to a colleague with a different Julia installation. In addition, the navigation process is identical across any project that uses DrWatson.
This can "just work" (TM) because of the following principles:
- Your science project is also a Julia project defined by a
Project.tomlfile. This way the project tracks the used packages (and their versions) and can be shared with any other Julia user.
- You first activate this project environment before running any code. This way you ensure that your project runs on the specified package installation (instead of the global one). See Activating a Project for ways to do this.
- You use the functions
datadir, etc. from DrWatson to navigate your project (see Navigating a Project).
Importantly, our suggested project setup was designed to be fully reproducible, see Reproducibility.
DrWatson suggests a universal project structure for any scientific project, which is the following:
│projectdir <- Project's main folder. It is initialized as a Git │ repository with a reasonable .gitignore file. │ ├── _research <- WIP scripts, code, notes, comments, │ | to-dos and anything in an alpha state. │ └── tmp <- Temporary data folder. │ ├── data <- **Immutable and add-only!** │ ├── sims <- Data resulting directly from simulations. │ ├── exp_pro <- Data from processing experiments. │ └── exp_raw <- Raw experimental data. │ ├── plots <- Self-explanatory. ├── notebooks <- Jupyter, Weave or any other mixed media notebooks. │ ├── papers <- Scientific papers resulting from the project. │ ├── scripts <- Various scripts, e.g. simulations, plotting, analysis, │ │ The scripts use the `src` folder for their base code. │ └── intro.jl <- Simple file that uses DrWatson and uses its greeting. │ ├── src <- Source code for use in this project. Contains functions, │ structures and modules that are used throughout │ the project and in multiple scripts. │ ├── README.md <- Optional top-level README for anyone using this project. ├── .gitignore <- by default ignores _research, data, plots, videos, │ notebooks and latex-compilation related files. │ ├── Manifest.toml <- Contains full list of exact package versions used currently. └── Project.toml <- Main project file, allows activation and installation. Includes DrWatson by default.
scripts folders have pretty similar functionality. However there is a distinction between these two. You can follow these mental rules to know where to put
- If upon
include("file.jl")there is anything being produced, be it data files, plots or even output to the console, then it should be in
- If it is functionality used across multiple files or pipelines, it should be in
srcshould only contain files that define functions or types but not output anything.
To initialize a project as described in the Default Project Setup section, we provide the following function:
initialize_project(path [, name]; kwargs...)
Initialize a scientific project expected by
path (directory representing an empty folder). If
name is not given, it is assumed to be the folder's name.
The new project remains activated for you to immidiately add packages.
readme = true: adds a README.md file.
authors = nothing: if a string or container of strings, adds the authors in the Project.toml file and README.md.
force = false: If the
pathis not empty then throw an error. If however
truethen recursively delete everything in the path and create the project.
git = true: Make the project a Git repository.
placeholder = false: Add hidden place holder files in each default folder to ensure that project is maintained when the directory is cloned. Should be used only when
git = true. Will throw a warning if used with
git = false
This part of DrWatson's functionality requires you to have your scientific project (and as a consequence, the Julia project) activated. This can be done in multiple ways:
- using the startup flag
--project pathwhen starting Julia
- by setting the
- using the functions
findprojectoffered by DrWatson.
We recommend the fourth approach, although it does come with a caveat (see the docstring of
quickactivate(path [, name::String])
Activate the project found by recursively searching the
path and its parents for a valid Julia project file. Optionally check if
name is the same as the activated project's name. If it is not, throw an error. See also
@quickactivate. Do nothing if the project found is already active, or if no project file is found.
using DrWatson quickactivate("path/to/project", "Best project in the WOLRLDD")
Notice that this function is first activating the project and then checking if it matches the
Note that to access
quickactivate you need to be
using DrWatson. For this to be possible
DrWatson must be already added in the existing global environment. The version of
DrWatson loaded therefore will be the one of the global environment, and not of the activated project. To avoid unexpected behavior take care so that these two versions coincide.
In addition please be very careful to write:
using DrWatson # YES quickactivate(@__DIR__) using Package1, Package2 # do stuff
instead of the erroneous:
using DrWatson, Package1, Package2 # NO! quickactivate(@__DIR__) # do stuff
This ensures that the packages you use will all have the versions dictated by your activated project (besides
DrWatson, since this is impossible to do using
If given a
Symbol then first
quickactivate(@__DIR__, string(ProjectName)), and then do
using ProjectName, as if the symbol was representing a module name.
This ties with Making your project a usable module functionality, see the docs for an example.
findproject(path = pwd()) -> project_path
path and its parents for a valid Julia project file (anything in
Base.project_names). If it is found return its path, otherwise issue a warning and return
The function stops searching if it hits either the home directory or the root directory.
Notice that to get the current project's name you can use:
Return the name of the currently active project.
Notice that the project initialized by DrWatson does not represent a Julia package. It represents a scientific project. That being said, it is often the case that you want to develop normal Julia Modules (and perhaps later publish them as packages) inside your project, so that you can later use them in your code with
using ModuleName. The proper way to do this is to initialize Julia packages, using the package manager, inside the
src folder, using these steps:
- Active your project that uses DrWatson.
- Change directory to the project's main folder (important!).
- Go into package mode and initialize a package with the name that you want:
devthe local path to
ModuleNameusing the package manager:
dev src/ModuleName. Notice that this command uses a local path, see this PR for more details.
- If you don't care to make this module a Julia package, simply delete its
- If you do care about publishing this module as a Julia package, then it is mandatory to keep it as git-repository. In this case it is sensible to put
src/ModuleName/.gitinto the main
- If you don't care to make this module a Julia package, simply delete its
Now whenever you do
using ModuleName, the local version will be used. This will still work even if you transfer your project to another computer, because the Manifest.toml file stores the local path.
To be able to navigate the project consistently, DrWatson provides the core function
Return the directory of the currently active project.
projectdir(args...) = joinpath(projectdir(), args...)
Join the path of the currently active project with
args (typically other subfolders).
Besides the above, the following derivative functions
datadir() srcdir() plotsdir() scriptsdir() papersdir()
behave exactly like
projectdir but have as root the appropriate subdirectory. These are also defined due to the frequent use of these subdirectories.
All of these functions take advantage of
joinpath, ensuring an error-free path creation that works across different operating systems. It is heavily advised to use
projectdir and derivatives by giving them the subpaths as arguments, instead of using multiplication between paths:
datadir("foo", "test.bson") # preferred datadir() * "/foo/test.bson" # not recommended
It is straightforward to make custom directory functions if there is a directory you created that you access more often. Simply define
customdir(args...) = projectdir("custom", args...)
to make the
customdir version that works exactly like e.g.
datadir but for
"custom" instead of
The project setup approach that DrWatson suggests is designed to work flawlessly with Julia standards, to be easy to share and to be fully reproducible. There are three reasons that true reproducibility is possible:
- The project's used packages are embedded in the project because of
- The navigation around the folders of the project uses local directories.
- The project is a Git repository, which means that it has a detailed (and re-traceable) history of all changes and additions.
If you send your entire project folder to a colleague, they only need to do:
julia> cd("path/to/project") pkg> activate . pkg> instantiate
to use your project (assuming of course that you are both using the same Julia installation and version). All required packages and dependencies will be installed and then any script that was running in your computer will also be running in their computer in the same way!
In addition, with DrWatson you have the possibility of "tagging" each simulation created with the commit id, see the discussion around
tag!. This way, any data result obtained at any moment can be truly reproduced simply by resetting the Git tree to the appropriate commit and running the code.
If you already have an existing project with scripts and data etc., then there is no reason to use the
initialize_project function. The only requirement is that everything that belongs to your project is contained within a single folder (which can have an arbitrary amount of subfolders). If your project is already a Julia project (which means it has its own Project.toml and Manifest.toml files), then there is nothing more necessary to be done, you can immediatelly start using DrWatson with it. Although we recommend following the Default Project Setup, you don't have to do this either, since you can create your own Custom directory functions.
If your project is not also a Julia project, the steps necessary are still quite simple. You can do:
julia> cd("path/to/project") pkg> activate . pkg> add Package1 Package2 ...
Julia will automatically make the Project.toml and Manifest.toml files for you as you add packages used by your project.