Saving Tools

This page discusses numerous tools that can significantly improve process of saving & loading files, always in a scientific context.

These tools are also used in the examples demonstrated in the Real World Examples page. After reading the proper documentation here it might be worth it to have a look there as well!

In DrWatson we save and load files with the functions wsave(filename, data) and wload(filename). These functions are further used in the tools below, like e.g. tagsave and can be overloaded for your own specific datatype.

In addition, wsave ensures that mkpath is always called on the path you are trying to save your file at. We all know how unpleasant it is to run a 2-hour simulation and save no data because FileIO.save complains that the path you are trying to save at does not exist...

To overload the saving part, add a new method to DrWatson._wsave(filename, ::YourType, args...; kwargs...) (notice the _!). By overloading _wsave you get all the extra functionality of tagsave, safesave, etc., for free for your own types (tagsave requires that you save your data as a dictionary, or extend tag! for your own type).

Saving and loading fallback

By default we fallback to FileIO.save and FileIO.load for and types. This means that you have to install yourself whatever saving backend you want to use. FileIO by itself does not install a package that saves data, it only provides the interface!

The suffix of the file name determines which package will be used for actually saving the file. It is your responsibility to know how the saving package works and what input it expects!

Safely saving data

Almost all packages that save data by default overwrite existing files (if given a save name of an existing file). This is the default behavior because often it is desired.

Sometimes it is not though! And the consequences of overwritten data can range from irrelevant to catastrophic. To avoid such an event we provide an alternative way to save data that will never overwrite existing files:

DrWatson.safesave — Function

safesave(filename, data...; kwargs...)

Safely save data in filename by ensuring that no existing files are overwritten. Do this by renaming already existing data with a backup-number ending like #1, #2, .... For example if filename = test.jld2, the first time you safesave it, the file is saved normally. The second time the existing save is renamed to test_#1.jld2 and a new file test.jld2 is then saved.

If a backup file already exists then its backup-number is incremented (e.g. going from #2 to #3). For example safesaving test.jld2 a third time will rename the old test_#1.jld2 to test_#2.jld2, rename the old test.jld2 to test_#1.jld2 and then save a new test.jld2 with the latest data.

Any additional keyword arguments are passed through to wsave (to e.g. enable compression).

Tagging a run using Git

For reproducibility reasons (and also to not go insane when asking "HOW DID I GET THOSE RESUUUULTS") it is useful to "tag" any simulation/result/process using the Git status of the repository.

To this end we have some functions that can be used to ensure reproducibility:

DrWatson.tagsave — Function

tagsave(file::String, d::AbstractDict; kwargs...)

First tag! dictionary d and then save d in file.

"Tagging" means that when saving the dictionary, an extra field :gitcommit is added to establish reproducibility of results using Git. If the Git repository is dirty and storepatch=true, one more field :gitpatch is added that stores the difference string. If a dictionary already contains a key :gitcommit, it is not overwritten, unless force=true. For more details, see tag!.

Keywords gitpath, storepatch, force, are propagated to tag!. Any additional keyword arguments are propagated to wsave, to e.g. enable compression.

The keyword safe = DrWatson.readenv("DRWATSON_SAFESAVE", false) decides whether to save the file using safesave.

source

DrWatson.@tagsave — Macro

@tagsave(file::String, d::AbstractDict; kwargs...)

Same as tagsave but one more field :script is added that records the local path of the script and line number that called @tagsave, see @tag!.

source

The functions also incorporate safesave if need be.

Low level functions

@tagsave internally uses the following low level functions:

DrWatson.tag! — Function

tag!(d::AbstractDict; kwargs...) -> d

Tag d by adding an extra field gitcommit which will have as value the gitdescribe of the repository at gitpath (by default the project's gitpath). Do nothing if a key gitcommit already exists (unless force=true then replace with the new value) or if the Git repository is not found. If the git repository is dirty, i.e. there are un-commited changes, and storepatch is true, then the output of git diff HEAD is stored in the field gitpatch. Note that patches for binary files are not stored. You can use isdirty to check if a repo is dirty. If the commit message is set to true, then the dictionary d will include an additional field "gitmessage" and will contain the git message associated with the commit.

Notice that the key-type of the dictionary must be String or Symbol. If String is a subtype of the value type of the dictionary, this operation is in-place. Otherwise a new dictionary is created and returned.

To restore a repository to the state of a particular git commit do:

checkout the relevant commit with git checkout xyz where xyz is the value stored
(optional) apply the patch git apply patch, where the string stored in the gitpatch field needs to be written to the file patch.

Keywords

gitpath = projectdir()
force = false
storepatch = DrWatson.readenv("DRWATSON_STOREPATCH", false): Whether to collect and store the output of gitpatch as well. By default it is false.
kw...: extra keywords are propagated to gitdescribe.

Examples

julia> d = Dict(:x => 3, :y => 4)
Dict{Symbol,Int64} with 2 entries:
  :y => 4
  :x => 3

julia> tag!(d; commit_message=true)
Dict{Symbol,Any} with 3 entries:
  :y => 4
  :gitmessage => "File set up by DrWatson"
  :gitcommit => "96df587e45b29e7a46348a3d780db1f85f41de04"
  :x => 3

source

DrWatson.@tag! — Macro

@tag!(d, gitpath = projectdir(), storepatch = true, force = false) -> d

Do the same as tag! but also add another field script that has the path of the script that called @tag!, relative with respect to gitpath. The saved string ends with #line_number, which indicates the line number within the script that @tag! was called at.

Examples

julia> d = Dict(:x => 3)Dict{Symbol,Int64} with 1 entry:
  :x => 3

julia> @tag!(d) # running from a script or inline evaluation
Dict{Symbol,Any} with 3 entries:
  :gitcommit => "618b72bc0936404ab6a4dd8d15385868b8299d68"
  :script => "test\stools_tests.jl#10"
  :x      => 3

source

DrWatson.gitdescribe — Function

gitdescribe(gitpath = projectdir(); dirty_suffix = "-dirty", warn = true) -> gitstr

Return a string gitstr with the output of git describe if an annotated git tag exists, otherwise the current active commit id of the Git repository present in gitpath, which by default is the currently active project.

If the repository is dirty when this function is called the string will end with the value of the keyword dirty_suffix. When this happens, the keyword warn = DrWatson.readenv("DRWATSON_WARN_DIRTY", true) will trigger a warning to be printed if it is true (the default).

Return nothing if gitpath is not a Git repository, i.e. a directory within a git repository.

The format of the git describe output in general is

`"TAGNAME-[NUMBER_OF_COMMITS_AHEAD-]gLATEST_COMMIT_HASH[-dirty]"`

If the latest tag is v1.2.3 and there are 5 additional commits while the latest commit hash is 334a0f225d9fba86161ab4c8892d4f023688159c, the output will be v1.2.3-5-g334a0f. Notice that git will shorten the hash if there are no ambiguous commits.

More information about the git describe output can be found on (https://git-scm.com/docs/git-describe)

Produce or Load

produce_or_load is a function that very conveniently integrates with savename to either load a file if it exists, or if it doesn't to produce it, save it and then return it!

This saves you the effort of checking if a file exists and then loading, or then running some code and saving, or writing a bunch of if clauses in your code. In addition, it attempts to minimize computing energy spent on getting a result.

DrWatson.produce_or_load — Function

produce_or_load(f::Function, config, path = ""; kwargs...) -> data, file

The goal of produce_or_load is to avoid running some data-producing code that has already been run with a given configuration container config. If the output of some function f(config) exists on disk, produce_or_load will load it and return it, and if not, it will produce it, save it, and then return it.

Here is how it works:

The output data are saved in a file named name = filename(config). I.e., the output file's name is created from the configuration container config. By default, this is name =savename(config), but can be configured differently, using e.g. hash, see keyword filename below. See also produce_or_load with hash codes for an example where config would be hard to put into name with savename, and hash is used instead.
Now, let file = joinpath(path, name).
If file exists, load it and return the contained data, along with the global path that it is saved at (file).
If the file does not exist then call data = f(config), with f your function that produces your data from the configuration container.
Then save the data as file and then return data, file.

The function f should return a string-keyed dictionary if the data are saved in the default format of JLD2.jl., the macro @strdict can help with that.

You can use a do-block instead of defining a function to pass in. For example,

produce_or_load(config, path) do config
    # code using `config` runs here
    # and then returns a dictionary to be saved
end

Keywords

Name deciding

filename::Union{Function, String} = savename : Configures the name of the file to produce or load given the configuration container. It may be a one-argument function of config, savename by default, so that name = filename(config). Useful alternative to savename is hash. The keyword filename could also be a String directly, possibly extracted from config before calling produce_or_load, in which case name = filename.
suffix = "jld2", prefix = default_prefix(config) : If not empty, added to name as name = prefix*'_'*name*'.'*suffix (i.e., like in savename).

Saving

tag::Bool = DrWatson.readenv("DRWATSON_TAG", istaggable(suffix)) : Save the file using tagsave if true (which is the default).
gitpath, storepatch : Given to tagsave if tag is true.
force = false : If true then don't check if file exists and produce it and save it anyway.
loadfile = true : If false, this function does not actually load the file, but only checks if it exists. The return value in this case is always nothing, file, regardless of whether the file exists or not. If it doesn't exist it is still produced and saved.
verbose = true : print info about the process, if the file doesn't exist.
wsave_kwargs = (;) : Keywords to pass to wsave (e.g. to enable compression). Defaults to an empty named tuple.
wload_kwargs = (;) : Keywords to pass to wload (e.g. to pass a typemap). Defaults to an empty named tuple.

source

DrWatson.@produce_or_load — Macro

@produce_or_load(f, config, path; kwargs...)

Same as produce_or_load but one more field :script is added that records the local path of the script and line number that called @produce_or_load, see @tag!.

Notice that path here is mandatory in contrast to produce_or_load.

source

DrWatson.istaggable — Function

istaggable(file::AbstractStrig) → bool

Return true if the file save format (file ending) is "taggable", i.e. allows adding additional data fields as strings. Currently endings that can do this are:

("bson", "jld", "jld2")

source

istaggable(x) = x isa AbstractDict

For non-string input the function just checks if input is dictionary.

source

See Stopping "Did I run this?" for an example usage of produce_or_load. While produce_or_load will try to by default tag your data if possible, you can also use it with other formats. An example is when your simulation function f returns a DataFrame and the file suffix is "csv". In this case tagging will not happen, but produce_or_load will work as expected.

Converting a struct to a dictionary

savename gives great support for getting a name out of any Julia composite type. To save something though, one needs a dictionary. So the following function can be conveniently used to directly save a struct using any saving function:

DrWatson.struct2dict — Function

struct2dict([type = Dict,] s) -> d

Convert a Julia composite type s to a dictionary d with key type Symbol that maps each field of s to its value. Simply passing s will return a regular dictionary. This can be useful in e.g. saving:

tagsave(savename(s), struct2dict(s))

source

DrWatson.struct2ntuple — Function

struct2ntuple(s) -> n

Convert a Julia composite type s to a NamedTuple n.

source