Restarting a Conformational Search

How to resume an interrupted iMTD-GC or entropy-mode conformational search from its last checkpoint.

CREST 3.1

The checkpoint mechanism

Conformational searches can run for a long time, and a run that is killed by a walltime limit, a node failure, or a stray Ctrl+C used to mean starting over from scratch. Since CREST 3.1, the iMTD-GC and the entropy mode workflows write a small checkpoint file called crest.restart at every stage boundary of the algorithm.

The checkpoint is a plain text file recording where the algorithm was, not what it found. No ensemble data is stored in it. The actual structures stay in the backup files (.cre_*.xyz) that CREST writes anyway after each metadynamics iteration:

# CREST restart checkpoint - do not edit manually
version 1
runtype 2
main_iter 0
mtd_iter 2
nmetadyn 12
stage mtd_loop
last_file .cre_2.xyz
elowest        -34.376274418000000
eprivious        -34.376274418000000

The stage entry is the important one. It can take the following values:

`stage`	meaning
`mtd_trj`	an MTD trajectory was written, but its (re-)optimization is incomplete
`mtd_loop`	an MTD iteration finished completely, including sorting
`post_collect`	the MTD loop is done and all ensembles were collected and sorted
`entropy_smtd`	entropy mode only: the static MTD (sMTD) part has finished
`done`	the run completed; the file is removed

Because the checkpoint refers to files in the working directory, a restart must be started from the same directory as the original run, with the same input structure.

Important: Do not edit crest.restart by hand and do not delete the .cre_*.xyz backup files. If the ensemble file referenced by last_file is missing, CREST will stop with an error and ask you to delete crest.restart and start over.

Example

We use the alanineglycine molecule from the standard conformational search example again. The search is started as usual:

crest struc.xyz --gfn2 --alpb h2o -T 4 > crest.out

Assume the job is killed during the third metadynamics iteration. The working directory then contains the backup ensembles of the completed iterations together with the checkpoint file:

$ ls
.cre_1.xyz  .cre_2.xyz  coord  crest.restart  crest.out  struc.xyz  ...

To resume, simply issue the same command again in the same directory:

crest struc.xyz --gfn2 --alpb h2o -T 4 >> crest.out

CREST detects the checkpoint and reports it right after the header:

  :::::::::::::::::::::::::::::::::::::::::::::::::::::::::
  RESTART DETECTED (crest.restart)
   runtype  : iMTD-GC
   stage    : mtd_loop
   MTD iter : 2 (MAINLOOP 0)
   last file: .cre_2.xyz
   elowest  :        -34.3762744180
  :::::::::::::::::::::::::::::::::::::::::::::::::::::::::

The already completed metadynamics iterations are skipped, the lowest structure found so far is restored as the new reference geometry, and the search continues with iteration 3. Once the run finishes successfully, crest.restart is deleted automatically, so no stale checkpoint is left behind for the next calculation in that directory.

Note: The restart resumes at the granularity of the algorithm’s stages. Work done within an unfinished stage (for example, the multilevel optimization of a trajectory that was interrupted halfway) is repeated. If the interruption happened after the MTD loop, at the post_collect stage, CREST picks up directly at the final optimization steps.

Disabling the restart

The checkpointing is active by default. It only writes a few hundred bytes and does not change the results of a completed run, but it can be turned off entirely with --norestart:

crest struc.xyz --gfn2 --alpb h2o -T 4 --norestart

With this flag no crest.restart file is written, and an existing one is ignored. Use it if you deliberately want to start a fresh search in a directory that still contains files from a previous, interrupted run.