Dear CTDAS users, We hope you’re all doing well! We would like to inform you that we have restructured our CTDAS code base on GIT. This new structure is completely in python3 and follows the setup of the 7 classes defined in the 2017 GMD paper. So e.g. instead of having a project-related folders like “TM5” or “STILT”, we now have folders for obsoperators and statevectors. In the current master code on GIT, we have included baseclasses and the code for the standard CarbonTracker Europe CO2 version (labeled cteco2).

Other previous projects have been moved to a temporary “archive” folder, from which each of you can take your project specific code and move it into the new structure. We would like to encourage all users to make use of the GITlab repository as much as possible, so that we can all benefit from each other’s work there. Note that moving your code to the new structure is very little work, it is a matter of moving files to the directories and changing the path for the import of the functions.

We have included further information below on the new structure and how to work with it on GIT. Please take a look, and let us know in case you have further questions. We hope that with this step, we can enhance the collaboration between all of us on this code base, and we will therefore also send you an invitation to our slack channel code-ctdas, so that we can keep in touch there with CTDAS code related questions.

Kind regards,

Ingrid, Wouter and Auke

===================================================

The new structure The code has been updated to python3, and the structure is as follows: The 7 components of the CTDAS system each have their own folder within the da folder.

• cyclecontrol - dasystems
• platform
• statevectors - observations - obsoperators - optimizers Each of the 7 main folders contains the cteco2 specific code, and in some cases, there is a baseclass module from which project specific code can inherit the properties (e.g. in optimizers). Further folders are:
• pipelines (this contains the pipelines that are called in the main ctdas.py script, currently implemented for the cteco2 setup)
• analysis (here we have standard analysis tools, plus a subfolder for cteco2)
• tools
• preprocessing
• rc
• doc
• archive (this contains all code that was previously in project-folders, this can be moved into the main new code, further details below). Furthermore, in the main CTDAS folder, there are the start_ctdas and clone_ctdas shell scripts, and the main ctdas.py, ctdas.jb and ctdas.rc files in the templates folder. Your project In case you already had your project specific CTDAS code on GIT, please read this section on how to update to the new structure. New (CTDAS and/or GIT) users can read the sections below. Note that we prefer users to work on branches in the main code, rather than on independent forks, so that we can profit for each other’s developments. Please follow these steps (two examples below for fork or branch):

1. Create a new temporary branch to make your project-specific code in the new structure
2. Get your changes from your original project-specific branch/fork into the new temporary structure branch
3. Merge request from temporary branch to the master You can check if you are on the main code or a fork using: git remote -v In case you are on the main code, you will get something like: origin https://git.wur.nl/ctdas/CTDAS.git In case you are on a fork, you will get something like: origin https://git.wur.nl/woude033/CTDAS.git (fetch) origin https://git.wur.nl/woude033/CTDAS.git (push) upstream https://git.wur.nl/ctdas/CTDAS.git (fetch) upstream https://git.wur.nl/ctdas/CTDAS.git (push) You can check on which branch you are as follows: git branch The output will look like: master * feature-STILT Example: if you are working on a fork, start at step 0, else, if you are working in a branch of the main code (https://git.wur.nl/ctdas/CTDAS.git) e.g. called feature-STILT start at step 1.
4. In case you are working on a fork: Make a new folder locally and get a new clone from the master of the main code: git clone https://git.wur.nl/ctdas/CTDAS
5. Make a new temporary branch on the main code: git checkout -b temp_stilt_new_structure remotes/origin/master
6. If you were working in a fork (else skip to step 3): merge your fork as a new temporary branch in the main code. Go to https://git.wur.nl/ctdas/CTDAS and go to ‘merge requests’, and ‘click new merge request’. On the left, fill in your fork and on the right the main CTDAS code and your new temporary branch temp_stilt_new_structure.
7. Copy a file from your previous branch to the new temporary branch: git checkout feature-STILT stilt/optimizer.py
8. Move this file to the correct place in the new structure: git mv stilt/optimizer.py optimizers/optimizer_stilt.py
9. Make sure everything is working with this new code, and then: git add . git commit -m “Added STILT optimizer” git push
10. After you finish to implement and test all your changes: Go to https://git.wur.nl/ctdas/CTDAS and go to ‘merge requests’, and ‘click new merge request’.

====================================================

Some useful GIT commands:

1. Check if you’re on the master branch:
   git branch
2. If you’re not on the master branch, switch to it:
   git checkout master
3. Switching to another existing branch:
   git checkout feature-STILT
4. If you want to compare your code to the master:
   git fetch
git status
5. To get the changes from the remote master:
   git pull
6. Moving a file:
   git mv stilt/optimizer.py optimizers/optimizer_stilt.py
7. Committing changes:
   git add .
git commit -m “A message” git push
8. If you want to replace your local master code with the remote master code, follow
   these steps (warning! Your previous version will be lost!): git fetch origin
   git reset --hard origin/master
9. Get changes to the master code in your branch:
   git merge origin/master

Experienced CTDAS users, but new to GIT:

Follow the first steps for new users (see below) to get your copy of the current CTDAS code from GIT in a new folder. Then compare the changes in your local code and merge your changes into the new structure (see description above) in your temporary branch. It might happen that you want to create a copy of an existing file on GIT, that you adapt to your project. There is no git cp command, so to create a copy with GIT history, follow these steps: cp optimizers/optimizer_cteco2.py optimizers/optimizer_ctesamco2.py git add optimizers/optimizer_ctesamco2.py git commit -m “Added CTE South America CO2 optimizer” git push

Python3

To update your code to python3, the tool 2to3 is very convenient. See for more details: https://docs.python.org/3/library/2to3.html New CTDAS users Make a folder, then: git clone https://git.wur.nl/ctdas/CTDAS Make your temporary branch to develop your code: git checkout -b temp_stilt_new_structure remotes/origin/master (Later on, you can do a merge request to merge your changes into the master branch). The main code is found in the da folder. You will find subfolders for each of the components of the CTDAS system (see information above about the structure). Furthermore, we have a script to start a new run, as follows: ./start_ctdas.sh /somedirectory/ name_of_your_run This will create a new folder for your run, and makes a copy of the da folder. Furthermore, the main ctdas.py will be created, named name_of_your_run.py, along with the main jb and rc file. These are based on the templates in the template folder. In this way, you will always have the current version of the code with your run. For further information, please refer to: https://www.carbontracker.eu/ctdas/

Installation as a site-package

In principle no installation is needed in order to do CTDAS runs. However, for testing purposes it may be desirable to be able to import this from another location in the filesystem, for example your project or anlysis folders. The following files

pyproject.toml
setup.cfg

allow you to do

cd CTDAS
pip install --user . 

Which will do an install of the current repository. Note the installed version is not updated automatically, so if you make changes in you CTDAS project, it will not be updated in the site-packages.

Which version does my CTDAS-run use, the pip installed version or the version coppied to the run directory start_ctdas.sh script? It will run the coppied version. Python traverses the PYTHONPATH searching for imports, and the current working directory comes first. This may look somewhat inefficient from a software perspective; and inconvinient as updating this directroy from the CTDAS repository needs to be done manually, but it is actually by design: the CTDAS code used by the run is automatically archived with the run. This means that the data-assimiliation expiriment performed is fully transparant and traceble. This is desired from a scientific reproducibility perspective.