Managing requirements
RASCIL requirements are stored in three files:
requirements.in
Python requirements for the main code base
requirements-test.in
Python requirements to run the tests
requirements-docs.in
Python requirements to build the documentation
pip-compile
is used to generate the corresponding .txt files. pip-compile
resolves
all dependencies and saves them with their resolved versions in the .txt files.
This method is used to make sure we do not update requirements with every build,
but rather install them from the .txt files, where they are pinned. We also have to
make sure we regularly update these versions, by running pip-compile
on the
.in files, which ideally do not contain version pins.
Manually updating the requirements
The Makefile
of RASCIL contains three options to work with requirements
on your local machine:
make requirements
This will update the requirements in the .txt file, but will not install them
make install_requirements
This will install the existing requirements from the .txt files, but not update them
make update_requirements
This will first update all requirements, then install them (i.e it runs the first two commands)
The first and third commands change the .txt files, but do not commit the changes. Still, it is worth running them from a branch, and not directly from master.
Process automation
Regularly updating the requirements manually is prone to be forgotten, which can result in packages being out-of-date very quickly. Hence we set up a semi-automatic process using the GitLab CI pipeline with a job run on a schedule.
The scheduled pipeline only runs one job, with the following steps:
run
make requirements
check if there are changes compared to the existing remote files
if there, create and check out a new branch
commit and push the changes to the new branch
create a Merge Request (MR) of the new branch into the source branch
assign the MR
if there aren’t any changes, do nothing
The tests are not run as part of this pipeline, because the MR created at the end of will have the tests run as part of its own pipeline.
The assignee now has the responsibility of keeping track how the pipeline of this new MR does. If it succeeds, then it should be merged to master. If it fails, then the failing tests should be checked and the reasons for failure should be fixed. Packages should not be pinned within the .in files, just because tests are failing, unless there is a very good reason for it. Packages pinned in the .in files should be regularly revisited and if possible, unpinned.