Configuration Management with Drupal Distributions

At 1xINTERNET we have been specializing in Drupal Distributions for many years. We have built several distribution products for clients, that allow them to independently operate websites on the same code basis. We build all new web projects off our internal distribution called 1xDX, which contains all features a modern digital experience platform needs.

The hard part about Drupal Distributions is not creating them, but allowing for project specific changes, and still being able to update the distribution.

For controlling updates we have decided to use the Configuration Synchronizer module and the family of its dependencies.

In this blog post we will show what approaches exist, and why we chose ours. We will then give an example how this works for updating a distribution with changed project configuration.

A simple use case

Let's look at the use case: a lot of projects have a content type for news. The requirements are typically the same: preview image, preview text, main content, publication date, some categorisation and author. To make this functionality reusable, the configuration could be added to a module in a distribution. As of Drupal 8 the installation of configuration from a module has become really easy. Simply add all necessary fields, form displays, view displays, node type configs, and image styles to module's config/install folder. Everytime a new project is installed with the distribution the functionality will be available.

Usually, after a while the requirements change. New fields need to be added to the content type for news for example.

Adding the new configurations to the config/install folder is still easy, and new installations would receive the new configuration. The difficulties arise when the existing projects also need this change.

Available options

Before we decided for our approach to updating Drupal Distributions we investigated the different options available. 

Write update/post update hooks: Adding fields programmatically is technically simple. However, looking at the use case above, this quickly becomes complicated, because it requires changing field storage, field instance, and form display and view display. With more configuration objects it could be even more complex. Also it somehow seems to be double work, because for clean installations the configuration management was already done, and now it needs to be implemented again.

Use drush/drupal console to partially load configs from the module's config/install folder: This method prevents double work, because updates are not programmed manually (yay!). But there is an important drawback. Partial configurations imports will replace the configuration in the active storage with the exact data from config/install folder. But what if there were changes made. Maybe the display of fields was changed to better support a different design? Maybe fields were added for a specific use case? Or some formatters/widgets were changed? All the customizations would be lost (nay!).

Use modules like Update Helper from Thunder: This module allows to create the updates with drush commands and place them into the config/update folder. There is also an integration with checklist API to have a visual representation of the updates and their statuses in case they were not applied as well as instructions on how to apply updates manually in case of failure. The module provides safe methods to update your configurations, but in case of the manual changes made in the project, the updates might fail and would need to be applied manually.

For us none of the approaches above met our requirements:

• We need a solution that allows us to not lose changes done on different projects.
• The updates should be easy to create and maintain, they should work the same for clean installations and existing sites.
• And we need logging of the update statuses with instructions on how to perform the updates manually in case of failures.

Configuration synchronizer

During our investigation we found the Configuration Synchronizer module and the family of its dependencies. This module provides plug 'n' play experience with config management for distributions, contributed or custom modules, and even themes and profiles. Detailed information about it can be found in this blogpost and in the rest of the series of stories by Nedjo Rogers, the creator and maintainer of the modules.

The module creates the snapshots of the normalized configurations provided by extensions, stores them for comparison during the updates, and finds the differences (with help of filters from Config Distro module). Then during an update configurations are processed with 3-way merge (using the Config Merge module) that allows to leave project specific changes untouched. Only in the case when the exact same configuration line changed there would be conflict (similar to Git merge procedures). In case of a conflict the configuration update is ignored and will not be applied.

The only thing not covered in this approach is how to deal with ignored updates. For this we use a similar approach like the Update Helper module described above. We also use a checklist, but instead of checking the config/update folder, we check the results from the configuration merge that was performed by Configuration Synchronizer. 

We decided to create update hooks for this. The modules that provide changes in the config/install folder provide update hooks that are responsible for checking the status of the configuration change.

A working example:

As in the use case described above, we want to add a new field to the news content type. Let’s call this field “News Example” with the machine name `field_news_example`.

After adding the field we add all configurations in the config/install folder of the module that provides the news content type. Now, we create the update.

First the update is documented:

/**
* Adds new field to news content type.
*
* The field needs to be added because editors need it.
*
* @success The field was added successfully.
* @failure The field was not added. Please try to add it yourself <a href="/admin/structure/types/manage/news">here</a>. Create a field of type text plain type and call it "News example".
*/


The structure of the php doc is following:

1. The first line contains a title that appears in the list.
2. The rest is a short and informative description.
3. Under @success tag a success message for the update is added.
4. Under @failure a description is added on how to manually do the update with a link where to add the configuration.

New tags can be easily added with phpdocumentor/reflection-docblock library.

Now the update hook is implemented:

function xi_news_update_8001() {
// If we are actually running the update.
if (\Drupal::state()->get('system.maintenance_mode')) {
// Create update status tracker.
\Drupal::service('xi_updater.check_updates')->createUpdate(__FUNCTION__);
} else {
// Perform the check here.
/** @var \Drupal\Core\Entity\EntityFieldManagerInterface $field_manager */
$field_manager = \Drupal::service('entity_field.manager');
$fields = $field_manager->getFieldDefinitions('node', 'news');
if (!empty($fields['field_news_example'])) {
return TRUE;
} else {
return FALSE;
}
}
}

Inside the update function it is important to separate real updates and the check of the status. This is done by checking if the site is in "maintenance mode" with this condition: `\Drupal::state()->get('system.maintenance_mode')`. 

If it is TRUE the updates are currently running. If it is FALSE a function is called to check the status, so it is important to return boolean value. In this function the checks whether the update worked as expected are performed. In the example above we check if the new field `field_news_example` was created.

If the checks returns TRUE, the update entity is marked as successful and checklist api is updated. In the administration UI you will see the status of the update with success message and checked item.

This update status check runs automatically on the event from the config distro module that is fired after successful configuration synchronization.

In case any of the updates failed, there is status indication in administration UI with instructions how to fix it. When fixes are done the corresponding checklist items can manually be marked as successful.

After that the module will check the status of the update again and either confirm or again show the message with more instructions. Additionally, it is also possible to mark the update as successful even though it was not applied, in case you don’t need it and don’t want to apply it manually.

Final thoughts and Drupal 9 porting weekend

We like the unobtrusive way to display successful and failed updates with Checklist API. By the way, any updates can be wrapped in a status checker as shown above. This approach is not related to configuration changes.

Of course our solution is not perfect and no panacea for every problem one might face with configuration management, but we constantly try to improve it and are open for suggestions.

As a part of Drupal 9 porting weekend 1xINTERNET became the official supporter of configuration synchronizer and its dependencies! Read more about our contribution efforts here. If you want to help and make updates even smoother get in touch through our website or the issue queues!