From idea to feature, what is the life cycle of an open source product?

As you know, Kuzzle is an open source product whose entire code is freely available on Github.

To continue this open approach, we have decided to make our working methods "open source" so that everyone can understand the processes we use every day to move our projects forward.

Throughout this article we will follow the steps of developing a real feature!

Find ideas

There is no shortage of ideas. In order not to overload Kuzzle with features that are rarely or never used, each idea must be accompanied by a concrete use case.

Everyone is free to offer a feature, whether it is a team member, a customer or the community.

It all starts with a request from one of our customers for Slack
that wanted a direct access to ES through Kuzzle.

All these ideas are then added to the backlog of our Product Workshop.

Study the need

Once every two weeks, the team meets to discuss the ideas submitted to the product workshop for 1 hour.

We use Trello to list ideas and a script to randomly draw a card.

Proposal of the feature in our Product Workshop.

The person who submitted the idea must then explain the potential functioning of the feature and the use case related to it.

The purpose of this workshop is to ensure two things: Utility and Feasibility.

If the feature meets these two criteria, the card is moved to the "Processed" column and a User Story is added to the Product Backlog in Jira.
Otherwise the card is moved to the "Won't Do" column.

A feature refused because of its non-feasibility in a cluster environment.

Archiving is important in order to keep track of which features have been approved or not!

Prioritize tasks

The third step in the life of a feature is its prioritization during Backlog Refinement.

Every two weeks, we analyze our Kanban board in Jira to prioritize the tasks to be performed in the next two weeks.

View of the Product Backlog. On the right, the task accepted from the Product Workshop

This prioritization is carried out in a collegial manner, in consultation with the entire team.

We take into account the needs of our customers but also those of the community through our public roadmap.

The tasks thus prioritized are moved to the "To do" column.

Develop the feature

Once a task has been prioritized in the "To Do" column, everyone is free to assign it to themselves to complete it.

This is part of our team's self-management approach, where everyone organizes their work as they wish, according to the needs around the product.

Create a feature branch

We use Gitflow for all our developments, which means that each feature is developed on a new branch created from the current development branch.

Architecture of a Git repository with Gitflow

Test Driven Development

The first step is to write the functional tests using Cucumber's Gherkin syntax.

Thus it will be possible to test the functioning of the code during the development phase without having to perform the actions by hand or via a "disposable" script”.

Write the code and the unit tests

Then, it is the writing of the code of the feature itself.

Special care must be taken when processing user input to prevent security breaches and also when modifying critical code areas that can potentially be executed several hundred times per second.

Finally, unit tests must be written to ensure maximum future maintainability.

Write the documentation

The documentation for each project is located within the repository and must therefore be updated at the same time as the feature is added.

If the new documentation includes code examples, they should also be tested with our documentation testing framework.

In addition to the ease of reviewing a pull request including code and documentation, this allows us to ensure that the deployment of a feature is always accompanied by the associated documentation.

Pull Request with Continuous Integration & Deployment

Once all the necessary work has been done, a pull request is created.

Pull Request of the feature on the Kuzzle repository.

It must follow the standard format to facilitate its review by team members.

Each pull request must be approved by 3 team members to ensure the best possible quality of the new product code.

Finally, each step of our CI must be validated in order for the pull request to be definitively validated.

Our CI works with Travis and includes the following steps for pull requests:

Build and deploy documentation on Netlify
Code quality analysis with ESlint, SonarQube and LGTM
Unit tests and code coverage measurement with Mocha and Codecov
Functional tests with Cucumber

Once the CI has passed and the team has approved, the pull request is merged into the development branch with a squash merge, as for description a summary of the feature which is used for the Git history.

Deploy to pre-production

The features delivered on the development branch are then deployed in pre-production.

This automatic deployment includes the following steps:

Build and push images on Dockerhub (kuzzleio/kuzzle:2-dev and kuzzleio/plugin-dev:2-dev)
Deployment of documentation on https://next-docs.kuzzle.io/

Release the new version

We regularly prepare the delivery of a new version including the latest features developed.

To do this, we use an internal tool that will automatically perform the different steps:

Generation of the changelog of the new version
Creation of a pull request from the development branch to the stable branch
Creating a tag with the new version and the associated Github release

Changelog of the release including the new feature.

Then, as during the pre-production deployment, our CI will take care of deploying the production images on Dockerhub as well as the documentation and NPM packages.