A bit of our experience trying to migrate pipelines from GoCD to Concourse-CI

Recently one of our clients asked us to set up a team to migrate all of their pipelines (which live in GoCD) to ConcourseCI. We won't go into the reasons behind that decision, but we'll tell you how the process we adopted went, what the challenges were, and why, after one month, we concluded that for their context the migration wasn't recommended.

Note: The team was made up of just three people: me, Melina Deraldo and Marco Valtas.

Note 2: We used version 3.9.1, so some of the problems we're going to describe may have been fixed or improved in newer versions.

The Beginning

Since we had very little time for this migration (the client wanted to go to production with their new platform already using the migrated pipelines, and that would happen in about 2 months) and none of us had worked with Concourse before, we decided to go as Lean as possible. For that we created a board on Trello and listed the main activities we had to do, such as:

1. List the pipelines to be migrated
2. Compare Concourse features with GoCD
3. Spin up a Concourse instance on AWS

• With the first activities we were able to list some of Concourse's characteristics and features:

  1. In Concourse everything is based on airline concepts, so some components are named things like ATC (Air Traffic Control), which is the web interface and build scheduler.
  2. It uses a CLI called fly, which is responsible for running builds on the ATC.
  3. EVERYTHING runs inside Docker containers, and I mean EVERYTHING.
  4. The web interface is easy to use and understand.
  5. Pipeline configuration is done entirely in YAML files.
  6. We can have pipelines separated by teams.
  7. Its documentation covers almost everything you need to build your pipelines.
  8. It already has some community-built plugins that make it easier to extend pipelines.

With a basic understanding of how Concourse works, we spun up two EC2 instances on AWS using Terraform. One instance with the web Docker container (it holds the whole Concourse interface and it's what the workers talk to) and another with a Docker container running a worker (responsible for running the pipelines, it talks directly to the web container).

The fact that everything uses Docker containers helped us a lot and made the process of getting it up on AWS fast.

With our CI up and running, we moved on to the next step: the pipelines.

Creating pipelines in Concourse

If you're already familiar with GoCD, you might know Gomatic. With it you can do all the configuration of your GoCD environment using Python.

In Concourse our only option was to work with YAML to create the pipelines, which by itself isn't a bad thing, since it's pretty easy to handle even for people who have never worked with this kind of file.

The only way to interact with and configure Concourse is through its CLI Fly, and that means you'll need to have the Fly binary somewhere, and it has to be run every time the pipelines change.

Creating a pipeline is pretty simple:

1. Spin up the Concourse server.
2. Create your pipeline YAML. (Pipeline Examples)
3. Use Fly to create your pipeline on the server.
4. Done, now you have a pipeline ready and running. Simple, right?

Well... now let's get to a few small problems with Concourse.

So... the problems.

Since life isn't all roses, we started having problems when our first pipeline grew into more than one and when we started having dependencies between pipelines.

Some small problems we ran into:

A special thanks to Melina, who put this list together with me.

• The only way to create pipelines in Concourse is through its CLI (fly cli).

• The only way to interact with and configure Concourse is also through fly, which connects to it and generates the configuration.

• Concourse has the concept of teams, which is a wonderful feature, but you can only create new teams using its CLI.

• When we create a new pipeline using the fly CLI, all of its content is shown in plain text, which means that if a credential is changed, the old one will be shown.

• Everything has to run in Docker containers, so when you need a pipeline to build a Docker image you'll have to work with Docker inside Docker.

• Its dashboards start getting very hard to understand when we have a large number of jobs. There's currently an effort to improve them, but it's still in development

• Concourse describes itself as a unified pipeline model, but the way it really works is as a series of independent jobs.

• You can't run a pipeline with an old version of your repository, unless you pin the repository version in the pipeline.yml file and reconfigure all your pipelines (https://github.com/concourse/concourse/issues/413).

• In Concourse we don't have the concept of upstream and downstream. The only way to run a job is manually or with some update (check in/commit) to a repository. You can set conditions so that a job runs as a dependency on the success of a previous job, but that job won't be triggered by the previous job.

• All build state has to be passed from job to job in the form of a * resource* that needs to be stored in an external resource (like git). Concourse is stateless; the only thing that keeps state in Concourse is resources, which, as mentioned, need to be versioned and stored in an external data store.

• A Concourse task has to run inside a Docker container (the official documentation says this isn't necessary when you specify the platform, but when we don't use a Docker image an error is thrown: https://github.com/concourse/concourse/issues/1154)

• Concourse "workers" often end up in a state called "stalled". We tried several kinds of configuration on our Amazon EC2 instances, but the problem kept happening (it might be due to a bad configuration in our Terraform files, but we couldn't identify the cause or fix the problem).

Conclusion

At the end of the experiment, my conclusion is that however promising the idea behind Concourse is, the tool itself isn't mature enough to be used in organizations that have many complex pipelines with dependencies between them.

Keep in mind that the tool is still relatively new and has an active community on GitHub and Slack, so it may evolve quickly.