Browse --- Chat --- Wekan

Skip to content

Fully fluid cluster first roadmap

Transition to ansible overall success.

Overall general things to do in no particular order:

  • First iteration Gitlab provisioning in ansible (mainly copy+paste shell scripts)
    • Install
    • Restore
    • Plugins
  • As many idempotent plays in ansible playbooks
  • Dynamically changing Kubernetes control plane/master node on server swapping
  • Using the consul and kubernetes cluster modules in ansible
  • Migrating as much installed software/services to containers
    • Consul
    • Databases - Redis/Mongo/Postgres
  • Sharding/Clustering databases
    • mongo / redis / postgres
  • Start using helm chart to provision Gitlab
    • Shooting for full 0 downtime gitlab migration with minimal configuration by doing 1 -> 2 -> 1 nodes
  • Integrate kubernetes with cloud kubernetes services for quick on-demand scaling

With the above list in mind, other general concepts/ideas that should naturally take shape.

  • Most "roles" should be gone as most things should be migrated to containers.
  • No such thing as "admin" role
    • Dynamic/multiple kubernetes control planes across nodes
    • Gitlab done via helm/kubernetes
  • No such thing as "db" role
    • Once DBs fully in containers with proper cluster data storage handling, provisioning should feel seamless
  • Either fully deprecate docker swarm or every node part of swarm

Naturally some tasks come first before others.

After looking at the requirements in order to run Gitlab using kubernetes/helm charts, it actually increases our cloud footprint 3x. The goal was to actually lower it by spreading it across smaller servers getting more cores and the same ram/customizable ram (3x 2core-2gb for example) and allowing for HA and easy migrations.

The omnibus installation requires
https://docs.gitlab.com/ee/install/requirements.html

  • 4 cores is the recommended minimum number of cores and supports up to 500 users
  • 4GB RAM is the required minimum memory size and supports up to 500 users

While the Gitlab helm chart installation requires
https://docs.gitlab.com/charts/installation/

  • A Kubernetes cluster, version 1.16 or higher. 8vCPU and 30GB of RAM is recommended.

30 GB ram is literally 3-4x the standard 4core-8gb most cloud providers provide.

EDIT: Hmmm, according to this page we can lower that for non-prod systems
Guess we can toy around with it in a minimal test cluster
https://docs.gitlab.com/charts/installation/deployment.html#cpu-and-ram-resource-requirements

This puts us in a rough spot in having 3 nodes our standard unless we can shrink our memory footprint below 4gb. Otherwise to have HA for our apps and DBs will require 2 additional 2core-2gb flat. A possibility is having the server with gitlab installed not be a kubernetes control plane. Along with other memory saving efforts, I believe we can get around 4gb, but staying below that will be a challenge.

We can now safely migrate mongo and our apps with 0 downtime across nodes seamlessly, but gitlab will still require some downtime. In this case we look into attempting the following for quick gitlab migrations:

  • Having it installed and ready to go on the 2nd node
  • Set read-only mode
  • Backup on primary
  • Restore on secondary
  • Change DNS
  • Allow writable again

Looks like gitlab provides instructions for updates. HA is premium so Helm chart becomes only option.

As much as I wanted to get gitlab into kubernetes sooner, its going to have to be a later. Theres still a lot of improvements to made across the project. I'd like servers booted up and ready to go in less than 5 minutes. This will require more baked-in installs/container downloads. 1-3 minutes for VM bootup, 2-3 minutes provisioning files and connecting services to cluster.

Edited by kc