How to Make Github and Travis Split Monorepo to Multiple Git Repositories for You

Do you use a monorepo? Then you know How to maintain multiple Git repositories with ease. If you're not there yet, you may wonder How to Merge 15 Repositories to 1 Monorepo and Keep their Git History.

Are you and your monorepo ready? Today we'll focus on fast, secured and outsourced monorepo auto split - all that under 10 minutes.

It's great to be alive in this era. We have solved maintaining multiple repositories, even merge migration of their git history to one repository. Creating huge code bases was more never cost effective and never had a steeper learning curve.

The same way it was never easier to drive an autonomous car. You just sit in the car, press the button of your destination, and Tesla will drive you there when you check all the news on /r/PHP.

Well, the monorepo paradigm is not there yet but it's getting there.

The Split Problem

One of the last problems I'm aware of is splitting the monorepo to particular packages. Imagine you have Symfony monorepo and you're trying to split it to all standalone packages like symfony/console, symfony/dependency-injection and so on.

Current Status

This whole "take a code from this directory and put it into this repository in master branch and last tag" process is now:

Instead, we want it to be:

Why? So you could amaze your friends at the party that you just set-up a monorepo split and they can enjoy merged PRs in a matter of minutes (even if you know almost nothing about git or PHP).

Do you think we can get there? You'll see.

The Best Solutions to Split (So Far)

Feel free to explore these following solutions. I did it for you and here are a few blockers that hold them from being massively adopted.

On the other hand, I'm grateful for each one of them, because they're pushing the evolution further and further. Thanks to them we don't have to reinvent the wheel and we can build on their shoulders.

1. splitsh/lite

2. dflydev/git-subsplit

What Would the Ideal State Look Like?

Before diving into solution and how to do it, I try to stop and go into a wonderland. What would the ideal solution look like? How would I use it? How would I explain it to others? How fast would it be? Try to break away from your know-how limits (because they're limiting your thinking) and be free to come up with absolutely non-sense answers:


If we put it in the code, it might look like:

composer require symplify/monorepo-builder --dev
# monorepo-builder.yml
parameters:
    directories_to_repositories:
        packages/MonorepoBuilder: '[email protected]:Symplify/MonorepoBuilder.git'
vendor/bin/monorepo-builder split

That could do, right? At least from a developer's experience view.


But what would security expert Michal Špaček say to lousy code like that?

How To Avoid Rape on Github and Travis?

"So, anyone can now push to your repository whatever he wants?"

This is a valid question that was probably scratching your mind when you saw Github + Travis + git + open-source combination. Travis is basically a terminal, that runs few command lines. What would prevent someone from using this to "play with" your repository?

Let's look at the repository address in our example:

[email protected]:Symplify/MonorepoBuilder.git

This basically means we need to make ssh key or username and a password public. Does that sound like a good idea to you?

Don't worry, Github and Travis thought about these cases - with a hashed GITHUB_TOKEN environment variable.

1. Create a Personal Access Token on Github

First, you need to create a custom token, that will authorize access to your Github repositories from any command line where it will be used.

Read the Github docs or use tl;dr;:

2. Add GITHUB_TOKEN in Repository Settings on Travis

Then we need to store this token to Travis build, so Travis can be authorized to manipulate with your repositories without any password or ssh key.

Read the Travis docs or use tl;dr;:

In the end, it should look like this:

If you got lost in this tl;dr;s, try this nice post with so many screenshots.

GitHub and Travis Protects You

Now the best part. If you accidentally commit your access token in .travis.yml (like I did while testing), GitHub will immediately disable it and sends you an email (to bad I found that out the next day after 4 hours of debugging with that token).

And if you add the token to your repository on Travis as above, it will hide it in all logs for you. No need to hash it.

So instead of insecure

[email protected]:Symplify/MonorepoBuilder.git
# or
https://[email protected]/Symplify/MonorepoBulder.git

everyone will see:

https://[secure]@github.com/Symplify/MonorepoBulder.git

Sound and safe!


Now we have:

Something is missing...

Oh right, when will the monorepo be split? Do we have to do it manually? How often? Should Travis CRON do it on daily basis?

When is the Best Time to Split our Monorepo?

In times like these, get back to our ideal product:

It often happens we merge fix or feature to monorepo and we want to try it before rushing to tagging a stable release. We want to do it as soon as possible, without manually triggering Travis to do it. Also, we don't want Travis to waste energy on pull-requests that are not merged to master. That would only slow the whole CI process down and frustrate contributors and maintainer.

Saying that how .travis.yml should look like?

language: php

# required for "git tag" presence for MonorepoBuilder split and ChangelogLinker git tags resolver
# see https://github.com/travis-ci/travis-ci/issues/7422
git:
  depth: false

matrix:
  include:
    - php: 7.2
      env: MONOREPO_SPLIT=true
    # ... other builds

install:
  - composer install

# ... other scripts

after_script:
  # split monorepo to packages - only on merge to master
  - |
    if [[ $TRAVIS_EVENT_TYPE == "push" && $MONOREPO_SPLIT == true && $TRAVIS_BRANCH == "master" ]]; then
      vendor/bin/monorepo-builder split -v
    fi

That way the split command is run only merge to master and exactly once after each merge. So you can test your feature in a matter of minutes...

Wait wait, no vague statements like a matter of minutes. How fast it really is? To give you an idea, this is Symplify Travis build with a split of 10 packages:

It takes under 7,5 minutes including all the tests, static analysis and code style validation.

That's all folks. You're ready to go and try it on your monorepo.


Are You Into Git Internals?

I knew you are, so here are few details.

All it' wrapped in a bash file at the moment. It could be done in symfony\process, but the original source subsplit.sh was in bash so I used it.

There are ~160 lines but most of them are arguments and options configuration, their resolving, preparing the repository and other boring stuff. The interesting part is really in this 1 and these 3 lines:

git remote add origin "[email protected]:Symplify/MonorepoBuilder.git"

git checkout -b "master"
git subtree split -q --prefix="/packages/MonorepoBuilder" --branch="master"
git push -q --force origin "master"

I used "real values" instead of $VARIABLES, so it's more clear to you.

In human words it works like this:

# in what repository should we push the code?
git remote add origin "[email protected]:Symplify/MonorepoBuilder.git"

# what branch do we push there?
git checkout -b "master"

# the split magic!
git subtree split -q --prefix="/packages/MonorepoBuilder" --branch="master"

# push this branch to remote branch
git push -q --force origin "master"

That is really it!

If you're git split geek (like me), feel free to explore the whole subsplit.sh script. There are many nice little details to learn from.


So, do you think you're ready to fascinate your friends tonight with all your brand new monorepo split setup?


  Continue Learning


What do you think?


GitHub RSS @votrubaT Runs on Statie Hosted on GitHub

Like what I write about? Hire me & we can work together