How I decreased CI/CD build time from 55 mins to just 5 mins!

How I decreased CI/CD build time from 55 mins to just 5 mins!

Intro

One of the achievements am proud of was to decrease the CI/CD pipeline of Camelan from 55 mins to 5 mins

Problem

In Camelan, we had to use the whole suite of Firebase for various stuff, real-time database for chat, Firestore for realtime updates for messaging and notifying the users if there are any new messages, performance benchmarking, some analytics, Crashlytics, and a lot more

This caused lots of overhead on our compile-time and of course, our CI/CD time

And of course, as any default dependency manager, your go-to dependency manager would be CocoaPods

Now, using CocoaPods is completely fine, until your build time expands to large numbers like 10-15 mins for clean builds, then you need to look for other solutions to handle your dependencies

We spent like 3 weeks trying to figure out how to cut our time and boost our iOS productivity as we were the slowest team regarding the feedback loop as any build we push over the CI/CD, we had to wait for at least 40 mins and at some time it goes to 55 mins.

Our CI/CD provider was AppCenter, so we had to do something about boosting our local compile-time and our pipeline

So let’s analyze our pain points:

1- Locally Xcode takes 15 mins to do a clean build

2- Any small change causes the whole project to rebuild (CocoaPods known setback)

3- CI takes a lot of time to finish the pipeline and send the build over to the QA team or to the store

Long build time and unnecessary rebuild

Our main pain points were due to that any small change in Camelan’s codebase triggered a recompile in CocoaPods, this caused us to wait for an unnecessary wait and caused us a lot of pain when tweaking things in the UI

So I decided to look for some solution to this issue, the key here was to convert the dependencies into binaries which are precompiled, and that would help Xcode to ignore those dependencies as they are already precompiled, so that would fix the issue of having to recompile dependencies with each small change

The question here is, what do we do to precompile the dependencies?

There were some options we tried like CocoaPods-binary but they were immature or didn’t work out at the end, so instead of twisting CocoaPods to fit our use case, we decided to go with Carthage since it was already designed to manage dependencies as precompiled frameworks and it was already supported by most of our used dependencies

So after we managed to migrate the project from CocoaPods to Carthage, we were enjoying far less compile and build times, locally it went from the 15 mins for clean builds to 3 mins, and on incremental builds, it was finishing in 24 seconds 🎉

So that helped us in our daily work and boosted our productivity, but we weren’t done yet

Our CI needed some love as well, but it was far trickier than the normal setup

Applying the same solution to App Center

Since App Center didn’t have much customizability, we had to somehow hack our way into customizing it with Bash scripts

In-app center, it detects what dependency manager you are using, is it CocoaPods or Carthage, then it reinstalls the whole dependencies again, we didn’t want that

What we wanted is to make App Center use our already precompiled dependencies without any work from his side, he would just go and compile the project without the need to compile any dependencies

So first thing to come to mind was to include the frameworks into the project’s repository on git

Of course, things didn’t go smoothly, as BitBucket had a limit of 2 Gigabytes on the size of the repository, and with Firestore only, we had around 800 Megabytes occupied (the whole repo size was around 1,800 MB)

So we had to think of another solution, we decided to upload the frameworks into another provider, We decided to go with Google Cloud hosting and that gave us exactly what we needed

Now that the frameworks are up and we clone them when needed

We had to somehow pull those frameworks when the CI pipeline works

Luckily, App Center allows for setting up a script after the project is cloned, which was a perfect place to pull the frameworks

But now there comes a question, how would the project compile after simply pulling the frameworks?

We had to reference the frameworks in the project without copying them, just set up a reference to their path and by that, Xcode expects the framework to be in that path, from this idea, we set up another script responsible for pulling the frameworks and putting them in the place where Xcode expects them to be, so that when it is time to compile, it can.

# GIT COOKIES GOES HERE # 1
git clone $frameworks_repo # 2
mv $frameworks_repo_local_name/Carthage Carthage # 3
rm -rf $frameworks_repo_local_name # 4
  1. We put here the git cookies of the Google Cloud Repository so that when the CI/CD fetches the frameworks, it doesn't get into any authorization errors
  2. we clone the frameworks' repo into the project's directory
  3. We extract the Carthage file into the project's directory in the path that Xcode expects the frameworks to be
  4. we remove the repository empty directory

Now we try to run a test build on App Center, we found that it redownloads the dependencies again, which is not what we wanted, we wanted it to skip any building or fetching to the project’s dependencies, to do so, we had to disable any Carthage command that run, so we set up a symbolic link to Carthage in App Center’s bin folder

Then make it execute a fake script, like so.

echo "creating mycarthage directory"
mkdir ~/mycarthage
cd ~/mycarthage
echo "creating fake file to intercept appcenter's call"
echo "echo fake" >> new_carthage
echo "making it executable"
chmod a+x new_carthage
echo "linking interceptor with bin/carthage"
ln -s new_carthage /usr/local/bin/carthage
chmod 0777 /usr/local/bin/carthage
PATH=~/mycarthage/:$PATH
echo "trying calling carthage like app center"
echo /usr/local/bin/carthage
./installation.sh

This worked great until we found out that the app was crashing with the QA

After some debugging, we realized that the frameworks were not getting attached to the project after compiling and thus causing the app to crash

This was caused due to our script that was disabling any Carthage command, which also disabled the command that attached the frameworks into the iOS app in the build phases

So we had to modify our script that hacks App Center’s Carthage Commands to conditionally allow the command responsible for attaching the frameworks

mv /usr/local/bin/carthage /usr/local/bin/carthage.bak
echo "\
if [[ \"\$1\" == \"copy-frameworks\" ]]
then
  /usr/local/bin/carthage.bak copy-frameworks
else
  echo "fake"
fi
" > new_carthage
chmod a+x new_carthage

cp new_carthage /usr/local/bin/carthage
z
echo "try new edits"
/usr/local/bin/carthage

./installation.sh

With that done, we finally enjoyed faster build times both locally and our testers now receive builds more frequently and the bottleneck was resolved!

Conclusion

I hope this little challenge becomes useful to anyone, as this allowed us to boost our productivity 11 times after lots of frustration with the build time and a frustrating business, we are more productive than ever and our users who love our product a lot are entertained with lots of updates every week (sometimes twice a week!)