Modularising the Badoo iOS app: dealing with knock-on effects In my previous article, I explained how we singled out the chat functionality in our app for modularisation. Everything had gone well and we were preparing to roll out this experience to start work on modularising all of Badoo iOS app development. We even did a presentation […]
In my previous article, I explained how we singled out the chat functionality in our app for modularisation. Everything had gone well and we were preparing to roll out this experience to start work on modularising all of Badoo iOS app development. We even did a presentation on the approach for the product, testing and continuous integration teams, before gradually starting to introduce modularisation into our processes.
We immediately realised there would be problems so we didn’t rush things. We rolled out the solution in stages and this helped us identify the problems we explore below.
In this article I explain:
Only once the number of modules really started to grow a lot, did we realise that linking and support for an extensive dependency graph is pretty complicated. What we had imagined 50 modules might be like, displaying an attractive hierarchy, was quickly dispelled by the harsh reality.
This is now the Badoo app dependency graph looked when we had about 50 modules:
No effort was spared on visualising this in a convenient and readable manner, but it came to nothing. The complicated graph helped us to realise several things:
But what we do know is that simple visualisation is really necessary, but that work with the dependency graph needs to be automated. This is what led us to create our internal utility, deps. Its purpose was to solve our new tasks: searching for problems in the dependency graph, directing developers to them and correcting typical linking errors.
The utility’s main characteristics are:
The final point is that it is not worth hoping for similar open-source utilities to appear. Today there is no universal, well-described one-size-fits-all structure for projects and their settings. Various companies may have a number of different graph parameters:
For this reason, if you are looking at having upwards of 100+ modules, at some stage you will most likely have to think about writing a similar utility.
So, in the interests of automating the work with the dependency graph, we have developed several commands:
2. Fix. This is an evolution of the “doctor” command. In automatic mode this command corrects problems found by the “doctor” command.
3. Add. This adds module-to-module dependency. When the app in question is simple and small, adding dependency between two frameworks appears to be a simple task. But when the graph is complex and multi-level, and you are working with explicit dependencies “on”, adding the necessary dependencies is something you don’t want to be performing manually each time. Thanks to the “add” command, developers can simply specify two framework names (the dependent one and the “dependee”) — and all the build phases will have the necessary dependencies, as the graph illustrates.
As a result, the script for creating a new module, as per the template, has also become part of the “deps” utility. What does this give us in the end?
Before we moved to modularisation we had several applications, and our strategy was simply to “check everything”. Irrespective of what has actually changed in our monorepository, we simply rebuilt all the applications and ran all the tests. If everything was fine, we allowed changes to make it into the main development branch.
Modularisation quickly made it clear that this approach is poor in terms of scalability. This is because there is a linear correlation between the number of modules and the number of CI-agents necessary for parallelising error checking: when the number of modules increases, the queues for CI also increase. Of course, up until a certain point in time, you can simply purchase new build agents, but we opt for a more rational path.
Incidentally, this wasn’t the only problem with the CI. We found that the infrastructure was also subject to problems: the simulator might not run, the memory might prove insufficient, the disk might get damaged etc. While these problems were proportionally small in the scale of things, as the number of modules to be tested (jobs run on agents overall) increased, the absolute number of incidents grew, and the CI team were no longer able to handle incoming requests from developers promptly.
We tried several approaches, all unsuccessful. Here’s more detail so that you can avoid making the same mistakes.
The obvious solution was to stop building and testing everything all the time. The CI needs to test what needs to be tested. Here is what failed to work:
As you will have understood, the compulsive testers kept the queues for the CI much the same, while the over-confident developers caused our main development branch to break.
In the end, we returned the idea of automated calculation of changes, but from a slightly different angle. We had the deps utility, which knew about the dependency graph and project files. And through Git we obtained a list of changed files. We extended deps using the “affected” command, which allowed us to obtain a list of affected modules based on changes shown by the version control system. Even more important was the fact that it took account of dependency between modules (if some modules depend on an affected module, then they have to be tested as well, in order that, for example, if the interface of a lower module changes, the upper one continues to build).
Example: changes to the “Registration” and “Analytics” blocks on our diagram point to the need to test the “Chat”, “Sign In with Apple” and “Video streaming” modules, as well as the app itself.
This was a tool for CI. But developers also had the option of viewing locally what could potentially be affected by their changes, in order to manually test the operating capability of the “dependee” modules.
This delivers a range of benefits:
For the Badoo application, we have over 2000 end-to-end tests which launch it and run, based on usage scenarios for the expected testing results. If you run all these tests on one machine, running all the scenarios takes about 60 hours. For this reason, on the CI all the tests are launched in parallel — insofar as the number of free agents permits.
After successful implementation of filtering based on changes for unit tests, we wanted to implement a similar mechanism for end-to-end tests. And this is where we encountered a clear problem: there is no direct correlation between end-to-end tests and modules. For example, a scenario for sending messages in the chat also tests the chat module, the module for loading images and the module which is the entry point for the chat. In actual fact, one scenario may indirectly test up to seven modules.
To resolve this problem, we have created a semi-automated mechanism underlying which is mapping between modules and sets of functional tests that we have.
Each new module has its own separate script in the CI which tests whether there is a module in this mapping. So that developers remember to synchronise with the testing team, tagging the module with the necessary groups of tests.
This sort of solution can hardly be described as optimal, but implementing it still brought tangible advantages:
2. The noise from infrastructure problems was reduced (the fewer tests run, the fewer crashes due to frozen agents, broken simulators, lack of space etc.).
3. The mapping of modules and their tests became a place where development and testing departments were able to synchronise. In the context of development, the programmer and the tester now discuss, for example, which of the groups of tests available may also be suitable for testing a new module.
Apple openly states that dynamic linking slows down the running of an application. However, it is precisely this which is the default option for any new project created in Xcode. This is what the real graph, showing the running time for our projects, looks like:
In the centre of the graph, you can see a sharp reduction in time taken. This is due to the transition to static linking. What is this connected with? The dyld tool for dynamically loading modules from Apple performs labour-intensive tasks but not in an entirely optimal manner. There is a linear correlation between the runtime for these tasks and the number of modules. This was the main cause for our application launching more slowly: the more new modules we added, the slower the dyld became (the blue plots shows the number of modules added).
After transitioning to static linking, the number of modules stopped slowing the application’s run speed and when iOS 13 Apple came out, it transitioned to using dyld3 to run applications which also helped speed up this process.
It should be noted though that static linking also carries with it a number of limitations:
And so, at this point we have dealt with the two main problems: we have moved resources out to separate bundles and we have corrected the build configuration. As a result:
We have considered global changes which we had to make for a “quiet life” when it comes to the modularisation process:
I have not said this directly but, as you may have noticed, we have graphs showing the time that tasks spend in the CI queue, the launch speed of the application etc. All the changes described above were necessary due to the values of metrics which are important when developing an application. However, we might consider transforming processes as well. It is essential you have an option of measuring their results. Without such an option, changes probably won’t make much sense.
In our case, we realised that changes would have a great impact on developers, so the main metric we kept our eye on, and continue to do so, is the build time for the project on the developer’s hardware. Yes, there are indicators for the full build time with the CI, but developers rarely perform full build; they use other equipment etc. This is to say that, both the build configuration and their environment differ.
For this reason, we have generated a graph showing the average build time for our applications on developers’ computers:
If the graph displays variations (a sharp decrease in the build speed for all applications), this most likely means that something has changed in the build configurations — and that won’t be good for developers. We have looked into these problems and tried to resolve them.
Another interesting conclusion we came to is that, having obtained similar analytics, modules building slowly is not always a reason to go for optimisation.
Take a look at this graph. The axes show average build duration and the number of builds per day, respectively. It is clear that, of the modules which take the longest to build, there are those which build just twice a day; overall, their impact on the general experience of working on the project is extremely small. On the other hand, there are modules which only take 10 seconds to build, but they are built more than 1500 times a day. We need to keep a careful eye on these. Basically, try not to limit monitoring to just one module, but see the bigger picture.
Moreover, it soon became clear which equipment was still suited for working on our project, and which was becoming obsolete.
For example, it is clear now that the 2017 iMac Pro 5K is no longer the best hardware for building Badoo, while the 2018 MacBook Pro 15 is still not bad at all.
But the main conclusion we drew was that it is essential to improve developers’ quality of life. In the process of modularisation you are likely to get so immersed in the technology that you forget the reason you are doing it for. So, it is important to keep in mind the basic motivations and to acknowledge that the repercussions will affect you.
In order to obtain data on the build duration on developers’ computers, we created a special macOS application called Zuck. It sits on the status bar and monitors all xcactivitylog files in DerivedData. xcactivitylogs are files containing the same information which we see in the Xcode build logs in Apple’s difficult-to-parse format. Based on these you can see when the build of an individual model began and finished, and the order they were built in.
The utility has white and blacklists so we only monitor working projects. If a developer has downloaded a demo project of a library from GitHub, we are not going to send data about its build anywhere.
We send information on building our projects to the internal analytics system, where there is a wide range of tools available for building graphs and analysing data. For example, we have the Anomaly Detection tool which predicts anomalies in the form of values which vary too much from the predicted values. If the build time has changed significantly compared with the previous day, the Core command receives notification and starts to investigate what has gone wrong and where.
Overall, measuring the local build time yields the following important benefits:
I will now give you an indication of the metrics which will need monitoring if you have started to move in the direction of modularisation:
2. Artifacts size. This metric helps us to identify linking problems quickly. There may be cases where the linker does not notify us, for example, when a given module is duplicated. However, this will be shown by the increase in size of the compiled application. It is worth paying close attention to this particular graph. The simplest way of obtaining information of this kind is from the CI.
3. Infrastructural indicators (build time, time tasks spent in the CI queue etc.). Modularisation will have an impact on many of the company’s processes but particularly on infrastructure is particularly important, because it has to be scaled. Do not fall into the same trap we did when, before merging to the master, changes were having to queue 5–6 hours.
Naturally, there will always be room for improvement but these are the main indicators which allow you to track the most critical problems and errors. Some metrics can also be collected locally if you are not able to invest in complex monitoring. Get local scripts and at least once a week look at the main indicators. This will help you to understand whether, overall, you are moving in the right direction.
If your impression is, “This is what I am doing now, and things will get better whatever happens,” but you are unable to measure what “better” means, it is best to wait before conducting an experiment along these lines. After all, you probably have a manager to whom you will have to report the results of your work.
After what I have told you, maybe your impression is that building a modularisation process at your company will require lots of energy and resources. This is true in part but let me give you some statistics. Actually, we aren’t that big.
The final figure is gradually increasing: old features and rewritten legacy code are slowly moving from the main target applications to modules.
In these two articles I have been “selling” modularisation to you but of course, it does have its downsides as well:
But the good news is that all these downsides are manageable. They are nothing to be afraid of since these are all things that are resolvable. You just need to be ready for them.
We have already looked at the considerable advantages of modularisation but allow me to reiterate them briefly here:
My parting words to you are that when you start implementing modularisation you don’t have to take on a large part of the application as your first module. when you start implementing modularisation. Take on smaller, simpler chunks and then experiment. It is not a good idea either to blindly follow any of the examples from other companies; everyone has their own approach to development and the chances of someone else’s experience suiting you perfectly are slim. Make sure you develop infrastructure and monitor your results.
I wish you every success and if you have any questions do not hesitate to add them to the comment section!