When developing software, there eventually comes a time when the team needs to start thinking about performance. Ideally this is done on a daily basis - most developers try to balance productivity, code readability and shipping features. But at some point of the life cycle of the application, the need to optimize for performance becomes evident.

Shipping new features makes users happy, user happiness will probably bring more users and eventually the performance of the application becomes more important: more users means more load, more load means a slower application and a slower application means unhappy users.

The first question we need to ask is why optimize? and the answer comes in many ways: we can say that the purpose of optimizing performance is the same as the purpose of writing software, that is to help users and make them happy. We also may want to be faster than our competitors or we may have read a recent study about how slow applications make for low conversion rates. In any case you have a real cause that is telling you that your application is slow and that you need to fix it.

To overcome that, one may be tempted to start focusing on performance at an early stage of application development or when you still have few users, but beware of this thought and ask yourself the question: Do we need better performance right now?. Optimizing for performance is not an easy task and requires lots of time and resources. If you start with it at an early stage you are risking your ability to ship new features or improve existent ones and your users will not be happy about it.

This answers the second question: when should I optimize? With a simple enough answer - only when you need to. They key is to have a good monitoring system in place which will tell you when it’s time to start optimizing things before your users notice.

The answer to the third question, how to optimize? is composed by a series of advices outlined below that will hopefully help you and your team to go through the optimization process without major issues.

It’s not all a bed of roses.

Be careful, optimizing for performance represents a series of challenges that your team will need to overcome in order to succeed in the process:

Don’t stop shipping features

Your application is at a point on its life when users are completely in love with it and competitors are probably out there trying to steal some of them, so you can’t stop shipping features to focus on performance. “Hey! another app launched this new awesome feature, but we are going to wait for your app to finish with optimizations and then launch that feature” is a phrase no one said, ever.

The problem with this is that your team will have a lot of resources (people or time) allocated to optimizations but you will need to find the correct balance.

Too many hands in the same plate

When developing features, developers probably will work on different files of the project and the process of merging changes on the repository should be straightforward enough. But what happens if one of the developers is, for example, adding a few new methods to your User model and another one is optimizing the existing methods? Be ready for merge conflicts and coordinating the tasks of different developers - a real challenge, especially for distributed teams.

You said concurrency? Hello race conditions!

One of the most used approaches is to start doing things concurrently, like for example running many instances of some background process in order to speed up data processing. You can achieve a great performance boost by doing things like that, but beware when one of your background processes depends on another one because things can get messy.

Technical debt

Sometimes you are in a rush to get your application optimized. Perhaps you are on a tight deadline to get things done. Like every high pressure situation you will find yourself in, you’ll need to find the correct balance between speed of work and technical debt. Try to write down those situations where you sacrificed the readability of your code and when things cool down a little, try to go back and fix that.

Ok, good, but you were going to talk about performance optimizations

Working on performance is not that different from other areas of software development. You just need to keep things organized and avoid rushing into optimizing little portions of your code without knowing exactly the outcome you are expecting. Here are a couple of key concepts that may be able to help you out along the way:

Measure, measure and measure

This one is perhaps the most important thing to keep in mind. Don’t optimize something you think is a bottleneck. You have to find a good way to measure and identify bottlenecks and when you identify one, find a good way to solve it and measure again. The key is to not optimize blindly but to have a reliable way of knowing that you are on the right path.

Knowing that requests to a certain endpoint takes 300 ms to complete is not enough, you also need to identify exactly where those milliseconds are being spent and beware if you take a look at the code and decide to optimize a lousy loop you found there, maybe the problem was in a proxy, in the database or in the network. A bottleneck can be anywhere and you need to positively identify it before working on a solution.

Take the following example: you find out that your application stops serving requests when 100 users are online at the same time, but you looked at your monitoring system and found out that actually the load in the servers is not high, so what’s going on? Perhaps the problem is just a configuration issue in your proxy server instead of the code.

So if you find a portion of your code that you are sure you can optimize by reducing the number of iterations or changing a third party library, first measure its behavior and then measure the improvement. There are a lot of benchmark tools out there that will allow you to obtain this kind of data.

Which tools to use depend a lot on the nature of the project and the programming language used, but you should take a look at Grafana which will allow you to show graphs based on data you can store in InfluxDB or Graphite. There are also cool data processing tools like Heka or Logstash, the data from which can be visualized using something like Kibana.

If you happen to work with Ruby, don’t forget to check out awesome tools that will help you discover what is going on in your code, like Stackprof and Benchmark-ips. Most languages have similar tools that can be used to deeply analyze your code and find optimization opportunities.

Set clear, measurable objectives

You already know that you are trying to speed up the application and which parts of it are not performing as well as you would like. The next step is to define how you would like - or better, how you need their performance to change. A good example of an objective would be “Endpoint X should be able to support 500 concurrent users and serve requests within 200 ms at that load”.

This step is really important because you may end up in an infinite loop of performance improvements. This is not necessarily bad if your team has the resources for that but this is not always the case and you don’t want to loose focus on what your users need.

Front end developers like to talk about “Performance Budgets” where they impose a limit on how much a web page should weight or how much time should it take to load on a certain type of connection and then they work around that objective. Any improvement that goes further than the objective is nice to have but not mandatory when it comes to planning.

Keep it simple

Performance optimization is not different than shipping new features and the simpler you keep your codebase and infrastructure, the easier it will be to iterate on improvements later. Don’t over complicate just to speed things up, it is highly likely that a simpler solution will achieve the same results with the added benefit of leaving room for further improvements when required.

Don’t reinvent the wheel

Many applications out there went through the same issues you are going through and managed to succeed, their developers also probably gave a talk or wrote a blog post about it. So don’t think you are alone in the world just because your application is different, go ahead and do some research on how X problem was already solved or about the tools other applications are using to scale. There is an enormous chance that the solutions and tool you need are already out there so try to investigate before coming out with a clever solution of your own which may need a lot of extra maintenance in the future.

You can take a look at the technical blog of big companies like Netflix or Spotify where they usually share stories about how they solved their scaling problems and contribute to OSS regarding the matter (and other cool geeky stuff also).

YAGNI

There’s no point on making your application support 100,000 concurrent users when you only need 500. There is a chance that when you need to handle such loads your application will be completely different and you will be much more experienced than right now to achieve that new objective.

Test the correct scenario

Knowing how your users are using the application is enormously helpful to determine how to perform load tests. Don’t go around guessing things like which are the most used endpoints, like when fixing a bug, you need to reproduce the exact same scenario your users are facing and optimize things based on their point of view and their usage patterns.

No silver bullets

You may have already read performance comparisons between languages and statements saying that X language is faster than Y because of a lot of technical reasons. You probably also read about object allocations in memory, garbage collectors and that kind of stuff.

The truth is that there isn’t a single solution that will fix all of your issues and most of the time you will find a lot of bottlenecks before you can even think about the internals of a language or the number of object allocations. Don’t rush into rewriting pieces of your application in a different, faster language until you are completely sure that the bottleneck is the programming language, you will go down a long road before you reach this point.

The big picture

Always keep in mind that when we talk about an application’s performance we need to see it as a whole, the source code is just a part of a much more complex ecosystem which may include database servers, caching mechanisms, file access, and web servers to name a few things.

I can’t emphasize enough how important is to have a monitoring system in place, it will give you all the information you will need in order to answer the questions about when, why, how and what to optimize. If you don’t have monitoring try to make it your first priority because it will open your eyes to what is going on inside your application.

Once you identify the problems, there is not a single solution that will make things go faster, optimizing for performance is a delicate task that will probably require a lot of iterations: “measure, improve, measure again”. That’s why it is so important to have metrics you can rely on and set an objective before you find yourself in “crazy optimizing mode”.

Finally, if you find yourself in a rush for optmization because you suddenly have a lot of users or you are expecting a huge client to start using your application next week, don’t panic and try to keep your team organized. You will have to work a lot but organized work yields much better results than rushing into changes that will come back to haunt you when you need to maintain them in the future.