500px users may notice new features on our site occasionally, but we put new code into production multiple times a day! How do we ensure that everything works after every change? Good question! I’m a member of the quality assurance (QA) team at 500px — we play a big part in this. Here are some of the […]
500px users may notice new features on our site occasionally, but we put new code into production multiple times a day! How do we ensure that everything works after every change? Good question! I’m a member of the quality assurance (QA) team at 500px — we play a big part in this. Here are some of the development techniques we use to ensure a smooth sail.
First of all, code changes are reviewed by other developers before they even reach the QA team. This often catches bugs and corner cases early in the process. Code review also helps reduce redundant code and encourages syntactic consistency, both of which keep our codebase clean.
Pull requests are reviewed by developers, but also reviewed by machines! We have a large suite of automated tests, which run when new pull requests are opened. These tests are a great way to ensure that new features work as expected and verify that these new changes do not break existing functionality. Given the variety of features on our site, it would be time-consuming to test all aspects by hand on every code change. Currently, we have about 4,000 automated tests that are separated into threads which run simultaneously. We use a continuous integration framework called Semaphore CI that runs these tests on every proposed change. The tests are randomly executed, which encourages the development of independent tests to ensure the order of execution does not impact the expected result. This helps us parallelize the test suite into different threads. Semaphore can also be integrated with Slack to inform developers about tests that have passed or failed. From this, developers are able to triage through and fix the code that broke things.
For a more in-depth overview of how we integrate and use Semaphore check out this link.
From my perspective as a QA developer, this automated testing allows us to focus on capturing edge cases of new product requirements and spend less time reviewing previously-tested functionality. The automated test cases are reusable: they eliminate the need to manually retest all features. We also create new tests and update older ones to keep the test specifications up-to-date and make sure they accurately reflect the behaviour of 500px.
But test automation does not cover everything! Each change is also manually tested to catch any remaining bugs. This includes positive testing (confirming that everything works when it should) and negative testing (confirming that errors are handled gracefully). Since 500px caters to many users, testing with a diversity of user accounts (e.g. both free users and paid members, or different language preferences) is crucial! For the QA team, testing with multiple user accounts helps us verify that features are accessible to the correct group of users and catch discrepancies between translations.
Generally, QA has the final say about when changes are ready to be deployed. Our Slack bot, BMO, can deploy changes to all our different testing and production environments. BMO encourages a process known as conversation-driven development and promotes greater transparency between team members. It’s also very easy for new team members to deploy changes! All they do is type a command such as “bmo deploy 500px” into Slack and BMO begins the deployment. BMO also notifies the user of any deployment errors as well as successful deployments and includes a log to help any debugging.
The QA team uses BMO’s ability to deploy to different environments to view code changes before they go live. Since we have access to multiple pre-production environments, BMO also tracks which ones are in use and notifies team members when someone is using an environment. This prevents people from inadvertently using the same environment and affecting their tests.To use an environment, we type “bmo lock environment” and “bmo release environment” in Slack.
To know more about the mechanics of BMO and how it was implemented, check out the ChatOps segment earlier on in the 500px developer blog.
Overall, that’s a quick summary of how we move fast and introduce new features without breaking things. If you have any questions, leave a comment below and check out our other blog posts for more details on how we work. And if this is interesting to you and have suggestions on how we can make our setup better: we’re hiring!