Getting Envato Market HTTPS everywhere

Last month we announced that we had finally completed the move to HTTPS everywhere for Envato Market. This was no easy feat since we are serving over 170 million page views a month that includes about 10 million products listed and are all user generated content. Along the way we have learnt many valuable lessons […]

Last month we announced that we had
finally completed the move to HTTPS everywhere for Envato
Market
. This was no easy feat since we are serving over
170 million page views a month that includes about 10 million products
listed and are all user generated content. Along the way we have learnt
many valuable lessons that we want to share with the wider community and
hopefully make other HTTPS moves easier and encourage a better adoption
of HTTPS everywhere.

Behind the scenes, the groundwork for the HTTPS rollout started back in
2014 with a couple of the engineers implementing a feature toggle which
allowed staff to opt-in for HTTPS. For a long time, this sat dormant and
unused by most staff until earlier this year when a few engineers got
together and decided it was time to give HTTPS everywhere another push
and get it to general availability.

But Why?

HTTPS isn’t just about the having a padlock or green indicator shown in
the browser. It’s about creating a trusted connection between the end
user and your services via three protection layers:

  • Encryption: Securing the exchanged data to prevent eavesdropping
    on the connections.
  • Data integrity: Confidence that the data has not been altered mid
    transit without being detected.
  • Authentication: Assuring the website you are connecting to is who
    you expect them to be.

An added side effect of migrating to HTTPS is that you can unlock HTTP/2
and features like request multiplexing and server push which are great
news for performance! Last year, Google announced HTTPS as a ranking
signal
so by migrating to HTTPS you get a boost in
your search results too!

User managed content

We have a lot of user managed content. The problem here is that many
of our authors don’t have time or additional funds to implement things
like content delivery network (CDN for short) caching so most of our
user managed content requests would end up needing to hit their origin
servers to fulfill requests. This was bad for a few reasons:

  • Many authors use shared hosting or very small instances for storing
    these assets. During the testing phase, we generated a low amount of
    traffic for a particular set of assets that would sometimes take over
    20 seconds to complete! The result of these slow load times is a very
    poor experience for buyers and would result in many people looking
    elsewhere because they couldn’t see previews or screenshots of the
    product quickly enough.
  • Very little HTTPS adoption. If we intended to serve our pages under
    HTTPS, we needed to ensure the assets on the page were also served
    securely. The issue here is that it’s very unrealistic for Envato to
    force users to spend time (and potentially money) on updating all of
    their assets to be served via HTTPS to avoid seeing mixed content
    warnings on the item pages.

To solve both of these issues, we decided to use an approach that
consisted of an image proxy and a CDN. The image proxy would rewrite all
of the non-secure links at render time to point at our CDN which would
help speed up response times and allow us to take some of the load off
author origins by caching the assets.

Initially we used camo which was built by Corey Donohoe
when he was at GitHub where they needed to solve a similar
issue
. This worked well for us until we started
trying to scale it to handle more traffic. GitHub solved the scaling
issue by adding more worker processes however we decided to try adding
clustering support
so that we could utilise more of
the hardware we already had in place. This didn’t solve the problem for
long and we eventually ended up back in the same position and needed to
resize our hardware to account for the additional load. We determined
this wasn’t going to be a viable path going forward and we needed a
better solution. After some looking around we found go-camo
which is a Go port of Corey’s original project. For a while we ran the
two implementations side by side and discovered that go-camo was able
to better utilise all of the existing hardware (due to its ability to
use more than a single operating system thread) and was easier to debug
when issues popped up. After a couple of weeks of load testing we
decided to completely swap to go-camo and start ramping up the number
of users who were using this.

Sharing cookies

As you may know, Envato Market is built using Ruby on Rails and out of the box Rails offers the ability to define how you
handle your cookies. To continue with our
incremental rollout, we needed to allow user cookies to be able to be
accessible on HTTP or HTTPS depending on which protocol their request
was served via. This was achieved by omitting the Secure flag on
cookies until we were confident post rollout that we were not going to
roll back.

Performance

One of the big concerns from teams looking to undertake HTTPS migrations
is that they will incur a performance hit once it’s live and in most
cases, it’s just not true. Deploying to modern hardware/software setup
and using a suitable cipher suite
mitigates many of the performance bottlenecks that used to be associated
with HTTPS. This definitely doesn’t mean you shouldn’t collect metrics
around these areas and monitor them, it just means that “HTTPS is slow”
is not a valid excuse.

In Envato’s case, we haven’t seen any performance impacts and our end
user time is consistent with the weeks prior to the HTTPS rollout.

Monitoring

One of the most important things you can do during a HTTPS migration is
Monitor All The Things. By having insight into the changes within your
stack during the migration you can quickly detect an issue before all
your users do. During our rollout some of the metrics we kept a very
close eye on were:

  • Exception rate
  • Time spent in network requests
  • End user response time
  • Application response time
  • Instance resource utilisation (CPU specifically)
  • Total number of requests
  • Edge network requests and the count by status code

During our rollout we identified a couple of issues, most notably a load
balancer misconfiguration. We were seeing a CPU spike on a small subset
of web instances that were missed in all of our testing which we managed
to catch before rolled it out to all of our users.

Here are two that we put together to keep everyone informed about how
far through the rollout we were. The top one was the initial rollout
(mostly just staff usage) and the second is when we decided to cut
everyone over to HTTPS.


SEO

2016 has been a big year for SEO at Envato. We’ve kicked off many
initatives targeting better visibility for search engines into our
author products and during early discussions it was decided we needed to
be extra careful during the migration not to undo all of the hard work
we’ve put into the last 7 months. To ensure we didn’t do go backwards we
took a couple of steps that has helped us stay on top and improve our
search engine rankings:

  • Submit both HTTP and HTTPS sitemaps to Google webmaster tools: In
    the week leading up to the swap over, we took a snapshot of our
    sitemaps and uploaded them into Google webmaster tools as a new set of
    sitemaps. This was done to ensure that when we swapped over to HTTPS
    Google would have access to a HTTP and HTTPS sitemap source and would
    allow Google to continue crawling the HTTP sitemap but at the same
    time be lead into the HTTPS version of the site.
  • Ensure we maintained 1:1 redirects: This helped ensure our users
    (and bots) still knew where to find us even though we moved to HTTPS.

In taking these steps, 61% of high volume terms we track have remained
stable or improved their rankings since the HTTPS release. The remaining
terms that have moved backwards were not on page 1 and have not actually
lost us traffic or revenue.

The migration wasn’t completed overnight and took longer than we would
have liked but we’ve managed to roll this out without any negative
impacts on our users or application which is something we are extremely
proud of. Hopefully by publishing our journey this will give others
information about migrating to HTTPS along with the added benefits that
come with it and remove the stigma that HTTPS is only a painful
experience.

Source: Envato