Report on Error-Free Users and Sessions with Stability Scores

We often hear from customers that their existing tooling for tracking crash and error-free statistics is too rigid. It does not provide them the flexibility they need to drill in releases that don’t follow standard versioning patterns, or to view crash-free stats by operating system, device type, or other custom metadata. Backtrace allows users to […]

We often hear from customers that their existing tooling for tracking crash and error-free statistics is too rigid. It does not provide them the flexibility they need to drill in releases that don’t follow standard versioning patterns, or to view crash-free stats by operating system, device type, or other custom metadata.

Backtrace allows users to compute error rates against any metrics you collect, such as “Sessions started”, “Minutes played”, “Unique Users”, and more. We call these Stability Scores. In the past, these metrics could be imported from raw data that lives in external systems like Graphite, Influx, Redshift, or Postgres. Now, we’re happy to announce that we are beginning to release a series of SDKs to support gathering this information directly from your game or app.

This means that out-of-the-box, you will be able to see session and user stability metrics like “Crash free sessions” in the Backtrace Web Console! Our APIs are super flexible, and allow you to create your own events or so you can track custom metrics and have them sent to your Backtrace instance as well. We started releasing this support with our backtrace-unity SDK, and will be expanding to others in the coming weeks.

Getting Started

By default, the backtrace-unity SDK is now configured to capture session and user metrics and use them to calculate and display the following on your project’s Overview page. No customization is needed at all!

  • Crash free session: What % of sessions went without crashing.  A great measure of your app’s aggregate session stability
  • Crash free users: What % of unique users have not experienced crashes.  Perfect for tracking user’s experiences
  • Crash free session % through time:  A line chart showing crash free session % through time to help you more closely monitor the evolution of your app’s stability

EventAgg1.png

Note that this functionality also works in conjunction with the Web Console’s filtering.  As such, you filter your data set to get a sense for these stability metrics within a more specific domain.  For example, if you wanted to see the stability of a specific release, you could add that to the filter bar as shown below!

Note: In order for filtering with a specific attribute to work, that attribute must be configured properly to be associated with the metric group in question (in the case of Crash free users / sessions, the “Application Launches” metric group). For more information on how to configure this, see here.

EventAgg2.png

Another common use case would be to exclude non-fatal errors to see these metrics with regard to only fatal errors (Crashes, hangs).  The exact filter you would use to do this depends on which attributes you are indexing / SDK you are using.  (errorType = “crash” for example).

Prioritizing Crash/Error Resolution

The Web Console’s Triage and Explore views allow you to normalize your fingerprints against these metrics as well.  This will allow developers, tech leads to prioritize resolving  crashes/errors which have the most impact on your app’s stability.

Once your SDK is sending these metrics, use the drop down at the top right of the Triage and/or explore view to normalize your errors by the metric of your choosing.EventAgg3.png

In this example, we have selected to normalize our errors by sessions.  Now, in the “Errors by sessions” column, you can see the impact of each fingerprint relative to the number of sessions it occurred in.  In the screenshot below you can see that even though the fingerprint “8206af2” had the most occurrences, it only manifested in 0.0034% of sessions (1 / 29,220 total sessions in this time period).  So it may make more sense for an engineer to investigate and resolve fingerprint “002e365” instead which is present in a higher percentage of sessions.

EventAgg4.png

Advanced Use Cases, Customization

We understand that different applications are likely to require different metrics to properly monitor stability.  While our SDKs provide session and unique user metrics out of the box, we built this set of features to allow users to define custom metric events to suit the needs of their applications.

For example, you could configure metrics to normalize for:

  • How many minutes of app usage were crash/error free
  • How many “levels” in your game a crash/error occurred in

Specific configuration steps for this functionality will vary by SDK so please refer to each supported SDK’s readme file for instructions to take advantage of this functionality.  See  Unity SDK – Stability Metrics Configuration Documentation for more information on customizing today!

The post Report on Error-Free Users and Sessions with Stability Scores appeared first on Backtrace.io.

Source: Backtrace