Testing

What do you need to test?

With my applications, I need to verify the following things:

Is my application configured correctly?
Does Sentry send events when errors are kicked up in various critical parts of my application?
Are the Sentry events scrubbed correctly?
Does scrubbing Sentry events kick up errors?
Has the shape of the event that Sentry sends changed because of a change in integrations, sentry-sdk client upgrade, or something else?

That results in a few different kinds of tests I run.

In local testing and CI:
- Integration tests to verify that Sentry is configured correctly, scrubbing works, and the shape of the Sentry event hasn’t changed since last time I examined it.
In production:
- A way to trigger a Sentry event to verify that Sentry is configured correctly and events reach the Sentry server.

Integration testing with SentryTestHelper 

Fillmore provides a SentryTestHelper to make it convenient to create new or reuse existing Sentry clients in a way that overrides the transport allowing you to inspect and assert things against Sentry events that were emitted.

The helper provides two ways to use it:

fillmore.test.SentryTestHelper.init() a new Sentry client with provided parameters
OR, fillmore.test.SentryTestHelper.reuse() an existing Sentry client that was configured in your application code

Both of those create new contexts and clear the event list. You can create multiple contexts in a single test.

Calling init or reuse returns an object that keeps track of what events were emitted and stores them as a list in the .events property.

Here’s an example test using unittest:

# myapp/test_app.py
import unittest

from fillmore.test import SentryTestHelper

from myapp.app import kick_up_exception


class TestApp(unittest.TestCase):
    def test_scrubber(self):
        # Reuse the existing Sentry configuration and set up the helper
        # to capture Sentry events
        sentry_test_helper = SentryTestHelper()
        with sentry_test_helper.reuse() as sentry_client:
            kick_up_exception()

            (event,) = sentry_client.events
            error = event["exception"]["values"][0]
            self.assertEqual(error["type"], "Exception")
            self.assertEqual(error["value"], "internal exception")
            self.assertEqual(
                error["stacktrace"]["frames"][0]["vars"]["username"], "[Scrubbed]"
            )

Fillmore also provides a pytest fixture.

Here’s an example test using pytest:

# myapp_pytest/test_app.py

from myapp.app import kick_up_exception


def test_scrubber(sentry_helper, caplog):
    # Reuse the existing Sentry configuration and set up the helper
    # to capture Sentry events
    with sentry_helper.reuse() as sentry_client:
        kick_up_exception()

        # Assert things against the Sentry event records
        (event,) = sentry_client.events
        error = event["exception"]["values"][0]
        assert error["type"] == "Exception"
        assert error["value"] == "internal exception"
        assert error["stacktrace"]["frames"][0]["vars"]["username"] == "[Scrubbed]"

        # Assert things against the logging messages created
        fillmore_records = [
            rec for rec in caplog.record_tuples if rec[0].startswith("fillmore")
        ]
        assert len(fillmore_records) == 0

Integration testing against Kent–a fakesentry service 

Kent is a service that you can run in CI or on your development machine which can accept Sentry event submissions and has an API to let you programmatically examine them.

Because Kent is keeping the entire event payload, you know exactly what got sent and you can hone your scrubbing accordingly.

This lets you write integration tests that run in CI in an environment that has multiple services.

For example, if you set Kent up at http://public@localhost:8090/0 and you had SENTRY_DSN set to that dsn, then you could access it like this:

import time

from fillmore.test import get_sentry_base_url, SentryTestHelper
import requests
# Use the werkzeug wsgi client because the Django test client fakes
# everything
from werkzeug.test import Client

from django.conf import settings

from myapp.wsgi import application


def test_sentry_with_kent():
    sentry_helper = SentryTestHelper()
    client = Client(application)
    kent_api = get_sentry_base_url(settings.SENTRY_DSN)

    # Flush the events from Kent and assert there are 0
    resp = requests.post(f"{kent_api}api/flush/")
    assert resp.status_code == 200
    resp = requests.get(f"{kent_api}api/errorlist/")
    assert len(resp.json()["errors"]) == 0

    # reuse uses an existing configured Sentry client, but mocks the
    # transport so you can assert things against the Sentry events
    # generated
    with sentry_helper.reuse() as sentry_client:
        resp = client.get("/broken")
        assert resp.status_code == 500

        # Give sentry_sdk a chance to send the events to Kent
        time.sleep(1)

        # Get the event list and then the event itself
        resp = requests.get(f"{kent_api}api/errorlist/")
        event_data = resp.json()["errors"]
        assert len(event_data) == 1
        error_id = event_data[0]

        resp = requests.get(f"{kent_api}api/error/{error_id}")
        event = resp.json()["payload"]

        # Assert things against the event
        assert "django" in event["sdk"]["integrations"]
        assert "request" in event
        assert event["request"]["headers"]["Auth-Token"] == "[Scrubbed]"

        # FIXME: Assert that Fillmore didn't log any exceptions

Check logging for errors 

If Fillmore raised an exception when scrubbing, it’ll log a message to the fillmore logger.

Your tests should assert that there are no messages at logging.ERROR level logged from the fillmore logger.

Test module API 

fillmore.test.get_sentry_base_url(sentry_dsn)

Given a sentry_dsn, returns the base url

This is helpful for tests that need the url to the fakesentry api.

Parameters:

sentry_dsn (str) – the sentry base url

Raises:

TypeError – when sentry_dsn is not a string
ValueError – when sentry_dsn is empty

Return type:

str

exception fillmore.test.ReuseException: Raised when there’s no sentry_sdk client configured to reuse

class fillmore.test.SentryTestHelper

Sentry test helper for initializing Sentry and capturing events.

This helper lets you create new sentry_sdk clients or reuse existing configured ones.

You can access emitted events with the .events attribute.

You can reset the event list with .reset().

property events: List[Dict[Any, Any]]: Access the event list.

reset()

Resets the event list.

Return type:: None

init(*args, **kwargs)

Create a new sentry_sdk client with specified args

This creates a new sentry_sdk client with the specified args and patches the client transport with one that captures Sentry events that are being emitted. This lets you assert things against events.

Arguments are the same as to sentry_sdk.Client.

Parameters:

args (Any) –
kwargs (Any) –

Return type:

Generator[SentryTestHelper, None, None]

reuse()

Re-use the current sentry_sdk client, but patch the transport

This clears the breadcrumbs of the current sentry_sdk scope and patches the client transport with one that captures Sentry events that are being emitted. This lets you assert things against events.

Raises:: ReuseException – if there’s no sentry client initialized
Return type:: Generator[SentryTestHelper, None, None]

exception fillmore.test.ConfigurationError

class fillmore.test.SaveEvents(wrapped_scrubber, outputdir)

Utility wrapper for saving Sentry events to files on disk.

This is for collecting Sentry event data to build tests to verify scrubbing is working as you need it to.

Note

Capturing Sentry event data and writing tests against that is fragile and not as good as writing integration tests that kick up Sentry events that then get scrubbed.

Make sure to update captured data periodically. This will avoid skew from sentry_sdk updates where they change the shape of the events or what’s included in events as well as changes to your code which changes frame-local vars, context data, and so on.

Usage:

scrubber = Scrubber( ... )
scrubber = SaveEvents(
    wrapped_scrubber=scrubber,
    outputdir="/some/path"
)

Parameters:

wrapped_scrubber (Scrubber) –
outputdir (str) –

fillmore.test.diff_event(a, b, path='')

Compares two Sentry event structures.

This supports unittest.mock.ANY which will always match.

Example:

# Get an event from the SentryTestHelper.events list and diff it
# against the expected event
differences = diff_event(event, expected)
assert differences == []

Parameters:

a (Dict[str, Any]) – first structure
b (Dict[str, Any]) – second structure
a –
b –
path (str) –

Returns:

list of differences each as a dict with “msg”, “a”, “b”, “path” keys

For example:

{
    "msg": "different types: a:<class 'int'> b:<class 'str'>",
    "path": "some.path",
    "a": 5,
    "b": "five",
}

Return type:

List[Dict[str, Any]]

Testing

What do you need to test?

Integration testing with SentryTestHelper

Integration testing against Kent–a fakesentry service

Check logging for errors

Test module API

Integration testing with SentryTestHelper 

Integration testing against Kent–a fakesentry service 

Check logging for errors 

Test module API 