Technology for Dogs

Build Time Optimization With Pip and Virtualenv

Using pip and virtualenv to manage your Python dependencies is a no brainer for development. The ability to ensure all developers use consistent versions, simplfy development environment bootstrapping, and quickly test new dependencies versions in an isolated environment are all huge wins for development. What about in a production environment or with continuous integration?

Pip presents certain difficulties when managing changing dependencies with the CI model of development. Different branches may have different dependencies, building a new virtualenv is not a trivial affair in that context and can easily take 10 or more minutes to download sources and compile distributions with binary dependencies. Adding an extra 10 minutes to the cycle time between creating a pull request and having the tests run against the branch was not acceptable for Rover.

A production environment poses additional challenges. How do you quickly roll out new environment changes to multiple instances? What about failed environment builds on particular machines? How to avoid rebuilding an environment if no requirements have changed?

We wanted the benefits of pip and virtualenv in production at Rover, and as a result we’ve solved a number of these problems.

Speeding up source downloads

Pip’s default configuration is to download a new copy of the source evertime it needs to be built. Excluding the compilation of binary packages (such as PIL) this is the most time consuming portion of building a new virtualenv.

So how to avoid it? Thankfully pip provides a solution. You can specifiy a directory to be used to store a cache of downloaded sources.

Create a file at:

~/.pip/pip.conf

The contents should be something like:

1
2
[global]
download-cache=/path/to/your/download-cache

You can use any location for your cache. I personally like nesting it under my .pip directory:

download-cache=~/.pip/download-cache

Speeding up binary compilation

So now you’re only downloading the source distributions for each package version once and reusing it each time you need it.

That’s much better. Still everytime you build a virtualenv you will need to spend time rebuilding packages such as PIL, psycopg2, and mysqldb that have binary components, and don’t change very often.

Pip doesn’t provide a great built in solution to this problem. However, pip-accel does a great job of managing this issue. Simply put it provides a thin wrapper around pip commands that manages the caching of the binary output of your builds resulting in massive speedups to your deployment process

Build one virtualenv for each requirements iteration

At this point, we had already improved our build time with pip and virtualenv by 80% or so. However, that still wasn’t good enough for us. We we’re still building a new virtualenv for every test suite run on CI and every new deployment build, whether or not the requirements have changed!

If the requirements haven’t changed, then neither has the virtualenv. So how to reuse virtualenv’s, but ensure new ones are made whenever the requirements change? Hashing was made for this. Simply hash the contents of the requirements file and look for existing environment matching that requirements version. The exact process will vary based on your system, but the gist of what we did:

  • Hash the contents of the requirements file
  • Look in a known location for a directory matching that hash
  • If it exists use it, if it doesn’t then create it with virtualenv
  • If you created it, install the requirements being sure to check that the installation succeeded

We use shell scripts to manage our deployment steps. Our virtualenv creation for CI looks roughly like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#!/bin/bash
set -e

function handle_dependency_errors() {
  venv_path=$1
  pip_error_code="$2"
  echo "Pip failed to build new virtual environment"
  rm -rf $venv_path
  exit $pip_error_code
}

venv_path="/var/cache/venvs/$(md5sum requirements.txt | awk '{ print $1 }')"

if [ ! -d "$venv_path" ]; then
  virtualenv $venv_path
  trap 'handle_dependency_errors ${venv_path} ${$?}' EXIT
  $venv_path/bin/pip install pip-accel
  $venv_path/bin/pip-accel install -r requirements.txt
  trap "" EXIT

fi

Shipping the virtualenv

If you’re running a limited number of servers it can be feasible to simple build the virtualenv on the boxes. However as your system grows in complexity it can be advantageous to have a single deploy machine that handles the entire build process, then simply copies the virtualenv to the production machines.

Doing this saved us significantly on build time, but it’s not without consequences. For deployment, we had to build our virtualenv with --relocatable. This causes several issues, the most obvious of which is the loss of activate.

More subtly, you’ll need to be on point with your configuration management. Any package that depends on system libraries (such as openssl, libjpeg, etc) requires that those libraries be installed on each machine, and at the same absolute path as the build machine. Depending on your needs it may be desirable to eat the build time hit and build it on each machine to avoid these potential complications.

Further Improvements

Hopefully this has given you some good idea for how to deal with building python dependencies quickly and effeciently. These changes made a massive improvement our build time, but there is still more we can do. While rolling out these changes we discovered one of our dependencies had been suddenly deleted from PyPi. This wouldn’t be an issue if we where running a local PyPi mirror. Doing so would improve our resiliency to changes outside our control as well as allow us to host internal packages.

Doggie Bag: Night Owl Edition

A weekly bag of snacks to take home after the meal, curated by Rover Tech.

  • Performing a Project Premortem — Unlike a typical critiquing session, in which project team members are asked what might go wrong, the premortem operates on the assumption that the “patient” has died, and so asks what did go wrong. The team members’ task is to generate plausible reasons for the project’s failure.

  • With Flextime, Bosses Prefer Early Birds to Night Owls — In three separate studies, we found evidence of a natural stereotype at work: Compared to people who choose to work earlier in the day, people who choose to work later in the day are implicitly assumed to be less conscientious and less effective in their jobs.

  • Git 2.0 is here and it’s full of goodies — This major release of git has been brewing for a long time and I am excited to go on the hunt in the Changelog to find cool bits of awesomeness.

  • Sass doesn’t create bad code. Bad coders do. — It’s all true. If you’re a poor developer. You know, one who would handcraft too specific selectors, 15MB sprites and doesn’t know how to cleanly structure a project.

“You will never change your life until you change something you do daily.” —Unknown

Doggie Bag: Guacamole Edition

A weekly bag of snacks to take home after the meal, curated by Rover Tech.

Googlebot’s recent improvements might revolutionize web development — “Googlebot is apparently much better at parsing JavaScript than many of us were aware – to the point where single page apps without a server-side HTML alternative now seem to be possible, even when SEO is important.”

Introducing Avocado — “A new toolbox for interaction designers”

Software is Eating Hardware – Lessons for Building Magical Devices — “Incredible industrial design is increasingly non-optional, what really matters is an equally beautiful software system that spans mobile, desktop, and more. It’s something that takes a lot of thought to get right.”

Experiments at Airbnb — “While the basic principles behind controlled experiments are relatively straightforward, using experiments in a complex online ecosystem like Airbnb during fast-paced product development can lead to a number of common pitfalls.”

Repl.it — “An online environment for interactively exploring programming languages. The name comes from the read-eval-print loop, the interactive toplevel used by languages like Lisp and Python.”

“We may fight, and you may win, but you’ll regret that you did.” —Brent Turner, on being relentless and tough competitor

Speeding Up Django Test Runs by Optimizing Factory Boy

From the very beginning at Rover, we’ve focused on making deploying code as fast and painlessly as possible. One important piece of our deployment infrastructure is Jenkins. As soon as we merge to master, (via a pull request) Jenkins runs our test suite—if the suite passes Jenkins automatically deploys the new version. As our app and our test suite have grown, these builds started taking longer than we’d like, so we decided to spend some time optimizing performance.

factory_boy is used very heavily in our test suite at Rover. In any given test, factories represent a meaningful portion of the total number of queries so improving factory_boy performance should improve the performance of the entire test suite.

This article will focus on how we minimized database queries when creating two of our most common models: Person and Dog.

Measure First

To make measurement easy (and to replicate the environment our factories will see in the test suite), I first wrote a throwaway test case:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
FACTORY_ITERATIONS = 1000

class FactoryCreateDurationTimingTests(TestCase):
    def _evaluate_factory_speed(self, factory_cls, attr='create', **kwargs):
        start = time.time()
        for i in range(FACTORY_ITERATIONS):
            getattr(factory_cls, attr)(**kwargs)
        end = time.time()
        duration_each = (end - start) / FACTORY_ITERATIONS
        print "\t{}.{}() takes ~{}ms.".format(
            factory_cls.__name__,
            attr,
            int(duration_each * 1000))

    def test_create_person(self):
        self._evaluate_factory_speed(PersonFactory)

    def test_build_person(self):
        self._evaluate_factory_speed(PersonFactory, attr='build')

    def test_create_dog(self):
        self._evaluate_factory_speed(DogFactory)

    def test_build_dog(self):
        self._evaluate_factory_speed(DogFactory, attr='build')

As the baseline, this test output:

  • DogFactory.create() takes ~100ms.
  • DogFactory.build() takes ~0ms.
  • PersonFactory.create() takes ~100ms.
  • PersonFactory.build() takes ~0ms.

Identify Queries

The easiest way I found to track down the source of queries is to insert a

1
import pdb; pdb.set_trace();

in the execute method of the appropriate backend.

This method lives in django/db/backends/(your database backend)/base.py.

Using standard pdb commands like w, you can see the call stack for each query that’s executed.

The Primary Culprits

In our case, excess queries fell into 3 buckets:

  1. Queries caused by factory_boy
  2. Queries caused by excessive work in .save() methods
  3. Queries caused by signal handlers or other code higher up the .save() call chain

Queries caused by factory_boy

The _setup_next_sequence method causes an additional query, but in our environment sequence value isn’t used, so we reimplemented the behavior of the parent class and return 0.

1
2
3
4
class YourFactory(DjangoModelFactory):
    @classmethod
    def _setup_next_sequence(cls):
        return 0

Any post_generation methods on a factory cause an additional .save() call.

These methods allow you manipulate the record in question, but if the methods only create or change associated records, you don’t need that extra .save().

Removing those from the results keyword argument to _after_postgeneration eliminates the extra query:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class YourFactory(DjangoModelFactory):
    @classmethod
    def _after_postgeneration(cls, obj, create, results=None):
        if results is not None:
            results.pop('create_a_related_model')
        super(YourFactory, cls).after_postgeneration(
            obj,
            create,
            results)

    @factory.post_generation
    def create_a_related_model(self, **kwargs):
        ...
        ...

Queries caused by work in .save() methods

By convention on our team, in our code base any additional work that is done in .save() methods can be disabled via flags passed to the .save() method in question.

An example is worth 1000 words, so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
class YourModel(models.Model):
    def save(self, do_something_expensive=True):
        if do_something_expensive:
            do_something_expensive()
        super(YourModel, self).save()


class SaveFlagsDjangoModelFactory(factory.DjangoModelFactory):
    """
    Allows a factory to handle accepting whitelisted kwargs to
    the ._create method.
    """
    @classmethod
    def _create(cls, target_class, *args, **kwargs):
        """
        Due to factory_boy passing through kwargs directly into the
        get_or_create call (which will query with them) we reimplement
        the functionality of the DjangoModelFactory class with slight tweaks.
        """
        save_flags = cls._get_save_flag_kwargs()
        save_flag_kwargs = {}
        for flag, default in save_flags:
            save_flag_kwargs[flag] = kwargs.pop(flag, default)

        if cls.FACTORY_DJANGO_GET_OR_CREATE:
            fields = cls.FACTORY_DJANGO_GET_OR_CREATE
            filter_data = {}
            for field in fields:
                if field in kwargs:
                    filter_data[field] = kwargs[field]
            try:
                return cls.FACTORY_FOR.objects.get(**filter_data)
            except cls.FACTORY_FOR.DoesNotExist:
                pass

        obj = cls.FACTORY_FOR(*args, **kwargs)
        obj.save(**save_flag_kwargs)
        return obj

    @classmethod
    def _after_postgeneration(cls, obj, create, results=None):
        """
        Duplicate behavior of DjangoModelFactory _after_postgeneration
        except to pass in the flags and defaults defined returned by
        _get_save_flag_kwargs().
        """
        if create and results:
            kwargs = dict(cls._get_save_flag_kwargs())
            obj.save(**kwargs)


class YourModelFactory(SaveFlagsDjangoModelFactory):
    FACTORY_FOR = YourModel

    @classmethod
    def _get_save_flag_kwargs(cls):
        return (
            ('do_something_expensive', False),
        )

Queries caused by signals, etc.

These are best worked around by patching the associated behavior during your test runs, then exposing the ability to re-enable the behavior via a decorator for individual tests that rely on that behavior or are actually testing that behavior. For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
from functool import wraps
from mock import patch


class Utility(object):
    def something_skippable(self):
        ...
        ...


class BaseTestCase(TestCase):
    def setUp(self):
        self._something_skippable_patcher = patch.object(
            Utility,
            'something_skippable')
        self._something_skippable_patcher.start()
        super(BaseTestCase, self).setUp()

    def tearDown(self):
        self._something_skippable_patcher.stop()
        super(BaseTestCase, self).tearDown()


def enable_something_skippable(func):
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        self._something_skippable_patcher.stop()
        try:
            result = func(self, *args, **kwargs)
        finally:
            self._something_skippable_patcher.start()
        return result
    return wrapper


class IndividualTestCase(BaseTestCase):
    @enable_something_skippable
    def test_where_we_need_something_skippable(self):
        ...
        ...

Results

Overall, we were able to massively speed up our factories.

  • DogFactory.create() takes ~3ms.
  • DogFactory.build() takes ~0ms.
  • PersonFactory.create() takes ~1ms.
  • PersonFactory.build() takes ~0ms.

Improving the speed of our factories ultimately resulted in a 50% decrease in our overall test runtime – exactly the kind of improvement we’d hoped to see!

JavaScript Error Reporting Using Source Maps and Sentry in Django

Sure, you have error reporting for when your app 500’s, but do you know when your latest JavaScript changes to your core funnel interactions block your users from doing anything at all? Errors are far easier to catch and test for on the backend using automated unit tests, but the front-end of the web is everything but predictable. You can test to your heart’s content with qUnit or Jasmine, and automate your testing in tools like Browserstack and DalekJS, but even our best efforts will have bugs falling through the cracks.

Doggie Bag: Free as in Internet Edition

Rover Tech’s weekly bag of snacks to take home after the meal.

“If all that’s between me and victory is a cement wall, then victory is assured”. —Brent Turner, on being relentless and never giving up

Doggie Bag: Programming Sucks Edition

Rover Tech’s weekly bag of snacks to take home after the meal.

The difference between what we do and what we are capable of doing would suffice to solve most of the world’s problems. —Mahatma Gandhi

How I Removed Email From My Life

Of the many things that are really great about Rover, one of them is that we have a very healthy “email culture”. We mostly talk face-to-face or in a chat room. Most mornings, I arrive at work with very few new emails in my inbox.

I surveyed my Rover email for the last week, and in that time I have been part of 118 email threads from people inside of Rover.[1] That’s 16.9 emails per day or 23.6 per work day.[2] I can tell you from past experience, that is virtually nothing; other places I’ve worked have been in the 100+ per day range.

With Rover’s email culture, I already was getting an “A” in not being overloaded by email. But, email is something that stresses me out, so I wanted to dial things up to an “A+”. Those few messages I did have sitting in my inbox were like a little devil on my shoulder whispering, “hey, look here…there are unattended things to do”. I didn’t like it.

So, I decided to get email out of my life for good.

PhantomJS, Selenium, and Django: Headless Browser Testing for the Rest of Us

Rover.com is a Django shop, but I personally come from a Rails background. The Rails world has great tooling and infrastructure for automated functional tests – capybara, capybara-webkit, and the new hotness poltergist. Underneath poltergeist lies PhantomJS, a headless webkit with very few dependencies, excellent for automated testing. Unfortuantely, PhantomJS version 1.5 dropped Python bindings, leaving us Djangonauts out to dry. There also isn’t a great capybara equivalent in the Python world (Ghost.py is the closest).

Thankfully, despite the roadblocks, there is a path forward! Django (starting with 1.4) comes with LiveServerTestCase to support our exact use case. After failed attempts to get Ghost.py up and running (due to dependencies and lack of documentation), I landed upon a solution that will start us Django developers down the path of being first class citizens once more (well, maybe not exactly first class, but at least those coach seats up front with a little extra legroom). I’ve also taken some baby steps to improve our testing assertion syntax, trying to fill the capybara void, which I’ll get into at the end of this post.