FEED Validator

for Atom and RSS and KML

Sorry

This feed does not validate.

line 18, column 14: Invalid email address: (102 occurrences) [help]

      <author></author>
              ^

line 1613, column 0: Invalid character in a URI: [here](https://github.com/AdRoll/rustenstein/blob/3c4452b38dad2ba0f5f3d2c07209b89bd61e50c2/src/main.rs#L99-L145) [help]
```
</description>
```

In addition, interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

line 4, column 11: title should not be blank [help]

    <title></title>
           ^

line 46, column 0: description should not contain relative URL references: /images/post_images/smartdoc.png (355 occurrences) [help]
```
</description>
```
line 944, column 0: description should not contain data-lang attribute (260 occurrences) [help]
```
&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;la ...
```
line 2260, column 0: description should not contain script tag (46 occurrences) [help]
```
&lt;script type=&quot;math/tex; mode=display&quot;&gt;
```

line 2883, column 0: description should not contain alg attribute (2 occurrences) [help]

    &lt;img alg=&quot;madvise usage&quot; src=&quot;/images/post_images/otp2 ...

line 6196, column 0: description should not contain style tag (3 occurrences) [help]

      <description>&lt;style type=&quot;text/css&quot;&gt;/*&lt;![CDATA[*/

line 7567, column 0: style attribute contains potentially dangerous content: max-width (10 occurrences) [help]
```
&lt;p&gt;&lt;img alt=&quot;Kelly&quot; src=&quot;/images/post_images/post_dd ...
```
line 7589, column 0: description should not contain iframe tag (29 occurrences) [help]
```
&lt;figure&gt;&lt;div class=&quot;youtube-wrapper&quot;&gt;&lt;iframe class= ...
```

line 18100, column 0: description should not contain data-height attribute [help]

  &lt;p data-height=&quot;402&quot; data-theme-id=&quot;348&quot; data-slug- ...

line 18100, column 0: description should not contain data-theme-id attribute [help]

  &lt;p data-height=&quot;402&quot; data-theme-id=&quot;348&quot; data-slug- ...

line 18100, column 0: description should not contain data-slug-hash attribute [help]

  &lt;p data-height=&quot;402&quot; data-theme-id=&quot;348&quot; data-slug- ...

line 18100, column 0: description should not contain data-user attribute [help]

  &lt;p data-height=&quot;402&quot; data-theme-id=&quot;348&quot; data-slug- ...

line 18100, column 0: description should not contain data-default-tab attribute [help]

  &lt;p data-height=&quot;402&quot; data-theme-id=&quot;348&quot; data-slug- ...

Source: http://tech.adroll.com/feed.xml

<?xml version="1.0"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title></title>
<link>https://tech.nextroll.com</link>
<atom:link href="https://tech.nextroll.com/feed.xml" rel="self" type="application/rss+xml" />
<description></description>
<language>en-us</language>
<pubDate>Fri, 15 Dec 2023 12:53:00 -0800</pubDate>
<lastBuildDate>Fri, 15 Dec 2023 12:53:00 -0800</lastBuildDate>
<item>
<title>Celebrating Innovation - NextRoll's Hack Week 2023 H2 Winners</title>
<link>https://tech.nextroll.com/blog/hackweek/2023/09/15/hack-week.html</link>
<pubDate>Fri, 15 Sep 2023 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/hackweek/2023/09/15/hack-week</guid>
<description>NextRoll’s Hack Week 2023 was a week filled with ingenuity, collaboration, and groundbreaking ideas. From AI-powered solutions to production optimization, our teams went above and beyond to showcase their talents. Without further ado, let’s dive into the winning projects!
<h1 id="aiml-award-the-future-is-now">AI/ML Award: The Future is Now!</h1>
<h3 id="smartdoc">SmartDoc</h3>
Winners: Federico Della Rovere, Lorenzo Savini
SmartDoc is more than an AI-powered document management system. By leveraging machine learning, it categorizes, sorts, and manages documents with unprecedented accuracy, significantly reducing manual labor and improving data retrieval processes. This innovation is a step forward in automating administrative tasks, allowing teams to focus on more strategic activities.
<img src="/images/post_images/smartdoc.png" alt="SmartDoc" />
<h2 id="product-award-figma-plugin-for-ad-production-optimization">Product Award: Figma Plugin for Ad Production Optimization</h2>
Winners: Jake Burroughs, Jose Hernandez, Joris Korbee, Aasif Shabbir, Eurico Nicacio
This Figma Plugin transforms ad production by enabling the generation of HTML5 ads in formats beyond the standard sizes. Its AI-supported animated ad templates not only enhance creativity but also speed up the production process, offering our clients more variety and quicker turnaround times. This is a significant leap in ad customization and efficiency.
<img src="/images/post_images/figma_plugin.png" alt="HTML5 Ad Generator Figma Plugin" />
<h2 id="technical-award-puertagma">Technical Award: Puertagma</h2>
Winners: Oscar Arbelaez, Peter Kuimelis, Cristian Rojas, Christopher Ramirez, Alvaro Tuso
Puertagma is not just another API. It’s the culmination of our learnings from past projects, crafted into a robust and efficient tool. Written in Go, this API is set to become the backbone of our future integrations, promising enhanced performance and scalability. This project represents a significant stride in our technical capabilities, paving the way for more seamless and powerful integrations.
Congratulations to all the winners and participants! Your hard work, innovative thinking, and focus on impactful solutions are what continue to drive NextRoll forward as a leader in the ad tech industry.
<img src="/images/post_images/puertagma.png" alt="Puertagma" />
</description>
</item>
<item>
<title>Exploring Monads with JavaScript</title>
<link>https://tech.nextroll.com/blog/dev/2022/11/11/exploring-monads-javascript.html</link>
<pubDate>Fri, 11 Nov 2022 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2022/11/11/exploring-monads-javascript</guid>
<description>I know that I’m some years (decades?) late, but here’s my take on writing a
monad tutorial.
When you start learning about Functional Programming, sooner or later, you will
hear about “monads” - the secret handshake for functional programmers.
For those coming from imperative languages, there’s a considerable
learning curve to conquer before venturing into dealing with monads.
Here we will see how the sausage is made in a mainstream language, JavaScript.
I hope that it will help you visualize how the data plumbing works.
After we understand the concept, we can also see how monads look in
functional languages.
Before we start, a warning: this is a more practical guide. We are going to skip
some of the math involved. If the concepts involved happen to light your
interest, the book “<a href="https://bartoszmilewski.com/2014/10/28/category-theory-for-programmers-the-preface/">Category Theory for Programmers</a>” is beneficial.
<h1 id="not-just-lists">Not just lists</h1>
One of the mistakes I see people making when they try concisely
explaining monads is comparing them to lists.
It is very tempting to use this example because lists and arrays are common
data structures and programmers coming from imperative languages are familiar
with at least one of these. The argument is quite appealing.
This line of thinking can make newcomers believe that if they <code class="language-plaintext highlighter-rouge">.flatMap</code>
a list, they explored all that monads offer. It goes pretty beyond
that.
Imagine someone asking what a machine is and then getting the response:
“a car is a machine.” While this is true, one must be aware that there are
many other types of machines: rockets, phones, clocks, etc.
<h1 id="maybe-javascript">Maybe JavaScript</h1>
For this tutorial, instead of lists, we will implement a data type
called <code class="language-plaintext highlighter-rouge">Maybe</code>.
This data type helps when working with values that may or may not exist, avoiding
common JS errors like <code class="language-plaintext highlighter-rouge">Cannot read properties of null</code>.
It is pretty common in functional languages - but if this is your first time
seeing it, don’t worry - we will use just a basic implementation.
Here are the minimal utility functions that we need to work with this data type:
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const maybe = (value) =&gt; {
if (value === null)
return {
tag: 'nothing',
};
else
return {
tag: 'just',
value,
};
};
const exists = (someMaybe) =&gt; someMaybe.tag === 'just';
</code></pre></div></div>
The code is pretty self-explanatory: some value comes in, and we check if it is
<code class="language-plaintext highlighter-rouge">null</code> and then we return a wrapper with a <code class="language-plaintext highlighter-rouge">tag</code> property that labels which
kind of data is being held. The <code class="language-plaintext highlighter-rouge">exists</code> function allows us to check the value
of the <code class="language-plaintext highlighter-rouge">tag</code>.
We can imagine that this code receives some nullable thing and returns a
maybe thing, where thing could be <code class="language-plaintext highlighter-rouge">string</code> or <code class="language-plaintext highlighter-rouge">number</code> or whatever.
When talking about the types involved in this operation, we can replace the
<code class="language-plaintext highlighter-rouge">string</code> or <code class="language-plaintext highlighter-rouge">number</code> part with <code class="language-plaintext highlighter-rouge">a</code>, and the maybe part with <code class="language-plaintext highlighter-rouge">m</code>. A function
returning something can be described as <code class="language-plaintext highlighter-rouge">-&gt;</code>.
By doing this, we can say that the type of the <code class="language-plaintext highlighter-rouge">maybe</code> function is <code class="language-plaintext highlighter-rouge">a -&gt; m a</code>.
Working with values wrapped in <code class="language-plaintext highlighter-rouge">Maybe</code> will look like this:
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const sum = (a, b) =&gt; a + b;
/** Receives two maybes (ma and mb) */
const process = (ma, mb) =&gt; {
if (exists(ma) &amp;&amp; exists(mb)) {
return sum(ma.value, mb.value);
} else {
return null;
}
};
process(2, 3);
// 5;
process(2, null);
// null;
</code></pre></div></div>
As you can see, we are not gaining much by using <code class="language-plaintext highlighter-rouge">Maybe</code> instead of checking if
a value equals <code class="language-plaintext highlighter-rouge">null</code> - we are just calling <code class="language-plaintext highlighter-rouge">exists</code> all the time.
Another code smell is that we are returning <code class="language-plaintext highlighter-rouge">null</code> in the “failure” path.
We threw away our safety harness and now we are letting <code class="language-plaintext highlighter-rouge">null</code>s wander inside
our app without adult supervision.
<h1 id="map-all-the-way">Map all the way!</h1>
This can be improved by using <code class="language-plaintext highlighter-rouge">map</code>, enabling us to provide the hypothetical
<code class="language-plaintext highlighter-rouge">m a</code> value to a function, if said function accepts an <code class="language-plaintext highlighter-rouge">a</code>:
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const map = (fn) =&gt; (ma) =&gt;
exists(ma) ? maybe(fn(ma.value)) : ma
const addThree = a =&gt; a + 3
map(addThree)(maybe(3))
// {
// tag: "just",
// value: 6
// }
</code></pre></div></div>
So far, so good. Now we can “inject” a function to operate on values that might
not exist.
We also get a <code class="language-plaintext highlighter-rouge">Maybe</code> back, instead of a raw value. It might be
tempting to “extract” the value after we are done, but keeping it “wrapped”
ensures that consumers of our code will be able to <code class="language-plaintext highlighter-rouge">map</code> their functions safely
on it. This also makes more sense than returning <code class="language-plaintext highlighter-rouge">null</code>: if we are adding a
number to something that might not exist, then the result might not exist as
well.
Tip: once you step into <code class="language-plaintext highlighter-rouge">Maybe</code> land, stay there (otherwise you are lying to
yourself and not handling all possible states properly).
<h1 id="lets-make-it-harder">Let’s make it harder</h1>
In the real world, things are usually more complicated than this example.
We are probably going to have multiple “maybe” values coming in.
Let’s see how we can sum two nullable values - for that, we can use a nested
<code class="language-plaintext highlighter-rouge">map</code> to access both values:
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const process = (ma, mb) =&gt;
map((a) =&gt;
map((b) =&gt;
sum(a, b)
)(mb)
)(ma);
process(maybe(2), maybe(3));
// {
// tag: 'just',
// value: {
// tag: 'just',
// value: 5,
// },
// };
</code></pre></div></div>
FP folks are probably screaming “Use apply!!”, but let’s restrict the scope
of this tutorial.
These nested <code class="language-plaintext highlighter-rouge">map</code>’s are hard to read - and there’s also an unfortunate effect:
We got a nested <code class="language-plaintext highlighter-rouge">Maybe</code> as a result!
If we add more variables (as in <code class="language-plaintext highlighter-rouge">map (map (map ...))</code>), more nested levels will
be added as well.
Working with this type of result is possible but burdensome:
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const mma = process(maybe(2), maybe(3));
if (exists(mma)) {
if (exists(mma.value)) {
console.log(mma.value);
}
}
//or by mapping:
map((ma) =&gt;
map((a) =&gt;
console.log(a);
)(ma);
)(mma);
</code></pre></div></div>
<h1 id="crunching-data">Crunching data</h1>
Let’s create a function that will help ease the pain of working with nested
values, as this nesting got out of hand quickly.
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const join = (mma) =&gt; (!exists(mma) ? mma : mma.value);
</code></pre></div></div>
This is a very simple function:
<ul>
<li>If we are in the “nothing” case, return it (as “nothing” can have no
children).</li>
<li>Otherwise, ignore the outer layer and return the inner, wrapped value.</li>
</ul>
The type of <code class="language-plaintext highlighter-rouge">join</code> is <code class="language-plaintext highlighter-rouge">m (m a) -&gt; m a</code>.
Now, for each variable that we add, a <code class="language-plaintext highlighter-rouge">join</code> call needs to be added as well:
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const process = (ma, mb, mc) =&gt;
join(map((a) =&gt;
join(map((b) =&gt;
map((c) =&gt;
a + b + c
)(mc)
)(mb))
)(ma));
// Notice that we have two `join(map(` segments
// and one `map(` segment. This will be revelant below.
</code></pre></div></div>
This is getting even worse to read! Let’s try refactoring this.
First, let’s replace the innermost, solitary, <code class="language-plaintext highlighter-rouge">map(</code> segment with
<code class="language-plaintext highlighter-rouge">join(map</code>. The result looks like this:
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const process = (ma, mb, mc) =&gt;
join(map((a) =&gt;
join(map((b) =&gt;
join(map((c) =&gt;
a + b + c
)(mc))
)(mb))
)(ma));
process(maybe(1), maybe(1), maybe(1));
// 3;
</code></pre></div></div>
This works, but the resulting value is not wrapped inside a <code class="language-plaintext highlighter-rouge">Maybe</code>. If our
inputs may not exist, coherence dictates that the result might not exist as well.
To fix this, we can add a <code class="language-plaintext highlighter-rouge">maybe</code> around the innermost expression (<code class="language-plaintext highlighter-rouge">a + b + c</code>):
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const process = (ma, mb, mc) =&gt;
join(map((a) =&gt;
join(map((b) =&gt;
join(map((c) =&gt;
maybe(a + b + c)
)(mc))
)(mb))
)(ma));
process(maybe(1), maybe(1), maybe(1));
// {
// tag: 'just',
// value: 3,
// };
</code></pre></div></div>
Now we have a working solution, but it looks like a letter soup.
Let’s try combining (composing) the repeated pattern <code class="language-plaintext highlighter-rouge">join(map</code> into a new
function:
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const bind = (ma) =&gt; (fn) =&gt; join(map(fn)(ma));
</code></pre></div></div>
As you can see <code class="language-plaintext highlighter-rouge">bind</code> equals “<code class="language-plaintext highlighter-rouge">map</code>, then <code class="language-plaintext highlighter-rouge">join</code>”.
Now let’s replace all <code class="language-plaintext highlighter-rouge">join(map</code> expressions with <code class="language-plaintext highlighter-rouge">bind</code>.
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const process = (ma, mb, mc) =&gt;
bind(ma)((a) =&gt;
bind(mb)((b) =&gt;
bind(mc)((c) =&gt;
maybe(a + b + c)
)
)
);
process(maybe(2), maybe(2), maybe(2));
// {
// tag: 'just',
// value: 6,
// };
</code></pre></div></div>
Now even if we add more variables, they will all collapse until a single, flat
maybe value, because we call <code class="language-plaintext highlighter-rouge">join</code> once per level.
The <code class="language-plaintext highlighter-rouge">bind</code> function allowed our nested structure to be executed as a sequence of
steps.
By the way, <code class="language-plaintext highlighter-rouge">bind</code>’s type signature is quite popular: <code class="language-plaintext highlighter-rouge">m a -&gt; (a -&gt; m b) -&gt; m b</code>.
The final code doesn’t look much pretty, as those nested functions can be
confusing to work with. Other languages have features that make working
with this type of expression easier.
<h1 id="entering-the-big-m-generalizing-plumbing">Entering the Big M: Generalizing Plumbing</h1>
Now you learned how to create a system that performs a sequence of computations
involving <code class="language-plaintext highlighter-rouge">Maybe</code>.
But now imagine Matrix’s Morpheus coming into the scene and saying:
“What if I told you that this pattern can be generalized to other data types?”
Remember when we saw <code class="language-plaintext highlighter-rouge">bind</code>’s type? That <code class="language-plaintext highlighter-rouge">m</code> doesn’t stand for <code class="language-plaintext highlighter-rouge">Maybe</code>. It’s
<code class="language-plaintext highlighter-rouge">Monad</code>.
When we say <code class="language-plaintext highlighter-rouge">m a</code>, we are not talking about a <code class="language-plaintext highlighter-rouge">Maybe a</code>, but any “<code class="language-plaintext highlighter-rouge">m</code>onadic” type
that holds an <code class="language-plaintext highlighter-rouge">a</code>. In the FP world there are many such types: a list (<code class="language-plaintext highlighter-rouge">List a</code>), a
side effect <code class="language-plaintext highlighter-rouge">IO a</code>, something that may contain an error (<code class="language-plaintext highlighter-rouge">Either err a</code>), etc.
This pattern is useful because we can generalize the act of “Work with this
data, running the functions in sequence. If something goes wrong, short
circuit”.
The necessary elements to create this protocol are:
<ul>
<li>the capability of “flattening” our data structure with <code class="language-plaintext highlighter-rouge">join</code>: <code class="language-plaintext highlighter-rouge">m (m a) -&gt; m a</code>;</li>
<li>when calling our sum (<code class="language-plaintext highlighter-rouge">a + b + c</code>) function, we need to place the result in
the same data structure that we are using. In our code we called it <code class="language-plaintext highlighter-rouge">maybe</code>, but
in the protocol, it has a different name: <code class="language-plaintext highlighter-rouge">return</code>. Its type is <code class="language-plaintext highlighter-rouge">a -&gt; m a</code>.
Depending on the context, <code class="language-plaintext highlighter-rouge">return</code> can also be called <code class="language-plaintext highlighter-rouge">pure</code>.</li>
</ul>
The catch is that each datatype handles “wrapping”, “collapsing” and “short
circuiting” differently, depending on what these actions mean for that type.
This is very important for FP, because this enables the <code class="language-plaintext highlighter-rouge">IO</code> monad to run
actions with side effects, then wrap the results back into lazy functions.
<h1 id="how-this-looks-outside-js-land">How this looks outside JS-land</h1>
I think that we pushed JavaScript as far as it can go. Working with this kind
of expression is easier for functional languages, as they eat “wrapped values” for
breakfast. (We can go further if we use a JS-library like fp-ts or Sanctuary).
In functional languages, you can either call <code class="language-plaintext highlighter-rouge">bind</code> directly or use it as an
operator <code class="language-plaintext highlighter-rouge">&gt;&gt;=</code>.
Our example from above:
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const process = (ma, mb, mc) =&gt;
bind(ma)((a) =&gt;
bind(mb)((b) =&gt;
bind(mc)((c) =&gt;
maybe(a + b + c)
)
)
);
</code></pre></div></div>
In a language like Haskell, the code becomes:
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>process ma mb mc =
ma &gt;&gt;= \a -&gt;
mb &gt;&gt;= \b -&gt;
mc &gt;&gt;= \c -&gt;
return (a + b + c)
</code></pre></div></div>
By the way, check out <a href="https://www.haskell.org/">haskell.org</a> and notice how the logo is similar to <code class="language-plaintext highlighter-rouge">bind</code>.
Those folks take <code class="language-plaintext highlighter-rouge">&gt;&gt;=</code> seriously.
Now let’s see how the same code looks like when using <code class="language-plaintext highlighter-rouge">do</code> notation, which is
syntax sugar over <code class="language-plaintext highlighter-rouge">bind</code>:
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>process ma mb mc = do
a &lt;- ma
b &lt;- mb
c &lt;- mc
return (a + b + c)
</code></pre></div></div>
The final result looks like an imperative language - one might argue that
when you type <code class="language-plaintext highlighter-rouge">;</code> in JS, you are, in fact, <code class="language-plaintext highlighter-rouge">bind</code>ing monadic code (I first
learned about this on Bartosz Milewski’s course <a href="https://www.youtube.com/watch?v=PlFgKV0ZXoE&amp;list=PLbgaMIhjbmEm_51-HWv9BQUXcmHYtl4sw">Parallel and Concurrent Haskell</a>.
The interesting thing about this snippet is that the same syntax for <code class="language-plaintext highlighter-rouge">process</code>
will work for other data types beyond <code class="language-plaintext highlighter-rouge">Maybe</code>. The only requirements for that
are:
<ul>
<li>the data type (the <code class="language-plaintext highlighter-rouge">m</code> in <code class="language-plaintext highlighter-rouge">m a</code>) “knows” how to <code class="language-plaintext highlighter-rouge">bind</code> (and to do that, the
data type needs to have some sort of <code class="language-plaintext highlighter-rouge">map</code> and <code class="language-plaintext highlighter-rouge">join</code>);</li>
<li>the data type “knows” how to wrap a value with <code class="language-plaintext highlighter-rouge">return</code>/<code class="language-plaintext highlighter-rouge">pure</code>.</li>
</ul>
And as we are performing a sum, there’s an additional requirement:
<ul>
<li>the contained data (the <code class="language-plaintext highlighter-rouge">a</code> in <code class="language-plaintext highlighter-rouge">m a</code>) “knows” what to do with <code class="language-plaintext highlighter-rouge">+</code>.</li>
</ul>
As we are using JavaScript, this means that we can use <code class="language-plaintext highlighter-rouge">+</code> to sum numbers or concatenate strings.
In the FP world, a function “knowing” what to do with a given type is handled
by type classes, let’s focus on this monadic stuff for now.
<h1 id="why-monads">Why Monads?</h1>
So this is the power that this abstraction provides: all the data plumbing and
small details handling are abstracted with the same interface for multiple
data types.
In Haskell, here’s how the definitions would look like (with types included):
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>process :: Monad m =&gt; Num a =&gt; m a -&gt; m a -&gt; m a
process ma mb = do
a &lt;- ma
b &lt;- mb
return (a + b)
-- Adding things that might not exist
processMaybe :: Maybe Integer
processMaybe = process (Just 3) Nothing
-- Adding values that might contain errors
processEither :: Either String Integer
processEither = process (Right 2) (Left "an error happened!")
-- Adding something that was obtained with a side effect
processIO = do
ma &lt;- getStuffFromDB
mb &lt;- getStuffFromFile
process ma mb
</code></pre></div></div>
If your function only cares about adding two numbers, let it just sum.
The error handling can be pushed outside of sum, allowing you to
separate the pure and impure parts of your code even further.
This abstraction also helps to test (free monads), as you can replace the monad
you are using during testing (like stubbing or mocking in some testing
frameworks). This allows you to replace functions that obtain their values with
side effects with an <code class="language-plaintext highlighter-rouge">Either</code> or <code class="language-plaintext highlighter-rouge">Maybe</code>, among others.
Monads also make refactoring easier - if you have a set of functions that handle
some <code class="language-plaintext highlighter-rouge">Maybe</code>s using <code class="language-plaintext highlighter-rouge">bind</code>, you should be able to replace the <code class="language-plaintext highlighter-rouge">Maybe</code>s with
<code class="language-plaintext highlighter-rouge">Either</code>, if the necessity arises.
And that’s it! I tried to keep it short, but the topic is quite extensive and I
barely scratched the surface. Now even if you don’t end up using monads, you
better know what kind of problems they help solve.
Now go ahead and write your monad tutorial as well!
<h2 id="other-resources">Other resources</h2>
If you are interested in using these data structures in JavaScript, alongside
their useful <code class="language-plaintext highlighter-rouge">map</code>, <code class="language-plaintext highlighter-rouge">bind</code> functions, some libraries that can help you:
<ul>
<li><a href="https://github.com/gcanti/fp-ts">fp-ts</a> - implementations of various data
types and utility functions in TypeScript;</li>
<li><a href="https://github.com/sanctuary-js/sanctuary">Sanctuary</a> - provides utilities
for writing code in an FP style and some type checking while in development mode;</li>
<li><a href="https://www.purescript.org/">PureScript</a> - not JavaScript, but it compiles to
JS and can easily interface with existing JS code.</li>
</ul>
</description>
</item>
<item>
<title>Scheduled Tasks with ECS and Step Functions</title>
<link>https://tech.nextroll.com/blog/dev/2022/08/03/step-functions.html</link>
<pubDate>Wed, 03 Aug 2022 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2022/08/03/step-functions</guid>
<description><h1 id="background">Background</h1>
Amazon Web Services allows you to <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduling_tasks.html">schedule tasks</a> using ECS and CloudWatch, and for noncritical jobs, it is enough to run these regularly. It is simple, and we can manage many scheduled tasks with code using Terraform without the overhead and complexity of cloud orchestration systems such as Kubernetes.
But there’s a caveat to task scheduling with ECS + CloudWatch. The duo is missing two essential features for our workflow: task locks and timeouts. This post will cover how we leveraged two other AWS services, Step Functions and DynamoDB, to add these missing features.
<h1 id="the-problem">The Problem</h1>
In the Real-time Bidding (RTB) team within NextRoll, we previously managed scheduled tasks using <a href="https://docs.celeryq.dev/en/stable/">Celery</a>, which is good in some respects, such as being flexible and integrating with Python (the language in which we wrote most of our tasks). But it also had significant drawbacks: it was hard to maintain and limited our programming language choices to Python. More importantly, our code deploys were tied to Celery deploys. That interaction caused minor but recurrent problems, such as task locks not getting released or multiple tasks of the same kind getting dispatched.
Because we were going to invest a significant amount of time fixing our problems with Celery anyway, we decided to invest some time investigating alternatives and comparing. So we defined some minimum requirements for the task management system to make the scope of our search smaller:
<ul>
<li>
Cron-like scheduling
The essential feature. It consists of running a task at fixed times, dates, or intervals.
</li>
<li>
Singleton behavior
We should be able to constrain task definitions so that only a single instance of it can run at any given time. This behavior helps prevent race conditions and resource starvation scenarios while producing repeatable side effects.
</li>
<li>
Timeouts
Tasks should also be constrained to only run for a limited and configurable interval. This behavior prevents runaway resource consumption and frozen tasks and helps identify bugs quicker.
</li>
</ul>
<h1 id="some-solutions">Some Solution(s)</h1>
We found a couple of options that met our minimum requirements, but they didn’t quite fit our taste:
<ul>
<li>
Code it into each of the tasks, perhaps via shared libraries.
Sometimes, timeouts and locking do not work when managed by the task itself. Funny bugs and unexpected situations are not uncommon, as experienced when attempting this solution using Python in a previous iteration.
We prefer a solution that manages the task from outside its container for reliability and ease of use. It also allows for more flexibility when choosing a programming language for a task because we don’t need to have existing libraries in the target language to manage it.
</li>
<li>
Use a cloud orchestration system such as Kubernetes or Mesos.
Orchestration systems have their complexities and management overhead. We didn’t have a Kubernetes expert in our team, for example. And other systems don’t have managed solutions in AWS. Although we prefer cloud orchestration systems to code-in-tasks, we continued to look for something simpler.
</li>
</ul>
While good in some aspects, these solutions drove us to keep looking for alternatives. One of those was ECS + CloudWatch scheduling, which we almost discarded because there is no way to achieve our minimum requirements (<a href="https://github.com/aws/containers-roadmap/issues/232">singleton behavior</a> or <a href="https://github.com/aws/containers-roadmap/issues/1313">timeout behavior</a>) with only ECS and CloudWatch configurations.
<h1 id="enter-step-functions">Enter Step Functions</h1>
<img src="/images/post_images/step_functions/gregorydickson.png" alt="gregorydickson" />
Looking past the discouraging GitHub issues, we found a suggestion in the comments that could perhaps solve our problem. We looked at the Step Functions service, and we found that in combination with the ECS and CloudWatch systems, this met our requirements:
<ul>
<li>A managed system with a low overhead configuration</li>
<li>Outside-the-container task management for singleton behavior and timeouts</li>
<li>Bonus: integrations with other AWS services</li>
</ul>
Although it is a novel approach, documentation is plenty, and it was easy to test it out before completely committing to the solution. We used the Step Functions integration with DynamoDB to implement the singleton behavior using per-task locks and the integration with ECS tasks to define timeouts.
But talk is cheap. On to the code!
<h1 id="defining-the-step-functions">Defining the Step Functions</h1>
<img src="/images/post_images/step_functions/step-function.png" alt="Step Function" />
<blockquote>
AWS Step Functions provides serverless orchestration for modern applications.
</blockquote>
Step Functions are a list of steps or states (as in state machine) defined via JSON. You can define those steps using the AWS console for a visual aid useful for prototyping or, as we prefer, using Terraform and keeping our infrastructure as code.
We start here intending to configure a single task with singleton behavior that times out after a configurable amount of time using Terraform. After setting this up, you should be able to trigger the task from the Step Functions section in the AWS console.
Using the <code class="language-plaintext highlighter-rouge">aws_sfn_state_machine</code> from the AWS library we define a step function that checks and adds a lock corresponding to the task. If successful, it will run the task and remove the lock after it finishes. Here’s the code for the step function itself, the “steps” go nested inside the <code class="language-plaintext highlighter-rouge">States</code> map as key/value pairs:
<div class="language-terraform highlighter-rouge"><div class="highlight"><pre class="highlight"><code>resource "aws_sfn_state_machine" "task_lock" {
name = var.task_name
role_arn = var.sfn_role
definition = jsonencode({
Comment = "Task Lock"
StartAt = "CheckLock"
States = {
</code></pre></div></div>
Now we’ll define the <code class="language-plaintext highlighter-rouge">CheckLock</code> step. This step will add an item that corresponds to the task to DynamoDB on the condition that it doesn’t exist, and if that succeeds, we run the ECS Task (<code class="language-plaintext highlighter-rouge">RunTask</code> step). Here’s the step definition :
<div class="language-terraform highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CheckLock = {
Type = "Task"
Resource = "arn:aws:states:::dynamodb:putItem"
Parameters = {
TableName = var.table_name
Item = {
Task = { S = var.task_name }
Locked = { BOOL = true }
}
ConditionExpression = "attribute_not_exists(Task)"
}
Next = "RunTask"
}
</code></pre></div></div>
Here are the details:
<ul>
<li><code class="language-plaintext highlighter-rouge">Type</code> is the <a href="https://states-language.net/spec.html#state-type-table">type of step</a> we’re defining.</li>
<li><code class="language-plaintext highlighter-rouge">Resource</code> is the <a href="https://docs.aws.amazon.com/step-functions/latest/dg/connect-supported-services.html">AWS resource</a> we’ll instantiate.</li>
<li><code class="language-plaintext highlighter-rouge">Parameters</code> are the parameters passed to the <code class="language-plaintext highlighter-rouge">Resource</code>, different for each kind of resource. In particular, <code class="language-plaintext highlighter-rouge">ConditionExpression</code> tells DynamoDB to return an error if the item key is in the table already.</li>
<li><code class="language-plaintext highlighter-rouge">Next</code> is the task that will get run after this.</li>
</ul>
For the <code class="language-plaintext highlighter-rouge">RunTask</code> step we’ll run an ECS task and wait for it to finish. After the task finishes, we’ll go to the <code class="language-plaintext highlighter-rouge">RemoveLock</code> step:
<div class="language-terraform highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RunTask = {
Type = "Task"
Resource = "arn:aws:states:::ecs:runTask.sync"
Parameters = {
Cluster = var.task_cluster
TaskDefinition = aws_ecs_task_definition.default.arn
}
TimeoutSeconds = var.timeout_minutes * 60
Next = "RemoveLock"
}
</code></pre></div></div>
In the snippet above, the key sections are the resource <code class="language-plaintext highlighter-rouge">runTask.sync</code>, where the <code class="language-plaintext highlighter-rouge">.sync</code> part means the step function will wait for the ECS task to complete, and the <code class="language-plaintext highlighter-rouge">TimeoutSeconds</code>, after which the step function will terminate the ECS Task resource if it hasn’t already.
Finally, we remove the lock:
<div class="language-terraform highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RemoveLock = {
Type = "Task"
Resource = "arn:aws:states:::dynamodb:deleteItem"
Parameters = {
TableName = var.locking.table.name
Key = {
Task = { S = var.name }
}
}
End = true
}
</code></pre></div></div>
Here, the only new parameter is the <code class="language-plaintext highlighter-rouge">End</code> parameter. It means no steps remain, and the step function finished successfully.
<h1 id="next--steps">Next = “Steps”</h1>
In this post, we’ve worked on a simplified version of a task-locking Step Function. It is missing a couple of important sections that will allow tasks and operations to fail gracefully, such as the <a href="https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html">Catch and Retry</a> sections. You will also need to define the permissions for the step function role and the task scheduling using CloudWatch. But those should be easy to figure out.
An additional recommendation: encapsulate these configurations in a Terraform module that allows easy reuse and shared resources (such as the DynamoDB table).
<h1 id="conclusion">Conclusion</h1>
Step Functions are an effective tool to integrate different services from AWS without writing code. The one we’ve defined for managing our scheduled tasks has run without issue, and its operation has been simple. We’ve found that Step Functions are a service we will keep close by in our dev toolbox.
</description>
</item>
<item>
<title>Runtime errors: Come again? Rust macros to the rescue.</title>
<link>https://tech.nextroll.com/blog/dev/2022/06/21/rust-lua.html</link>
<pubDate>Tue, 21 Jun 2022 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2022/06/21/rust-lua</guid>
<description>We admit that this article is blatant Rust propaganda. If you are rust-curious, this is the place for you.
We want to show you how the benefits of using Rust go beyond the language itself and trickle down to your whole ecosystem.
Dynamic languages can undoubtedly be fun and convenient.
However, always being dynamic is not necessarily a good thing.
The word on the street is that unit testing is driven by dynamic languages precisely because they do not have a static type system.
We can eliminate whole classes of bugs by using a statically typed language. So, we shall use it.
Rust’s metaprogramming features allow us to be dynamic while preserving stability and safety.
If you are unfamiliar with Rust macros and have a macro-phobia from C++, they are not the same.
You should check them out <a href="https://doc.rust-lang.org/book/ch19-06-macros.html">here</a>.
<h1 id="background">Background</h1>
We are using Aerospike (distributed key-value store). It has a feature where you can register and execute UDF (user-defined functions) written in Lua.
The caveat is that Aerospike has some restrictions on the language, like reserved keywords, disallowing globals, and such.
So even with a modern IDE and linter, you do not get proper validation until you have registered the code in your Aerospike node.
You might say, “Hold on, what does Lua have to do with Rust?”.
Well, if you happen to use it from a Rust application, you can write a little macro that enforces the Aerospike rules on Lua and runs the Lua interpreter to catch the relevant errors at compile time.
<h1 id="what-we-want-to-achieve">What we want to achieve</h1>
At this point, we do not even need <code class="language-plaintext highlighter-rouge">.lua</code> files.
Here is how it would look.
Notice the Lua source code inside the macro invocation.
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code>use aerospike_code_gen::define;
fn main() {
define! {
function my_func(record)
end
};
}
</code></pre></div></div>
Since <code class="language-plaintext highlighter-rouge">record</code> is a reserved Aerospike identifier, we get a compilation error.
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code>error: local name reserved: `record`
--&gt; tests/main.rs:5:24
|
5 | function my_func(record)
| ^^^^^^
|
= note: this error originates in the macro `define` (in Nightly builds, run with -Z macro-backtrace for more info)
</code></pre></div></div>
<h1 id="development">Development</h1>
How can we make this work?
We need to create a Rust proc macro project and define our dependencies.
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[dependencies]
quote = "1"
proc-macro2 = "1.0"
syn = "1.0"
rlua = "0.19.1"
luaparse = "0.2.0"
</code></pre></div></div>
We need <code class="language-plaintext highlighter-rouge">rlua</code> to run the interpreter through and <code class="language-plaintext highlighter-rouge">luaparse</code> to parse the code to enforce the Aerospike requirements.
The other dependencies are the default libraries one would use to build macros.
Now, it is only a matter of parsing the macro input via the <code class="language-plaintext highlighter-rouge">luaparse</code> crate, recursively checking the definitions, and we are done.
In the actual argument of kings, Linus Torvalds said: “Talk is cheap. Show me the code”.
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#[proc_macro]
pub fn define(input: TokenStream) -&gt; TokenStream {
let s = input.to_string();
let mut lua_err = None;
Lua::new().context(|lua| {
let chunk = lua.load(&amp;s);
let r = chunk.exec();
if let Err(err) = r {
lua_err = Some(err.to_string());
}
});
if let Some(err) = lua_err {
return SynError::new(Span::call_site(), err)
.into_compile_error()
.into();
}
}
</code></pre></div></div>
This was easy enough, and it will give us the errors from the Lua interpreter.
The next step involves a little more work: validating the Aerospike requirements.
First, we create a function that will be called from the macro.
<code class="language-plaintext highlighter-rouge">luaparse::parse</code> is going to give us the entire AST.
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fn validate_aerospike(s: &amp;str) -&gt; Vec&lt;String&gt; {
let mut errs = vec![];
match luaparse::parse(s) {
Ok(block) =&gt; {
loop_statements(&amp;block.statements, &amp;mut errs, true);
}
Err(e) =&gt; panic!("{:#}", LError::new(e.span(), e).with_buffer(s)),
}
errs
}
</code></pre></div></div>
<code class="language-plaintext highlighter-rouge">loop_statements</code> is where our real work begins.
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fn loop_statements(stmts: &amp;Vec&lt;Statement&gt;, errors: &amp;mut Vec&lt;String&gt;, is_global: bool) {
for statement in stmts {
recurse(statement, errors, is_global);
}
}
</code></pre></div></div>
We need to go through the entire tree. For your sanity, the <code class="language-plaintext highlighter-rouge">match</code> statement in this function is shortened for this example.
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fn recurse(stmt: &amp;Statement, errors: &amp;mut Vec&lt;String&gt;, mut is_global: bool) {
let mut allow_vars = true;
if is_global {
is_global = false;
allow_vars = false;
}
match stmt {
Statement::FunctionDeclaration(func) =&gt; match func {
FunctionDeclarationStat::Local { body, .. } =&gt; {
validate_func(&amp;body, errors, is_global);
}
FunctionDeclarationStat::Nonlocal { body, .. } =&gt; {
validate_func(&amp;body, errors, is_global);
}
},
Statement::LocalDeclaration(ld) =&gt; {
let names = ld.names.pairs.iter().map(|a| a.0.clone()).collect();
validate_names(&amp;names, errors, allow_vars);
}
_ =&gt; {}
}
}
</code></pre></div></div>
Here is where we check for the reserved Aerospike names.
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const AEROSPIKE_NAMES: [&amp;str; 9] = [
"record",
"map",
"list",
"aerospike",
"bytes",
"geojson",
"iterator",
"list",
"stream",
];
fn is_reserved(s: &amp;str) -&gt; bool {
for aerospike_name in AEROSPIKE_NAMES {
if aerospike_name == s {
return true;
}
}
false
}
fn validate_names(names: &amp;Vec&lt;Name&gt;, errors: &amp;mut Vec&lt;String&gt;, allow_vars: bool) {
for param in names {
let name = param.to_string();
if is_reserved(&amp;name) {
errors.push(format!(
"aerospike reserved identifier: `{}`. consider renaming your variable",
name
));
}
if !allow_vars {
errors.push(format!("global variables are not allowed: `{}`", name));
}
}
}
fn validate_func(body: &amp;FunctionBody, errors: &amp;mut Vec&lt;String&gt;, is_global: bool) {
let names = body.params.list.pairs.iter().map(|a| a.0.clone()).collect();
validate_names(&amp;names, errors, true);
loop_statements(&amp;body.block.statements, errors, is_global);
}
</code></pre></div></div>
You can view a complete working example <a href="https://github.com/Adroll/aerospike-code-gen">here</a>.
This tremendously improves the development process, as one does not need to run code to register it on the Aerospike cluster to see an error that should have been caught at compile time.
<h1 id="conclusion">Conclusion</h1>
We all know the Rust trifecta - safe, fast and concurrent.
Rust allows us to write the code we want while preserving those three inherent qualities.
However, there is no free lunch in this world.
While there have been many improvements in compilation speed, adding more macros will exacerbate the problem if you have an extensive project.
</description>
</item>
<item>
<title>Embracing ARM</title>
<link>https://tech.nextroll.com/blog/dev/2022/05/16/embracing_arm.html</link>
<pubDate>Mon, 16 May 2022 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2022/05/16/embracing_arm</guid>
<description><center style="margin-bottom: 30px">
<img alt="man hugging computer" src="/images/post_heroes/embracing.jpg" /> 
</center>
<h1 id="introduction">Introduction</h1>
From AWS Graviton instances to M1 MacBooks, ARM processors are everywhere these days. This blog post illustrates the experiences, challenges, and solutions encountered in the Data Science Engineering team at NextRoll as we have come to embrace the ARM ecosystem.
<h1 id="docker--containerization">Docker / Containerization</h1>
For many years, the Data Science Engineering team has used Docker to create a consistent running environment for our code: Once code is working on one developer’s laptop, it should work on their teammate’s laptops, in our CI test pipelines, and when deployed to our cloud infrastructure.
Everything has worked great until recently when a developer on our team upgraded their work laptop to an M1 MacBook (which uses an ARM CPU architecture). Suddenly many of our images would not build or run on their laptop, preventing them from developing locally.
While the developer could develop, e.g., remotely on an X86 machine, more team members would be upgrading their laptops over time. So as a team, we decided to make our Docker images compatible with ARM.
<h1 id="first-approach">First Approach</h1>
The first approach we tried was to use simple emulation: Docker provides an emulation layer via <a href="https://wiki.qemu.org/Main_Page">QEMU</a>, which should allow a developer on an ARM laptop to build and run a docker image targeted for X86.
There were some images where this worked (albeit at a performance hit) but other images failed to run entirely. Of particular note, in our <a href="https://github.com/AdRoll/fledge_lab">Fledge Lab</a> project, headful Chrome would immediately segfault when run in an X86 docker container on an ARM host machine.
<h1 id="solution-multi-arch-images">Solution: Multi-Arch images</h1>
For cases where emulation failed, we went the route of multi-arch images: We would build both an X86 and ARM version for each image, leading to several challenges.
<h2 id="differing-build-steps">Differing Build Steps</h2>
Some packages or tools we installed in the image required different build steps in our Dockerfile. We found the most elegant way to address this was via <a href="https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/">RUN heredocs</a>: We use one RUN heredoc per package/tool we install and switch on the <code class="language-plaintext highlighter-rouge">$TARGETPLATFORM</code> variable provided by Docker.
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">// Install a given tool/package
RUN &lt;&lt;EOF
if [ $TARGETPLATFORM == 'linux/arm64' ]
then
// Build steps for ARM
else
// Build steps for X86
fi
EOF</code></pre></figure>
<h2 id="code-compatibility-issues">Code Compatibility Issues</h2>
We ran into several issues getting code optimized for X86 to compile and run on ARM.
We had some places in our code where we used X86-specific instructions, including X86 inline assembly and Intel intrinsics. We replaced these with a mix of LLVM IR code, which is architecture agnostic, and code sections where we switch on the target platform.
We also ran into a compatibility issue using epoll: On x86 machines, the epoll_event struct is expected to be packed, whereas, on ARM machines, it is not, leading to odd runtime behavior where threads would try to read data from the wrong client connection and took a while to debug. Once we diagnosed the issue, the fix was simple: We just needed to switch the struct alignment on the machine architecture.
<figure class="highlight"><pre><code class="language-dlang" data-lang="dlang">struct EpollEvent
{
uint events = EPOLLIN | EPOLLONESHOT | EPOLLRDHUP;
version(X86_64)
align(4) ulong ID; // User data, packed
else
ulong ID; // User data, not packed
}</code></pre></figure>
<h2 id="memory-model-issues">Memory Model Issues</h2>
One of the most subtle issues we ran into was the difference between the X86 and ARM memory models. There is an excellent introduction to memory models and code reordering <a href="https://preshing.com/20120612/an-introduction-to-lock-free-programming/">here</a>, but the rough idea is as follows:
Code that is written in a high-level language (Dlang in our case) can be reordered in many places:
<ul>
<li>At compile time: The compiler outputs assembly in a different order than the original code.</li>
<li>At runtime: The CPU executes the assembly/machine instructions out of order.</li>
</ul>
The CPU’s memory model determines what sort of reorderings are allowed at runtime. X86 has a strong memory model that prevents many reorderings of memory access instructions. ARM has a weak memory model, which gives the CPU much more liberty to rearrange memory access instructions.
In order to achieve optimal performance for our ad pricing engine, we relax the memory ordering constraints for atomic operations in our multi-threaded code in several places (e.g., in collecting statistics for the 100k ads we price per second). When targeting ARM, we needed to audit our code to ensure we were not implicitly relying on the stronger memory guarantees of X86, lest we introduce a subtle race condition. (As an interesting aside, Apple sidestepped this issue with Rosetta 2 by letting it use <a href="https://twitter.com/ErrataRob/status/1331735383193903104">X86 memory-ordering</a>)
<h3 id="a-concrete-example">A Concrete Example</h3>
The following example shows how the different memory models of X86 and ARM can lead to different behavior at runtime.
<figure class="highlight"><pre><code class="language-dlang" data-lang="dlang">import core.atomic;
import core.thread;
align(128) __gshared bool mutex1;
align(128) __gshared bool mutex2;
void do_work()
{
while(true)
{
while(atomicExchange(&amp;mutex1, true)){} // block for mutex1
while(atomicExchange(&amp;mutex2, true)){} // block for mutex2
mutex2 = false; // release mutex2
mutex1 = false; // release mutex1
}
}
void main(string[] args)
{
auto thread = new Thread(&amp;do_work);
thread.start();
while(true)
{
while(atomicExchange(&amp;mutex1, true)){} // block for mutex1
if(atomicLoad(mutex2) == true)
assert(false);
atomicStore(mutex1, false); // release mutex1
}
}</code></pre></figure>
Here a worker thread acquires two mutexes in a nested manner, while the main thread checks if <code class="language-plaintext highlighter-rouge">mutex1</code> is free while <code class="language-plaintext highlighter-rouge">mutex2</code> is held, violating the nesting. When I compile and run this on a pre-M1 MacBook (X86), it runs indefinitely, whereas when I compile and run it on an M1 MacBook (ARM) it hits the assertion almost immeadiately.
What’s going on? The issue lies in the lines
<figure class="highlight"><pre><code class="language-dlang" data-lang="dlang">mutex2 = false; // release mutex2
mutex1 = false; // release mutex1</code></pre></figure>
which compile to
<figure class="highlight"><pre><code class="language-asm" data-lang="asm">strb wzr, [x10]
strb wzr, [x8]</code></pre></figure>
and
<figure class="highlight"><pre><code class="language-asm" data-lang="asm">mov byte ptr [rcx], 0
mov byte ptr [rax], 0</code></pre></figure>
on <a href="https://godbolt.org/z/s67Mc5add">ARMv8</a> and <a href="https://godbolt.org/z/YeMoETjnM">X86</a> respectively.
Even though these instructions are still in order at compile time, the ARM memory model allows these stores to be run out of order whereas the X86 memory model prevents the reordering of these stores. To fix this, we can instead use
<figure class="highlight"><pre><code class="language-dlang" data-lang="dlang">atomicStore(mutex2, false); // release mutex2
atomicStore(mutex1, false); // release mutex1</code></pre></figure>
which compiles to
<figure class="highlight"><pre><code class="language-asm" data-lang="asm">stlrb wzr, [x10]
stlrb wzr, [x8]</code></pre></figure>
and
<figure class="highlight"><pre><code class="language-asm" data-lang="asm">xor edx, edx
xchg byte ptr [rcx], dl
xor edx, edx
xchg byte ptr [rax], dl</code></pre></figure>
on <a href="https://godbolt.org/z/9sbf6YKfb">ARMv8</a> and <a href="https://godbolt.org/z/44bdYY3KG">X86</a> respectively. In ARM, the <code class="language-plaintext highlighter-rouge">strb</code> instructions are now <code class="language-plaintext highlighter-rouge">stlrb</code> instructions. These cannot be reordered (they each provide a one-way fence from above), so the intended nesting is guaranteed.
Even for X86 I believe the latter code is more correct. Even though the <code class="language-plaintext highlighter-rouge">mov</code> instructions are replaced with heavier <code class="language-plaintext highlighter-rouge">xchg</code> instructions, we are expressing our intention that these lines not be rearranged in the dlang code itself and leaving it to the compiler to ensure our intentions are met. In particular, in the former code, I think the compiler could have rearranged the order of these two lines, even before the runtime ordering guarantees of X86.
<h2 id="conclusion">Conclusion</h2>
While it took some work (and a fair bit of debugging), we successfully made all of our Data Science Engineering images compatible with both X86 and ARM. And along the way we got to learn some of the subtle differences between the two CPU architectures! If optimizing machine learning systems at a low level catches your interest, check out our <a href="https://www.nextroll.com/careers">careers page</a>.
</description>
</item>
<item>
<title>Introducing Rebar3 TypEr</title>
<link>https://tech.nextroll.com/blog/dev/2022/04/27/rebar3-typer.html</link>
<pubDate>Wed, 27 Apr 2022 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2022/04/27/rebar3-typer</guid>
<description><center style="margin-bottom: 30px">
<img alt="a typewriter" src="/images/post_heroes/typewriter.jpg" /> 
Introducing Rebar3 TypEr
</center>
Do you have an Erlang codebase that’s a little short on type specs? Maybe you know <a href="https://www.erlang.org/doc/man/typer.html">TypEr</a> exists, but wouldn’t it be nice to:
<ul>
<li>incorporate it into your build system?</li>
<li>configure it only once?</li>
<li>have it ✨ magically✨ figure out all the command line arguments for you?</li>
</ul>
<h2 id="good-news">Good news</h2>
We built that! It’s called <code class="language-plaintext highlighter-rouge">rebar3_typer</code>, and it (as the name implies) plugs into rebar3. With that rebar3 integration comes understanding the rest of your configuration. It’s <a href="https://hex.pm/packages/rebar3_typer">available on hex.pm</a>.
One of the first things we had to do was refactor the original upstream TypEr a bit. TypEr was always intended to be run from the command line. It didn’t have (or need) an API available from within the BEAM. Well, until now. We needed that. So, we split out a “core” module <a href="https://github.com/erlang/otp/pull/5660">and sent that upstream</a>. That’ll be in OTP 25.
On that note, contributions welcome. You can <a href="https://github.com/AdRoll/rebar3_typer/issues">file bugs with us</a>, <a href="https://github.com/AdRoll/rebar3_typer/pulls">submit improvements to our repo</a> (improvements to the hexdocs especially welcome), or <a href="https://github.com/erlang/otp">help out upstream</a>.
We have to thank Viacheslav Katsuba for his <a href="https://github.com/vkatsuba/rebar3_plugin">rebar3_plugin template</a>. It was very helpful for getting us a framework to start from so we could dive in. And of course, thanks to <a href="https://github.com/kostis">Kostis Sagonas</a> for all his work on Dialyzer and TypEr.
<h2 id="how-to-use-rebar3_typer">How to use rebar3_typer</h2>
For a quick up-and-running, just add this to your <code class="language-plaintext highlighter-rouge">rebar.config</code> (either in your project or globally in <code class="language-plaintext highlighter-rouge">~/.config/rebar3/rebar.config</code>):
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{plugins, [rebar3_typer]}.</code></pre></figure>
Then run…
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 dialyzer
rebar3 typer</code></pre></figure>
…and wait.
Without further configuration, it will:
<ul>
<li>look for source files in the usual places</li>
<li>look for source files wherever they’ve been configured in <code class="language-plaintext highlighter-rouge">rebar.config</code> for the project in general</li>
<li>look for Dialyzer’s PLT file in the usual place</li>
<li>look for Dialyzer’s PLT file wherever it’s been configured in <code class="language-plaintext highlighter-rouge">rebar.config</code> for the project in general</li>
<li>emit a list of all the specs you can add to those modules, based on the types inferred by Dialyzer</li>
</ul>
Or if you want to take advantage of our enhancement that automatically annotates your source code directly for you, try:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 dialyzer
rebar3 typer --annotate-in-place</code></pre></figure>
(Yes, this new feature has been <a href="https://github.com/erlang/otp/pull/5802">sent upstream</a>.)
<h2 id="options">Options</h2>
All options are available on the command line or in the <code class="language-plaintext highlighter-rouge">rebar.config</code> file. The command-line options always override the options in the config file.
The basic config starts out:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, []}.</code></pre></figure>
<h3 id="choose-the-files">Choose the files</h3>
If source file auto-detection doesn’t work for your situation, you’ll need to specify either a list of files (<code class="language-plaintext highlighter-rouge">-f</code>) or a list of directories (<code class="language-plaintext highlighter-rouge">-r</code>) (or both!) to be analyzed.
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 typer -r d1,d2 -f foo.erl</code></pre></figure>
Or
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, [
{recursive, ["d1/", "d2/"]},
{files, ["foo.erl"]}
]}.</code></pre></figure>
<h3 id="provide-the-plt">Provide the PLT</h3>
If <code class="language-plaintext highlighter-rouge">rebar3_typer</code> can’t find your PLT file (and you’re sure it’s already been generated), you can give it some hints.
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 typer --plt myfile.plt</code></pre></figure>
Or
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, [
{plt, "/path/to/plt"}
]}.</code></pre></figure>
<h3 id="show-or-annotate">Show or annotate</h3>
Do you want it to only print the type specs to stdout, or do you want it to put its output in files?
If you want them printed to stdout, do you want it for all functions?
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 typer --show</code></pre></figure>
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, [
{mode, show}
]}.</code></pre></figure>
Or maybe only for exported (public) functions?
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 typer --show-exported</code></pre></figure>
(<code class="language-plaintext highlighter-rouge">--show_exported</code> is also accepted)
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, [
{mode, show_exported}
]}.</code></pre></figure>
If you’d like the results to go to files, use an <code class="language-plaintext highlighter-rouge">annotate</code> option instead of a <code class="language-plaintext highlighter-rouge">show</code> option.
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 typer --annotate</code></pre></figure>
Or
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, [
{mode, annotate}
]}.</code></pre></figure>
Do you want to annotate <code class="language-plaintext highlighter-rouge">include()</code> files? (Warning, though: there is an <a href="https://github.com/erlang/otp/issues/5653">open bug</a> upstream for this feature.)
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 typer --annotate-inc-files</code></pre></figure>
(<code class="language-plaintext highlighter-rouge">--annotate_inc_files</code> is also accepted)
Or
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, [
{mode, annotate_inc_files}
]}.</code></pre></figure>
By default, annotations go into separate <code class="language-plaintext highlighter-rouge">.ann.erl</code> files in a <code class="language-plaintext highlighter-rouge">typer_ann/</code> directory. Would you like it to instead put the annotations directly into your code for you? Check out <code class="language-plaintext highlighter-rouge">--annotate-in-place</code>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 typer --annotate-in-place</code></pre></figure>
(<code class="language-plaintext highlighter-rouge">--annotate_in_place</code> is also accepted)
Or
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, [
{mode, annotate_in_place}
]}.</code></pre></figure>
By default, only functions with missing specs will be addressed, in any of these modes.
<h3 id="what-format-to-use">What format to use</h3>
By default, <code class="language-plaintext highlighter-rouge">rebar3_typer</code> will use <code class="language-plaintext highlighter-rouge">-spec</code> syntax. If you’d prefer Edoc syntax (<code class="language-plaintext highlighter-rouge">@spec</code>), you can specify that:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 typer --edoc</code></pre></figure>
Or
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, [
{edoc, true}
]}.</code></pre></figure>
<h3 id="how-to-handle-pre-existing-type-specs">How to handle pre-existing type specs</h3>
You have some options for what to do when it finds type specs already in place. If you have particular files with specs, and you want those specs to serve as the basis for additional type specs:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 typer -T f1,f2</code></pre></figure>
Or
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, [
{typespec_files, ["f1", "f2"]}
]}.</code></pre></figure>
(This accepts file names or directory names. But another warning: this <a href="https://github.com/erlang/otp/issues/5657">also has an open bug upstream</a>.)
If you’d rather existing type specs be completely ignored:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 typer --no_spec</code></pre></figure>
Or
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, [
{no_spec, true}
]}.</code></pre></figure>
Finally, TypEr has an option to show Dialyzer’s success typings, which is reachable through this plugin. This is undocumented upstream.
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 typer --show_success_typings</code></pre></figure>
Or <code class="language-plaintext highlighter-rouge">--show-success-typings</code>.
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{typer, [
{show_succ, true}
]}.</code></pre></figure>
<hr />
This project was part of Hack Week, the twice-yearly event where employees get to decide what we build. We learn things, try things out, and get to work with different folks than usual. You can <a href="https://tech.nextroll.com/blog/culture/2019/11/26/hackweek-at-nextroll.html">read more about Hack Week</a>.
</description>
</item>
<item>
<title>Rustenstein 3D: Game programming like it's 1992</title>
<link>https://tech.nextroll.com/blog/dev/2022/02/02/rustenstein.html</link>
<pubDate>Wed, 02 Feb 2022 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2022/02/02/rustenstein</guid>
<description>Twice a year, NextRoll celebrates Hack Week, where employees get to work for a week on a project of their choice. It’s an excellent opportunity to experiment, learn new technologies and team up with people from across the company. You can learn all about Hack Week <a href="https://tech.nextroll.com/blog/culture/2019/11/26/hackweek-at-nextroll.html">here</a>.
As NextRoll increasingly adopts the Rust programming language, it’s common for engineers to use Hack Week as an opportunity to gain experience with it. Another popular choice is to work on video games and, as you may have guessed, we frequently see them combined in Rust video game projects. Last year, a group of us worked on extending my <a href="https://github.com/facundoolano/rpg-cli/">rpg-cli game</a>. This time, though, we wanted to step it up a notch with a project that would exercise some of Rust’s strengths: low-level programming, intense computations and C data interoperability. And so we decided to port the classic <a href="https://en.wikipedia.org/wiki/Wolfenstein_3D">Wolfenstein 3D game</a> to Rust.
<h1 id="background">Background</h1>
id Software was famous for pushing the envelope of PC game programming: first by implementing NES-like side-scrollers on hardware that wasn’t prepared for it, then practically inventing and dominating the 3D first-person shooter genre, then making network and internet multiplayer a reality. Along the way, they also popularized the shareware distribution method, encouraged community modding and open-sourced all of their hit titles. <a href="https://en.wikipedia.org/wiki/Masters_of_Doom">Masters of Doom</a> by David Kushner tells the story; Fabien Sanglard’s <a href="https://fabiensanglard.net/gebb/index.html">Game Engine black books</a> explains the technical details.
<center>
<img alt="The Game Engine Black Book" src="/images/post_images/rustestein/book.png" style="width: 50%" />
 
</center>
Perhaps less notorious than its successors Doom and Quake, Wolfenstein 3D is a big milestone in id Software’s evolution and PC gaming in general. In addition, because its technology is more primitive, the source code is more approachable for study and implementation. The game doesn’t have a real 3D engine but rather simulates a 3D world from a 2D map using a technique called Ray Casting. All the drawing is done by directly putting pixels on the screen.
A few years ago, after reading the Wolfenstein black book, I spent some time trying to <a href="https://github.com/facundoolano/wolf4py">port it to Python</a>, based on another modern port, <a href="https://github.com/facundoolano/wolf4sdl">wolf4sdl</a>. I tried to remain as close as possible to the original source, which proved to be very difficult, so I eventually dropped the project. More recently, <a href="https://github.com/Oppen">Mario Rugiero</a>, who also read the book, proposed a Rust port as a project for this Hack Week. Several people jumped in, and so did I; although, based on my previous experience, the enterprise still seemed daunting: some of us were new to Rust, some had never played Wolf, some hadn’t read the book yet, and none had implemented ray casting before. We started without much hope of having something to show by the end of the week, but we saw enormous learning opportunities, so we dove right in.
<h1 id="development">Development</h1>
We roughly identified some components of the game that could be tackled separately, so each member picked one and went to work on it:
<ul>
<li>graphic files decompression and parsing</li>
<li>map files decompression, parsing and interpretation</li>
<li>SDL graphic manipulation and texture rendering</li>
<li>ray casting</li>
<li>game loop and input management</li>
<li>world rendering</li>
</ul>
In the cases where the output of one component was required as input for the next, we used pre-parsed or hard-coded data, extracted from the reference wolf4py and wolf4sdl implementations: decompressed binary dumps of assets, hardcoded maps and walls, etc. This allowed us to make progress in parallel.
<h2 id="assets">Assets</h2>
The first task of porting the game is to read its data. Wolfenstein ships with a set of files for its different assets: graphics (images, textures and sprites), audio (music and sound effects) and maps. One of the complications is that each version of the game has slightly different files, with different offsets and, in some cases, using different compression methods. For Rustenstein, we used the .WL1 files of the shareware version, which we <a href="https://github.com/AdRoll/rustenstein/tree/main/shareware">included in the repository</a>.
Each file uses a different combination of several decompression algorithms, all of which we had to port to Rust:
<ul>
<li>the traditional <a href="https://moddingwiki.shikadi.net/wiki/Huffman_Compression">Huffman compression</a></li>
<li><a href="https://moddingwiki.shikadi.net/wiki/Id_Software_RLEW_compression">RLEW compression</a>, a run-length encoding algorithm that works at the word level</li>
<li>a <a href="https://moddingwiki.shikadi.net/wiki/Carmack_compression">“Carmack”</a> compression, which is John Carmack’s variant of the LZ (Lempel-Ziv) method. According to the Black Book, without much access to the literature, Carmack would “invent” an algorithm to later find out that someone else had done it before.</li>
</ul>
The original Wolf engine has a Memory Manager component to handle memory allocation and compacting (instead of the traditional C <code class="language-plaintext highlighter-rouge">malloc</code>) as well as a Page Manager to move assets from disk to RAM. Both components are unnecessary in modern hardware as we can safely assume that we can fit all assets in memory, so we did not include them in our port.
Parsing and decompression code can be found <a href="https://github.com/AdRoll/rustenstein/blob/3c4452b38dad2ba0f5f3d2c07209b89bd61e50c2/src/map_parser.rs">here</a> for maps, and <a href="https://github.com/AdRoll/rustenstein/blob/3c4452b38dad2ba0f5f3d2c07209b89bd61e50c2/src/cache.rs">here</a> for the rest of the assets.
<h2 id="maps">Maps</h2>
Wolfenstein 3D maps are defined as 64x64 grids of tiles. Each map has two layers of tiles: one for walls and doors, and another to place the player, enemies, and bonus items. The different tile values determine what texture to render on walls, what locks are required for doors, what direction the player is facing, etc. All walls have the same height and since they are represented as blocks in the tile grid, all intersections are rectangular; while this constrains the level designs, it dramatically simplifies the ray casting algorithm for drawing the 3D world.
Below is the first map of the first episode, as seen in a Wolfenstein map editor:
<center>
<img alt="E1M1 viewed on an editor" src="/images/post_images/rustestein/map1.png" style="width: 50%" />
 
</center>
And the same map as ASCII, as printed by our debugging code:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWW
WWWWWW WWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWW
WWWWWW WWWWWWWW W W WWWWWWWWWWWWWWWWWWWW
WWWWWW WWWWWWW | | W WWWWWWWWWWWWWWWWWWWW
WWWWWW WWWWWWW W W WWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWW WWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWW
WWWWWW WWW WWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWW
WWWWWW WWW WWWWWWWWWWWWW-WWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWW W WWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWW | WWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWW W WWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWW WWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
W WW WWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
W WWWWWW-WWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
W WWWWW WWWWWWWWWWWWWWWW WW WWWWWWWWWWWWWWWWWWWWWWWWWW
WW-WWWWWW WWWWWWWWWWWWWWWW WWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
W WWWWW WWWWWWWWWWWWWWWW WWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
W W W WWWWWWWWWWWWWWWW WWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
W W WWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
W W W WWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
W WWWWWW-WWWWWWWWWWWWWWWWWWWWWWW-WWWWWWWWWWWWWWWWWWWWWWWWWWWWW
W WW WWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWW WW
W WW WWWWWWWWWWWW WWWWWWWWWWWW WW
W WW WWWWWWWWWWWW WWWWWWWWWWWW WW
W W WWWWWWWWWWWW W W WW
W | WWWWWWWWWWWW | | WW
W W WWWWWWWWWWWW W W WW
W WW WWWWWWWWWWWW WWWW WWWW WW
W WW WWWWWWWWWWWW WWWWW WWWWW WW
W WW WWWWWWWWWWWWWWWWW WWWWWWWWWW WWWWW WW
W WWWWWW-WWWWWWWWWWWWWWWWWWWWWWW-WWWWWWWWWWWW WWWWWWW WW WWWW
W WWWWW WWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWW WWWWWWWWWWWWWWW
W WWWWW WWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWW WWWWWWWWWWWWWWW
W W W WWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWW WWWWWWWWWWWWWWW
W W WWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWW WWW W W W WWWWW
W W W WWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWW W WWWW
W WWWWW WWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWW | WWWW
W WWWWW WWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWW W WWWW
W W WWWWWWWWW WWWWWWWWWWWWWWWW W W W WWWWW
W | W WWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
W W WWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWW W W WWWWWWWWWWWWWW-WWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWW W W W WWWWWWWWW W W WWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWW WWWWWWWWWWW | | WWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWW W W WWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWW WWWWWWWWWWWWW W W WWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWW WWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWW W W WWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWW P | | WWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWW W W WWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
</code></pre></div></div>
 
<h2 id="pixel-drawing">Pixel drawing</h2>
For the graphic assets, decompressing and loading the data to memory is just half of the story. The binary chunks that make each graphic (image, sprite or texture) are arranged explicitly for fast rendering on the VGA displays the game was initially designed for. This means that the graphics are rotated to be drawn in columns, and the columns themselves appear interleaved in the file since VGA allowed for parallel writing to four different video memory banks.
Each byte in the graphic binary chunks is an index to the 256 color palette used in Wolfenstein 3D. The reference wolf4sdl implementation would write those chunks to an SDL surface, which would, in turn, be translated to RGB colors before being copied to the screen, as described in <a href="http://sandervanderburg.blogspot.com/2014/05/rendering-8-bit-palettized-surfaces-in.html">this blog post</a>. Since the <a href="https://github.com/Rust-SDL2/rust-sdl2">Rust bindings for SDL</a> use a different set of abstractions (and, in particular, they don’t expose the <a href="https://wiki.libsdl.org/SDL_ConvertPixels">SDL_ConvertPixels</a> function), we opted for converting from palette index to RGB colors on the fly, writing directly to an RGB texture that then gets copied to the renderable canvas. This means that the rendering routines need to be adapted to write red, green and blue bytes to form each pixel, instead of the single palette index byte.
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fn put_pixel(buffer: &amp;mut [u8], pitch: usize, x: u32, y: u32, color: (u8, u8, u8)) {
let (r, g, b) = color;
let offset = y as usize * pitch + x as usize * 3;
buffer[offset] = r;
buffer[offset + 1] = g;
buffer[offset + 2] = b;
}
</code></pre></div></div>
 
The two graphic rendering routines we implemented were directly ported from the wolf4py implementation, which in turn was ported almost line by line from the wolf4sdl reference fork. The first routine handles displaying a full image directly to the screen. This is used for the title screen as well as the player status bar at the bottom of the in-game view:
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fn draw_to_texture(texture: &amp;mut Texture, pic: &amp;Picture, color_map: ColorMap) {
texture.with_lock(None, |buffer: &amp;mut [u8], pitch: usize| {
for y in 0..pic.height {
for x in 0..pic.width {
let source_index =
(y * (pic.width &gt;&gt; 2) + (x &gt;&gt; 2)) + (x &amp; 3) * (pic.width &gt;&gt; 2) * pic.height;
let color = pic.data[source_index as usize];
put_pixel(buffer, pitch, x, y, color_map[color as usize]);
}
}
});
}
</code></pre></div></div>
 
<center>
<img alt="The Game Engine Black Book" src="/images/post_images/rustestein/titlepic.png" style="width: 100%" />
 
</center>
The second routine, a much more complex one, is in charge of drawing sprites and is currently used to display the player weapon. A similar but even more complicated function is left to be ported: the one that draws scaled images such as wall textures and enemy sprites.
<center>
<img alt="Doge weapon" src="/images/post_images/rustestein/doge.png" style="width: 100%" />
An unexpected sprite appeared instead of the weapon during development.
 
</center>
It would be desirable to improve this implementation such that most of the processing is done once as part of the asset loading step, and the binary chunks are kept in memory, ready to be written to the screen.
The related code can be found <a href="https://github.com/AdRoll/rustenstein/blob/3c4452b38dad2ba0f5f3d2c07209b89bd61e50c2/src/main.rs#L194-L332">here</a>.
<h2 id="ray-casting">Ray Casting</h2>
At the heart of the Wolfenstein 3D engine is the Ray Casting algorithm. This routine allows us to project a 2D world (defined by the 64x64 tilemap) into a 3D view, solely based on 2D operations. The algorithm can be summarized as follows:
<ol>
<li>Cast a ray from the player’s current position for each pixel column in the screen width. For example, the classical Wolfenstein 3D resolution is 320x200, so this means casting 320 rays to draw a frame.</li>
<li>Extend the ray in a direction determined by the current horizontal pixel, the player’s position and its field of view, until it hits a wall in the map. Because the walls are rectangular, the calculations to extend the rays are greatly simplified, since there’s a constant distance between a tile and the next.</li>
<li>Once the ray intersects a wall, calculate the distance from the player to that wall, using trigonometry.</li>
<li>Set a height for the wall, inversely proportional to the calculated distance. This is: the further the wall the ray hits is from the player, the smaller the wall looks from the player’s perspective (and the smaller the column of pixels we will need to draw on the screen).</li>
</ol>
<center>
<img alt="Sample raycast map" src="/images/post_images/rustestein/raycast1.gif" style="width: 50%" />
 
</center>
Below is a simplified JavaScript version of the algorithm, based on <a href="https://github.com/vinibiavatti1/RayCastingTutorial">this tutorial</a>:
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>function rayCasting(screen, map, player) {
let precision = 64;
let incrementAngle = player.fieldOfView / screen.width;
let wallHeights = [];
let rayAngle = player.angle - player.fieldOfView / 2;
for(let rayCount = 0; rayCount &lt; screen.width; rayCount++) {
// start the ray at the player position
let ray = {
x: player.x,
y: player.y
};
// the ray moves at constant increments
let rayCos = Math.cos(degreeToRadians(rayAngle)) / precision;
let raySin = Math.sin(degreeToRadians(rayAngle)) / precision;
// advance the ray until it finds a wall (a non zero tile)
let wall = 0;
while(wall == 0) {
ray.x += rayCos;
ray.y += raySin;
wall = map[Math.floor(ray.y)][Math.floor(ray.x)];
}
// calculate the distance from the player to the wall hit
let distance = Math.sqrt(Math.pow(player.x - ray.x, 2) + Math.pow(player.y - ray.y, 2));
// calculate height at current x inversely proportional to the distance
wallHeights.push(Math.floor(screen.halfHeight / distance));
// increment the angle for the next ray
rayAngle += incrementAngle;
}
return wallHeights;
}
</code></pre></div></div>
 
For a ray casting implementation closer to the original Wolfenstein 3D one, <a href="https://lodev.org/cgtutor/raycasting.html">this series of tutorials</a> is recommended.
This routine was clearly the most challenging we tackled during this Hack Week, but we made a couple of decisions early on that reduced the complexity enough to be able to deliver something on time. First, we went with the most basic version of the algorithm that supports walls of solid colors instead of textures. Second, <a href="https://github.com/qhool">Josh Burroughs</a> figured out ray casting separately, based on tutorials, instead of trying to make a line-by-line port of the original Carmack implementation (which, quoting Sanglard, is a “fully-handcrafted 740 lines of highly unorthodox and super efficient assembly code”) or its wolf4sdl counterpart (which is C but still relied heavily on <code class="language-plaintext highlighter-rouge">goto</code> statements and had a lot of global side-effects in addition to calculating wall heights).
Here’s what the top-down view of the first Wolf map looked like after integrating it into the ray casting routine:
<center>
<img alt="E1M1 as viewed from the ray casting routine" src="/images/post_images/rustestein/mapcast.png" style="width: 100%" />
 
</center>
The full implementation can be found <a href="https://github.com/AdRoll/rustenstein/blob/3c4452b38dad2ba0f5f3d2c07209b89bd61e50c2/src/ray_caster.rs">here</a>.
<h2 id="world-rendering">World Rendering</h2>
The 3D world is displayed by first splitting the screen horizontally in two halves, painting the upper half with a ceiling solid color and the lower half with a floor color. After that, a pixel column needs to be drawn with the height received from the ray casting algorithm for each horizontal coordinate. While the algorithm was still in development, we tested the rendering code with hard-coded walls:
<center>
<img alt="Doge weapon" src="/images/post_images/rustestein/hardcoded.png" />
 
</center>
Once the ray casting routine was implemented and fed with an actual Wolfenstein map, we got an array of wall heights for each pixel column in the screen, and we started to see the world:
<center>
<img alt="Doge weapon" src="/images/post_images/rustestein/noshadow.png" />
 
</center>
Although we haven’t implemented texture rendering, there are a couple of tricks that improve the appearance of the scene: using different colors for the horizontal and vertical faces of a wall, and making the r, g, b components of each pixel inversely proportional to the distance to the player (which we know from the height of the wall), to generate a darkness effect:
<center>
<img alt="Doge weapon" src="/images/post_images/rustestein/start.png" />
 
</center>
The <a href="[here](https://github.com/AdRoll/rustenstein/blob/3c4452b38dad2ba0f5f3d2c07209b89bd61e50c2/src/main.rs#L99-L145)">world rendering code</a>, then, looks like this:
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code>texture
.with_lock(None, |buffer: &amp;mut [u8], pitch: usize| {
// draw floor and ceiling colors
let floor = color_map[VGA_FLOOR_COLOR];
let ceiling = color_map[VGA_CEILING_COLOR];
let vm = view_height / 6;
for x in 0..pix_width {
for y in 0..pix_height / 2 {
let ceilings = darken_color(ceiling, vm - y, pix_center);
put_pixel(buffer, pitch, x, y, ceilings);
}
for y in pix_height / 2..pix_height {
let floors = darken_color(floor, y - vm, pix_center);
put_pixel(buffer, pitch, x, y, floors);
}
}
for x in 0..pix_width {
// use different colors for horizontal and vertical wall faces
let mut color = if ray_hits[x as usize].horizontal {
color_map[150]
} else {
color_map[155]
};
let current = min(ray_hits[x as usize].height, pix_center);
color = darken_color(color, current, pix_center);
for y in pix_center - current..pix_center + current {
put_pixel(buffer, pitch, x, y, color);
}
}
})
fn darken_color(color: (u8,u8,u8), lightness: u32, max: u32) -&gt; (u8,u8,u8) {
let (r,g, b) = color;
let factor = lightness as f64 / max as f64 / DARKNESS;
let rs = (r as f64 * factor) as u8;
let gs = (g as f64 * factor) as u8;
let bs = (b as f64 * factor) as u8;
(rs, gs, bs)
}
</code></pre></div></div>
 
<h1 id="putting-it-all-together">Putting it all together</h1>
A day before the demo, we only had pieces of the game: asset loading wasn’t finished, we had show-stopper bugs in map parsing and sprite rendering, and the ray casting engine was working in a 2D hardcoded map, separate from the rest of the project. In an amazing few final hours, everything just fell into place: the bugs were ironed out, the different components fit together and, with a few hacks and a lot of ugly code, we managed to put together an impressive-looking video, just in time for the Hack Week demo session. We even had time to throw in a last-minute face animation of the character! The whole experience reminded me of those stories about video game companies putting together one-off builds in a hurry, just to make it to E3 demos.
This is still far from a functioning game, but it surpassed our most optimistic predictions from a few days before. During this week, we learned a fair deal about Rust, and we went further than we could have if we were working on our own. And the project ended up winning the technical award of the event!
<center>
<img alt="Doge weapon" src="/images/post_images/rustestein/gameplay.gif" />
 
</center>
The prototype is now <a href="https://github.com/AdRoll/rustenstein">published as open-source</a>, although, as said, the code still needs significant clean-up. Since the project was a lot of fun and, in this first week, we managed to solve some of the most challenging parts (assets loading, ray casting, sprite and wall rendering), we’re eager to continue working on it. Some of the features that we could tackle next are:
<ul>
<li>Rendering wall textures</li>
<li>Show and pick up items</li>
<li>Add enemies to the map, implement combat and enemy AI</li>
<li>Implement doors and keys</li>
<li>Implement push walls</li>
</ul>
</description>
</item>
<item>
<title>Stay on top of your Erlang deps with our latest rebar3 plugin</title>
<link>https://tech.nextroll.com/blog/dev/2021/09/01/erlang-rebar3-depup.html</link>
<pubDate>Wed, 01 Sep 2021 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2021/09/01/erlang-rebar3-depup</guid>
<description>Over the past few years, we have spent several <a href="/blog/culture/2019/11/26/hackweek-at-nextroll.html">hackweeks</a> writing, improving, and maintaining multiple open-source rebar3 plugins for the Erlang community, such as <a href="/blog/dev/2020/02/25/erlang-rebar3-format.html">the formatter</a>, <a href="https://hex.pm/packages/rebar3_lint">Elvis</a>, or <a href="/blog/dev/2021/01/06/erlang-rebar3-hank.html">Hank</a>. Particularly for Hank, we even wrote an <a href="https://arxiv.org/abs/2107.08699">academic paper</a> that, with the help of <a href="https://twitter.com/lauramcastro">Laura M. Castro</a>, I recently presented at the latest edition of the <a href="https://icfp21.sigplan.org/home/erlang-2021">ICFP Erlang Workshop</a>.
The most recent HackWeek was not an exception. That’s why I’m writing this article to introduce you to our latest plugin: <a href="https://hex.pm/packages/rebar3_depup">rebar3_depup</a>.
<code class="language-plaintext highlighter-rouge">10 minute read</code>
<hr />
<center>
<img alt="Cinderella" src="/images/post_images/cinderella.gif" /> 
Make your projects magically shine!
</center>
<h2 id="tldr">TL;DR</h2>
In the same way as any other <code class="language-plaintext highlighter-rouge">rebar3</code> plugin, you can skip this whole article, add <code class="language-plaintext highlighter-rouge">{plugins, [rebar3_depup]}.</code> to your <code class="language-plaintext highlighter-rouge">rebar.config</code> and run the following command:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ rebar3 help update-deps
…
A rebar plugin to update dependencies
Usage: rebar3 update-deps [-r [&lt;replace&gt;]] [-c [&lt;rebar_config&gt;]]
[-a [&lt;update_approx&gt;]] [-d [&lt;just_deps&gt;]]
[-p [&lt;just_plugins&gt;]] [-h [&lt;just_hex&gt;]]
-r, --replace Directly replace values in rebar.config. The
default is to just show you what deps can be
updated because this is an experimental feature and
using it can mess up your formatting and comments.
[default: false]
-c, --rebar-config File to analyze [default: rebar.config]
-a, --update-approx Update requirements starting with '~&gt;' as well as
the ones with a specific version. [default: true]
-d, --just-deps Only update deps (i.e. ignore plugins and
project_plugins). [default: false]
-p, --just-plugins Only update plugins and project_plugins (i.e.
ignore deps). [default: false]
-h, --just-hex Only update hex packages, ignore git repos.
[default: false]</code></pre></figure>
Or you can be brave and just run <code class="language-plaintext highlighter-rouge">rebar3 update-deps</code> (or even <code class="language-plaintext highlighter-rouge">rebar3 update-deps --replace</code> 😱). That will produce an output like the one below, listing all the dependencies you can update in your <code class="language-plaintext highlighter-rouge">rebar.config</code> to get them to their latest versions.
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">…
===&gt; rebar3_lint can be updated from 0.4.0 to 0.5.0
===&gt; rebar3_hank can be updated from 1.1.2 to 1.1.4
===&gt; rebar3_gpb_plugin can be updated from 2.21.0 to 2.22.0
…</code></pre></figure>
And that’s it! That’s our new plugin. It tells you what dependencies in your project have new versions available either on <a href="https://hex.pm">hex.pm</a> or <a href="https://github.com">Github</a>, so you can always stay up-to-date effortlessly.
<hr />
If you want to know more, keep reading. In the following sections, I’ll tell you:
<ul>
<li>The background story behind it (i.e., what itch does it scratch for us).</li>
<li>The things that it can and cannot do since it’s not perfect yet.</li>
<li>How we automated its usage for all of our projects.</li>
</ul>
Let’s dive into it, shall we?
<h2 id="background-story">Background Story</h2>
Within NextRoll, the RTB and Supply teams are responsible for maintaining more than 30 Erlang repositories in our organization’s <a href="https://github.com/AdRoll">Github account</a>. Some of them are open-sourced, some are private. Most of them depend on one another. They form a quite complex multi-leveled dependency tree.
These repositories have somewhat strict CI pipelines to check pull requests, and we want those pipelines to be predictable. That’s why, four our dependencies and even for project plugins, we use specific versioning like <a href="https://github.com/AdRoll/mero/blob/8c526966c487bdd0b5d04d69092e4315f666692e/rebar.config#L43-L47">this one</a>:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{deps, [{dynamic_compile, "1.0.0"}]}.
...
{project_plugins,
[{rebar3_hex, "~&gt; 6.11.6"},
{rebar3_format, "~&gt; 1.0.1"},
{rebar3_lint, "~&gt; 0.5.0"},
{rebar3_hank, "~&gt; 1.1.4"}]}.</code></pre></figure>
That guarantees that a new version of any of those plugins won’t introduce new warnings/reports in any pull request, unless the PR also updates the plugin version.
But, on the other hand, we also maintain most of those plugins ourselves. And we do release new versions of them fairly regularly.
So, whenever we shipped, for instance, a new version of Hank, we needed to manually go through all of our 25 repositories and generate a PR with the updated version in the <code class="language-plaintext highlighter-rouge">rebar.config</code> file. More often than not, those PRs included no changes outside of <code class="language-plaintext highlighter-rouge">rebar.config</code>, but they were necessary since we wanted to check future PRs with the latest version of Hank.
That process usually took us approximately 6 hours, it happened typically once a week, and it was a fully manual (i.e., error-prone) task.
That’s why we decided to automate it as much as we could. Our very ambitious original goal was to build something along the lines of Github’s <a href="https://dependabot.com/">Dependabot</a> for Erlang. It’s fair to say that we’re still far away from that, but we’re certainly moving in that direction.
<h2 id="caveats-and-limitations">Caveats and Limitations</h2>
Initially, we split the problem into two pieces:
<ol>
<li>A <code class="language-plaintext highlighter-rouge">rebar3</code> plugin to detect (and possibly update) dependencies in Erlang projects.</li>
<li>An automatic pipeline that would run that plugin in all of our repositories periodically.</li>
</ol>
The first item is <code class="language-plaintext highlighter-rouge">rebar3_depup</code>, and the second one would initially be just a <a href="https://buildkite.com/">Buildkite</a> pipeline for now (It’s a Hack week, after all). The main idea here is that eventually, using <code class="language-plaintext highlighter-rouge">rebar3_depup</code>, somebody can extend Github Dependabot to review Erlang projects, too. At that time, we will be able to throw away our poor-man version of the bot with no regrets.
The plugin idea didn’t seem too complex at first glance. In a nutshell, its logic boils down to:
<ol>
<li>Read <code class="language-plaintext highlighter-rouge">rebar3.config</code> using Erlang’s <a href="https://erlang.org/doc/man/file.html#consult-1"><code class="language-plaintext highlighter-rouge">file:consult/1</code></a>.</li>
<li>Check for <code class="language-plaintext highlighter-rouge">deps</code>, <code class="language-plaintext highlighter-rouge">plugins</code>, <code class="language-plaintext highlighter-rouge">project_plugins</code> lists.</li>
<li>For each of their elements, verify if there is a newer version of the package/library.</li>
<li>Either report the list of new versions or update <code class="language-plaintext highlighter-rouge">rebar.config</code> with them.</li>
</ol>
But, as we found out multiple times while working on the formatter, Elvis and Hank, parsing and particularly modifying Erlang files automatically is never as easy as it seems.
<h3 id="parsing-and-formatting-config-files">Parsing and Formatting Config Files</h3>
Being able to read config files using just <code class="language-plaintext highlighter-rouge">file:consult/1</code> is one of my personal favorite magic tricks from Erlang/OTP. I still remember my days as a C# engineer parsing extremely convoluted <code class="language-plaintext highlighter-rouge">ini</code> files with great sadness.
With just a tiny extra bit of complexity, you can also easily print out an config file full of Erlang terms to use for configuration:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">% Read the file
{ok, Sections} = file:consult("rebar.config"),
% Process it
NewSections = do:your_magic(Sections),
% The expected format includes a row for each section
Format = lists:flatmap(fun(_) -&gt; "~p.\n\n" end, NewSections),
% Print it out
ok = file:write_file("rebar.config", io_lib:format(Format, Sections)).</code></pre></figure>
That’s good, but it has two limitations that would become obvious the first time you try to test that procedure:
<ol>
<li>It doesn’t preserve anything beyond Erlang terms. Therefore, if you had comments in your config files… They’ll be gone.</li>
<li>It doesn’t preserve any formatting decisions. It prints out the new <code class="language-plaintext highlighter-rouge">rebar.config</code> file in the standard way that <code class="language-plaintext highlighter-rouge">io_lib</code> uses to format expressions which is typically not the nicest one to read them.</li>
</ol>
To alleviate the first problem, <code class="language-plaintext highlighter-rouge">rebar3_depup</code> uses OTP’s <a href="https://erlang.org/doc/man/erl_comment_scan.html"><code class="language-plaintext highlighter-rouge">erl_comment_scan</code></a> and <a href="https://erlang.org/doc/man/erl_recomment.html"><code class="language-plaintext highlighter-rouge">erl_recomment</code></a>, but those tools are far from perfect. That’s why <code class="language-plaintext highlighter-rouge">rebar3_depup</code> will emit some warnings if comments are removed or misplaced.
For the second issue, well… We decided to ignore it entirely. We considered that every well-maintained Erlang project these days will be using a formatter, and therefore, developers can always run…
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ rebar3 do update-deps --replace, format</code></pre></figure>
<h3 id="semantic-versioning">Semantic Versioning</h3>
The next issue we faced was figuring out what it actually means to have a newer version of a library. There are multiple ways to define newer in this context. For instance, we could have checked if there was a more recently published version of the package in hex.pm or a more recently pushed tag for the repository in Github. But, in hopes of achieving more accuracy, we decided to trust <a href="https://semver.org/">SemVer</a>, instead.
For that, we used <a href="https://github.com/starbelly">Bryan Paxton</a>’s <a href="https://hex.pm/packages/verl"><code class="language-plaintext highlighter-rouge">verl</code></a>. A simple library that can be used to compare two SemVer-compatible versions and, among other things, tell you if one is greater than the other. So that’s what <code class="language-plaintext highlighter-rouge">rebar3_depup</code> considers as a newer version of a library.
That, in turn, means that the updater can only be used with properly-versioned hex.pm packages or Github tags. Since hex.pm also enforces semantic versioning, we think that this should be enough for most well-maintained projects, anyway.
<h2 id="automation">Automation</h2>
So… How do we use this new tool here? And how can you use it in your own company?
We use Buildkite pipelines for many automation tasks, and therefore writing a recurring pipeline to run the updater on all of our Erlang repos was the natural (i.e., easier) thing to do in this case. This is the main part of our pipeline now…
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml">- label: 'Check Dependencies'
key: "check"
command:
# Move to a well-known path that has a rebar.config file
# with the updater in project_plugins
- pushd .buildkite/support
# Download rebar3
- curl https://rebar3.s3.amazonaws.com/rebar3 -o ./rebar3
- chmod +x rebar3
# git clone all the repositories we want to check
- ./clone-repos.sh
# Run the updater in each one of them and push the changes to github
- for REPO in $$(ls repos); do ./update-repo.sh $${REPO}; done</code></pre></figure>
This is the core of <code class="language-plaintext highlighter-rouge">update-repo.sh</code>:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">#!/bin/bash
set -eu
# Generate a unique branch name
BRANCH_NAME=update-deps-$(date "+%Y-%m-%d")
# Get into the project folder
pushd repos/$1
# Checkout the new branch from main
git checkout main
git checkout -b ${BRANCH_NAME}
# Go back to the root folder to use the main rebar.config file
popd
# Update deps in the project's rebar.config file
./rebar3 update-deps --rebar-config=repos/$1/rebar.config --replace
# Get into the project folder again
pushd repos/$1
# Run the formatter, if possible
../../rebar3 format || echo "No formatters here :("
# If there are changes, generate a new pull request with them
git diff --exit-code || ../../push-changes.sh ${BRANCH_NAME}
# Go back to the root folder again
popd</code></pre></figure>
And, as you might have guessed, this is <code class="language-plaintext highlighter-rouge">rebar.config</code> file on the root folder:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{plugins, [rebar3_depup]}.</code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
With this new tool and our very simple CI pipeline, we turned what used to be a 6 hours weekly manual task into a 15 minutes weekly pull request review and merge process. To be clear: the review and merge process was already part of those 6 hours before, too. But it was not just 15 minutes: since the process was manual, the review had to be more thorough and careful back then.
Of course, blindly updating libraries and plugins may break stuff. That’s why we don’t just merge these updates. Instead, we generate pull requests that are reviewed by our CI pipelines and our developers as well. To be fair, that was exactly how we did this before the automation, so that part didn’t change. Only the review and pull request generation was automated. Which is great, because that’s what you can also automate in your own projects.
And then… what do we do with the extra hours that we gained? Well… we use them to write blog posts like this one or create new <code class="language-plaintext highlighter-rouge">rebar3</code> plugins, of course!
<center>
<a href="https://xkcd.com/1319/">
<img alt="XKCD" style="width: 75%" src="/images/post_images/xkcd-automation.png" />
 
XKCD knows what we're talking about, as usual
</a>
</center>
<h2 id="contributing">Contributing</h2>
As usual, this project is publicly available in Github. Please try it out on your projects and let us know about any <a href="https://github.com/AdRoll/rebar3_depup/issues">issues</a> you find. We also gladly accept pull requests from the community :)
</description>
</item>
<item>
<title>MaskedLARK Explainer</title>
<link>https://tech.nextroll.com/blog/data-science/2021/08/03/masked-lark.html</link>
<pubDate>Tue, 03 Aug 2021 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data-science/2021/08/03/masked-lark</guid>
<description><h1 id="background">Background</h1>
<a href="https://github.com/WICG/privacy-preserving-ads/blob/main/MaskedLARK.md">MaskedLARK</a> is the latest in a number of proposals for Google’s <a href="https://www.chromium.org/Home/chromium-privacy/privacy-sandbox">Privacy Sandbox</a> initiative. This initiative aims to increase user privacy in the browser by removing third-party cookies while still preserving the ad-tech ecosystem that has evolved on top of those cookies.
A key part of the current ecosystem are machine learning models, which determine how valuable a given ad is to be shown to a given user: Third-party cookies are used to track to whom ads are shown and whether these ads were effective. This data is then used to train machine learning models to better allocate future advertising dollars.
Privacy advocates don’t want private companies to be able to track the individual history of users, while ad-tech companies want to efficiently spend advertising dollars with effective machine learning models. MaskedLARK provides a compromise: The machine learning models can be trained on individual user data, but neither the ad-tech companies nor any other third party will have access to this raw data. This post will go over the details using one of our NextRoll machine learning models as an example.
<h1 id="the-problem-and-solution">The Problem and Solution</h1>
The gold standard of ad-tech metrics today is Click-Through-Conversions. A Click-Through-Conversion occurs when a user clicks on an ad and then goes on to “convert” (purchase from the advertiser) within a given time span (usually 30 days). This metric is popular with advertisers as the conversion represents an actual sale while the click shows that the ad influenced the user and led to the conversion. In fact, the Click-Through-Conversion metric is so popular that most browsers promise to continue supporting it even in the absence of third-party cookies.
To optimize Click-Through-Conversions, at NextRoll we train a model for <code class="language-plaintext highlighter-rouge">P(Convert | Click)</code>, the probability that a user converts given that they clicked on an ad.
The training data for this model consists of rows where each row represents a click and has a positive or negative label depending on whether that user went on to convert (Our actual model is a bit more complicated and uses <a href="https://en.wikipedia.org/wiki/Survival_analysis">survival models</a> but I’ll ignore that in this post for simplicity).
When a user clicks an ad we log a “click event” and when a user converts with one of our advertisers we log a “conversion event”. Since both of these events have access to and log the third party cookie, we can join the click and conversion events on the third party cookie to build a timeline of the user’s ad interactions and purchases to build our training data:
<center>
<img alt="User Browsing Timeline" src="/images/post_images/user_timeline.png" />
</center>
The MaskedLARK proposal creates this same training set but keeps it within the browser to ensure privacy. Then “helpers” (trusted third parties) perform the computations needed to update our model. We are able to train the model we need while never having access to the raw user data!
But now won’t the helpers have access to the data? To solve this, MaskedLARK proposes a protocol using <a href="https://medium.com/dropoutlabs/secret-sharing-explained-acf092660d97">secret sharing masks</a> where data is split among multiple helpers such that no individual helper can recover the raw data yet their combined computations still produce the desired model update. This will be explained in more detail below but at a high level the flow of MaskedLARK is as follows:
<ol>
<li>We tell the browser which features <code class="language-plaintext highlighter-rouge">x</code> we care about.</li>
<li>The browser keeps track of which ads the user clicks on and whether the user converts. It thereby creates the training data <code class="language-plaintext highlighter-rouge">(x, y)</code> for our model – one row for each click with features <code class="language-plaintext highlighter-rouge">x</code> and label <code class="language-plaintext highlighter-rouge">y</code> determined by whether the user goes on to convert or not.</li>
<li>Using secret sharing masks, the browser splits this training data into <code class="language-plaintext highlighter-rouge">H</code> chunks, one for each of <code class="language-plaintext highlighter-rouge">H</code> helpers.</li>
<li>Each helper computes the model update over the training data <code class="language-plaintext highlighter-rouge">(x, y)</code> it receieves.</li>
<li>We receive the <code class="language-plaintext highlighter-rouge">H</code> computations and combine them into a single computed model update with which we update our model.</li>
</ol>
<h1 id="how-secret-sharing-masks-work">How Secret Sharing Masks Work</h1>
Let’s take a brief detour to understand the idea behind secrete sharing via masks.
Suppose I have a secret number, say <code class="language-plaintext highlighter-rouge">17</code>, and a collection of numbers which sum to <code class="language-plaintext highlighter-rouge">1</code>, say <code class="language-plaintext highlighter-rouge">-4</code>, <code class="language-plaintext highlighter-rouge">2</code>, and <code class="language-plaintext highlighter-rouge">3</code>. We will call these numbers “masks”. Suppose now I give each of three “helpers” the secret number multiplied by one of the masks. That is,
<ul>
<li>Helper 1 receives <code class="language-plaintext highlighter-rouge">17 * (-4) = -68</code></li>
<li>Helper 2 receives <code class="language-plaintext highlighter-rouge">17 * 2 = 34</code></li>
<li>Helper 3 receives <code class="language-plaintext highlighter-rouge">17 * 3 = 51</code></li>
</ul>
Then none of the three helpers knows my secret number and yet if these three numbers are added together they recover my secret number <code class="language-plaintext highlighter-rouge">17</code>! Thus a collection of masks that <code class="language-plaintext highlighter-rouge">sum_to_1</code> can be used to split a secret into pieces such that no party knows the secret and yet the secret can be recovered from the pieces.
Suppose now I take a different number, say <code class="language-plaintext highlighter-rouge">13</code>, and a collection of “masks” which sum to <code class="language-plaintext highlighter-rouge">0</code>, say <code class="language-plaintext highlighter-rouge">-5</code>, <code class="language-plaintext highlighter-rouge">2</code>, and <code class="language-plaintext highlighter-rouge">3</code>. Again I give each helper the number multiplied by the corresponding mask. That is,
<ul>
<li>Helper 1 receives <code class="language-plaintext highlighter-rouge">13 * (-5) = -65</code></li>
<li>Helper 2 receives <code class="language-plaintext highlighter-rouge">13 * 2 = 26</code></li>
<li>Helper 3 receives <code class="language-plaintext highlighter-rouge">13 * 3 = 39</code></li>
</ul>
Now if we add these three numbers we get <code class="language-plaintext highlighter-rouge">0</code>. And so a collection of masks that <code class="language-plaintext highlighter-rouge">sum_to_0</code> can be used to create data that vanishes when combined together.
Finally, we can use both these strategies to not only hide secret numbers but also how many secret numbers there even are: Suppose we have some secret numbers – say just one, <code class="language-plaintext highlighter-rouge">17</code>. We can create some fake numbers – say just one, <code class="language-plaintext highlighter-rouge">13</code> – and pass to each helper the masked numbers using <code class="language-plaintext highlighter-rouge">sum_to_1</code> masks for the secret numbers and <code class="language-plaintext highlighter-rouge">sum_to_0</code> masks for the fake numbers. Continuing the example above and shuffling the passed data we could have e.g. that
<ul>
<li>Helper 1 receives <code class="language-plaintext highlighter-rouge">[-68, 65]</code></li>
<li>Helper 2 receives <code class="language-plaintext highlighter-rouge">[26, 34]</code></li>
<li>Helper 3 receives <code class="language-plaintext highlighter-rouge">[51, 39]</code></li>
</ul>
Now each helper doesn’t know how many real secret numbers there are or even whether each number they receive corresponds to a real or fake secret number. And yet when all of these numbers are added together we get <code class="language-plaintext highlighter-rouge">-68 + 65 + 26 + 34 + 51 + 39 = 17</code>. That is we are still able to recover the sum of all the true secret numbers!
<h1 id="back-to-maskedlark">Back to MaskedLARK</h1>
With the idea of masks in mind, let’s return to MaskedLARK. After step 2. the browser has the true training data <code class="language-plaintext highlighter-rouge">(x_i, y_i)</code> and to update our model we need to compute
<figure class="highlight"><pre><code class="language-python" data-lang="python">sum(G(x_i, y_i) for (x_i, y_i) in true_data)</code></pre></figure>
where G is the gradient for the loss function for our model (c.f. <a href="https://en.wikipedia.org/wiki/Gradient_descent">gradient descent</a>).
For each true data point <code class="language-plaintext highlighter-rouge">(x_i, y_i)</code>, the browser concocts a number of fake data points. Then for each true data point, it computes a collection of masks that <code class="language-plaintext highlighter-rouge">sum_to_1</code>, while for each fake data point, it computes a collection of masks that <code class="language-plaintext highlighter-rouge">sum_to_0</code>. It then prepares for each helper a shuffled collection of data <code class="language-plaintext highlighter-rouge">[(x_i, y_i, mask)]</code> which is a mix of true and fake data points.
Each helper <code class="language-plaintext highlighter-rouge">h</code> receives its share of data and computes
<figure class="highlight"><pre><code class="language-python" data-lang="python">sum(mask * G(x_i, y_i) for (x_i, y_i, mask)
in mixed_data_for_helper_h)</code></pre></figure>
Finally we (the Ad Server) receieve each of these <code class="language-plaintext highlighter-rouge">H</code> computations, and sum them to recover the full model update! This works because the fake data masked with <code class="language-plaintext highlighter-rouge">sum_to_0</code> masks drop out while the true data masked with <code class="language-plaintext highlighter-rouge">sum_to_1</code> combine back into the original data. In mathematical terms,
<figure class="highlight"><pre><code class="language-python" data-lang="python">sum(sum(mask * G(x_i, y_i) for (x_i, y_i, mask)
in mixed_data_for_helper_h) for h in helpers)
== sum(G(x_i, y_i) for (x_i, y_i) in true_data)
== desired gradient computation</code></pre></figure>
As the raw history of the cookie is only contained in the browser, and each helper only gets masked information (including fake data), neither the helpers nor the Ad Server (NextRoll) get access to the user’s private browsing history. And yet, we still get the full power of a machine learning model trained on individual browsing history to allocate future advertising spend.
<h1 id="extensions">Extensions</h1>
The preceding sections explain the core idea of MaskedLARK in the context of computing a gradient update for a machine learning model. But the same framework could be used to calculate any aggregated quantity over individual user browsing histories. For example, we could compute “the total predicted number of conversions in California” and compare that to “the actual number of conversions in California” to see if our machine learning models are well-calibrated for California. The only limitation is that the computed quantity exhibit the desired interaction with the masks: fake data drops out, and the true data is recovered (this is expressed as a bilinearity condition in <a href="https://github.com/WICG/privacy-preserving-ads/blob/main/MaskedLARK.md#framework">this section</a> of the official README)
The MaskedLARK framework can also be augmented with additional privacy-preserving measures such as adding noise to the helpers’ aggregated quantities or using <a href="https://en.wikipedia.org/wiki/K-anonymity">k-anonymity</a>
<h1 id="conclusion">Conclusion</h1>
NextRoll Engineering has been an active participant in the privacy sandbox meetings, which will transform the ad-tech ecosystem in the near future. By keeping at the forefront, we ensure that our machine learning models and systems will continue to be first in class, no matter which proposals are ultimately adopted. If working with large-scale real-time machine learning systems in an evolving environment catches your interest, be sure to check out our <a href="https://www.nextroll.com/careers">careers page</a>.
</description>
</item>
<item>
<title>UI smoke-testing with Cypress</title>
<link>https://tech.nextroll.com/blog/dev/2021/05/11/frontend-smoke-testing-with-cypress.html</link>
<pubDate>Tue, 11 May 2021 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2021/05/11/frontend-smoke-testing-with-cypress</guid>
<description>In NextRoll, many teams are continuously working on different micro-frontend applications, many little bricks of our dashboards and products.
To help these teams building homogeneous [and awesome] interfaces, we maintain a library of UI components. You can read about it in this <a href="/blog/frontend/2015/11/05/rollup-shared-ui-components.html">blog post</a>.
On the wave of this philosophy of DRY and shared tools, we decided to build a tool to simplify smoke testing of our UIs.
A tool that we wanted to be portable, capable of running both on the developer machine and our CI infrastructures (<a href="https://buildkite.com/">BuildKite</a> and <a href="https://www.jenkins.io/">Jenkins2</a>), and integrated with our incident escalation and monitoring systems.
<h2 id="smoke-testing">Smoke testing</h2>
According to Wikipedia:
<blockquote>
Smoke testing is preliminary testing to reveal simple failures severe enough to, for example, reject a prospective software release.
</blockquote>
First, let’s define the type of tests we wanted to run with this tool.
Our goal was to run simple tests on our UI to detect significant issues that would disrupt the user experience on our platform.
Examples of these major issues can be an application unable to load or a business-critical flow not working (e.g., creating a shining new campaign).
We already have many different ways to ensure the stability of our services:
<ul>
<li>Unit and functional tests of our code, even some complex end-to-end tests that we run on pull requests.</li>
<li>HTTP API testing to guarantee backward compatibility of our interfaces and monitor their availability.</li>
<li>Monitoring of different layers of our infrastructure, from the databases to the rate of errors of our Production API.</li>
</ul>
All of these methods have always been fundamental to detect production issues or to prevent bad deploys. Still, most of them are responsible for testing or monitoring specific parts of the system without looking at the broader picture.
One of the common issues with UIs is that they may be powered by many HTTP APIs, and this set of dependencies changes over time. Also, some specific sections of the same dashboard may rely on different APIs, or someone may enable A/B testing that introduces new dependencies.
At a certain point, keeping track of the impact of your back-end systems’ deployments becomes impossible.
Yeah, this should never happen due to the “agreements” that an HTTP API should maintain (no breaking changes), so your API testing should cover you. But having redundancy even on testing is still a good practice because you may not be able to catch all the issues with a single type of testing.
Eventually, having fast and straightforward tests that would help to detect both a bad deployment and major outages of the UI was a double achievement for our goal.
<h2 id="finding-the-best-framework-for-our-needs">Finding the best framework for our needs</h2>
To achieve our goal, we started by studying the best UI testing frameworks on the market to choose the perfect one for our needs.
We wanted a framework that didn’t require learning new languages or complex API, possibly based on technologies/libraries already known to our teams, to reduce the learning curve to adopt our tool.
A reliable framework to allow us implementing stable tests, with the possibility to visualize the test execution.
Many of us already had experiences with <a href="https://www.selenium.dev/">Selenium</a> and the effort to maintain it and ensure our tests’ stability. This time, we wanted stable tests!
We also used services like <a href="https://newrelic.com/lp/synthetic-monitoring">New Relic Synthetics</a> and also the DataDog alternative, for a while.
They actually seem more stable than Selenium and with some excellent features, but, eventually, we were not happy to write our tests on their UIs without the ability to store them on our repositories.
This also required an extra effort during the deployments because we weren’t able to automatically update the tests, and we needed to jump on their UIs to address any change on our applications. After this experience, we decided that having the ability to adopt versioning of the code and automatically update our tests after each deployment were two key features for us to simplify the operations around the release of a new version of our product.
After some experiments, we found our best candidate: Cypress!
<h2 id="cypressio">Cypress.io</h2>
Cypress is an open-source framework for end-to-end testing that you can find on <a href="https://www.cypress.io/">https://cypress.io</a>.
It has a brilliant architecture that does not rely on Selenium. Instead, Cypress is executed in the same run loop as your application, which allows the framework to have native access to the DOM, the window, your application. This approach could also allow us to intercept and modify HTTP requests efficiently, and directly connect Cypress to your Redux store.
You can read more about how Cypress works <a href="https://www.cypress.io/how-it-works">on their website</a>.
Users can write tests in JavaScript or TypeScript, and Cypress already provides well-known testing libraries like <a href="https://mochajs.org/">Mocha</a>, <a href="https://www.chaijs.com/">Chai</a>, and <a href="https://sinonjs.org/">Sinon</a>.
It supports Mocha reporters, like <a href="https://github.com/adamgruber/mochawesome">Mochawesome</a>:
<figure class="highlight"><pre><code class="language-json" data-lang="json">{
"reporter": "mochawesome",
"reporterOptions": {
"reportDir": "output/reports",
"overwrite": false,
"html": false,
"json": true
}
}</code></pre></figure>
Part of our cypress.json
It also records each test to a different mp4 file, so you can watch the test execution and spot any UI issue.
<h2 id="writing-a-test">Writing a test</h2>
Cypress API are very simple, you don’t even need to have a look at their documentation to understand this test:
<figure class="highlight"><pre><code class="language-typescript" data-lang="typescript">describe("An example", () =&gt; {
before(() =&gt; {
// Here you can setup your tests.
// As example, you could log in to your application.
prepareYourTest();
});
it("Should load", () =&gt; {
cy.visit(
`${Cypress.env('HOST')}/test-url`
);
// After .visit(), we want to check if the next page H1
// contains "Hi!".
// As you can see, we don't need to wait for
// the page to be ready, this is on Cypress which
// will automatically wait for your H1 to be visible
// (or, if your H1 doesn't appear, it will fail after a timeout).
cy.get("h1").should("contain", "Hi!");
// Let's also confirm that we are on the right URL.
cy.url().should("include", "test-url");
// Now, let's find an entry on our navbar,
// and let's click on it.
cy.get("div.main-navbar")
.contains("Section 1")
.click();
// Here too, we don't need to write code to
// wait for our application to be
// ready: Cypress will take care of it.
cy.get("div.main-page&gt;h2").should(
"contain",
"It works!"
);
});
});</code></pre></figure>
See? No “wait until visible” commands!
<h2 id="distribute-to-our-teams">Distribute to our teams</h2>
Finding the framework was not the end of our journey. We needed to make the setup as easier as possible and abstract some Cypress complexities.
<h3 id="one-docker-image-to-rule-them-all">One Docker image to rule them all</h3>
We decided that distributing our tool as a Docker image would simplify the maintenance and the adoption of the tool.
Also, Cypress requires some setup and tuning, and we wanted to abstract this to other teams.
Shipping our tool as a Docker image also allows us to solve this complexity by encapsulating the Cypress instrumentation and post-processing logic. Engineers will need to provide the required config files and run the Docker command they find on the documentation.
Our base image is built from one of the official Docker images for Cypress, <code class="language-plaintext highlighter-rouge">cypress/browsers:node12.19.0-chrome86-ff82</code>. You can find the complete list of images they prepare here: <a href="https://github.com/cypress-io/cypress-docker-images">https://github.com/cypress-io/cypress-docker-images</a>.
Let’s have a look at the lifecycle of a test execution:
<center>
<img alt="Flow of the smoke testing tool" src="/images/post_images/smoketester/smoketester_flow.png" />
</center>
This approach requires teams to have a minimal boilerplate into the application:
<ul>
<li>A Dockerfile, to download our Docker image and load their tests into it;</li>
<li>Two docker-compose.yml files, one for the developer machine and another for the CI;
<ul>
<li>The main difference between these two files is <code class="language-plaintext highlighter-rouge">ipc: host</code> on the developer configuration (and a volume so they can see the output of their tests).</li>
</ul>
</li>
<li>A JSON file representing the main configuration, which could be different on each environment to allow configuring different hosts and incident escalation policies (well, I don’t think that a failure on Staging would be worth paging an engineer during the night);</li>
<li>The test files.</li>
</ul>
This allows us to store our tests on the repository and edit them using the editor we use every day. Our CI can automatically use the correct version of the tests without requiring us to perform manual adjustments after the deployment.
Another benefit is that our Docker image is fully extensible, just by overriding the files contained into the base Docker image. This allows other teams to build their own Cypress tasks, fixtures, and install more plugins.
<h3 id="test-utilities">Test utilities</h3>
Most of our UIs are accessible only after signing in to our application, so we built a simple utility that can be run from the <code class="language-plaintext highlighter-rouge">before</code> hook to instrument the session by signing in using the user that is available in the Cypress’ environment.
For teams that want to use TypeScript, we also published a little NPM package to enable type hinting for Cypress and these utilities on their editor.
<h2 id="monitoring-and-incident-escalation">Monitoring and incident escalation</h2>
Having a smoke testing tool not being integrated with the services we use every day would have been pretty useless.
For this reason, we enable the Mochawesome reporter support in Cypress, which creates a JSON file per test file.
Then, we built a simple Python script to post-process these reports:
<ol>
<li>It uploads all the artifacts (reports, MP4 videos, and screenshots) to a secure storage;</li>
<li>It emits key metrics (test duration and result) to DataDog;</li>
<li>In case of failures on tests integrated with PagerDuty, it triggers an incident to escalate the issue quickly.</li>
<li>Optionally, it posts a message on Slack with a summary of the results:</li>
</ol>
<center>
<img alt="Example of a Slack notification" src="/images/post_images/smoketester/smoketester_slack.png" />
</center>
<h3 id="who-is-monitoring-the-smoke-tester">Who is monitoring the smoke tester?</h3>
Having a smoke-tester running on our infrastructure means that we should monitor its stability, too, to ensure it is appropriately testing our applications.
This was as easy as setting a <a href="https://www.datadoghq.com/">DataDog</a> monitor that would alert if the smoke testing tool doesn’t emit metrics for a while.
<h3 id="reducing-the-noise">Reducing the noise</h3>
Nobody wants to wake up in the middle of the night because they have been paged for a false alarm. It’s a pain, which reduces the confidence in the smoke testing. 
We can mitigate this situation by applying a retry logic, and we are doing this by rerunning the whole suite if it fails. In the future, we plan to retry just the tests that failed to reduce the notification delay in case of serious issues.
The tool also takes care not to repeatedly send the same notifications if a specific test keeps failing. It’s capable of resolving the PagerDuty incident once the test succeeds again.
Also, it’s not very nice when you’ve been paged for an issue that is not under your domain, and you just need to point to the right team. This can easily happen on UIs that have multiple ownerships.
In our case, a team is usually in charge of the application, and various teams own sub-sections or specific features. 
Mitigating this was a little bit harder than implementing a retry logic. 
We chose to resolve this by implementing a hierarchy model that prevents incident escalation to the branches of a failed node.
<h3 id="hierarchy-of-tests">Hierarchy of tests</h3>
Let’s see an example:
<center>
<img alt="Hierachy of the smoke testing tests" src="/images/post_images/smoketester/smoketester_hierarchy_1.svg" />
</center>
We have three teams:
<ul>
<li>Team A is the team owning the application. They are responsible for any significant outage that causes the UI not to load at all due to, for instance, infrastructure issues.</li>
<li>Team B and C own sub-sections of the UI, either critical (P0) or non-critical (P1 or lower) sections.</li>
</ul>
Let’s assume the whole UI doesn’t load at all because the web service is down.
If the root fails, our tool doesn’t notify the other teams:
<center>
<img alt="Hierachy of the smoke testing tests - failed test" src="/images/post_images/smoketester/smoketester_hierarchy_2.svg" />
</center>
Team A will be alerted of the issue, while team B and C will sleep peacefully because the failures on their tests were caused by the application outage.
This concept of hierarchy allows us to define custom notification rules for each different team.
<h2 id="case-study-smoke-testing-the-email-verification">Case study: smoke testing the email verification</h2>
As we recently started to roll out an email verification process during the signup flow, we wanted to ensure its stability over time.
To properly test this flow, we set up a smoke testing instance that’s able to connect to the email server to grab the verification token and complete the process. This was as easy as declaring a new <a href="https://docs.cypress.io/api/commands/task">Cypress task</a> that relies on the NPM package <a href="https://github.com/levz0r/gmail-tester">gmail-tester</a> to return the email messages from our Gmail-powered account.
As this new flow has been implemented on various back-end and UIs, this smoke testing tool is a nice solution for us to ensure that the interactions between these systems will continue to be stable over time.
<h2 id="conclusions">Conclusions</h2>
Almost one year has passed since the first release of this tool, and various teams are currently using it for their own applications.
<h3 id="was-cypress-a-good-choice">Was Cypress a good choice?</h3>
Yes, for many reasons:
<ul>
<li>We managed to easily integrate it into our workflow.</li>
<li>Teams are adopting our tool with low friction, and some of them also made some contributions to add new Cypress plugins or features to it.</li>
<li>Overall, our simple tests look stable.</li>
</ul>
Also, the ability to easily see the test executions by looking at the recordings is a key feature for us, and it actually helped us during nightly incidents.
<h3 id="would-you-expand-the-scope-of-the-tool">Would you expand the scope of the tool?</h3>
Right now, we are only using it for smoke testing, and we don’t plan to support other types of tests like E2E or acceptance testing.
Even though Cypress is far above Selenium in terms of stability, it’s not so easy to define new tests for complex scenarios. So, following the <a href="https://martinfowler.com/articles/practical-test-pyramid.html">test pyramid</a> and creating tests only for the critical UI paths is, in our experience, the best approach.
<h2 id="next-steps">Next steps</h2>
We always want to improve our tools, so we already have some ideas:
<ul>
<li>Rework the retry logic (we mentioned this before) only to retry the failed tests, as right now, we are retrying the entire suite.</li>
<li>Play with the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Performance_API">Performance API</a> to extract more metrics regarding our UIs, like the Resource Timing, which allows us to retrieve network timing data regarding the loading of the application’s resources.</li>
</ul>
</description>
</item>
<item>
<title>The Winner's Curse</title>
<link>https://tech.nextroll.com/blog/data-science/2021/03/30/winners-curse.html</link>
<pubDate>Tue, 30 Mar 2021 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data-science/2021/03/30/winners-curse</guid>
<description><h1 id="introduction">Introduction</h1>
Anytime you have an auction, you have the potential for the
<a href="https://en.wikipedia.org/wiki/Winner%27s_curse">winner’s curse</a>,
a simple, yet surprising, statistical phenomenon. Despite its
simplicity, the effect can have significant financial implications
for anyone participating in auctions.
We care about the winner’s curse at NextRoll because we
participate in about 100,000 ad auctions per second. In this
post, we explain the curse and explore the way it manifests
itself in our ad-buying systems.
<h1 id="understanding-the-curse">Understanding the Curse</h1>
Let’s pretend Alice is auctioning off a jar containing exactly
$25 of spare change she has collected. $N$ of Alice’s friends
participate in the auction, hoping to purchase Alice’s jar of
change. No one can open the jar before the auction, but each of
Alice’s friends can inspect the jar to estimate the total worth
of the coins contained within. After completing their valuations,
each of Alice’s friends submit a bid.
Alice’s friends assume one of them can make a few dollars here.
However, it may be the case that one of them overestimates the
value of the coins in the jar. As a result, that person may bid
too much into Alice’s auction. If this happens, that person may,
depending on the auction dynamics and the other bids, end up
paying Alice more for the jar than the value of the coins
contained within.
It turns out that the winner of Alice’s auction will end up
overpaying for the jar of change much more frequently than we
would naively expect; this is the winner’s curse. To see how
this works, let’s denote Alice’s $i$th friend’s valuation as
follows:
<script type="math/tex; mode=display">
%&lt;![CDATA[
\begin{equation}
V_i = 25 + \epsilon_i
\end{equation}
%]]&gt;
</script>
where $\epsilon_i$ is the error each friend makes in their
valuation. The highest valuation of the jar of coins among
Alice’s $N$ friends will be:
<script type="math/tex; mode=display">
%&lt;![CDATA[
\begin{equation}
\max(V_1, ... ,V_N) = 25 + \max(\epsilon_1, ... , \epsilon_N)
\end{equation}
%]]&gt;
</script>
This highest valuation is very likely to be larger than $25. To
see why, let’s assume Alice’s friends are equally likely to
overestimate the value of the coins as they are to underestimate
it. In this case, for the highest valuation to be less than $25,
all $N$ of Alice’s friend’s bids have to underestimate the value
of the jar of coins. This happens only $\frac{1}{2^N}$ of the
time.
This discussion of the highest valuation is important because,
typically, the person with the highest valuation will win the
auction. Let’s call this person with the highest valuation, whom
we expect to win the auction, Bob. If Bob wins the jar of change
and tallies up the money, he is probably in for a nasty surprise.
Almost every time, the jar will have less money than he anticipated
it having. Many times, this effect is so pronounced that not only
will Bob have not made as much profit as he had anticipated, he
will have lost money! Just like that, Bob has fallen victim to
the curse!
<h2 id="simulating-the-curse">Simulating the Curse</h2>
To further illustrate this effect, we can make some concrete
assumptions and run a simulation. Let’s pretend the error each
of Alice’s friends make when estimating the jar of coins’ value
is distributed normally with a mean of $0 and a standard
deviation of $3. That is, $\epsilon_i \sim \mathcal{N}(0, 3^2)$.
Let’s pretend Alice is running a second-price auction, in which
the bidders naively assume bidding their noisy valuation is
optimal. Below we see what happens with 4, 7, and 10 of Alice’s
friends bidding into her auction.
<img src="/images/post_images/winners_curse/winners_curse_simulation.png" alt="" />
In this simulation, Bob usually paid more for the jar of coins
than it is worth. This happened almost every time with just ten
of Alice’s friends participating in the auction. Also notice,
the more participants the auction has, the worse the winner’s
curse becomes.
This shows that when you’ve estimated the value of an item in an
auction, you should reduce your bid, not just to account for
auction dynamics, but also to account for uncertainty in your
valuation. In particular, despite conventional wisdom saying to
bid your true valuation in a second-price auction, you should
actually reduce your bid a bit in most real-world situations.
<h1 id="observing-of-the-curse">Observing of the Curse</h1>
Now that we understand the winner’s curse, we can see what it
looks like in some real data. There are two places we observe
the winner’s curse at NextRoll. When an ad exchange notifies
us of an ad opportunity, we have to select an ad for which to
bid. This is done via an internal auction and is the first place
this effect manifests. Next, we submit the highest internal bid
to the ad exchange, and an auction is run amongst bidders like
NextRoll. This external auction is the second place we observe
the winner’s curse.
To be clear, I highlight these instances of the winner’s curse
because the effect is interesting, not because these cases are
particularly problematic. Unlike in the example of Bob losing
money by purchasing Alice’s jar of coins, the impact of the
winner’s curse on NextRoll’s bidding systems is minor overall.
<h2 id="the-internal-auction">The Internal Auction</h2>
The internal auction is where we decide which ad from our
thousands of advertisers will be submitted to the external
auction. Some of this decision is made for us by simply applying
the advertiser’s targeting criteria. Selecting among the remaining
advertisers’ eligible ads is done with an auction that has two steps.
First, we select the most valuable eligible ad for each advertiser.
As you can see below, the more ads from which we have to select for
a given advertiser, the more overvalued the selected ad is. This is
analogous to the winner of Alice’s auction paying more for the jar
of coins when there are more bidders. This is the winner’s curse in
action!
<img src="/images/post_images/winners_curse/ad_selection.png" alt="" />
The second step in the internal auction is to choose the most
valuable ad amongst each advertisers’ best candidates. As in the
case above, the more advertisers targeting the opportunity, the
more overvalued the selected ad is. Again, this is the winner’s
curse.
<img src="/images/post_images/winners_curse/advertiser_selection.png" alt="" />
<h2 id="the-external-auction">The External Auction</h2>
Once we have decided which ad to select and what to bid for it, we
send that information to the ad exchange. The exchange runs another
auction, and based on the results, makes the final decision on which
ad to show. Unlike in our internal auction, we are not the
auctioneer in the external auction, which means we don’t have
complete information about the auction’s bids. This makes confidently
observing the winner’s curse more challenging.
The first sign of the winner’s curse in the external auction is that
we consistently predict ad impressions to be slightly more valuable
than they turn out to be. To have confidence that these
overvaluations really are the winner’s curse, rather than an
undiscovered bug, we will need to understand a bit about the
interaction between our machine learning and the external auction.
The key thing to understand is that the valuations predicted by our
machine learning system are very closely related to when we win the
external auction. In particular, when our models overvalue a
potential impression, we tend to bid higher, which causes us to win
the auction more often. In other words, we win a disproportionate
amount of the bids whose value we overestimate. This selection bias
is visualized below.
<img src="/images/post_images/winners_curse/selection_bias.png" alt="" />
Fortunately, when we win the external auction and show an ad, we
receive feedback on the ad’s correct value. This, in turn, feeds
into our machine learning models as training data. As a result,
mistakes made by previous iterations of the model are corrected.
Unfortunately, no model is perfect, and our newer model iteration
will make new mistakes.
The crucial insight for our investigation is that by comparing the
predictions (and mistakes) made by older and newer model iterations
over the same slice of backtesting data, we can learn whether the
overvaluations we observe are, in fact, caused by the winner’s
curse, or more menacingly, an undiscovered bug.
To make this concrete, let’s say that we trained model A ten days
ago, and that we trained model B yesterday. Consider what we would
expect to happen if we used both models A and B to predict over
yesterday’s ad impression data that model B bid on and purchased.
If the winner’s curse were the cause of the overvaluations in the
external auction, we’d expect model B to overvalue these impressions
that it purchased because of the selection bias discussed above.
We’d also expect model A to value these impressions accurately.
This is because this selection bias affecting model B does not
apply to model A, since it was not used to bid on and purchase
these impressions.
<img src="/images/post_images/winners_curse/external_auction.png" alt="" />
The plot above shows this experiment run over real data on 11
different days. As you can see, model A does not overvalue the ad
impressions purchased by model B. This means we really are
observing the winner’s curse in the external auction!
<h1 id="conclusion">Conclusion</h1>
You’re more likely to win an auction when you bid too high. That is
the essence of the winner’s curse, a surprisingly simple, yet
counter-intuitive, statistical effect. In this post, we learned how
the curse works in theory, and we observed it in some real data.
Often, those of us interested in auctions ignore the fact that
bidders estimate the value of an item, assuming instead that they
know it exactly. However, bidders having uncertain valuations is all
that’s needed for the winner’s curse to appear. When the curse strikes,
buyers realize less profit than expected from the auctions in which
they participate. In extreme cases, this effect can even result in
the buyer losing money, as we saw with Bob and Alice. Beware of the
curse!
<hr />
</description>
</item>
<item>
<title>Ship it! with ecs-ship</title>
<link>https://tech.nextroll.com/blog/dev/2021/02/23/ship-it-with-ecs-ship.html</link>
<pubDate>Tue, 23 Feb 2021 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2021/02/23/ship-it-with-ecs-ship</guid>
<description>Having a fast and reliable way to update your deployment environments is a game changer in
terms of team velocity and confidence. <code class="language-plaintext highlighter-rouge">ecs-ship</code> is a simple yet powerful tool that can
help your team ship faster.
<code class="language-plaintext highlighter-rouge">10 minute read</code>
<hr />
<h2 id="introduction">Introduction</h2>
Most of our teams at NextRoll leverage the power of <a href="https://aws.amazon.com/ecs/" title="Elastic Container Service">AWS ECS</a> for container
orchestration. Each team manages their <a href="https://aws.amazon.com/" title="Amazon Web Services">AWS</a> resources, providing them with
ultimate freedom when choosing technologies or what or when to deploy. In this article,
we want to share <code class="language-plaintext highlighter-rouge">ecs-ship</code>, a small tool the likes of <a href="https://github.com/fabfuel/ecs-deploy" title="ECS Deploy"><code class="language-plaintext highlighter-rouge">ecs-deploy</code></a> that
we wrote to solve some of our common workflows and will soon be open source (Update: it’s now opensource yay!!)
<h2 id="some-typical-workflows">Some typical workflows</h2>
Let’s see by example which workflows you might want to use <code class="language-plaintext highlighter-rouge">ecs-ship</code> for by looking at
two typical tasks we can accomplish with it:
<ul>
<li>Deploying a new version of an application.</li>
<li>Changing the resource allocation of an application (aka right-sizing).</li>
</ul>
<h3 id="deploying-with-ecs-ship">Deploying with <code class="language-plaintext highlighter-rouge">ecs-ship</code></h3>
When deploying a new version of a containerized application to <a href="https://aws.amazon.com/ecs/" title="Elastic Container Service">AWS ECS</a>, there
are three resources you need to account for: container images, ECS task definitions, and
ECS services. <a href="https://www.docker.com/resources/what-container" title="What's a container">Container images</a> are a pretty common artifact these
days. They represent a “unit” containing your application and all the
dependencies it might need to run correctly. <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html" title="Task definitions">ECS Task definitions</a>
represent a containerized workload; essentially, they let ECS know what’s required to run
your tasks. For example, I need three containers running my application image and 1 running
a database image. You express these requirements via container definitions inside the task
definition. Finally, <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html" title="ECS Services">ECS services</a> reference a version of a task
definition and are responsible for letting AWS know “we do need these containers running
now”.
In this framework, to deploy a new version of your application, you need to follow these
steps:
<ul>
<li>Create and push a new version of your application’s image to a registry.</li>
<li>Create a new version of your task definition that references the freshly uploaded image.</li>
<li>Finally, update your service to the newly created task definition.</li>
</ul>
Let’s go through them step by step.
Building and pushing your images to a registry would usually look like this:
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export VERSION=`git rev-parse HEAD`
docker build -t your.registry.io/yourusername/yourapp:$VERSION .
docker push your.registry.io/yourusername/yourapp:$VERSION
</code></pre></div></div>
Once you have your new image (or images) up on the registry and ready to use, <code class="language-plaintext highlighter-rouge">ecs-ship</code>
comes in to help you with the rest:
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat &lt;&lt; EOF | ecs-ship your-cluster yourapp-service
containerDefinitions:
yourapp-definition:
image: your.registry.io/yourusername/yourapp:$VERSION
environment:
VERSION: $VERSION
SHIPPED_BY: ecs-ship
EOF
</code></pre></div></div>
As you can see, <code class="language-plaintext highlighter-rouge">ecs-ship</code> takes the cluster you want to update and the service you want
to update in that cluster. Furthermore, it takes a configuration file to
non-destructively modify some aspects of the chosen service, including the container
definitions. Here inside the container definition, we have decided to update the image to
refer to the new <code class="language-plaintext highlighter-rouge">VERSION</code> and upserted a couple of environment variables to later check on
the task definition. You can check the <a href="https://github.com/AdRoll/ecs-ship#usage" title="ecs-ship usage">README on ecs-ship’s repo</a> to get
familiar with which aspects of ecs services you can change.
Running this command will perform these steps for you:
<ul>
<li>Pull the task definition of the specified cluster and service.</li>
<li>Create a copy of this task definition performing the updates you requested on the input
file (only if anything is updated).</li>
<li>Update the specified service to point to the newly created version of the task
definition.</li>
<li>Wait for your service to reflect the changes you requested correctly.</li>
</ul>
If for some reason, your service doesn’t reach stability with your new configuration,
<code class="language-plaintext highlighter-rouge">ecs-ship</code> will automatically:
<ul>
<li>Revert to the last task definition (if it was stable).</li>
<li>De-register the new bugged task definition.</li>
</ul>
<h3 id="right-sizing-your-services">Right-sizing your services</h3>
We just saw how <code class="language-plaintext highlighter-rouge">ecs-ship</code> would help you in deploying new versions of ECS services. Now
let’s explore another simple yet powerful <code class="language-plaintext highlighter-rouge">ecs-ship</code> that can help you change the
number of resources a particular service requests from your ECS cluster.
One essential part of your ECS service configuration is the amount of resources a given
container or a given service requires. These resources are controlled by the <code class="language-plaintext highlighter-rouge">cpu</code>,
<code class="language-plaintext highlighter-rouge">memory</code>, and <code class="language-plaintext highlighter-rouge">memoryReservation</code> settings. <code class="language-plaintext highlighter-rouge">cpu</code> represents the amount of “shares” of a
virtual CPU your service needs. Simultaneously, <code class="language-plaintext highlighter-rouge">memory</code> and <code class="language-plaintext highlighter-rouge">memoryReservation</code> are hard and
soft caps to the amount of memory your service can use, respectively. These variables
cost you money in the following sense: if the sum of either the memory or the cpu
required by the services running in your cluster surpasses the number of actual
resources in the cluster, you need to spin up a new instance incurring in some
new costs.
However, not always do your services use as many resources as you reserve for them, and
since you’re charged for the amount of resources you do request, it is a good idea to
keep those as close as possible to the actual usage values. To work out the
amount of resources your ECS services need, you can enable and monitor <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html" title="CloudWatch metrics">CloudWatch
metrics</a>, and once you have worked out the number of resources you need,
you can update the configuration using <code class="language-plaintext highlighter-rouge">ecs-ship</code>:
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat &lt;&lt; EOF | ecs-ship your-cluster yourapp-service
containerDefinitions:
yourapp-definition:
cpu: 123
memoryReservation: 123
EOF
</code></pre></div></div>
As a side note, if you specify <code class="language-plaintext highlighter-rouge">memory</code>, you need to make sure the value is
comfortably higher than <code class="language-plaintext highlighter-rouge">memoryReservation</code>.
<h2 id="how-ecs-ship-came-to-be">How <code class="language-plaintext highlighter-rouge">ecs-ship</code> came to be</h2>
Those familiar with <a href="https://github.com/fabfuel/ecs-deploy" title="ECS Deploy">fabfuel’s ecs-deploy</a> will notice that there’s no much
difference between it and <code class="language-plaintext highlighter-rouge">ecs-ship</code>. <code class="language-plaintext highlighter-rouge">ecs-ship</code> can be seen as a new take on the
already popular project, with just a few different opinions. <code class="language-plaintext highlighter-rouge">ecs-ship</code> focuses only on
updating a given service with a new version of its task definition, leaving out other
add-on features like slack reporting or newrelic metrics for deployments.
In terms of new features, <code class="language-plaintext highlighter-rouge">ecs-ship</code> can change resource allocations and
could be extended to change other aspects of a task definition. Furthermore, and we will
see an application of this on a further post, <code class="language-plaintext highlighter-rouge">ecs-ship</code>’s input file-based API makes it
easier to interoperate with other third-party or homebrewed tools.
Besides these minor differences, one of the primary motivations for building <code class="language-plaintext highlighter-rouge">ecs-ship</code>
was our bi-annual <a href="https://tech.nextroll.com/blog/culture/2019/11/26/hackweek-at-nextroll.html" title="Hack Week">HackWeek</a> events. These events give us some time to explore
technologies, work on a little side-project or solve issues in our workflow that are not
directly related to our backlog. In our case, with <code class="language-plaintext highlighter-rouge">ecs-ship</code>, we experimented with
<a href="https://golang.org/" title="GO Programming Language">golang</a> (our team working mostly with Python and JavaScript) and solved the
need to update resource allocations for our services.
<h2 id="conclusion">Conclusion</h2>
<code class="language-plaintext highlighter-rouge">ecs-ship</code> is a little tool you can use either as a static linked binary or as a
containerized tool to manage your <code class="language-plaintext highlighter-rouge">ECS</code> deployments from your local machine or your
continuous delivery pipelines.
<h2 id="update-2021-04-08">Update (2021-04-08)</h2>
As of yesterday <code class="language-plaintext highlighter-rouge">ecs-ship</code> is now open source! You can grab the source code, binaries
and docker images following the instructions at <a href="https://github.com/AdRoll/ecs-ship#usage" title="ecs-ship usage">https://github.com/AdRoll/ecs-ship</a>.
</description>
</item>
<item>
<title>rebar3_hank: The Erlang Dead Code Cleaner</title>
<link>https://tech.nextroll.com/blog/dev/2021/01/06/erlang-rebar3-hank.html</link>
<pubDate>Wed, 06 Jan 2021 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2021/01/06/erlang-rebar3-hank</guid>
<description>From the creators of <a href="https://tech.nextroll.com/blog/dev/2020/02/25/erlang-rebar3-format.html">rebar3_format</a>, here comes… <a href="https://github.com/AdRoll/rebar3_hank">rebar3_hank</a>, a powerful but simple tool to detect dead code around your Erlang codebase (and kill it with fire!).
Developers can use this rebar plugin in addition to a linter (<a href="https://github.com/inaka/elvis_core">Elvis</a>), <a href="https://erlang.org/doc/apps/tools/xref_chapter.html">Xref</a>, and <a href="https://erlang.org/doc/man/dialyzer.html">Dialyzer</a>; they complement each other perfectly.
<code class="language-plaintext highlighter-rouge">10 minute read</code>
<hr />
<center>
<img alt="Hank Scorpio - Kill it with fire!" src="/images/post_heroes/hank_scorpio.jpg" /> 
Hank Scorpio - Kill it with fire!
</center>
<h2 id="introduction">Introduction</h2>
In NextRoll’s RTB Team, we have two passions while we maintain our codebases: killing dead code and automating things! That’s why we thought that a tool for automating this process would be handy for the community and us. So, we started thinking more seriously about <code class="language-plaintext highlighter-rouge">Hank</code>, and we decided to spend our Winter HackWeek time to make it possible!
Nobody wants to maintain dead code. In fact, most of us are huge fans of negative PRs. Hank can help you with that by traversing your project, analyzing every .erl and .hrl file in it (optionally skipping some folders/files if you want to), applying the rules, and producing a list of all the code that you can effectively delete and/or refactor.
The best thing is that you can be sure that the dead code is, in fact, dead since Hank is built with Dialyzer levels of certainty™️.
<h2 id="differences-between-hank-and-other-existing-tools">Differences between Hank and other existing tools</h2>
You might be thinking: Why Hank? That’s a job for my linter!
The answer is: No.
We use <a href="https://github.com/inaka/elvis_core">Elvis</a> for code linting; it reviews our Erlang code style like function naming, nesting level, line length, variable naming convention, etc.
Hank doesn’t do that.
<a href="https://erlang.org/doc/apps/tools/xref_chapter.html">Xref</a> is a cross-reference tool that can be used for finding dependencies between functions, modules, applications, and releases. It does so by analyzing the defined functions and function calls.
So it will warn us about a defined function that is never used around our source code.
Hank doesn’t do that either.
And <a href="https://erlang.org/doc/man/dialyzer.html">Dialyzer</a>?
Dialyzer is a static analysis tool that identifies software discrepancies, such as success type errors, code that has become dead or unreachable because of a programming error, and unnecessary tests, among other things.
It bases its analysis on the concept of success typings.
Hank does not rely on specs nor evaluates the “semantics” in functions params/returns.
<h3 id="so-what-exactly-does-hank-do">So what exactly does Hank do?</h3>
Hank will detect and warn you about valid parts of your code that could potentially be deleted or at least refactored based on rules.
It works on entire projects (as opposed to Elvis, which works on individual files), on source code (as opposed to Xref, which works on compiled code), and on individual projects (as opposed to Dialyzer, which analyzes entire systems - including OTP and your dependencies).
The current version while writing this post is <code class="language-plaintext highlighter-rouge">0.2.1</code>. It’s a minor version, but we’re already using it in our systems, and it’s practically ready for production usage.
<hr />
<h2 id="how-to-use-rebar3_hank">How to use rebar3_hank</h2>
Just add this to your <code class="language-plaintext highlighter-rouge">rebar.config</code> (either in your project or globally in <code class="language-plaintext highlighter-rouge">~/.config/rebar3/rebar.config</code>):
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{plugins, [rebar3_hank]}.</code></pre></figure>
Then run…
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 hank</code></pre></figure>
…and kill it with fire!
<h3 id="ignoring-rules">Ignoring rules</h3>
There are cases where you need to ignore some rules, like when developing libraries, where you can define hrls or modules which will be consumed by other projects. In those cases, you’ll possibly need to ignore some rules (like <code class="language-plaintext highlighter-rouge">single_use_hrl_attributes</code>). Something similar happens when using Xref.
For this purpose, you can ignore hank at the module level:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">% ignoring all the rules for this module
-hank ignore
% or ignoring specific rules
-hank [single_use_hrl_attributes]</code></pre></figure>
Or add this configuration in your <code class="language-plaintext highlighter-rouge">rebar.config</code>:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{hank, [{ignore, [
{"test/*.erl", unused_ignored_function_params}
]}]}.</code></pre></figure>
<hr />
<h2 id="the-rules">The Rules</h2>
Here you can see the rules we’ve already created, and you can use them with Hank directly.
<h3 id="unused_ignored_function_params">unused_ignored_function_params</h3>
Functions evolve, and some parameters that were used before may no longer be needed. A typical easy solution could be just ignoring them and forgetting about the issue.
Hank detects ignored parameters in the same position for all function clauses and lets you know that you can delete those parameters and refactor the places where the function is invoked, thus making your code cleaner. 😉
For instance, when analyzing this module…
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">-module(my_module).
-export([external_fun/1]).
external_fun(X) -&gt;
multi_fun(X, rand:uniform(), undefined).
%% A multi-clause function with unused 3rd param
multi_fun(undefined, _, _) -&gt;
ok;
multi_fun(Arg1, Arg2, _Arg3) when is_binary(Arg1) -&gt;
Arg2;
multi_fun(Arg1, _, _) -&gt;
Arg1.</code></pre></figure>
Hank will output…
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ rebar3 hank
===&gt; Looking for code to kill with fire...
===&gt; The following pieces of code are dead and should be removed:
src/my_module.erl:9: Param #3 is not used at 'multi_fun/3'</code></pre></figure>
To avoid this warning, remove the unused parameter(s).
<h3 id="single_use_hrls">single_use_hrls</h3>
Sometimes you put some code in a header file that’s supposed to be shared among multiple modules, but you end up writing just one module that uses it. In this case, it would be better to directly put the header file’s contents in the module itself. And Hank has a rule for that!
Assuming header.hrl:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">-define(APP_HEADER, "this is a header from an app that will be used in just one module").
-define(SOME_MACRO(A), A).</code></pre></figure>
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">-module(app_include_lib).
-include("header.hrl").
-export([my_function/0]).
my_function() -&gt;
% those are only used here!
 ?SOME_MACRO(?APP_HEADER).</code></pre></figure>
It will output:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ rebar3 hank
===&gt; Looking for code to kill with fire...
===&gt; The following pieces of code are dead and should be removed:
header.hrl:0: This header file is only included at: src/app_include_lib.erl</code></pre></figure>
Move the hrl file’s contents directly to the module that uses them, and you’ll not see this warning again.
See a complete example <a href="https://github.com/AdRoll/rebar3_hank/tree/main/priv/test_files/single_use_hrls">here</a>.
<h3 id="single_use_hrl_attrs">single_use_hrl_attrs</h3>
Sometimes it’s more subtle, tho. It’s not that the whole file is used in just one module; it is shared among many modules. But some attributes (like macros or records) are not. They are defined in the header file but only used in a single module. Hank has a rule that will suggest you to place those attributes inside the module to limit the amount of stuff that’s shared unnecessarily.
Given the previous files and including the hrl in another file:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">-module(app_include_lib_2).
-include("header.hrl").</code></pre></figure>
It will output:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ rebar3 hank
===&gt; Looking for code to kill with fire...
===&gt; The following pieces of code are dead and should be removed:
include/header.hrl:2: ?SOME_MACRO/1 is used only at src/app_include_lib.erl</code></pre></figure>
See a complete example <a href="https://github.com/AdRoll/rebar3_hank/tree/main/priv/test_files/single_use_hrl_attrs/lib/app">here</a>
<h3 id="unused_hrls">unused_hrls</h3>
Sometimes the situation is even worse: You might have hrl files that are not included in any module. Hank will detect those and let you know that you can remove them entirely since they’re virtually useless.
Adding a header_2.hrl file which is not included, the output will be:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ rebar3 hank
===&gt; Looking for code to kill with fire...
===&gt; The following pieces of code are dead and should be removed:
include/header_2.hrl:0: This file is unused</code></pre></figure>
See an example <a href="https://github.com/AdRoll/rebar3_hank/tree/main/priv/test_files/unused_hrls/lib">here</a>
It’s worth mentioning that <a href="https://erlang-ls.github.io/">erlang-ls</a> already provides a similar functionality.
<h3 id="unused_macros">unused_macros</h3>
Hank also has a rule that will detect unused macros around the project. Those macros could be defined in any file within the source code but used in none of them. Therefore, they are effectively unnecessary and can be deleted.
See an example <a href="https://github.com/AdRoll/rebar3_hank/tree/main/priv/test_files/unused_macros">here</a>
<h3 id="unused_record_fields">unused_record_fields</h3>
A fascinating one! With this rule, Hank will spot record declarations with fields that are defined (even giving them default values) but never used. Hank considers that you are using a record field when it is accessed or written.
You can use this warning to reduce your records’ size by removing the unused fields from them.
See an example <a href="https://github.com/AdRoll/rebar3_hank/tree/main/priv/test_files/unused_record_fields">here</a>
<hr />
<h2 id="extensibility">Extensibility</h2>
Following the lead of Elvis and rebar3_format, we built this project with extensibility in mind. Anybody can write their own rules for their projects by just implementing the <code class="language-plaintext highlighter-rouge">hank_rule</code> behavior.
But if you feel like sharing your new rules with the world, we are eager to get community contributions in the <a href="https://github.com/AdRoll/rebar3_hank">rebar3_hank GitHub</a>! Check out the <a href="https://github.com/AdRoll/rebar3_hank/issues">open issues</a>, and feel free to open new ones!
You can also use the <a href="https://github.com/AdRoll/rebar3_hank/discussions">discussions page</a> to get in touch with us.
<hr />
<h2 id="testing-hanks-power">Testing Hank’s Power</h2>
To see how powerful Hank was, we decided to test it in a very large codebase.
We decided to try with <a href="https://github.com/erlang/otp">Erlang/OTP</a> itself. Since it’s mainly composed by libraries, we had to limit the rules to apply to avoid some bogus results. We used this configuration:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{hank, [
{ignore, ["**/test/**"]}, %% Just "production" code, no tests
 {rules, [
unused_ignored_function_params,
unused_hrls,
unused_macros,
unused_record_fields
]}
]}.</code></pre></figure>
We hoped to find a large number of warnings, but never as large as what we found. Hank found more than 4000 pieces of dead code in OTP’s production code (i.e., we didn’t check the tests).
Indeed, not all of them are supposed to be removed, but to give you a taste of the stuff that Hank found, check out the following warnings…
<h3 id="unused-fields-in-records">Unused Fields in Records</h3>
Hank found 130 unused fields in records, like <a href="https://github.com/erlang/otp/blob/6378a0c825db64df91e01ee39e3a268f4ba050b7/lib/syntax_tools/src/erl_tidy.erl#L954">this one in <code class="language-plaintext highlighter-rouge">erl_tidy</code></a> or <a href="https://github.com/erlang/otp/blob/6378a0c825db64df91e01ee39e3a268f4ba050b7/lib/kernel/src/logger_server.erl#L45"><code class="language-plaintext highlighter-rouge">remote_logger</code> here</a>.
<h3 id="unused-macros">Unused Macros</h3>
Hank found more than 1000 unused macros in OTP, most of them in large modules of the <code class="language-plaintext highlighter-rouge">megaco</code> application and others like <a href="https://github.com/erlang/otp/blob/6378a0c825db64df91e01ee39e3a268f4ba050b7/lib/xmerl/src/xmerl_uri.erl#L349">this one in <code class="language-plaintext highlighter-rouge">xmerl_uri</code></a>.
<h3 id="unused-parameters">Unused Parameters</h3>
Hank also found more than 2000 functions with unused params. Some of them are not actually errors, like <a href="https://github.com/erlang/otp/blob/6378a0c825db64df91e01ee39e3a268f4ba050b7/erts/preloaded/src/erlang.erl#L370-L371">this one</a> that’s masking a NIF function (<a href="https://github.com/AdRoll/rebar3_hank/issues/65">Which will be fixed soon</a>). But others are worth checking, like <a href="https://github.com/erlang/otp/blob/6378a0c825db64df91e01ee39e3a268f4ba050b7/lib/inets/src/http_lib/http_uri.erl#L257-L266">this non-exported function</a> that never uses its first argument.
</description>
</item>
<item>
<title>Chasing a Performance Regression with Erlang/OTP 22</title>
<link>https://tech.nextroll.com/blog/dev/2020/11/03/chasing-a-perf-regression-erlang.html</link>
<pubDate>Tue, 03 Nov 2020 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2020/11/03/chasing-a-perf-regression-erlang</guid>
<description>Updating the underlying systems that our service depends on (be them operating system, VMs, core libraries, databases, or other components) is a regular part of our systems’ lifecycle. In this post, we’ll discuss how we found a performance regression when updating to a newer erlang OTP release, the steps we took to investigate it, and how we worked around the specific issue at hand.
<h2 id="otp-update-cadence">OTP Update Cadence</h2>
For a big part of our core tech running on Erlang, years ago we settled on a policy to run on the second to last OTP release. The reasoning behind this was to avoid being subject to bugs or performance issues that might still exist on the freshest release. The idea was also to wait a bit till new features settle down before rushing to adopt them. It is not written in stone, but it is a rule we try to respect.
Following this workflow, we first tried to migrate from OTP21 to OTP22 some months ago, as OTP23 was released. But much to our dismay, this wasn’t as smooth as we would have liked.
<h2 id="performance-drop">Performance Drop</h2>
As part of our release process, we ran the new release parallel with the previous one and looked for any anomaly or performance degradation that could impact us. In this case, it was immediately noticeable that we were hitting some problems when running on OTP22, as the service’s production throughput was lower. Noticeably lower: our system was running at ~0.75x of what it was before.
<center>
<img alg="madvise usage" src="/images/post_images/otp22-throughput.png" /> 
Throughput when switching to OTP22 (yellow) vs. throughput on our production system running OTP21 (blue) and a control group (purple)

</center>
The first thing we tried was to look at the <a href="http://erlang.org/download/otp_src_22.0.readme">release notes</a>. Was there any change that could have this kind of unexpected impact on our system? We were looking for things we thought could have this kind of significant impact, like…
<ul>
<li>Changes to BEAM startup parameters and default values</li>
<li>Changes to socket/networking implementation</li>
<li>Changes related to literal terms (we have tons of data that is preprocessed and backed in <a href="https://erlang.org/doc/man/persistent_term.html"><code class="language-plaintext highlighter-rouge">persistent_term</code></a> at startup time)</li>
</ul>
Initially, nothing stood out as suspicious.
<h2 id="the-plot-thickens">The Plot Thickens</h2>
Then we looked at what the system was actually doing, and it was evident that it was spending much more time on system-cpu than before:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">[ec2-user@ip-172-26-77-44 ~]$ mpstat -P ALL 5
Linux 4.14.177-107.254.amzn1.x86_64 (ip-172-26-77-44) 06/08/2020 _x86_64_ (16 CPU)
07:44:51 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
07:44:56 PM all 62.28 0.00 16.24 0.00 0.00 0.61 0.41 0.00 20.46
07:44:56 PM 0 21.22 0.00 4.69 0.00 0.00 0.00 0.61 0.00 73.47
07:44:56 PM 1 62.37 0.00 17.71 0.00 0.00 4.63 0.40 0.00 14.89
07:44:56 PM 2 65.09 0.00 17.25 0.00 0.00 0.00 0.41 0.00 17.25
07:44:56 PM 3 65.58 0.00 16.50 0.00 0.00 0.00 0.41 0.00 17.52
07:44:56 PM 4 66.94 0.00 15.70 0.00 0.00 0.00 0.41 0.00 16.94
07:44:56 PM 5 66.67 0.00 16.36 0.00 0.00 0.00 0.41 0.00 16.56
07:44:56 PM 6 65.79 0.00 16.70 0.00 0.00 0.00 0.40 0.00 17.10
07:44:56 PM 7 68.78 0.00 14.69 0.00 0.00 0.00 0.20 0.00 16.33
07:44:56 PM 8 64.62 0.00 15.95 0.00 0.00 0.00 0.61 0.00 18.81
07:44:56 PM 9 67.69 0.00 14.93 0.00 0.00 0.00 0.41 0.00 16.97
07:44:56 PM 10 53.54 0.00 24.65 0.00 0.00 4.85 0.40 0.00 16.57
07:44:56 PM 11 61.89 0.00 19.47 0.00 0.00 0.00 0.41 0.00 18.24
07:44:56 PM 12 62.68 0.00 19.38 0.00 0.00 0.00 0.62 0.00 17.32
07:44:56 PM 13 64.21 0.00 17.79 0.00 0.00 0.00 0.61 0.00 17.38
07:44:56 PM 14 74.49 0.00 10.93 0.00 0.00 0.00 0.40 0.00 14.17
07:44:56 PM 15 64.24 0.00 17.67 0.00 0.00 0.00 0.42 0.00 17.67</code></pre></figure>
It’s clear from the <code class="language-plaintext highlighter-rouge">%sys</code> column that more than 15% of the CPU time is in kernel tasks across all cores (core 0 is irrelevant to our discussion here since our erlang system doesn’t use it; it’s reserved for other short-lived time-sensitive tasks). The system also had a high % of idle time, so it was clear that it was waiting for something.
Since kernel tasks where up, we compared vmstat output between the two systems. The number of interrupts and context switches was almost double on OTP22…
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">[ec2-user@ip-172-26-77-44 ~]$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
24 0 0 24724364 63472 1242540 0 0 12 57 1466 981 59 9 31 0 1
19 0 0 24773600 63480 1244352 0 0 0 98 271077 62547 74 12 13 0 0
15 0 0 24930996 63488 1246924 0 0 0 359 248553 61832 74 12 13 0 0
15 0 0 24840760 63496 1247752 0 0 0 1405 250937 48011 74 12 13 0 0
13 0 0 24858236 63504 1249876 0 0 0 664 247081 48183 75 12 12 0 0
17 0 0 24855240 63512 1251484 0 0 0 35 252037 47864 76 12 12 0 0
19 0 0 24911248 63520 1253176 0 0 0 18 243274 48386 75 12 13 0 0
14 0 0 24803668 63528 1253780 0 0 0 22 236502 47642 76 11 12 0 0</code></pre></figure>
…than on OTP21:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">[ec2-user@ip-172-26-77-50 ~]$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
16 0 0 22068060 63404 1410956 0 0 9 54 1218 513 70 8 21 0 2
17 0 0 22078024 63412 1413436 0 0 0 674 124876 27720 84 7 9 0 0
15 0 0 22120716 63412 1414376 0 0 0 267 119251 26972 84 7 10 0 0
6 0 0 22107168 63420 1415960 0 0 0 602 117711 25838 85 7 8 0 0
17 0 0 22124960 63428 1418760 0 0 0 1528 121784 26970 83 7 10 0 0
12 0 0 22149468 63436 1420380 0 0 0 18 124217 30018 82 7 11 0 0
15 0 0 22135184 63444 1422208 0 0 0 115 121552 27490 86 7 6 0 0</code></pre></figure>
Next, we looked at network activity: could it be that the pattern of socket writes/read was different?
Even if the data sent and received was the same, were we using more IP packets than before? Were those interrupts network interrupts? The answer to all of those was “NO”. The problem was not on the IO side.
Instead, as <a href="https://linux.die.net/man/1/sar">System Activity Report</a> told us, we had a huge number of page faults:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ sudo sar -B 10 10
Linux 4.14.177-107.254.amzn1.x86_64 (ip-172-26-68-64) 08/03/2020 _x86_64_ (16 CPU)
09:21:55 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
09:22:05 PM 0.00 243.64 140891.62 0.00 128650.61 0.00 0.00 0.00 0.00
09:22:15 PM 0.00 103.24 135143.62 0.00 129283.20 0.00 0.00 0.00 0.00</code></pre></figure>
Note that these were minor page faults. We weren’t thrashing on disk, and we weren’t short on memory.
With this new info, we went back to look at changelogs and release notes, this time, knowing a bit more about what we were looking for.
We were looking for any change in allocator settings/workings that could lead to what we were seeing.
The “Memory optimization” section at the very bottom of <a href="http://blog.erlang.org/OTP-22-Highlights/">this article</a> caught our attention, and we thought we had a <a href="https://github.com/erlang/otp/pull/2046">winner</a>. It seemed like a strong possibility that a change having to do with freeing memory could be related to our increasing page faults.
At this point, we rushed and made a quick and dirty patch disabling that new behavior (or that’s what we thought). We retried the test to confirm this was the cause before digging deeper into how to solve it in a more civilized way. Adding to our frustration, this didn’t make a dent in the performance drop, so we decided this particular patch probably wasn’t our issue and continued digging elsewhere.
<h2 id="git-bisect">Git Bisect</h2>
While investigating this early on, one problem we had was that we have good, fast ways to generate new releases of our systems based on a given (fixed) OTP release. But generating releases using different OTP versions was a painful and tedious process. This is simply something that our dev infra hadn’t been optimized for. We had to change several configuration and packing files, as well as redeploy our CI pipelines.
This hadn’t been a problem so far as switching OTP releases is usually a once-in-a-year thing, but now it was slowing us down as we wanted to try different OTP versions (we tried other minor releases of the OTP22 family, for instance) as well as BEAM patches. So we went on and solved this issue, making it possible for us to build and test deploy releases with different underlying OTP versions easily and fast, even in parallel.
Armed with this new tool, we went with the brute-force approach: <code class="language-plaintext highlighter-rouge">git-bisect</code> the changes between OTP21 and OTP23, until we found the commit where our problem was introduced. The investigation was surprisingly fast to do, and after one afternoon, we saw what looked like the smoking gun. Surprisingly it brought us back to <a href="https://github.com/erlang/otp/pull/2046">OTP’s PR #2046</a>. It was a bit confusing; hadn’t we ruled that out already?
Looking at the kind of memory-related syscalls we were doing on the newer BEAM, we saw:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ sudo strace -f -e trace=memory -c -p $pid
% time seconds usecs/call calls errors sys-call
------ ----------- ----------- --------- --------- ----------------
86.91 0.622994 74 8371 madvise
7.39 0.053005 71 746 munmap
5.70 0.040851 62 656 mmap</code></pre></figure>
Comparing that with the same tracing on OTP21, the difference was that there were no <code class="language-plaintext highlighter-rouge">madvise</code> calls on OTP21, while the number of memory allocations was about the same.
Where were we calling <code class="language-plaintext highlighter-rouge">madvise</code>?. We used <a href="http://www.brendangregg.com/perf.html">perf</a> to find that out.
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ sudo perf record -e syscalls:sys_enter_madvise -a -g -- sleep 10</code></pre></figure>
<center>
<img alg="madvise usage" src="/images/post_images/madvise.png" /> 
Snapshot of madvise calls, zooming in just one scheduler.

</center>
<h2 id="the-madvise-patch">The <code class="language-plaintext highlighter-rouge">madvise</code> Patch</h2>
At this point, we were pretty sure <a href="https://github.com/erlang/otp/pull/2046">#2046</a> was related to our problem, and so we did what we should have done the first time we suspected it: We tried to understand what the patch was doing and what <code class="language-plaintext highlighter-rouge">madvise</code> actually means.
The description on the pull request states its intention pretty clearly:
<blockquote>
This PR lets the OS reclaim the physical memory associated with free blocks in pooled carriers, reducing the impact of long-lived awkward allocations. A small allocated block will still keep a huge carrier alive, but the carrier’s unused parts will now be available to the OS.
</blockquote>
We won’t explain here how the memory allocation is structured on the BEAM. For that, refer to the <a href="https://erlang.org/doc/man/erts_alloc.html">alloc framework docs</a> and <a href="https://erlang.org/doc/apps/erts/CarrierMigration.html">carrier migration docs</a>. There you’ll find an in-depth explanation of pooled carriers.
<a href="https://www.man7.org/linux/man-pages/man2/madvise.2.html"><code class="language-plaintext highlighter-rouge">madvise</code></a>, the way it’s used here, is a hint to the OS about the usage of a given block of memory. In particular, the BEAM (since OTP22) calls that function to tell the OS that some memory blocks are likely not going to be needed in the near future.
Now that we understood what we were looking at, our performance measurements made sense. Our system was returning carriers to the pool too often and then picking them again shortly after. When returning the carriers to the pool, the new behavior on OTP22 was supposedly hinting to the OS that the free memory there was not expected to be needed again soon.
The question is, why did <code class="language-plaintext highlighter-rouge">madvise (MADV_FREE)</code> usage lead to page faults? It is supposed to be a hint to the OS, so it should choose to unmap that memory over others in case of memory pressure. But we never were under memory pressure. As usual, <a href="http://erlang.org/pipermail/erlang-questions/2020-August/099911.html">Lukas was very helpful</a>, suggesting that we check if <code class="language-plaintext highlighter-rouge">MADV_FREE</code> was actually available on our system. We run on AWS, and we were still using an older version of Amazon Linux which doesn’t support <code class="language-plaintext highlighter-rouge">MADV_FREE</code>. So the BEAM falls back to using <code class="language-plaintext highlighter-rouge">MADV_DONTNEED</code>, and in that case, the OS was very eager to unmap these memory blocks from our processes, even when memory wasn’t needed elsewhere.
If you want to check if your environment supports <code class="language-plaintext highlighter-rouge">MADV_FREE</code>, you can grep for it on .h headers provided by your OS. Or in a perhaps more civilized way, you can compile and run a test program like:
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code> #include &lt;sys/mman.h&gt;
 #include &lt;stdio.h&gt;
 int main() {
#ifdef MADV_FREE
 printf("Congrats, MADV_FREE flag is available\n");
#else
 printf(":(\n");
#endif
 return 0;
}
</code></pre></div></div>
Note: the newer Amazon Linux 2 does support MADV_FREE.
<h2 id="how-we-solved-it">How we solved it</h2>
Why do we have such a repeated pattern of leaving and picking carriers constantly?
This is an interesting question, and it is, in fact, the root of the problem. It means our allocation setting wasn’t appropriate. The settings had not changed in years, while the system had evolved and grown quite a bit during that time. This misusage of carrier pools was present when running on OTP21 as well, of course. It’s just that it was less taxed there.
The solution was to go and improve them, a thing much easier said than done :) (if in doubt, a good starting point is to use <a href="https://erlang.org/doc/man/erts_alloc_config.html"><code class="language-plaintext highlighter-rouge">erts_alloc_config</code></a> to get an initial config and iterate from there).
We did several iterations adjusting our settings and improved our memory handling and performance quite a bit, but couldn’t completely eliminate the overhead as carriers were still abandoned too often. This is something we need to investigate further. Carriers aren’t supposed to be pushed in and out of the pool at such a pace. As a temporary workaround that would let us update to OTP22, we disabled carrier migration entirely (this is done by setting the <code class="language-plaintext highlighter-rouge">+M&lt;S&gt;acul</code> system flag to <code class="language-plaintext highlighter-rouge">0</code>, replacing <code class="language-plaintext highlighter-rouge">&lt;S&gt;</code> with the desired allocators you want to tweak).
<h2 id="a-note-about-the-process">A note about the process</h2>
As mentioned earlier, we did consider the <code class="language-plaintext highlighter-rouge">madvise</code> change as a possible cause of our issue when we started to investigate it. So why did we rule it out when we were initially suspicious about it?
As it turns out, that quick-and-dirty little patch to disable it discussed above was wrong, and it didn’t really disable the thing. This was a regrettable incident and revealed two errors on our side, both caused by rushing to advance on the investigation without taking time to ask why things where the way they were:
<ul>
<li>We rushed to do a quick patch to test the hypothesis without fully understanding the code at hand. We disabled one code path calling <code class="language-plaintext highlighter-rouge">madvise</code> but missed another.</li>
<li>When the hypothesis failed, we again rushed to move on with another work. We didn’t take the time to analyze why our theory was wrong. We didn’t question the experiment’s validity, which would have led us to realize our patch was wrong early on.</li>
</ul>
There is a broader lesson here: When investigating any complex problem, it’s easy to get overwhelmed by the number of moving parts to consider, but it is crucial not to rush to premature conclusions. You should take your time to understand the problem at hand and how each step moves you a bit closer to the resolution. Knowing what is not the cause of the problem is good progress!
</description>
</item>
<item>
<title>How NextRoll is Supporting Our Rollers During the COVID-19 Pandemic</title>
<link>https://tech.nextroll.com/blog/culture/2020/07/28/supporting-rollers-during-covid.html</link>
<pubDate>Tue, 28 Jul 2020 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/culture/2020/07/28/supporting-rollers-during-covid</guid>
<description>Our Rollers (employees) are the most essential “ingredient” to our success and culture. This pandemic is like nothing any of us have been through before. So, as we navigate this time with our Rollers, we want to ensure they are being supported… but we can’t read their minds! The best way to understand their needs and concerns is to ask them via a survey, interpret the results, and then take action. That’s just what we did and will continue to do.
<h1 id="ask-the-right-questions">Ask the right questions</h1>
It’s important to ask the right questions to get Rollers’ sentiment and feedback. Since 2013, we have used <a href="https://www.cultureamp.com/">Culture Amp</a>, an employee engagement platform, for almost all of our internal surveys. They have a library of research-backed surveys designed by organizational psychologists and data scientists. Think onboarding, exit, engagement, and diversity &amp; inclusion. Recently, they have added surveys to support our current challenging environment, such as a COVID-19 response survey and return to the workplace survey. We use their off-the-shelf survey templates as a starting point and tailor them to include questions we want to ask. Our recent sentiment survey focused on how to best support Rollers through the changing environment, their sentiment on the future of remote work, and their return to workplace readiness.
We always include a few open-ended questions too. They are important because they lead to longer, more insightful responses and not just one-word answers. Open-ended questions are great to use to gain understanding of the sentiments of employees as they tend to give more context and information on their thoughts and concerns.
<h1 id="understand-the-results">Understand the results</h1>
For our COVID-19 Sentiment Survey, we had a participation rate of 83% with almost 1,800 comments to read. Culture Amp’s platform helps to easily review and navigate the quantitative data. We were able to drill down by specific demographics like location and department. This was especially helpful when reviewing the return to workplace readiness questions. We found our US offices had a 50% or higher sentiment of not wanting to return to the office until a vaccine or therapeutic becomes available.
It’s important to understand that in reviewing results, the data and bubbled up themes are from a point in time (and we know things will change yet again). Therefore we took action on the most immediate short-term areas, and will spend more time gathering insights externally and internally on the longer-term policies and programs. We care a great deal about the employee experience for all of our Rollers and will continue to bring employee sentiment into the equation.
<h1 id="themes-from-roller-feedback">Themes from Roller Feedback</h1>
This is a summary of the Themes we gathered from our survey:
<ul>
<li>Continued employee support by:
<ul>
<li>Sending timely communications about company updates</li>
<li>Leaders being visible/accessible</li>
</ul>
</li>
<li>Opportunities to enhance support to employees by:
<ul>
<li>Providing a work from home stipend</li>
<li>Helping manage/provide resources for the increased cognitive load due to this challenging time, and overall mental health and wellness</li>
</ul>
</li>
<li>Return to workplace insights for consideration:
<ul>
<li>International locations have larger percentages of employees who are feeling more ready to return to the workplace in 2020 than other locations</li>
<li>All US offices have 50% or higher sentiment of not returning to the workplace until a vaccine/therapeutic is available</li>
<li>Do not pressure employees to return</li>
</ul>
</li>
</ul>
<h1 id="take-action">Take action</h1>
Based on the survey results and the themes that bubbled up, we were able to take action on the following areas right away:
<ul>
<li>Communication: Live stream was the most popular communication method being used to share company updates. Based on Roller’s feedback, we will continue with our weekly Town Hall meetings. We are also committed to providing regular updates through email, video, and Slack as we navigate through these challenging times.</li>
<li>Work From Home (“WFH”) Stipend: Rollers will receive a monthly stipend to offset WFH expenses through the end of 2020.</li>
<li>Summer Recharge Days: We heard from many Rollers that they are experiencing an increased cognitive load as they navigate extended WFH conditions, health and safety concerns, and feel they need further time off to recover and/or avoid burnout, and boost productivity. Given the uniqueness of this time, we decided to provide one Friday off during the months of June - September.</li>
</ul>
These are just some of the immediate actions we were able to take. The more feedback we collect, the more perspectives we will receive to help us better understand the varied experiences Rollers are having, and come up with inclusive solutions during this time. It’s a top priority for us to help everyone navigate this situation with the information, resources, tools, and flexibility that we’re able to offer.
Surveys are the fastest way to see where our current resources and programs are effective, and where we might need to lean in more. It’s important that we support our Rollers through these unprecedented times, so we will continue to survey them and ask for continuous feedback.
</description>
</item>
<item>
<title>Coordinated Cost Savings</title>
<link>https://tech.nextroll.com/blog/costs/2020/07/01/coordinated-cost-savings.html</link>
<pubDate>Wed, 01 Jul 2020 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/costs/2020/07/01/coordinated-cost-savings</guid>
<description>
Reducing costs on infrastructure can often feel like a chore when done in isolation. In this post, we’ll
discuss the coordinated process that NextRoll is using to make the effort feel like a full-team
project with the satisfaction of having a job well done.
Costs have a nasty habit of accumulating over time, usually because of a wide variety of decisions
made by disparate teams that were made in the past and never revisited. It’s an ancient problem.
Aristotle writes:
<blockquote>
For that which is common to the greatest number has the least care bestowed upon it. Every one thinks chiefly
of his own, hardly at all of the common interest; and only when he is himself concerned as an individual.
For besides other considerations, everybody is more inclined to neglect the duty which he expects another
to fulfill…
–Aristotle
</blockquote>
Of course, the difficulty is that many small problems demand many small solutions. As such, significantly
reducing your costs can seem like an exceptionally burdensome task.
Here are our tips if you’re looking to save a chunk out of your budget.
<h4 id="1-local-vs-global-optimization">1. Local vs. Global Optimization</h4>
Often, teams are asked to track costs on their own and apply some common sense rules to keep
them down. This works ok in practice, but it is far from optimal. One of the main issues is that
it’s very easy for a team to look at a relatively small cost-saving opportunity and it gets pushed down to
the bottom of their priority list. I mean, in the grand scheme of things, does $1000 per month
seem like that big of a deal?
<center>
<img alt="Metaphor for a non-right-sized instance." src="/images/post_images/leak.jpg" /> 
Metaphor for a non-right-sized instance.
</center>
 
Well, probably every engineering team is making similar calls. NextRoll Engineering has dozens
of teams as a moderately-sized company. Forty-two teams each saving $1000 every month results
in an extra half million dollars in the company coffers. And the thing is, $1000 per month is
not an onerous amount to be saving when you deal with data at the scale that we do; more on that
later, but really, this represents a lower bound on what can be saved.
Beyond this, operating costs affect the bottom line of any company. Ideally, this money would
be invested, so that returns can be made: hiring more people, spinning up new products, etc.
that can meaningfully drive the top line revenue. Instead, in the best case, these costs are just
wasted, but in the worst case, actually compound over time because not investing the money elsewhere
is an opportunity cost. Also, as the company scales, these wasted costs will only increase.
Getting your unit economics in line is critical.
To this end, we recommend creating a task force, with representatives across the engineering
organization, that is responsible for talking through all the cost-saving initiatives. This
team doesn’t need to meet frequently, but tracking and accountability are key. It also helps
everyone get on the same page about how systems are built and interoperate, revealing more
opportunities.
<h4 id="2-solicit-ideas">2. Solicit Ideas</h4>
No one knows the code better than the engineers who work on their systems every day. Some in
management may be tempted to set arbitrary cost saving goals, but this is extremely
counterproductive. It’s also not enough to demand some vague sense of “cost savings” as a
project. In reality, each individual cost-saving initiative is unique in its own right and has
its own set of tradeoffs that the engineers are best equipped to evaluate.
To this end, ask every engineering team to submit their ideas. Each team can meet amongst
themselves, spend an hour or two just brainstorming, and write everything down. At
NextRoll, we take an attitude of every idea being valid in this stage, no matter how difficult
to accomplish. In some ways, this is similar to a design sprint: the ideas must be on the table
before further discussions around feasibility and prioritization can happen. There are side
benefits as well: small ideas can be enhanced or compounded; new product ideas can emerge; wild
ideas can inspire other, more feasible ones.
One of NextRoll’s core values is, “Do more with less.” We try to consistently uphold this,
but these cost-saving efforts provide a good opportunity for us to reflect on that value. It’s
important during brainstorming that engineers ask themselves not, “What do we want,” or “What
is convenient,” but rather, “What do we need?”
Everyone is contributing. This creates a shared sense of ownership and engineers
become invested in their own ideas. Finger-pointing at the teams with the largest cost centers
melt away, because, hey, we already know the little stuff adds up and we’re all in this
together. Every team is trying to reduce their footprint to only what’s required.
Every team lists out some potential objectives and it’s clear how these relate to
the overarching theme and goal of the project.
Once all of the ideas are generated, they need to be submitted to the aforementioned task force.
It’s up to the task force to meet and decide which tasks will be tackled. Hopefully the task
force has enough representation across engineering that they know enough context to discuss the
proposals intelligently. But just in case, these ideas should contain documentation that includes:
<ul>
<li>The estimated amount of savings, in a standard unit (such as dollars/month)</li>
<li>The estimated amount of engineering effort, in a standard unit (such as eng-weeks)</li>
<li>A high-level description of the proposal</li>
<li>A list of any open questions that would need to be resolved before tackling the project</li>
<li>A list of impacted teams</li>
</ul>
Estimates do not need to be super accurate. The idea is to get a rough cut of what is possible.
If you have enough ideas submitted, it is likely that some estimates will be over and some will
be under. Statistically, over all submissions, you’ll probably hit a reasonable total
estimate of how much money is on the table for the taking.
From here, the task force can compile the list of items in a centralized location, assess the
total opportunity on each team and across the organization, and be prepared for their first meeting.
Another good tip for soliciting ideas is to set a reasonable minimum on how much a submission
can save. This will keep brainstorming meetings focused and reduce the number of items the
task force has to prioritize. What that minimum should be is going to depend on any number of
variables, such as revenue, current total costs, number of teams, desired savings goals, etc.
Pick something that fits your company’s situation.
<h4 id="3-task-force-prioritization">3. Task Force Prioritization</h4>
The task force finally needs to meet and discuss all items. This will likely be a
time-consuming meeting, but it will be valuable. We’ll talk about side benefits later, but
for now, let’s talk about how to prioritize.
One thing that’s critical to keep in mind is that individual teams still have product roadmaps
to execute on. Cost reduction doesn’t happen in a vacuum. Representatives should be able to
explain to the group what the current workload for each individual team is, and how much
engineering time is available to spare. But time isn’t free, so product management needs to
be on board with the overall effort, and recognize that some time is going to be spent on
getting unit costs down.
At this point, it’s common sense to tackle the biggest bang-for-the-buck items. As mentioned
prior, NextRoll is fundamentally a data company. A really easy place for things to build up
are S3 costs. My background is data science; my personal inclination is to keep plenty of data
around so I can look at historical models, their performance, and so on. But how often do we
really need to go back, say, 90 days? Is 45 enough? These are the questions we were asking
ourselves on my teams, and we realized there were a lot of five-minute tasks to set some TTLs
that slashed our budgets pretty significantly.
<center>
<img alt="Metaphor for a typical S3 bucket." src="/images/post_images/junkyard.jpg" /> 
Metaphor for a typical S3 bucket.
</center>
 
Right-sizing instances is another big opportunity; as products develop over time, the needs
for their servers can shift. It’s also possible to identify less-used features in the product
that are supported by data and infrastructure and to just delete and shut off those services
entirely. For high-volume systems, even a 10% improvement in efficiency can meaningfully impact
your budget.
A common question at NextRoll was, “What about AWS reservations? If we’ve already paid for
servers, why should we shut them down?” Sure, but this is something that should largely
factor into prioritization. The reservations buy you time to reduce the number of servers
you need. Try to avoid falling for a sunk cost fallacy: once the money is spent it’s gone.
Those reservations could be used for other services that need them more, or could be used to
spin up new product features. Also, reservations will eventually expire. Even though you’ve
bought some time, try to get ahead of the problem and don’t lose focus on the end goal. Chances
are, that expiration date will sneak up on you, and you’ll be in a new product development
cycle that may be hard to find the time. The point is to strike while the iron’s hot.
Once the task force has balanced every team’s roadmaps and selected a set of appropriate items,
they organize their selections into a centralized document that all teams can reference. The
selected items get pushed down to the individual teams for implementation with some expectation
on when the items will be completed. At NextRoll, we’ve opted for a quarterly deadline. Like any
engineering project, deadlines can slip, but what’s important is to be on top of the tracking.
Some of the side benefits of this task force meeting are that participants gain a better sense
of how other systems they aren’t responsible for work and interact with each other. They also
get a better sense of other teams’ product roadmaps. If they’re technical leaders for their
respective teams, they can bring that knowledge back. On top of all this, the task force
meeting is yet another opportunity to brainstorm and share notes. As the team goes through all
the items, new opportunities and solutions may present themselves that can be discussed with
the implementing teams.
<h4 id="4-tracking">4. Tracking</h4>
I’m a big fan of this quote, probably apocryphally attributed to Karl Pearson, and likely
expressed by others before him:
<blockquote>
“That which is measured improves. That which is measured and reported improves exponentially.”
–Karl Pearson
</blockquote>
After each team implements an item, they should spend some time to measure the actualized
savings and report back. The task force can log these data as they come in, and track how
well estimates line up with reality. This feedback loop will hopefully lead to better
estimates in the future.
The most obvious benefit here is that these numbers can help FP&amp;A teams with their jobs.
Understanding the overall impact on the company justifies the effort spent achieving the
results. Measuring the return on investment will also help in the future when it’s time
to tackle further initiatives.
And, of course, measuring the results provides engineers with some actual satisfaction by
seeing the results of their labors. Since we approached this collectively, every engineer
contributed to the outcome. The flipside of the “death by a thousand cuts,” is that the
thousand bandages add up to something big and meaningful. The task force should take some
time to communicate the summarized results to not only the whole engineering team, but the
whole company. Whatever medium works for you is fine; at NextRoll we report on our results
at our All Hands meetings.
At NextRoll, the task force meets monthly to quickly go over updates. We chat about progress,
blockers, and celebrate successes. It’s not particularly different from a scrum of scrums.
With the distributed contributions of every engineer, the task force meeting is not
particularly onerous because they’ve been provided with the information they need ahead of time.
<h4 id="5-lather-rinse-repeat">5. Lather, Rinse, Repeat</h4>
Once the original deadline is passed, the task force reviews unfinished items, reevaluates
items that weren’t prioritized in the first round, and continues to prioritize and push
things down to the teams for the next period. This ensures that harder, but perhaps bigger,
opportunities don’t become forgotten and dropped. Recall that the first pass was about the
best bang-for-the-buck projects. Things will get harder, but with an appropriate balance,
the payoffs should be higher in the next round.
Momentum shouldn’t be lost. Cost saving should not be a one-off initiative because, as
stated, costs tend to accumulate over time. In some sense, it’s easier to
tackle things as they come up; on the other hand, it’s not easy to change culture.
Explicitly maintaining cost reduction as a priority sends a strong message that the value
of “Do more with less” is not empty.
One thing that I’ve noticed at NextRoll that’s been great is that individual engineers
continue to ping me about new opportunities they’ve managed to uncover, even outside of
brainstorming sessions. We add these items to our overall tracking and make sure these
ideas don’t go unnoticed or unprioritized. Sometimes I even get pinged with cost reduction
measures that have already been completed, and I make sure even those things are
documented and tracked to recognize what these engineers have accomplished.
And this is really the attitude that we’re looking for. Yes, it’s a decent amount of
upfront effort to document and coordinate everything. But once the process is in place
to solicit creative ideas from all sides, the tracking mechanisms are in place to measure
accomplishments, and recognition is provided, cost savings becomes less of a chore and
hopefully a rewarding process unto itself.
<h4 id="conclusion">Conclusion</h4>
I hope this provides a glimpse into our cost saving process at NextRoll. Obviously, not
all of these processes are necessarily applicable to your own company or situation. Adapt
as you see fit, but whatever process you decide to implement, I strongly recommend
adhering to the following principles:
<ul>
<li>Small stuff adds up. Think globally, act locally, as they say.</li>
<li>Put the idea generation in the hands of the engineers. They know what they’re doing.</li>
<li>Crazy ideas are fun and can spur further innovation.</li>
<li>Centralize coordination and prioritization.</li>
<li>Estimate. Measure. Report.</li>
<li>Recognize.</li>
<li>Continue.</li>
</ul>
Thanks for reading!
</description>
</item>
<item>
<title>How to save a lot of money with a Baker in the spot market?</title>
<link>https://tech.nextroll.com/blog/dev/2020/06/16/how-we-saved-with-spot-market.html</link>
<pubDate>Tue, 16 Jun 2020 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2020/06/16/how-we-saved-with-spot-market</guid>
<description>OMFG (one-minute file generator) is a service that reads log lines from many Kinesis streams and saves them to S3, creating one compressed file every minute.
Its first version was written long ago using <a href="https://storm.apache.org/">Apache Storm</a>, but it became expensive and sometimes unstable over time.
Our target was to reduce the service cost significantly and create a 0-downtime infrastructure.
We decided to completely replace the software using Baker, our in-house data processing software (written in Go). We deployed it across all AWS regions by using spot instances only.
The final result exceeded expectations, as we reduced the overall service cost by over 85%.
<code class="language-plaintext highlighter-rouge">15 minute read</code>
<hr />
<h2 id="omfg-overview-and-improvement-ideas">OMFG, overview and improvement ideas</h2>
<center>
<img alt="The old OMFG version" src="/images/post_images/omfg_old_infra.jpg" /> 
The old OMFG version in the NextRoll infrastructure

</center>
Bidders and AdServers produce many terabytes of log lines every day. Those logs are available for consumers through hundreds of kinesis streams spread across 5 AWS regions. Part of those log lines (around 8TB/day) are useful to other services and should be quickly available on S3. Here OMFG enters the scene.
Every minute OMFG takes what has read from Kinesis during the previous minute and compresses it into a single <a href="https://facebook.github.io/zstd/">zstandard</a> file per stream, immediately uploaded to S3.
The <a href="https://storm.apache.org/">Storm</a> version used 6 <code class="language-plaintext highlighter-rouge">m5ad.4xlarge</code> EC2 reserved instances plus 2 additional <code class="language-plaintext highlighter-rouge">c5d.large</code> machines. We thought that using Baker, which is heavily optimized to process log lines, we would have immediately been able to shut down some machines. Moreover, we also decided to move to use EC2 instances from the spot market as <a href="https://tech.nextroll.com/blog/dev/ops/2018/10/15/x-marks-the-spot.html">previous experiences</a> told us it’s an excellent way to save some money.
A possible alternative to Baker could have been <a href="https://aws.amazon.com/kinesis/data-firehose/">Kinesis Data Firehose</a>, whose job is exactly to “reliably load streaming data into data lakes, data stores and analytics tools”, including S3, our target. Unfortunately, it doesn’t support zstd compression and, moreover, its costs would have been too high: the data streams consumed by OMFG produce more than 1 PetaByte of log lines each month, which means more than $30k/month (not including S3 and data transfer costs). That’s way more than what we ended spending with our custom solution.
Analyzing the service’s detailed costs we realized that only ~25% was spent on the instances, while the data transfer across regions cost around 70%. 
That was somewhat surprising and it seemed clear that it should be the first area to focus on to make savings.
<h2 id="baker-our-in-house-data-processing-software">Baker, our in-house data processing software</h2>
Back in 2016 we decided to try to write a data processor pipeline using <a href="https://golang.org/">Go</a>. The results were so good that the “experiment”, now called Baker, has been highly successful here at NextRoll during the past years and it’s currently used by many teams to process data at PetaByte scale.
Baker works on a “topology”, the description of a pipeline including one input component (where to read the loglines from), zero or more filters, which are functions that can modify loglines (changing fields, dropping them, or even “splitting” them into multiple loglines), and one output component, defining where to send the resulting loglines to (and which columns), also optionally adding a sharding rule. Input/output components support the main AWS services (Kinesis, DyanomDB, S3, SQS) as well as relational databases and in-house products (like the open-sourced <a href="http://traildb.io/">TrailDB</a>).
Baker is fully parallel and maximizes the usage of both CPU-bound and I/O bound pipelines. On an AWS <code class="language-plaintext highlighter-rouge">c3.4xlarge</code> instance, it can run a simple pipeline (with little processing of log lines) achieving 30k writes per seconds to DynamoDB in 4 regions, using ~1GB of RAM in total, and only a portion of the available CPU (so with room for scaling even further if required).
It is very stable and so optimized for memory and CPU consumption that replacing an existing pipeline with Baker often means greatly reducing the number of servers needed. 
This has also been the case for OMFG: with Baker we only use 6 <code class="language-plaintext highlighter-rouge">c5.xlarge</code> instances. Compared to the old OMFG version it means 24 vCPUs and 48 GB of RAM instead of 100 vCPUs (-76%) and 392 GB of RAM (-87%).
We are actively working to make Baker an open source software. It will be publicly available over the next months, so <a href="https://github.com/adroll/">stay tuned</a>!
<h2 id="aws-data-transfer-between-regions">AWS data transfer between regions</h2>
<center>
<img alt="AWS datacenter are spread all around the globe" src="/images/post_images/omfg_aws_regions.png" /> 
AWS datacenter are spread all around the globe

</center>
As I mentioned previously, the data transfer to our main region <code class="language-plaintext highlighter-rouge">us-west-2</code> from the other regions represented by far the most significant costs of OMFG.
The original OMFG service was deployed entirely in <code class="language-plaintext highlighter-rouge">us-west-2</code> while the log lines were produced (and thus read by OMFG) in all the 5 regions used by NextRoll. The log lines, produced inside Kinesis, are uncompressed, and transferring all those bytes to <code class="language-plaintext highlighter-rouge">us-west-2</code> was very expensive. In fact, AWS charges for outbound or inter-region bandwidth, while it doesn’t charge for data transfer within the same availability zone or data transfer between some services in the same region (like Kinesis to EC2).
For this reason we decided to create a split infrastructure, deploying OMFG on multiple ECS clusters across the globe (one per region), reading uncompressed data from the Kinesis in the same region (with no costs) and transferring the files to <code class="language-plaintext highlighter-rouge">us-west-2</code> after compression.
Fun fact about <a href="https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer">EC2 data transfer prices</a> is that, while outbound bandwidth costs decrease when the volume increases, data transfer prices between AWS regions are constant. The latter is generally lower than the former, but it can be unexpectedly higher in some cases. For example, data transfer from EC2 to the internet in Tokyo costs $0.084 per GB when transferring more than 150TB/month, while transferring data to another region costs $0.09 per GB. In Singapore, it’s even more: $0.09 per GB between regions and $0.08 per GB to the internet.
<center>
<img alt="The new OMFG" src="/images/post_images/omfg_new_infra.jpg" /> 
The new OMFG, deployed in multiple regions

</center>
The compression ratio was the key to this idea of distributed deployment. An overly low ratio would not have balanced the distributed deployment costs of a centralized solution. 
The default zstd compression level (3) is good enough (~9x compression ratio), but after some testing we found out that it’s easy to obtain better results using the <a href="https://github.com/facebook/zstd/releases/tag/v1.3.2">long range mode</a>. 
In the chart below we can see an example of compression ratios on one of our log line types. Also without the long range mode (red line) it’s possible to increase the default compression ratio using higher values, but the cost in terms of lost speed is not particularly low. Level 10 seemed a good compromise for us because it performs almost like smaller levels but has a good jump in the ratio (&gt;10x).
Maintaining the same compression level (10) but using the long range mode (enabled with the <code class="language-plaintext highlighter-rouge">--long</code> option) at level 27 (the default, with a 128MB window size) we see a big jump to &gt;13x ratio (blue line) with a small decrease of speed. Larger window sizes still slightly increase the compression ratio. Still, they have a significant drawback: the default zstd decompression parameters allow for using values of the <code class="language-plaintext highlighter-rouge">--long</code> option up to 27 without having to add any new option to the decompressor. Beyond that number, the decompressor must use an equivalent parameter, which would require updating each reader, which was incompatible with our use case.
<center>
<img alt="Zstd --long option values" src="/images/post_images/omfg_zstd_long_option.png" /> 
How the zstandard long range mode impacts the compression ratio

</center>
So, ultimately, we used the compression level 10 with the default long range mode window size. 
The resulting compression ratios range from 4x to 30x on average, with peaks at 100x (performances depend on log lines type and sizes).
<h2 id="ecs-and-spot-market-a-not-so-smooth-relationship">ECS and spot market, a not-so-smooth relationship</h2>
<center>
<img alt="ECS and spot market, a not-so-smooth relationship" src="/images/post_images/omfg_troubles.png" /> 
</center>
The new OMFG runs on <a href="https://aws.amazon.com/ecs/">Elastic Container Service</a> clusters as <a href="https://www.docker.com/">Docker containers</a>. We decided to use <a href="https://aws.amazon.com/ec2/spot/">EC2 spot instances</a> to provide computational power because they can ensure significant savings (&gt;65% on the instance types we use). But by its very nature, a spot instance <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html">can be stopped</a> by Amazon at any time, and the availability of the chosen instance types can change based on the fluctuation in the market demand.
To address the change in availability we started introducing a long list of possible instance types, decreasing the chance of running out of instances. 
But spot interruptions will still happen, and OMFG must provide an almost realtime service, with a maximum delay in uploading the files to S3 of ~5 minutes. In small clusters such as ours (1-2 instances per region), an interruption has a vast impact, stopping 40-100% of the running docker containers. The Auto Scaling Group (ASG) we use to manage the active instances in the cluster requires some time to perceive the interruption and replace the missing machine, with an average downtime of 8 minutes, too high for us.
When AWS decides to stop a spot instance, 120 seconds are conceded before the operating system enters the shutdown phase. During these 2 minutes, it is necessary to perform a graceful shutdown saving all data currently been processed by Baker. Otherwise, it will be lost (there is, in any case, a way to recover data if using non-“volatile” disks, a subject that we wrote about <a href="https://tech.nextroll.com/blog/dev/ops/2018/10/15/x-marks-the-spot.html#spot-savior-a-log-data-recovery-system">in a past article</a>).
Since an official solution doesn’t exist to address the spot instance eviction, we had to craft our own.
<h4 id="solution">Solution</h4>
The solution we have found is to check whether a server eviction is happening and, where necessary, to perform a controlled shutdown of the containers. AWS provides notification of imminent shutdown through the <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html">Instance Metadata Service</a> at <code class="language-plaintext highlighter-rouge">http://169.254.169.254/latest/meta-data/spot/instance-action</code>, which returns a <code class="language-plaintext highlighter-rouge">200</code> HTTP code only when the instance is in the interruption phase. That’s the signal that triggers our cleanup.
This excerpt from the script that we use is executed as a daemon by the operating system (we forge our AMIs with <a href="https://www.packer.io/">Packer</a>). It checks the spot interruption endpoint every 5 seconds and, when the interruption is intercepted, it increases the number of desired instances in the ASG, triggering an immediate launch of a new server. Moreover, it sends a <code class="language-plaintext highlighter-rouge">SIGINT</code> signal to the Docker containers, which is captured by Baker to perform a graceful shutdown, stop the Kinesis consumers and upload all the files before quitting (this generally requires less than 1 minute):
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">#!/bin/bash
REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document|grep region|awk -F\" '{print $4}')
ASG="&lt;asg_name&gt;"
while true; do
CODE=$(curl -LI -o /dev/null -w '%{http_code}\n' -s http://169.254.169.254/latest/meta-data/spot/instance-action)
if [ "${CODE}" == "200" ]; then
# ...
# Kill gracefully all OMFG containers
for i in $(docker ps --filter 'name=omfg' -q)
do
docker kill --signal=SIGINT "${i}"
done
# Immediately increase the desired instances in the ASG
CURRENT_DESIRED=$(aws autoscaling describe-auto-scaling-groups --region "${REGION}" --auto-scaling-group-names ${ASG} | \
jq '.AutoScalingGroups | .[0] | .DesiredCapacity')
NEW_DESIRED=$((CURRENT_DESIRED + 1))
aws autoscaling set-desired-capacity --region "${REGION}" --auto-scaling-group-name "${ASG}" --desired-capacity "${NEW_DESIRED}"
# ...
fi
sleep 5
done</code></pre></figure>
Together with this script, the ECS agent in the instance is configured to set the instance status to <code class="language-plaintext highlighter-rouge">DRAINING</code> when it receives a spot interruption notice (automatically done by ECS when <code class="language-plaintext highlighter-rouge">ECS_ENABLE_SPOT_INSTANCE_DRAINING=true</code> is set in <code class="language-plaintext highlighter-rouge">/etc/ecs/ecs.config</code>, see the <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html">official documentation</a> for details).
Setting the instance status as <code class="language-plaintext highlighter-rouge">DRAINING</code> results in stopped containers not starting again in that server but, instead, being launched in the new instance when ready.
<center>
<img alt="OMFG spot interception" src="/images/post_images/omfg_spot_intercept.jpg" /> 
How OMFG manages a spot interruption

</center>
This solution permits a final downtime of less than a minute.
There’s only a small downside that is shown in the picture above: adding a desired instance to the ASG during the eviction means that another instance is launched when the interrupted machine is stopped, resulting in an exceeding server compared to the previous desired number of instances. 
This additional machine though will only live for a few minutes, until the ASG will shut it down because its usage is too low, bringing the ASG configuration to its default values. 
Not perfect but still a good and working solution.
<h2 id="containers-autobalance">Containers autobalance</h2>
ECS lacks one important feature: balancing the docker containers across the available EC2 instances. 
When a cluster is freshly launched the containers are created in a balanced manner (same number on each server), but when an instance interruption occurs, the tasks are stopped and started again into the available machines (and this happens in seconds, before the new one starts). Then the new machine will be empty or at most with the tasks remaining from the previous operation (because they couldn’t find sufficient free resources on the existing servers to be launched).
As ECS does not provide tools to balance the tasks in the cluster automatically, we, therefore, need to do it ourselves.
<h4 id="solution-1">Solution</h4>
The solution we have found is to run a scheduled task every 15 minutes checking the number of tasks in each <code class="language-plaintext highlighter-rouge">RUNNING</code> instance and balancing them if the difference is excessive. 
The job “counts” the number of containers per server and moves them if the cluster is unbalanced (count difference &gt;35%).
<center>
<img alt="OMFG autobalancing" src="/images/post_images/omfg_balancing.jpg" /> 
How the autobalancing scheduled task works

</center>
<h2 id="replacing-all-instances-without-downtime">Replacing all instances without downtime</h2>
Sooner or later it will be necessary to replace the instances of the cluster with new versions, typically because the AMI has been updated. 
In itself, the operation is not particularly complicated, essentially only manually turn off the machines and they will be replaced with new units that use the updated version of the AMI. 
The problem is that this operation leads to a downtime of the service. That is between the moment in which the instances are turned off and when, after restarting, all the tasks have been automatically launched (more than 8 minutes in our tests). We then found a solution that allows us to launch all the tasks in the new machines before turning off the old ones and, thus, the old instances, taking advantage of the operation of the autoscaling groups.
<h4 id="solution-2">Solution</h4>
The trick is to implement the following steps, which can be performed manually as well as by an automatic script that makes calls to AWS APIs (the latter is our solution):
<ol>
<li>Set all instances of the cluster to the <code class="language-plaintext highlighter-rouge">DRAINING</code> state. Nothing will happen as there are no other <code class="language-plaintext highlighter-rouge">ACTIVE</code> instances on which to move tasks.</li>
<li>Double the number of desired instances in the autoscaling group. These new requests are launched in the <code class="language-plaintext highlighter-rouge">ACTIVE</code> state and all the tasks are started on them thanks to the previous step. As they are launched, similar tasks are also turned off in the <code class="language-plaintext highlighter-rouge">DRAINING</code> units. Baker, as mentioned above, performs a graceful shutdown saving all its current work.</li>
<li>Wait until all the tasks have been turned off on old instances, which means they are active on new ones.</li>
<li>Return the ASG number to the previous value. The ASG shuts down the excess machines bringing the situation back to its initial state.</li>
</ol>
This latter point deserves a broader focus. Turning off excess machines does not in itself guarantee that the machines just launched are not turned off again. 
The key is to set the termination policy of the autoscaling group to <code class="language-plaintext highlighter-rouge">OldestInstance</code>. We are telling the ASG that, if a unit has to end, the choice must fall on the oldest one, precisely what we want.
<h2 id="conclusion">Conclusion</h2>
What is the final outcome of our work?
<ul>
<li>We saved 85% on EC2 costs migrating to Baker (and thus reducing the number and size of instances) and moving to the spot market.</li>
<li>We also saved 90% on data transfer costs compressing all records in their origin regions before sending them to S3.</li>
<li>We incurred some minor additional costs on DynamoDB due to <a href="https://github.com/vmware/vmware-go-kcl/">how Baker works with Kinesis</a>.</li>
</ul>
The final saving was 85% compared to the cost of the previous version, a great result. 
We have learned a lot from this experience and we are now replaying it on other similar services.
</description>
</item>
<item>
<title>How I became the first Latina remote engineer</title>
<link>https://tech.nextroll.com/blog/careers/2020/04/21/first-latina-remote-engineer.html</link>
<pubDate>Tue, 21 Apr 2020 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/careers/2020/04/21/first-latina-remote-engineer</guid>
<description>It’s been 2 years working remotely as a Software Engineer at NextRoll, a tech company in Silicon Valley. When I started the journey to find a new job, I’ve never aimed to be working in the U.S. center of innovative technology companies. I didn’t think I could be the first Latina working remotely in one of these companies either. I was simply motivated to be open to new challenges.
<code class="language-plaintext highlighter-rouge">25 minute read</code>
<hr />
<h2 id="what-if">“What if?”</h2>
Since the beginning of my career, I’ve always worked for people that knew me before, or I’ve applied to positions that a friend was recommending. I never said to myself “now it’s time to get the job you want”. Since then, the perfect job to me is one where I can consider the team members as my family, too. After all, we spend hours in the office. I had the luck to have many families in my professional life.
Once in the past, a friend told me he would start to work from home for a company in the U.S., and that story sounded pretty crazy to me. We live in Brazil, why someone would have interests to contract people so far away? What about the language? You have to convince interviewers you are good enough for some position speaking another language. That wasn’t for me. I had barely traveled abroad at that time to try my English skills that I’ve learned in a classroom. Even though my skeptical ideas, I felt like that was the wonderland of jobs that I would never be part of, a completely new and wonderful world. That was my first “What if?” moment.
Years later, a second friend asked me if I would like to go to Ireland. He and his wife had applied to jobs there and she was accepted. He would finish his last interviews after moving. I answered back immediately: “Why not?”. What if I try the same? I was working for the same company for 8 years, the contracts were ending, the new contracts were paying lower salaries and the remaining people would have to fit the way of work of the new contracts. As an Agilist, I knew that I wouldn’t fit that job anymore, all the agile culture our team had built before wouldn’t be a priority in the new era. From that moment on, I decided to look for the job I really wanted.
When we start to ask questions, things begin to happen. Combined “What if?” and “Why not?” questions are the trigger to make us overcome periods of inertia in our lives and achieve our goals. You start to change the way you think and many possibilities present themselves when you keep your mind open.
<img src="/images/post_images/many_directions.jpg" alt="open_sourced_projects" />
<h2 id="making-things-happen">Making things happen</h2>
After the popularization of the Web, the way people look for a job has changed a lot. I remember when I needed an internship to get my data processing certificate. My father had to drive me to a place that use to concentrate all the open internship positions in the town, a wall full of papers describing these positions. I was 16 years old, all the options I’ve seen on that wall were requiring men. I’ve never forgotten my father’s face realizing that his daughter wouldn’t have a chance in that place. Anyway, I’ve never had to look at a wall like that anymore.
The motivation to keep going in this journey was coming from a requirement list I’ve made based on what I wanted for my future:
<ol>
<li>I wanted to have an international experience, to work with people from different countries and to live abroad for a while.</li>
<li>I’ve been working with agile methodologies since ever, a company with an agile culture suited me well.</li>
<li>I couldn’t wait for new challenges, I wanted to try new programming languages, cloud solutions and microservices.</li>
<li>I wanted to improve my English skills.</li>
</ol>
Virtual life came to make things much easier, but more competitive too. The first step was creating my online CV. I reviewed the description of my professional experiences countless times. It’s not forbidden to spy other online profiles to have some ideas. I’ve learned that it’s important to be specific describing your experiences, it cannot be too short. I used to write on my CV only some topics and technologies I’ve used and that is not enough if you want to find an international opportunity. The first call I’ve answered was a recruiter asking me to be more detailed, that 5 minutes call helped me a lot. These are some things I learned on how to use the Web in your favor:
<blockquote>
<ul>
<li>Emphasize those aspects that will help you to get the job you want and hide those to avoid what you don’t want.</li>
<li>There are many tools and social networks available aiming the professional life and it’s a good practice to keep your personal life out of this. Connect to recruiters in these networks.</li>
<li>Post and share those articles you get interested in, it helps to understand and know you better.</li>
<li>Try some online courses or study to get a certification.</li>
<li>Keep an online portfolio. It can be a simple code you’ve done as a test, a course exercise or an open-source contribution.</li>
<li>Practice your algorithm skills to be prepared for practical tests, there are tools online to do that too.</li>
</ul>
</blockquote>
<h2 id="dealing-with-rejection">Dealing with rejection</h2>
In the beginning, I was focusing on getting a job in Ireland, so I was applying only to positions there. Soon after, I realized that many companies don’t contract people outside Europe, it’s expensive. And then I started to think that I could check in any place in Europe to expand my chances. As I had made my online work right, recruiters began to find me. That’s why it is so important to give as much information as you can in your profile. I’ve attended some interviews and it helped me to improve my English skills during the process.
After repeating the story about your experiences, you feel more confident for a while. Each time you tell the story again, it’s a chance to make it better. However, after hearing some No’s, your confidence changes. You start to feel like a fraud. Your mind begins playing tricks with yourself, making you doubt your competence. I’ve repeated so many times the same story that I wasn’t sure what was real anymore. I’ve found out then that this phenomenon has a name, it’s the Impostor syndrome. Indeed, I was feeling like an impostor. Was I expecting too much of a job? Was I prepared for that?
While I was in those circumstances, my contract had ended. I was unemployed and I’ve got pregnant. I felt like I was 18 years old and my future was compromised. I was not willing to remove any item from my requirement list. So I changed the perspective: I could keep the international experience without living abroad. I added a fifth item: to work remotely. I even convinced another friend, another woman in tech, that she should start to search for remote jobs. Six months later, she was working from home. I was thrilled that I could plant that “What if?” in her mind.
My situation had changed, I needed a break. I was full of concerns on how to become a mom, how to start a new job being pregnant, how to work having a newborn besides some existential crises. I’ve seen in this period dozens of work colleagues going to live abroad after a few interviews. And I had to prepare myself for the most challenging task in my life. The job search was put on standby.
When I returned to my job search, and I started to attend some interviews again, I had a gap of 2 years in my professional life. I’ve been working for 15 years in software development, but the interviewers were worried about those last 2 years. It wasn’t enough to them saying that I was studying during that time, I was updating myself trying new technologies, I became an undesirable candidate. I even heard the question “Who will take care of your baby while you are at work?”. It sounded like it would be impossible to work again after all I had done. That requirement list of the perfect job didn’t matter anymore, I had to concentrate my efforts to get a job.
I still don’t have a tip for people in the same situation of having an empty lapse of time on their CVs. I’ve always told the truth and I had to live with the consequences. Fortunately, some interviewers evaluate your capacity for growth in the company. In all of the following experiences I describe in the next section, I didn’t hear the question “What were you doing in the last 2 years?”.
<h2 id="point-of-seeing-the-results">Point of seeing the results</h2>
Looking for remote jobs, I found out that most offers come from the U.S. and most of them only accept applications from U.S. citizens. I was doing well in a selection process and I couldn’t go ahead because of the bureaucracy of contracting a foreigner. Each country has its labor laws and it can be complicated to deal with it for many companies.
I applied to a position at a company in the south of Brazil, but it would require relocation to another city. It was an amazing opportunity, after many steps in the selection process they offered me the job. What they were offering me, in terms of the culture of the company and technologies, totally fit my ideas for my new job except for not being an international experience. It was really hard to say “no” to them, after many calls and texts messages, I realized that my family wasn’t prepared for that relocation. I’m grateful for that offer, I stopped feeling like an impostor.
In the meantime, I was referred to a position in a DevOps team. The company didn’t have an agile culture, but it was starting to put in place some agile techniques. It was a huge challenge, they needed to apply automation in all main company systems and I had the required experience to work on that. I was happy to start working again, but I felt like I had failed with my initial goals. That was a really important opportunity in many ways, I had my confidence again.
One month after I accepted that regular job, I answered an unexpected call. People say that we don’t find the best opportunities in job lists. Usually, they find you. After some interviews with the DC team in NextRoll, they offered me a remote job. I didn’t know too much about the company at that time and I didn’t know what to expect, but this was the international experience I was pursuing so long. I was afraid to quit my regular job, I was afraid of an unknown field, I almost said “no”, but I didn’t. On my first day in the company, I found out that it suited all 5 requirements on my list: NextRoll has an amazing agile and diverse culture and it’s committed to the best solutions.
I’ve taken some time to realize the importance of my achievements. I was in the center of technology in the U.S. and I was the first Latina working remotely as a Software Engineer at NextRoll, a Silicon Valley company. If I wanted new challenges I was in the right place, not because the office was in San Franciso, but the complexity of the solutions I was facing was impressive. Until now, I had the opportunity to write code in Java, Python, Javascript, Ruby, Golang, and Erlang. I also have to read data from SQL and NoSQL databases. The systems deal with a high volume of data in the cloud, I could explore many AWS services. All of these systems are part of a service architecture that depends on different teams to be maintained. It means that we always have to be in contact with different people in the company, in a different part of the world and from a different culture. Every day is a day to learn something new and that’s why I’ve chosen this profession, you never get bored.
<h1><img src="/images/post_images/desk.jpg" alt="open_sourced_projects" /></h1>
<h2 id="conclusion">Conclusion</h2>
I wanted to share this experience to motivate people to chase their goals and to put into words some difficulties that stood in my way. Usually, we share the successful results we had, but we forget the bad moments and hard decisions we’ve made that guide us there. I believe it can help someone else that is facing some setbacks and need that extra push to keep going.
Finally, I’d like to inspire other women in tech and to remind us again of the importance of our stories.
</description>
</item>
<item>
<title>Squeezing the most out of the server: Erlang Profiling</title>
<link>https://tech.nextroll.com/blog/dev/2020/04/07/erlang-profiling.html</link>
<pubDate>Tue, 07 Apr 2020 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2020/04/07/erlang-profiling</guid>
<description>NextRoll’s <a href="http://tech.nextroll.com/blog/dev/2018/01/08/quaff-that-potion-saving-millions-with-elixir-and-erlang.html#real-time-bidding">real-time bidding</a> (RTB) platform has been featured several times in this tech blog: we run a fleet of Erlang applications (the bidders) that typically ranges between one and two
thousand nodes. As described in <a href="http://tech.nextroll.com/blog/dev/ops/2018/10/15/x-marks-the-spot.html">past articles</a>, an ongoing goal of the RTB
team —as well as a source of interesting technical problems— is to minimize operational costs
as much as possible.
An obvious way to reduce costs is to make the system more efficient and this
means entering the <a href="http://wiki.c2.com/?RulesOfOptimization">hazardous land of software optimization</a>. Even for experienced programmers,
identifying bottlenecks is a hard enough problem when using the right tools;
trying to guess what could make the code run faster will not only waste time
but is likely to introduce unnecessary complexity that can cause problems down the line. The cousin of
premature optimization is necessary optimization <a href="http://wiki.c2.com/?ProfileBeforeOptimizing">without profiling first</a>
While Erlang is famously known for its concurrency model and fault-tolerant design,
one of its biggest strengths is the level of live inspection and tuning it offers,
often with little or no setup and runtime cost. In this article, we outline
how we leverage those features to profile our system, driving the optimizations that
can lead to cost reductions.
<code class="language-plaintext highlighter-rouge">10-15 minute read</code>
<hr />
<h2 id="infrastructure">Infrastructure</h2>
An interesting aspect of real-time bidding is that it is fairly low-risk to test in production. Even if the new code is slow or contains errors, the bidders are architected to just send a no-bid response whenever a request can’t be fulfilled.
Taking advantage of this, we incorporate canary deploys to our day-to-day development workflow. In the context of optimizations, this means we can quickly test our performance hypothesis by updating the code and testing with live traffic. We have metrics and dashboards that give feedback on common metrics like timeouts, errors and amount of requests processed, making it obvious when a change is beneficial.
<h2 id="request-level-timers">Request-level timers</h2>
Bid request processing is the fundamental operation of the bidder application. Any improvement in the amount of time it
takes us to send a response to an ad exchange means we can process more requests per server, requiring fewer servers to handle
our traffic and ultimately saving us money.
The work involved in a bid request can be broadly divided into a series of tasks such as payload parsing, selection of matching ads, pricing a particular ad, and response encoding.
A common practice is to periodically measure the time invested in each of those phases to make sure they don’t degrade over time and to help provide a frame of
reference to use when we look for areas of the code that are worthy of our optimization efforts.
<center>
<img alt="A sample of bid request timings per phase" src="/images/post_images/erlang_requests.png" /> 
A sample of bid request timings per phase.

</center>
The most basic way of profiling consists of timing the calls to
a given piece of the code (perhaps one that was deemed suspicious by one of the methods described in the next sections):
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">%% Evaluates Fun() and reports its evaluation time as a histogram
%% metric.
-spec time_call(name(), fun(() -&gt; Result)) -&gt; Result.
time_call(Metric, Fun) -&gt;
Start = erlang:system_time(microsecond),
Result = Fun(),
End = erlang:system_time(microsecond),
Diff = End - Start,
update(histogram, Metric, Diff),
Result.</code></pre></figure>
The above helper is used to wrap calls to the function we want to measure.
A canary deploy of the timed code to production will generate average, median and percentile metrics that we can then compare to the overall request time to identify bottlenecks.
<h2 id="recon">recon</h2>
Timing request operations can be a very useful technique for understanding the specifics of a request flow but gives us a limited perspective of the entire system. Most of the bid request phases are handled on a single process and some of them involve idle time waiting for external systems. There are many periodic tasks and long-lived support processes in the bidders, and we can benefit from system-wide profiling that looks beyond bid request processing. This is where the Erlang toolbox comes into play.
The first valuable resource is not a piece of software but a little book by <a href="https://twitter.com/mononcqc">Fred Hebert</a>: <a href="https://www.erlang-in-anger.com/">Erlang in Anger</a>. This guide is the perfect reference because it describes methods for gaining insight and optimizing production systems, backed by real-world experience.
The book’s companion library, <a href="http://ferd.github.io/recon/">recon</a>, provides safer, friendlier and more productive interfaces to
Erlang’s powerful inspection tools. What follows are some simple examples mostly derived from the book.
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">%% get node general stats
(bidder_dev@node)&gt; recon:node_stats_print(1, 100).
{[{process_count,3061},
{run_queue,31},
{error_logger_queue_len,0},
{memory_total,6141339952},
{memory_procs,2623297976},
{memory_atoms,967320},
{memory_bin,610903880},
{memory_ets,2078839480}],
[{bytes_in,729538},
{bytes_out,2722704},
{gc_count,9475},
{gc_words_reclaimed,28611695},
{reductions,699610880},
{scheduler_usage,[{1,0.9840756659394524},
{2,0.9760797698171886},
{3,0.9891411385108696},
{4,0.9817415742856814},
{5,0.985099143110567},
{6,0.9862095293650041},
{7,0.9759060081154223},
{8,0.9926624828687856}]}]}
%% get top 3 memory consumers
(bidder_dev@node)&gt; recon:proc_count(memory, 3).
[{&lt;0.1597.0&gt;,986767172,
[bidder_banker,
{current_function,{gen_server,loop,7}},
{initial_call,{proc_lib,init_p,5}}]},
{&lt;0.344.0&gt;,915466692,
[bid_guardian,
{current_function,{gen_server,loop,7}},
{initial_call,{proc_lib,init_p,5}}]},
{&lt;0.15784.2618&gt;,41052056,
[{current_function,{prim_inet,recv0,3}},
{initial_call,{proc_lib,init_p,5}}]}]
%% get top 3 cpu consumers
(bidder_dev@node)&gt; recon:proc_count(reductions, 3).
[{&lt;0.7011.0&gt;,2464069696,
[{current_function,{exometer_probe,loop,1}},
{initial_call,{proc_lib,init_p,3}}]},
{&lt;0.6730.0&gt;,2422804659,
[{current_function,{exometer_probe,loop,1}},
{initial_call,{proc_lib,init_p,3}}]},
{&lt;0.6740.0&gt;,2402511307,
[{current_function,{exometer_probe,loop,1}},
{initial_call,{proc_lib,init_p,3}}]}]
%% get top cpu consumers during a time window
(bidder_dev@node)&gt; recon:proc_window(reductions, 3, 1000).
[{&lt;0.20046.2631&gt;,688443,
[{current_function,{views_bid,process,1}},
{initial_call,{proc_lib,init_p,5}}]},
{&lt;0.29341.2629&gt;,547335,
[{current_function,{timer,sleep,1}},
{initial_call,{proc_lib,init_p,5}}]},
{&lt;0.7687.2632&gt;,386187,
[{current_function,{prim_inet,recv0,3}},
{initial_call,{proc_lib,init_p,5}}]}]
%% get process stats, including stacktrace
(bidder_dev@node)&gt; recon:info(&lt;0.1368.0&gt;).
[{meta,[{registered_name,bidder_banker},
{dictionary,[{'$initial_call',{bidder_banker,init,1}},
{'$ancestors',[bidder_sup,&lt;0.340.0&gt;]}]},
{group_leader,&lt;0.339.0&gt;},
{status,running}]},
{signals,[{links,[&lt;0.342.0&gt;]},
{monitors,[]},
{monitored_by,[]},
{trap_exit,true}]},
{location,[{initial_call,{proc_lib,init_p,5}},
{current_stacktrace,[{bidder_banker,'-handle_info/2-lc$^0/1-0-',1,
[{file,"/bidder/src/pacing/bidder_banker.erl"},
{line,337}]},
{bidder_banker,'-handle_info/2-lc$^0/1-0-',1,
[{file,"/bidder/src/pacing/bidder_banker.erl"},
{line,337}]},
{bidder_banker,'-handle_info/2-lc$^0/1-0-',1,
[{file,"/bidder/src/pacing/bidder_banker.erl"},
{line,337}]},
{bidder_banker,handle_info,2,
[{file,"/bidder/src/pacing/bidder_banker.erl"},
{line,337}]},
{gen_server,try_dispatch,4,
[{file,"gen_server.erl"},{line,637}]},
{gen_server,handle_msg,6,
[{file,"gen_server.erl"},{line,711}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,249}]}]}]},
{memory_used,[{memory,916065740},
{message_queue_len,0},
{heap_size,75113},
{total_heap_size,114508086},
{garbage_collection,[{max_heap_size,#{error_logger =&gt; true,kill =&gt; true,size =&gt; 0}},
{min_bin_vheap_size,46422},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,3}]}]},
{work,[{reductions,1961022577}]}]</code></pre></figure>
As suggested by the book, a good method is to run <code class="language-plaintext highlighter-rouge">recon:proc_window</code> repeatedly and try to identify patterns, e.g. a process that frequently ranks among the top CPU consumers.
The process id can then be passed to <code class="language-plaintext highlighter-rouge">recon:info</code> to get useful information (such as the stacktrace) in order to understand what the process is doing.
Using this method we quickly found that a commonly accessed data structure contained ~100kb of debug data which was being copied thousands of times per second.
This approach, though, will tend to highlight long-running busy processes over short-lived ones (that could be called a lot and account for bigger overhead overall).
This can be partially overcome by running <code class="language-plaintext highlighter-rouge">proc_window</code> repeatedly and aggregating the results by location rather than process id. However, there are better tools to look at aggregated process times.
<h2 id="redbug">redbug</h2>
Strictly speaking, <a href="https://github.com/massemanet/redbug">redbug</a> is not a profiling tool, but it’s so useful for debugging live systems that it deserves a mention in this article.
It allows you to safely trace functions from the shell in a very intuitive yet sophisticated way (as opposed to the rougher Erlang built-in tracing modules).
It can be very handy to get a quick notion of how functions are being called, how frequently, and what production data looks like:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">(bidder_dev@node)&gt; redbug:start("jiffy:encode/1").
% 00:11:03 &lt;0.18718.7151&gt;({erlang,apply,2})
% jiffy:encode(1)

(bidder_dev@node)&gt; redbug:start("jiffy:encode-&gt;return").
% 00:11:03 &lt;0.18718.7151&gt;({erlang,apply,2})
% jiffy:encode([&lt;&lt;"USD"&gt;&gt;])
% 00:11:03 &lt;0.18718.7151&gt;({erlang,apply,2})
% jiffy:encode/1 -&gt; &lt;&lt;"[\"USD\"]"&gt;&gt;

(bidder_dev@node)&gt; redbug:start("jiffy:encode([N]) when is_integer(N)-&gt;return").
% 00:12:20 &lt;0.1535.7173&gt;({erlang,apply,2})
% jiffy:encode([300])
% 00:12:20 &lt;0.1535.7173&gt;({erlang,apply,2})
% jiffy:encode/1 -&gt; &lt;&lt;"[300]"&gt;&gt;</code></pre></figure>
Including recon and redbug in an Erlang application release has no cost and can be a real life-saver
when diagnosing production issues. These tools promote a flow that’s more powerful
than adding prints to the code and feels more natural than step debuggers
which wouldn’t be that useful in a highly concurrent world anyway.
<h2 id="erlang-easy-profiling-eep">Erlang Easy Profiling (eep)</h2>
<a href="https://github.com/virtan/eep">eep</a> allows for a more “traditional” approach to profiling by using Erlang tracing to take a snapshot of the system operation with function call counts, execution times and inter-dependencies.
It requires a bit more effort to use and it is not as safe as the rest of the tools described in the article.
It will slow down the system potentially even killing it if used carelessly. Its output file can eat up a lot of disk space (a 100 ms snapshot takes about 300mb for our system).
Depending on the nature of the application, it may not make sense to run it directly in production.
Here’s an example tracing session using eep:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">(bidder_dev@node)&gt; eep:start_file_tracing("file_name"),
timer:sleep(100),
eep:stop_tracing().</code></pre></figure>
Note that we start, sleep and stop tracing in the same line. Don’t rely on the shell being responsive during tracing! You could send a message or call a function in between, as well, to force a certain part of the code to be executed while taking the snapshot.
The instructions above will output a <code class="language-plaintext highlighter-rouge">file_name.trace</code> file in the release directory. The file then needs to be moved out of the production server and processed on a local Erlang shell:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">(local)&gt; eep:convert_tracing("file_name").
working
38436 msgs (38366 msgs/sec), 0.106996 secs (9x slowdown)
39367 msgs (37807 msgs/sec), 0.106996 secs (10x slowdown)
done</code></pre></figure>
This will, in turn, produce a <code class="language-plaintext highlighter-rouge">callgrind.out.file_name</code> that can be input to <a href="http://kcachegrind.sourceforge.net/html/Home.html">Kcachegrind</a> (Qcachegrind in macOS). Note that by default the tracing will discriminate entries per process id. This would yield a similar situation to what we saw with <code class="language-plaintext highlighter-rouge">recon:proc_window</code>. A more interesting view is to merge function calls of all processes, which can be accomplished by stripping the pid:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ grep -v "^ob=" callgrind.out.file_name &gt; callgrind.out.merged_file_name
$ qcachegrind callgrind.out.merged_file_name
</code></pre></div></div>
QCachegrind presents the snapshot in a very sophisticated UI that can be used to spot the most frequently called functions, where most time is spent, etc.
<center>
<img alt="eep output to QCachegrind" src="/images/post_images/erlang_qcachegrind.png" /> 
eep output to QCachegrind.

</center>
Since eep is based on Erlang tracing, it will add an overhead to all code and may comparatively
misrepresent the work done by Built-in functions (BIFs) and Native-implemented functions (NIFs),
so the timings shown in the snapshot need to be taken with a grain of salt. Nevertheless, it is
still a great exploratory tool to understand how the different components of the system interact,
how dependencies are used, and learn about obscure or suspicious areas that can be hard to spot by just looking at the code.
Note that there are other Erlang profiling libraries (which we haven’t tried yet), that produce
callgrind output: <a href="https://github.com/isacssouza/erlgrind">erlgrind</a> and
<a href="https://github.com/rabbitmq/looking_glass">looking_glass</a>.
<h2 id="erlangsystem_monitor">erlang:system_monitor</h2>
Yet another way of looking at the application is the <a href="http://erlang.org/doc/man/erlang.html#system_monitor-2"><code class="language-plaintext highlighter-rouge">erlang:system_monitor/2</code></a> BIF.
It allows you to set up a process to receive a message every time a certain condition is met. It was particularly helpful for us
when examining long garbage collections and schedules of long duration, the latter of which can surface issues with NIFs that
would go unnoticed with other methods.
Here’s an example of its use in the shell (based off a snippet from Erlang in Anger):
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">(bidder_dev@node)&gt; Loop = fun F() -&gt;
receive
{monitor, Pid, Event, Info} -&gt;
ReconLocation = recon:info(Pid, location),
io:format("monitor=~p pid=~p ~n info=~p ~n recon=~p~n",
[Event, Pid, Info, ReconLocation])
end,
F()
end.
(bidder_dev@node)&gt; spawn(fun() -&gt;
register(temp_sys_monitor, self()),
erlang:system_monitor(self(), [{long_schedule, 30}, {long_gc, 30}]),
Loop()
end),
timer:sleep(10000),
exit(whereis(temp_sys_monitor), kill).</code></pre></figure>
This will monitor the system for 10 seconds and output process information to the shell every time there’s a garbage collection or schedule that takes more than 30ms:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">monitor=long_gc pid=&lt;7011.2176.5384&gt;
info=[{timeout,33},
{old_heap_block_size,2984878},
{heap_block_size,514838},
{mbuf_size,0},
{stack_size,35},
{old_heap_size,1535726},
{heap_size,51190}]
recon={location,
[{initial_call,{proc_lib,init_p,5}},
{current_stacktrace,
[{bidder_stat,update,4,
[{file, "/bidder/src/bidder_stat.erl"}, {line,202}]},
{views_bid,bid,1,
[{file, "/bidder/src/views/views_bid.erl"}, {line,79}]},
{views_bid,bid_request,1,
[{file, "/bidder/src/views/views_bid.erl"}, {line,17}]},
{erl_stat,time_call,3,
[{file, "/bidder/src/bidder_stat.erl"}, {line,42}]},
{bidder_web_handler,dispatch,2,
[{file, "/bidder/src/bidder_web_handler.erl"}, {line,211}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,249}]}]}]}</code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
This article is by no means an exhaustive list of Erlang diagnosing tools; there’s <a href="https://erlang.org/doc/man/observer.html">the
observer</a>, <a href="https://erlang.org/doc/man/eprof.html">eprof</a>, <a href="https://erlang.org/doc/man/fprof.html">fprof</a>, <a href="https://github.com/proger/eflame">eflame</a>, <a href="https://github.com/jlouis/eministat">eministat</a> and the list goes on. The Erlang documentation
itself has a nice <a href="https://erlang.org/doc/efficiency_guide/users_guide.html">efficiency guide</a> with an overview of the <a href="http://erlang.org/doc/efficiency_guide/profiling.html">built-in profiling modules</a>.
Since we started with this effort, we consistently reduced request times (and operational costs), month over month.
To a large extent these gains came thanks to the advanced tools Erlang and its ecosystem have to offer.
What’s most interesting is that we achieved this by getting to know our systems better,
fixing bugs, and often removing rather than adding specialized code.
<center>
<img alt="Make it beautiful" src="/images/post_images/erlang_joe.jpg" />
</center>
</description>
</item>
<item>
<title>
Women in Tech Spotlight: Larissa Licha
</title>
<link>https://tech.nextroll.com/blog/women_in_tech/2020/03/31/women-in-tech-spotlight-larissa.html</link>
<pubDate>Tue, 31 Mar 2020 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/women_in_tech/2020/03/31/women-in-tech-spotlight-larissa</guid>
<description>Women in Tech - Spotlight: Larissa Licha
<code class="language-plaintext highlighter-rouge">6 minute read</code>
<hr />
March is Women’s History Month and in honor of this occasion, NextRoll is launching a month-long Women in Tech Spotlight – where we highlight a few of the women here at NextRoll. This will be a limited three-part series of interviews, showcasing three of our extraordinary women in technology. This week, I had the pleasure of interviewing Larissa Licha!
<h2 id="larissa-licha">Larissa Licha</h2>
<h3 id="title-director-of-product-platform-services">Title: Director of Product, Platform Services</h3>
<img src="/images/post_images/larissa.jpg" alt="open_sourced_projects" />
<h2 id="can-you-share-a-little-bit-about-what-it-is-that-you-do-and-what-a-typical-day-for-you-is-like">Can you share a little bit about what it is that you do and what a typical day for you is like?</h2>
I am the Director of Product Management for Platform Services, our newest business unit. My focus is heavily tied to strategy and the overarching vision for the business unit and ensuring that product and development execute towards that vision. I also work closely across departments such as Business Development, Finance, Legal and end customers. On a typical day, I have to context switch a lot between working with engineers to make sure we get key tickets done while prioritizing the right tasks (our priorities change a lot), to delegating across cross-functional teams or impacted engineering teams, to talking to the end customer ensuring we meet their needs, keep them in the loop on progress or key updates, to engaging with new prospects, to monitoring revenue numbers and help forecasting, and staying on top of what’s going on in the industry.
As the Director of Product Management, I need to make sure I can unblock where I can to set others up for success while driving our revenue forward. At the same time, I try to stay sane while having 100 TODOs every day. Meetings are definitely a big portion of my day, but it is a very exciting role because it involves greenfield technical architecture, pricing and packaging, machine learning to general product management and more. To be exposed to a lot of different areas across the business makes this a very exciting role and business unit.
<h2 id="what-influences-you-to-pursue-a-career-in-technology">What influences you to pursue a career in technology?</h2>
This is a really weird one on my end. I grew up in a village with 300 people in the middle of nowhere in Germany. Our paths in the village are very set on going to school, apprenticeships in banks or insurance companies, then you stay at the bank or insurance company, you get married, have kids and that’s it, you never really look to go outside of these set boundaries. Even going to university was never really something I was being made aware of. I never knew I could be a Data Scientist or an Engineer. I knew I didn’t want to follow the path I was told to follow, but I wasn’t aware of what other options I had.
Being a teenager was an interesting time (especially for my mom) since I was trying to find my path which resulted in some ‘rebellious’ behavior (based on village standards). I ended up becoming a tattoo artist for a big chunk of time, traveling a lot all over the world, living in different countries and cities but at 19 I had a life event and was sick for a year, and I realized I didn’t want to be the ‘rebel’ that people expected me to be. I stopped following my ‘self-fulfilled prophecy’ after reading a book about a woman that got stuck in a rut and then ‘life visited her’, she ended up moving to Dublin, Ireland to start over. I thought ‘I can move to Dublin, Ireland’. I started applying to random jobs and ended up getting a contracting job for a vendor that works for Google to do Customer Service for Google Ads. My first job in tech.
When I moved to Ireland, I didn’t know anyone, lived in a hostel, but just having this job seemed enough for me to go on. Starting at the company in customer services, only 6 months in I became a team lead and moved into operations. A year later, I had the opportunity to join NextRoll and something told me that I had to take that leap. NextRoll has been a defining part of my career and sometimes I still wonder how I got here. I barely made it through math in school, and now I’m reading machine learning papers about causal inference which just shows how life is never a straight line or predictable. But I really found my place here – in this weird non-linear way.
<h2 id="what-is-it-like-to-be-a-woman-working-in-tech-for-you">What is it like to be a woman working in Tech for you?</h2>
Being a woman in tech during a time where there’s a focus on having more women in tech has been encouraging. Concepts like advocates and allies becoming the norm, alongside technological changes in how we hire, or coding boot camps for women (and girls) all strive to help other women to go into technology, which is exciting. As a hiring manager, it’s my job to focus on growing a diverse team and make sure that my team – man or woman – have an equal voice. We value everyone’s opinion and ensure everyone is heard no matter the gender or how you identify.
Besides hiring more diverse teams and ensuring belonging, it’s important to have someone you can look up to that you can see yourself in. I have always had people I looked up to, but oftentimes those people have been men. This shows that there’s still a lot of work to be done to grow the representation of women in tech and leadership.
I am happy to see that in NextRoll Platform Service we built a more diverse team. There’s still a lot of progress to be made but in product management, we’re solely women, and on the business development side our lead is a woman as well. Meanwhile, on the engineering side, we still got ways to go.
It’s been encouraging to be right at the point of change and to be able and empowered to influence that change so we can all pave a path for women in tech.
<h2 id="is-there-one-piece-of-advice-you-wish-somebody-gave-you-at-the-beginning-of-your-career">Is there one piece of advice you wish somebody gave you at the beginning of your career?</h2>
It’s very important to just go for it! This is my 7th job at the company in 5 ½ years. I started in sales development and ultimately ended up in product leadership but I never proactively pursued any of the roles I’ve held. I was fortunate enough to have had people that told me to go for it and believed in me when I didn’t. I still get moments of imposter syndrome where I think ‘I shouldn’t be here’. Without the people that invested in me and trusted me to do bigger things than what I have trusted myself to do, I don’t think I’d be where I am today. So, don’t be like me, you gotta be braver and just grab opportunities when they arise since having people nudge you or waiting for them to do so may never come around.
<h2 id="who-has-been-your-biggest-advocatementor-in-the-workplace-and-why">Who has been your biggest advocate/mentor in the workplace and why?</h2>
I’ve had many but if I had to pick one that made the biggest impact on my career, it’s <a href="https://www.linkedin.com/in/dialtone/">Valentino</a> (our CTO). I don’t know if Valentino actually knows this but back when I was an account manager one year into the company, he set up a meeting with me. I was terrified wondering why he’d want to talk to me of all people. During our meeting, he walked me through a dashboard with cool performance stats giving me insight into our underlying infrastructure. He helped me realize I really liked looking at data and understanding the underlying technical pieces. He really pushed me, maybe even unintentionally, to care about the technology and infrastructure and brought me closer to product and engineering. He’s always been someone I could go to and ask questions and then come back with a really deep understanding of the answer. He’s also been a sounding board and great mentor during challenging times, or times where I was unsure of myself or skills. Lastly, he’s been a huge advocate for me throughout my career believing I can do more and trusting me to take bigger leaps. He took down the barriers I’ve built allowing me to be where I am today and I couldn’t be more grateful.
<h2 id="who-are-your-role-models-in-women-in-tech">Who are your role models in women in Tech?</h2>
I nerd out about Susan Athey. She did a lot of interesting research in economics and especially causal inference. I have a strange obsession with causal inference so I can geek out about her content for hours. Once I reached out to her asking whether she’d be willing to hop on a call and when she said yes, I nearly lost it.
I also look to people I work with or observe in my day to day, how they bring themselves to work, and the impact they’re having. I look to people like <a href="https://www.linkedin.com/in/jgrist/">Jessica Grist</a> who is someone that paves the path for other women in tech every day and I admire her drive and passion, she’s also just so smart! Another person is <a href="https://www.linkedin.com/in/prathibha-deshikachar-7435852/">Prathibha</a>, who no longer works here but left her mark to this day. Prathibha is a total badass. She always fought for things to be better than they were and held people accountable. She never accepted mediocrity and pushed everyone to do their best work while taking a no-nonsense approach. There are a lot more people I could mention since my immediate environment influences who I am as a person, co-worker, peer and leader every day.
<h2 id="any-advice-for-women-in-techallies-you-would-give">Any advice for women in tech/allies you would give?</h2>
Now is the time where things are finally shifting. It’s time for everyone to do their part to contribute whether it’s contribution time to areas like ‘<a href="https://girlswhocode.com/">Girls who code</a>’ or talking to kids about technology/computer science and showing them that this is something they can pursue. It’s time to make it normal for girls and women to know what they have a place here in technology.
Among women, we are honestly the hardest on one another. We are raised to be competitive and often bitter about other women’s achievements rather than celebrating them. I ask other women to celebrate women that take on leadership roles, support and help other people grow where they can instead of getting into a competitive mindset. Maybe the competitiveness among women in tech is even higher since there’s so few but only by uplifting each other, we can change that going forward.
For allies, it’s the same thing – elevate people and enable them. If you need to kick someone in the butt for them to take a leap and you’re in a position to do so, do it. Don’t doubt yourself or think you’re not the right person to help them find their way, take the leap to lift someone up. Without the allies along my journey, I don’t know where I’d be today. I needed someone to push me out of my comfort zone, be that person to someone.
</description>
</item>
<item>
<title>
Women in Tech Spotlight - Celebration of Women History Month - Elena Rose
</title>
<link>https://tech.nextroll.com/blog/women_in_tech/2020/03/24/women-in-tech-spotlight-elena.html</link>
<pubDate>Tue, 24 Mar 2020 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/women_in_tech/2020/03/24/women-in-tech-spotlight-elena</guid>
<description>Women in Tech - Spotlight: Elena Rose
<code class="language-plaintext highlighter-rouge">6 minute read</code>
<hr />
March is Women’s History Month and in honor of this occasion, NextRoll is launching a month-long Women in Tech Spotlight – where we highlight a few of the women here at NextRoll. This will be a limited three-part series of interviews, showcasing three of our extraordinary women in technology. This week, I had the pleasure of interviewing Elena Rose!
<h2 id="elena-rose">Elena Rose</h2>
<h3 id="title-manager-data-science-engineer">Title: Manager, Data Science Engineer</h3>
<img src="/images/post_images/elena.png" alt="open_sourced_projects" />
<h2 id="can-you-share-a-little-about-what-you-do-at-nextroll-and-what-your-typical-day-is-like">Can you share a little about what you do at NextRoll and what your typical day is like?</h2>
I have migrated to a manager position a couple of years ago. My day changed quite a bit – before it was mostly coding, now it is primarily connecting teams together, understanding the strategic way to develop our product, cross-team dependencies, and try to compensate for that. This is where all where my brainpower goes to.
<h2 id="what-do-you-like-most-about-your-tech-career">What do you like most about your tech career?</h2>
I enjoy how tech people know what they want: they have the right expertise and a good definition of what they want to do. They have a clear path, are logical, and have structure. The types of people who pursue tech career – engineers joining engineering workforce – join for a reason. These are the people who like to poke around with different ideas. They are curious, and being around with these people is very interesting.
<h2 id="what-would-you-say-was-the-reason-for-you-to-pursue-a-career-in-sciencetechnology">What would you say was the reason for you to pursue a career in science/technology?</h2>
My father. He is an engineer. He created 50 patents in night visions. Growing up, I wanted to be an archeologist. I was looking into the history and participated in archaeology session during summer. Then my father told me that archeologists are people who find something, and then they spend time looking at that stuff. You can build them devices and make their workflow much more efficient, and I just swallowed that idea and decided to go into physics instead.
<h2 id="is-there-one-piece-of-advice-you-wish-somebody-gave-you-at-the-beginning-of-your-career">Is there one piece of advice you wish somebody gave you at the beginning of your career?</h2>
Don’t skip programming classes. Focus on programming more because it will pay off. Start working hard as early as possible. Every minute you spent developing your brain will pay off in the future. That is definitely what I’d like to hear.
<h2 id="who-has-been-your-biggest-advocatementor-in-the-workplace-and-why">Who has been your biggest advocate/mentor in the workplace, and why?</h2>
So I was not always in tech. I started as a physicist and migrated to finance, and I’ve grown emotionally and matured quite a bit during that period, and I had great mentors in the finance field. Working with the bankers was a real good time for me because right now it’s paying off: I now developed intuition to know how to tackle open-ended questions.
Secondly, My husband is probably the biggest influence for me in tech. He started working at Google being one of the early engineers. Through his years he touches very successful companies at the beginning of the stages. He developed a very deep understanding of approaches, best practices, and focuses for me to grow in the early days that shaped the way that I focus. I benefited quite a bit from listening to him. When he starts talking, I have a pen and paper writing it down because it is something that he should be teaching/have starting training materials on.
<h2 id="is-there-anything-that-nextroll-does-to-support-women-in-technology-that-other-companies-should-do-too">Is there anything that NextRoll does to support women in technology that other companies should do too?</h2>
Historically, women are not frequently joining in engineering. Women think differently. Women pay attention to different things. What attracts women in positions needs to be pitched differently between men and women. Women are very cautious about which job opportunity she would pursue. If she does not fit in any of the job descriptions, she would not apply.
Also at the workplace, there should be some set of mindful places for women to be comfortable among a purely men’s culture. We need to develop certain islands to be comfortable to think and iterate. This is hard actually. I am a woman, I see women around. I am trying to understand how to formalize this. It’s not purely just diversity and inclusion. It is certain details that make a change.
Women in tech meetings here at NextRoll are really good. They are talking about formalizing what is good for the women, and how the group should act, and what is the beneficial way for women to express her idea.
<h2 id="who-are-your-role-models-in-women-in-tech">Who are your role models in women in Tech?</h2>
People who are forward-thinking are the most interesting and influential to me. I don’t have anybody among women that is my role model, but I have quite a few among men. Right now, I am quite fascinated with Elon Musk. He has an amazing ability to work, focus, extraordinary thinking, and also he is capable of encapsulating himself in his current life. I just read a book about his life, and it was amazing. I think we will say in the future that we lived in the Elon Musk era.
<h2 id="any-advice-for-women-in-techallies-you-would-give">Any advice for Women in Tech/Allies you would give?##</h2>
Througout my tech career, I don’t sense any difference between being a man or a woman. My class at the university was 130 people and only 8 women. At that point, I would prefer more women. But after that, I don’t think that I had any disadvantages during my life because I was a woman. I would say try to gather as many mentors as possible, no matter what the gender is. Because among men, you might get some interesting ideas that could enhance your ability. Try to catch attention to those that are most knowledgable, and have the most expertise in whatever field you want to pursue right now. Usually, those people don’t need you. You need them and spend time to make yourself interesting to them. It doesn’t matter who they are, but the right person and the right time would make tons of differences.
</description>
</item>
<item>
<title>Reflections on Remote Work at NextRoll</title>
<link>https://tech.nextroll.com/blog/culture/2020/03/17/remote-work.html</link>
<pubDate>Tue, 17 Mar 2020 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/culture/2020/03/17/remote-work</guid>
<description><h1 id="introduction">Introduction</h1>
As COVID-19 continues to spread and cause anxiety, many <a href="https://www.usnews.com/news/national-news/articles/2020-03-06/apple-employees-join-twitter-facebook-amazon-in-remote-work-as-coronavirus-spreads">tech companies have begun
encouraging or mandating that employees work from home</a>.
For the time being, NextRoll is among these companies. Fortunately, NextRoll engineering
has long had a remote-friendly work culture; I myself have been working remotely
for the last three years after two years at our San Francisco office. In this post,
I’ll share my thoughts on the benefits of remote work, and some tips from NextRoll’s
remote employees intended for anyone who has recently started working remotely.
<h1 id="benefits-of-remote-work">Benefits of Remote Work</h1>
From a personal perspective, remote work has greatly increased both my quality of life
and my productivity. When I worked in San Francisco, I had almost an hour-long commute
to work each way. Now, I have an extra two hours of every weekday that I can spend working,
relaxing at home, or playing basketball. The extra time I am able to allocate to my personal
life allows me to be less stressed and better rested, which makes me more focused and
productive when it’s time to work. In other words, one of the major efficiencies of remote
work is that remote employees have more time for both their personal and professional lives.
From a broader perspective, the option of remote work has helped enable NextRoll to hire
and retain a large number of very talented engineers. Some of our brightest, most impactful,
and most experienced engineers work remotely. In some of their cases, the option to work
remotely was the key differentiator between a NextRoll job offer, and offers from other
companies with more rigid work policies. In other cases, like my own, the option to work
remotely allowed NextRoll to retain engineers who would otherwise have had to leave after
deciding to move. In these ways, remote work can be a huge win for companies willing to
accommodate flexible work arrangements.
<h1 id="tips-for-remote-work">Tips for Remote Work</h1>
If you find yourself working remotely, and are looking for some tips on how to make it work,
this section is for you. These tips and suggestions come from several of my fellow remote
NextRollers.
<h2 id="logistical-tips">Logistical Tips</h2>
For your remote working arrangement to succeed, the first step is setting up your home
workstation. Of course, a reliable internet connection (with VPN) is a must. Beyond that,
messaging services like Slack and teleconferencing solutions like Google Meet help bridge
the physical gap between the office and your home office. Be careful though, these tools
can quickly become distractions as Matt Burch, one of our Software Engineering Team Leads,
points out. He recommends disabling notifications at times to give yourself an opportunity
to focus on the task at hand.
It is very important to give your workstation its own, dedicated space, away from wherever
you unwind. This helps keep your work from bleeding into your personal life, and vice versa.
It pays dividends to invest in your workstation, especially for long-term remote working
situations. I learned this the hard way in my first year working remotely after developing
pain in my elbow from bad ergonomics. For this reason, I recommended a dedicated desk with
an adjustable chair, and at least one external monitor. A wireless keyboard and mouse are also
well worth the investment. Several of my colleagues, including Senior Software Engineer Jose
Hernandez, suggest purchasing a nice pair of speakers as well; instrumental background music
seems to be popular among the remote folks at NextRoll.
As far as office attire, there are differing schools of thought. For some people, like Senior
Software Engineer Danny Fowler and Senior Staff Engineer Brian Ecker, it’s best to keep a
consistent schedule as if you were at the office. In particular, this means getting dressed
as if you were going into the office. Others, such as Staff Engineer Tyler Brown, believe
in embracing the flexibility that remote work offers by working in sweatpants.
Another logistical aspect unique to remote work is the need to over-communicate with teammates.
Effective communication takes a more conscious effort when you are remote. This communication
is particularly challenging when teammates are spread across the world, in multiple timezones.
In situations like this, Staff engineer Brujo Benavides recommends carefully thinking through
questions you may have for people in other timezones, and being patient because their responses
may take a few hours. The good news is sometimes this extra thought can help you answer your
own question.
<h2 id="tips-for-well-being">Tips For Well-Being</h2>
As important as having the right work setup is having the right mentality and approach.
When working remotely, it is very easy to feel isolated or overworked. Technical Talent
Partner Haley Scruggs suggests scheduling meetings with teammates for the sole purpose
of catching up personally. Senior Software Engineer Corey Shott says taking walks, and
breaks away from the computer are particularly important when working remotely. For example,
it’s best to eat meals away from the computer.
Remote work provides a lot of flexibility in terms of when you work, which is both a perk
and a challenge. On one hand, it’s easy to start a load of laundry during the workday, and
on the other hand, it’s easy to send just one more slack message after dinner. For this
reason, setting boundaries is particularly important when you work from home. Security
Engineer Nicolas Valcárcel suggests separating home and work entirely. To make this work
in practice, Senior Staff Engineer Mike Watters recommends developing routines that start
and end the day to enforce a mental separation between home and work.
I’ve found a workout to end the day is particularly effective for me in this regard. On that
note, regular exercise is extremely important, as Director of Software Engineering Paul Huff
told me, because it’s so easy to become sedentary when working remotely. Not only does
exercise make you physically healthier, it will make you feel better mentally too.
<h1 id="conclusion">Conclusion</h1>
Remote work creates unique challenges. If you are working remotely for the first time, it takes
some getting used to both in terms of logistics, and in terms of your approach to work. For
example, you may have to create a dedicated space for your workstation or find a way to set
boundaries between your home and work.
Nonetheless, at NextRoll we’ve found that these challenges are outweighed by the significant
benefits of remote work. Flexible work arrangements give employees better quality of life
while often increasing, rather than sacrificing, productivity, and give employers an advantage
in recruiting and retaining talent. If working at a remote-friendly company sounds appealing
to you, please take a look at our <a href="https://www.nextroll.com/careers">careers page</a>.
</description>
</item>
<item>
<title>
Women in Tech Spotlight: Iva Ivanova
</title>
<link>https://tech.nextroll.com/blog/tech,%20diversity,%20inclusion/2020/03/03/women-in-tech-spotlight.html</link>
<pubDate>Tue, 03 Mar 2020 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/tech,%20diversity,%20inclusion/2020/03/03/women-in-tech-spotlight</guid>
<description>Women in Tech - Spotlight: Iva Ivanova
<code class="language-plaintext highlighter-rouge">5 minute read</code>
<hr />
March is Women’s History Month and in honor of this occasion, NextRoll is launching a month-long Women in Tech Spotlight – where we highlight a few of the women here at NextRoll. This will be a limited three-part series of interviews, showcasing three of our extraordinary women in technology. This week, I had the pleasure of interviewing Iva Ivanova!
<h2 id="iva-ivanova">Iva Ivanova</h2>
<h3 id="title-data-scientist-ii">Title: Data Scientist II</h3>
<img src="/images/post_images/ivanova.jpg" alt="open_sourced_projects" />
<h2 id="can-you-share-a-little-about-what-you-do-at-nextroll-and-what-your-typical-day-is-like">Can you share a little about what you do at NextRoll and what your typical day is like?</h2>
I am a Data Scientist on the Analytics team. My day starts with a strong cup of tea. I generally spend half my day meeting and connecting with product managers and engineers in person or over Slack where we brainstorm about experimentation design, feature launches, data infrastructure, reporting, forecasting and more. The other half of my day is usually spent executing on those ideas.
<h2 id="what-do-you-like-most-about-your-tech-career">What do you like most about your tech career?</h2>
I like problem solving with technology. I also like the fact that I am always learning something new. If you like troubleshooting, learning, and being challenged, this is the career for you.
<h2 id="what-would-you-say-was-the-reason-for-you-to-pursue-a-career-in-technology">What would you say was the reason for you to pursue a career in technology?</h2>
I was working as a Sales Finance Analyst when a friend floated the idea of doing a 3-month data science bootcamp; I signed up and was instantly hooked. As an Analyst I was already pretty comfortable writing advanced SQL queries but had to learn Python in a matter of weeks as it was a prereq for the bootcamp. The bootcamp was my initial exposure to machine learning.
I was fascinated by the sheer power that Python and machine learning put at my fingertips.
Using Python, I was suddenly able to perform tasks that used to take me hours in Excel in a matter of seconds with one line of code.
About a year later, I applied and was accepted into a part-time M.S. in Business Analytics program while keeping my full-time role at NexRoll.
<h2 id="what-is-it-like-to-be-a-woman-working-in-tech-for-you">What is it like to be a woman working in tech for you?</h2>
While it may still feel like an uphill battle as women are still a minority in the field, there are thankfully folks that are working to make things better by mentoring and advocating for women in engineering. And those are the folks that inspire me and keep me going!
<h2 id="is-there-one-piece-of-advice-you-wish-somebody-gave-you-at-the-beginning-of-your-career">Is there one piece of advice you wish somebody gave you at the beginning of your career?</h2>
Buckle up, you are in for quite the ride! Haha. Just kidding, sort of. I would tell myself to be patient, enjoy the journey, and don’t stress over what others are doing. Some of us are meant to take a short way to fulfill our dream, others the long way and what matters is not how long it takes but that you are enjoying the process and find what you do to be fulfilling.
<h2 id="who-has-been-your-biggest-advocatementor-in-the-workplace-and-why">Who has been your biggest advocate/mentor in the workplace and why?</h2>
I have been fortunate to have several great mentors over the years that have helped me navigate the reality of being a woman in tech. I had one manager early on in my career that always steered me toward technical projects even though I was not in a technical role at the time because they recognized I was good at teaching myself technical skills.
I think it is important to find good mentors early on in our careers because they are often able to see potential in us long before we are able to realize it and steer us in the right direction.
<h2 id="is-there-anything-that-nextroll-does-to-support-women-in-technology-that-other-companies-should-do-too">Is there anything that NextRoll does to support women in technology that other companies should do too?</h2>
I can’t speak much about other companies but NextRoll allowed me to transition from a career in Sales Finance to Analytics and supported my decision to complete a master’s degree while working full-time. Companies like NextRoll that truly strive to not just hire great people but help them grow are at an advantage when it comes to the talent war. That’s why the Owl is my favorite culture creature. That is the kind of support system I have found here.
<h2 id="who-are-your-role-models-in-women-in-tech">Who are your role models in women in Tech?</h2>
Marie Curie - she was the first woman to win the Nobel Prize not once but twice in two separate sciences - Physics and Chemistry. How awesome is that?
<h2 id="any-advice-for-women-in-techallies-you-would-give">Any advice for Women in Tech/Allies you would give?##</h2>
Women in Tech:
For those early on in your career, have a growth mindset and keep your eyes on the prize. You are going to fail and it is ok, just don’t let that discourage you along the way. But keep in mind that you have to put in the work as well.
Allies:
For those further ahead looking to mentor and advocate for women in tech, start a conversation, welcome questions, listen, and offer advice when asked. A small effort on your part might bring a lot of value to someone’s life.
</description>
</item>
<item>
<title>
The new code formatter for Erlang: rebar3 format
</title>
<link>https://tech.nextroll.com/blog/dev/2020/02/25/erlang-rebar3-format.html</link>
<pubDate>Tue, 25 Feb 2020 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2020/02/25/erlang-rebar3-format</guid>
<description>In recent years, many language ecosystems have developed automatic code formatters to reduce the mental overhead of code readers and therefore to share code more easily. These tools work by ensuring that all code written in the same language looks the same. Some examples of these tools include <a href="https://golang.org/cmd/gofmt/">gofmt</a> for Go or <a href="https://hexdocs.pm/mix/master/Mix.Tasks.Format.html">mix format</a> for Elixir. The Erlang community was lacking a tool like this, so we created a rebar3 plugin just to automatically format code.
In this article we’ll discuss the history of the Erlang parsing and formatting tools, the challenges of developing a formatter and the resulting tool that we created. Learn how you can use it and customize it to your needs.
<code class="language-plaintext highlighter-rouge">10-15 minute read</code>
<hr />
<center>
<img alt="Bugs Bunny - The Rabbit of Seville." src="/images/post_images/rabbitofseville.jpg" /> 
Bugs Bunny - The Rabbit of Seville.
</center>
<h1 id="introduction">Introduction</h1>
NextRoll’s RTB Team devotes quite a bit of our efforts to make our codebase as mature and maintainable as we can.
For our Erlang code, we started working on this task years ago. We trim dead code using <a href="/blog/dev/2018/10/09/remove-erlang-dead-code-xref.html"><code class="language-plaintext highlighter-rouge">xref</code></a>. We remove discrepancies with the help of <a href="/blog/dev/2019/02/19/erlang-dialyzer.html"><code class="language-plaintext highlighter-rouge">dialyzer</code></a>. We make sure our code is well behaved using <a href="https://github.com/okeuday/pest"><code class="language-plaintext highlighter-rouge">PEST</code></a>. We let <a href="https://hex.pm/packages/elvis">Elvis</a> find stylistic anomalies…
But there was one tool that was missing: a code-formatter. We were using a code formatter for Go, Elixir, and Python. But there weren’t any (or barely any) for Erlang. So, we decided to use one of our <a href="/blog/culture/2019/11/26/hackweek-at-nextroll.html">HackWeeks</a> to create one.
<h2 id="a-bit-of-history">A Bit of History</h2>
<h3 id="other-code-formatters">Other Code Formatters</h3>
Code formatting is certainly nothing new, it’s been around for ages with several very interesting papers written about it. But recently, there’s been a tendency in all modern languages to include one (and only one) formatter, like <a href="https://golang.org/cmd/gofmt/"><code class="language-plaintext highlighter-rouge">gofmt</code></a> and, of course, the one that influenced us the most: <a href="https://hexdocs.pm/mix/master/Mix.Tasks.Format.html"><code class="language-plaintext highlighter-rouge">mix format</code></a>.
We’re by no means experts in this area and therefore we wanted to rely on existing efforts as much as we could. We took inspiration from all of them, but we tried to use as many already written components as possible, and (as usual) there are a bunch of them already baked into OTP…
<h3 id="parsing-and-formatting-erlang-code">Parsing and Formatting Erlang Code</h3>
Our first initiative was to try to find existing tools to format Erlang code.
We found a ready-to-use solution: <a href="https://hex.pm/packages/rebar3_fmt"><code class="language-plaintext highlighter-rouge">rebar3 fmt</code></a>. The problem is that, as it clearly states in its description, it requires emacs, which is something most of us don’t use. But it pointed us to what is generally recognized as the de facto standard for Erlang formatters: <a href="https://erlang.org/doc/apps/tools/erlang_mode_chapter.html">erlang-mode for Emacs</a>. That’s what the OTP Team considers the standard way of formatting Erlang code and what the other tools included in OTP are loosely based on.
What are these tools you say? That was our follow up question as well! And these are the ones we found:
<ul>
<li><a href="http://erlang.org/doc/man/erl_tidy.html"><code class="language-plaintext highlighter-rouge">erl_tidy</code></a>: The closest thing to an automatic code formatter for Erlang. It uses many of the other modules in this list to parse and rewrite Erlang code. It’s a bit old and it has a bunch of well-known deficiencies, including but not limited to its lack of proper support for macros and comments in code.</li>
<li><a href="http://erlang.org/doc/man/erl_prettypr.html"><code class="language-plaintext highlighter-rouge">erl_prettypr</code></a>: This is a pretty printer – it takes an AST (Abstract Syntax Tree) as an input and it prints it out in a pretty way. It’s the standard Erlang pretty printer. Our original intention was to use it, but its extensibility support is complex and poorly documented at best. So, like many others before us (e.g. <code class="language-plaintext highlighter-rouge">wrangler</code>, <code class="language-plaintext highlighter-rouge">erlang-ls</code>, etc.), we just copied it into our project and started from there.</li>
<li><a href="http://erlang.org/doc/man/epp.html"><code class="language-plaintext highlighter-rouge">epp</code></a>: The other side of the story. This is the Erlang parser that comes with OTP. It also has some limitations, most importantly when it comes to macros and comments, since it’s intended to be used primarily by the compiler.</li>
<li><a href="http://erlang.org/doc/man/epp_dodger.html"><code class="language-plaintext highlighter-rouge">epp_dodger</code></a>: A module explicitly created to work like <code class="language-plaintext highlighter-rouge">epp</code> but bypassing macros and preprocessor directives. It’s also a bit buggy and it has limited support for extensibility. Luckily, <a href="http://twitter.com/jfacorro">Juan Facorro</a> already copied it and improved it in <a href="https://hex.pm/packages/katana_code"><code class="language-plaintext highlighter-rouge">katana-code</code></a>.</li>
</ul>
So, what we ended up using is <code class="language-plaintext highlighter-rouge">ktn_dodger</code> (from <code class="language-plaintext highlighter-rouge">katana-code</code>) to parse the code and turn it into an AST and then our version(s) of <code class="language-plaintext highlighter-rouge">erl_prettypr</code> (more on this below) to output the formatted code.
<h3 id="choosing-the-right-format">Choosing the Right Format</h3>
Once we had the tools, we developed our very first version of the formatter (you can still find it on hex.pm as <code class="language-plaintext highlighter-rouge">0.0.1</code>). That version simply passed the provided Erlang code through the tools and generated the code formatted with <code class="language-plaintext highlighter-rouge">erl_prettypr</code>.
For a while, we considered that to be the canonical formatting since, you know, it comes with OTP, right? But we (and many others like us) didn’t want our code formatted strictly as <code class="language-plaintext highlighter-rouge">erl_prettypr</code> outputs it.
So, we started adding configuration options to be able to adjust the formatter to our tastes. After a while, though, we realized that what we wanted was not an extension of <code class="language-plaintext highlighter-rouge">erl_prettypr</code>, it was a different formatter. That’s when we moved from one formatter with lots of options, to a formatting behavior with multiple implementations.
Our favorite way of formatting Erlang code is now encoded in the <code class="language-plaintext highlighter-rouge">default_formatter</code>, but we kept the OTP-approved way alive as the <code class="language-plaintext highlighter-rouge">otp_formatter</code>. And now you can define your own, too.
<h3 id="were-not-alone">We’re not Alone</h3>
While doing our research, we found that we’re not the only ones who saw the need for an Erlang formatter. As a matter of fact, several people are working on different code formatting tools for Erlang these days:
<ul>
<li>As we mentioned before, if you’re an emacs user you already have a <code class="language-plaintext highlighter-rouge">rebar3</code> plugin that you can use: <a href="https://hex.pm/packages/rebar3_fmt"><code class="language-plaintext highlighter-rouge">rebar3 fmt</code></a>.</li>
<li>A while back, <a href="http://zalastax.github.io/">Pierre Krafft</a> tried to just improve <code class="language-plaintext highlighter-rouge">erl_tidy</code> and wrote <a href="https://github.com/erlang/otp/pull/2451">a PR</a> for that.</li>
<li>In that thread, we learned that <a href="https://michal.muskala.eu/">Michał Muskała</a> was also working on a formatter, probably called <code class="language-plaintext highlighter-rouge">erlfmt</code>.</li>
<li>Later on, we found <a href="https://elixirforum.com/t/steamroller-an-opinionated-erlang-code-formatter/27532"><code class="language-plaintext highlighter-rouge">steamroller</code></a>, by <a href="https://oldreliable.tech/">Daniel Tipping</a>.</li>
</ul>
Both <code class="language-plaintext highlighter-rouge">steamroller</code> and <code class="language-plaintext highlighter-rouge">erlfmt</code> are much more opinionated than our formatter, mostly because their authors aim at having consistent formatting across all Erlang codebases in the world, much like the goal of <a href="https://hexdocs.pm/mix/master/Mix.Tasks.Format.html"><code class="language-plaintext highlighter-rouge">mix format</code></a>.
We see that as a great goal, but we have a fairly smaller one: What we want is consistent formatting within all Erlang codebases in the world. In other words: We want all modules in each project to be consistently formatted, even if they don’t share the same formatting rules with other projects.
After all, at this point it’s hard to even define the canonical formatting for all Erlang code in the wild. And if we achieve our goal and some developers format their code using <code class="language-plaintext highlighter-rouge">rebar3 format</code> and you (another developer) can’t read their code because it is using a different formatter than the one you’re used to (e.g. they use a comma-first ROK-style formatter), all you need to do is switch an option in the <code class="language-plaintext highlighter-rouge">rebar.config</code>, run <code class="language-plaintext highlighter-rouge">rebar3 format</code> and magically… that ugly code now looks your way.
But enough with the history, let’s see what you can do with this tool.
<hr />
<h1 id="how-to-use-rebar3-format">How to Use <code class="language-plaintext highlighter-rouge">rebar3 format</code></h1>
<h2 id="quick-start">Quick Start</h2>
Just add this to your <code class="language-plaintext highlighter-rouge">rebar.config</code> (either in your project or globally in <code class="language-plaintext highlighter-rouge">~/.config/rebar3/rebar.config</code>):
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{plugins, [rebar3_format]}.</code></pre></figure>
Then run
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rebar3 format</code></pre></figure>
and enjoy.
<h2 id="configuration">Configuration</h2>
If you don’t really like the default formatting as-is, <code class="language-plaintext highlighter-rouge">rebar3 format</code> can be configured using the <code class="language-plaintext highlighter-rouge">format</code> section of the <code class="language-plaintext highlighter-rouge">rebar.config</code>. There are three main options you can specify.
<h3 id="what-to-format">What to Format</h3>
To determine what files the formatter should format, you use the <code class="language-plaintext highlighter-rouge">files</code> parameter:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{format, [{files, ["src/*.erl", "test/*.erl"]}]}.</code></pre></figure>
<h3 id="what-formatter-to-use">What Formatter to Use</h3>
Unless you specify otherwise, <code class="language-plaintext highlighter-rouge">rebar3 format</code> will always use the default formatter that’s baked into it. But if you want you can use the <code class="language-plaintext highlighter-rouge">otp_formatter</code> or your one, like this:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{format, [
{files, ["src/*.erl", "test/*.erl"]},
{formatter, otp_formatter}
]}.</code></pre></figure>
<h3 id="how-to-configure-the-formatter">How to Configure the Formatter</h3>
Finally, you can also set up individual options for the formatter you want to use. For instance, for the <code class="language-plaintext highlighter-rouge">otp_formatter</code> you can change <code class="language-plaintext highlighter-rouge">paper</code> (i.e. the expected max width of the formatted code):
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{format, [
{files, ["src/*.erl", "test/*.erl"]},
{formatter, otp_formatter},
{options, #{paper =&gt; 150}}
]}.</code></pre></figure>
To find out the options for each provider, check out the docs that are available <a href="https://github.com/AdRoll/rebar3_format/#configuration">online</a>.
<h2 id="a-proposed-workflow">A Proposed Workflow</h2>
Drawing from the Smalltalk formatter experience that I had (where the code was formatted only when presenting it to the developer but not when stored in the image itself), I want to propose a workflow for teams where each member has its preferred style for code formatting.
The idea is to take advantage of <code class="language-plaintext highlighter-rouge">rebar3</code> profiles and write the following on your <code class="language-plaintext highlighter-rouge">rebar.config</code> file:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">%% The canonical format used when pushing code to the central repository
{format, [
{files, ["src/*.erl", "include/*.hrl", "test/*.erl"]},
{formatter, default_formatter},
{options, #{paper =&gt; 100}}
]}.
{profiles, [
{brujo, [
{format, [
{files, ["src/*.erl", "include/*.hrl", "test/*.erl"]},
{formatter, rok_formatter}, % I prefer comma-first formatting
 {options, #{paper =&gt; 100}}
]}
]},
{miriam, [
{format, [
{files, ["src/*.erl", "include/*.hrl", "test/*.erl"]},
{formatter, default_formatter},
{options, #{
inline_clause_bodies =&gt; false, % she doesn't like one-liners
 inline_items =&gt; all % but she doesn't like long lists of items
 }}
]}
]}
]}</code></pre></figure>
Then whenever you’re about to work on something, follow this ritual:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">git checkout master
git checkout -b my-branch
rebar3 as brujo format
# I work on my code normally
# Run tests and what-not
# Until I'm ready to commit
rebar3 format
git commit -am "Apply my changes"
git push origin my-branch --set-upstream</code></pre></figure>
Miriam does the same but using <code class="language-plaintext highlighter-rouge">as miriam</code> instead of <code class="language-plaintext highlighter-rouge">as brujo</code>.
That way each one of us can read code in the way we understand it better, write code exactly how we like to write it, etc. Then publish it in a consistent way that matches the style of the rest of the project.
<h1 id="examples">Examples</h1>
If you want to see what the formatter can do to your code, the best place to go is the sample project on <a href="https://github.com/AdRoll/rebar3_format/tree/master/test_app">the repo itself</a>. All sorts of examples are there with as many edge cases as we could create and find. If you know of others, please contribute by adding them there or writing issues so we can add them.
Even though we’re still in the process of testing and improving our tool, we already started using it for several of our repositories. You can see the formatter in action in <a href="https://github.com/AdRoll/spillway/pull/4">spillway</a> and <a href="https://github.com/AdRoll/mero/pull/54">mero</a>.
As the best code reviewers out there may notice, those PRs required some manual adjustments after processing the code with the formatter. We consider that a feature: That’s the formatter allowing us to spot very ugly pieces of code (e.g., too deeply nested structures) that we should refactor.
To be clear: the formatter didn’t break our code (it has built-in verification for that), it just made it look extremely ugly, therefore prompting us to beautify it.
<center>
<img title="Sorry, it's just too good!" src="/images/post_images/rabbit-hair.gif" /> 
Bugs Bunny - The Rabbit of Seville.
</center>
<hr />
<h1 id="what-now">What Now?</h1>
We’re releasing the formatter as early as we can to catch as many bugs and nuances as possible. Please try it in your code and report any bugs and new ideas you have <a href="https://github.com/AdRoll/rebar3_format/issues">here</a>.
We plan to keep using this ourselves, but if anybody feels like making this tool official (if you’re a member of the OTP Team, this message is 100% for you 🙄), that would be amazing.
<hr />
<hr />
<h1 id="appendix-a-beautiful-code">Appendix A: Beautiful Code</h1>
As a bonus track, I wanted to know how does <a href="https://medium.com/erlang-battleground/beautiful-code-254a5f8ef958?source=friends_link&amp;sk=a7c610b2e13ed4aa72797832cd953ec6">my favorite piece of code</a> look like when formatted by <code class="language-plaintext highlighter-rouge">rebar3 format</code>. Let’s see…
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">-module(in).
-author(john).
-author(paul).
-author(george).
-author(ringo).
-export([my_life/1]).
my_life(NewPlaces) -&gt;
Places = db:get_all(places),
UpdatedPlaces = [NewPlace
|| NewPlace &lt;- NewPlaces,
lists:member(NewPlace, Places)],
lists:foreach(fun (Place) -&gt;
db:insert(places, Place)
end,
UpdatedPlaces),
DeletedPlaces = [Place
|| Place &lt;- Places,
not lists:member(Place, NewPlaces)],
db:delete(places, DeletedPlaces),
Moments = [Moment
|| Place &lt;- Places,
Moment &lt;- places:moments(Place)],
People = [Person
|| Moment &lt;- Moments,
Person
&lt;- moments:lovers(Moment) ++
moments:friends(Moment)],
{Dead, Living} = lists:partition(fun person:is_dead/1,
People),
lists:foreach(fun person:love/1, Dead ++ Living),
You = db:get_first(people),
[] = [Person
|| Person &lt;- People,
person:comparable(Person, You)],
ok = love:update(),
UpdatedMemories = [moments:meaning(Moment, null)
|| Moment &lt;- Moments],
db:update(moments, UpdatedMemories),
my_life(You, People, UpdatedMemories).
my_life(You, People, Things) -&gt;
case rand:uniform(5) of
1 -&gt;
timer:sleep(rand:uniform(100) + 100),
person:think_about(People);
2 -&gt;
timer:sleep(rand:uniform(100) + 100),
moments:think_about(Things);
_ -&gt;
dont_stop_now
end,
person:love(You),
my_life(You, People, Things).</code></pre></figure>
Not bad, huh?
<center>
<img title="Last time, I promise" src="/images/post_images/rabbit-eyes.gif" /> 
Bugs Bunny - The Rabbit of Seville.
</center>
</description>
</item>
<item>
<title>
How NextRoll encourages Engineering and Friends to take a week off to build amazing stuff!
</title>
<link>https://tech.nextroll.com/blog/culture/2019/11/26/hackweek-at-nextroll.html</link>
<pubDate>Tue, 26 Nov 2019 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/culture/2019/11/26/hackweek-at-nextroll</guid>
<description>NextRoll Hack Week
<code class="language-plaintext highlighter-rouge">5 minute read</code>
<hr />
Hello! Alex here from NextRoll!
Don’t you ever wish there was a time reserved to build any idea you had in mind? Maybe it’s some technical debt that you’re hoping to clean up, some new language you’re hoping to play with, or just collaborating with other engineers in the organization to build something fun.
Well…NextRoll has the solution for that…and it is HACK WEEK! Yes, we get a full week of hacking! Twice a year!!
<h1 id="what-is-hack-week">What is Hack Week?!</h1>
Hack Week is a twice-yearly engineering event where engineers and friends can work on a project of their own choice.
There are events (like a kickoff, a wrap-up with demos, and a fabulous dinner for the winners) and some rules, but outside of that, what you work on is up to you!
We encourage anyone to join Hack Week! People from Marketing, Operations, Account Management, and more have all participated in the past. Projects have addressed pain points, technical debt house-cleaning, and MVPs of new ideas. In the past, we’ve seen folks make a video game emulator, an MVP for the next NextRoll product - or people just take the opportunity to play with new frameworks, languages, ML models, and more!
The event is also quite a logistical challenge! Coordinating Hack Week between our local Engineering group in San Francisco, remote engineers, and the Product organization takes multiple coordinators. Fortunately, we have a team of Hack Week coordinators – with Jose Hernandez coordinating for Remote Engineering, Zachary Swetz coordinating from the product perspective, and our advisor Jessica Grist to help us ensure everything is done smoothly.
<h1 id="what-do-i-get-from-participating-in-hack-week">What do I get from participating in Hack Week?</h1>
When folks join Hack Week, they not only get the excitement of working with new people for a short amount of time and the ability to work on something they have wanted to work on, but they also get SWAG! Yes! We give out cool swag shirts like these in the summer:
<img src="/images/post_heroes/hackshirt.png" alt="Hack shirt!" />
And for the winter we give out cool holiday swag like these:
<img src="/images/post_images/hackweek_mugs.png" alt="hackswag" />
Other swag we have given in the past includes winter socks, finger-less gloves, beer stein, etc.
<h1 id="what-if-i-dont-know-what-to-build">What if I don’t know what to build?</h1>
Need ideas to figure out what to build? We have solutions for that as well! Introducing Bounty Awards and Brainstorming session!
<h2 id="bounty-awards">Bounty Awards</h2>
Different organizations at NextRoll have different needs. They have their pain points, and also their own ideas of what they would like to see built during hack week. To encourage engineers and friends to work on the project, organizations can set a bounty price. They offer prices given the project that is built to solve their primary pain point.
For instance, Marketing, inspired by our <a href="https://www.nextroll.com/careers/culture">culture creature</a>, sponsored the “Most Customer-Centric Award” to encourage projects to directly benefit our customers.
<img src="/images/post_images/dog.png" alt="dog" />
Account Management sponsored the “Do More With Less Award” – also inspired by our culture creature Beaver.
<img src="/images/post_images/beaver.png" alt="beaver" />
These awards help provide ideas for what engineering and friends would like to build for Hack Week. Again, the decision to choose which project to build is really up to you!
<h2 id="brainstorming-jam-sesh">Brainstorming Jam Sesh!</h2>
A month before each event, we host two lunch meetings for Hack Week brainstorming – where people can come in and share ideas that they would like to build for Hack Week. The key here is freely let the ideas flow, so people can hear different proposals and build on top of one another. To allow more opportunites for folks to speak their mind, we split the local and remote engineers into smaller groups for brainstorming.
After the groups have spent 20 minutes in the discussion, we then regroup for the last 10 min to share as a group. The goal is to get to hear as many ideas as we can.
This is where crazy ideas get shared. For example:
<ol>
<li>Food recommendation around the Mission District (where our office is at)</li>
<li>Bathroom playlist recommendation.</li>
<li>Free up our meeting rooms with some Google Calendar/Slack integration.</li>
</ol>
Of course, not all of these ideas get to be implemented, but it’s impressively cool to see how creative people can get when they’re given the freedom to think outside the box.
<h2 id="hack-week-begins">Hack Week Begins!</h2>
<h3 id="kick-start">Kick Start</h3>
Kick start is where people share the idea they would like to work on. It usually is a 1-minute description of what the project is, why it’s interesting, and a call for members to join the team’s Slack channel to participate in hacking.
People usually have a 1 slider similar to this:
<img src="/images/post_images/batchie-stats.png" alt="slide" />
Again, it’s more about gaining interest for more people to participate in the project.
<h3 id="readysethack">Ready…set…HACK!</h3>
During hacking, we ask each team to create a Slack channel for their project, and we also ask hackers to make a daily stand up to be accountable for the project they work on.
<h3 id="hack-demos-and-party">Hack Demos and Party!</h3>
<img src="/images/post_images/hw_jump.gif" alt="Alt Text" />
Demo is where people showcase their creations! This is a time where we celebrate all the projects that are built. Everyone gets to share the stuff they have been working on. Along with the presentation, we also have wine, beer, and finger food to celebrate! We also have judges representing Product, Engineering, Marketing, and Remote to evaluate the winners.
Cool Projects that came out along the way that we can share (since they’re open sourced!)…
<img src="/images/post_images/small_collage_hw.png" alt="open_sourced_projects" />
<h4 id="erlang-formatter">Erlang Formatter</h4>
<blockquote>
A Rebar3 plugin for code formatting! (<a href="https://github.com/AdRoll/rebar3_format">repo here</a>)
</blockquote>
<h4 id="batchie-patchie">Batchie Patchie</h4>
<blockquote>
A service built on top of AWS batch for jobs monitoring! (<a href="https://github.com/AdRoll/batchiepatchie">code here</a> / Previous Blog about batchie patchie <a href="http://tech.nextroll.com/blog/data/2018/08/08/running-jobs-with-aws-batch.html">here</a>)
</blockquote>
<h4 id="traildb">TrailDB</h4>
<blockquote>
An efficient way to store event based data (<a href="https://github.com/traildb/traildb">code here</a>)
</blockquote>
<h4 id="python-hyperloglog">Python HyperLogLog</h4>
<blockquote>
Python implementation of hyperloglog (<a href="https://github.com/AdRoll/python-hll">code here</a>)
</blockquote>
<h3 id="winners">Winners</h3>
In the end, everyone is a winner! Everyone had fun building something awesome, and we all had wine and cheese and swag to celebrate it. To put an icing on top of the cake, we have our Hack Week judges get together to decide the winners. Prizes are given to the most product-driven project, the most technical-driven project, and the bounty prices!
<h3 id="wrap-up">Wrap up!</h3>
And that’s how we spend our time two weeks per year! We hope it provides a glimpse of the engineering life that is happening around in NextRoll!
</description>
</item>
<item>
<title>
How NextRoll leverages AWS Batch for daily business operations
</title>
<link>https://tech.nextroll.com/blog/dev/2019/11/19/aws-batch-at-nextroll.html</link>
<pubDate>Tue, 19 Nov 2019 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2019/11/19/aws-batch-at-nextroll</guid>
<description>Describing how NextRoll is handling processing petabytes of data using AWS Batch.
<code class="language-plaintext highlighter-rouge">15 minute read</code>
<hr />
In this blogpost, we are going to review how NextRoll is using AWS Batch for processing data, and how we benefit from this platform. We will start with an overview of the company and explain our needs for data pipelines. Then we’ll talk about how we are utilizing Batch, reviewing the advantages and disadvantages of this technology. Finally, we will show you Batchiepatchie, the product that we have introduced to overcome the challenges presented by Batch.
<h1 id="about-nextroll">About NextRoll</h1>
NextRoll is an active company in Digital Marketing and Advertisement. Currently, we are serving more than <a href="https://www.adroll.com/why-adroll-digital-marketing-platform">37,000 brands</a>. These brands from all over the world have trusted us to run marketing campaigns for them.
Our IntentMap features 1.2 billion shopper profiles and trillions of intent data points. Let me emphasize: It’s 1.2 billion profiles for different shoppers across the web. The number of profiles is growing every day.
To keep up with the demand, our machine learning engine is doing 80,000,000,000 predictions every single day. Most machine learning systems would not do billions of predictions in their lifetime, but our system needs to scale up quickly to keep up with the demands, and that number above is not for busy days like Christmas or thanksgiving.
To serve our customers, we are connected to 500 different supply sources across the internet. The sources provide a range of ways to reach customers through banner ads, social media, email, and onsite personalization.
We have three different business units. Each business unit is targetting a different type of customers and
fulfilling different needs.
<h1 id="why-batch">Why Batch?</h1>
Given the numbers provided above, we can see that the data that is being generated by digital advertisement is a lot more than anyone can fit in their laptop. Unless you have a massive laptop with terabytes of disk space. That is the first reason that pushed us to look for cloud solutions to orchestrate our tasks.
Over 2019 alone, Batch and Spot market usage has saved us $4,000,000. That saving comes from comparing our model to the usage of EC2 on-demand instances to run our jobs. The saving would be even bigger if you consider the other alternative that is having the servers on-premise. Saving would be less if you compare Batch with EMR or other big data processing that also uses Spot markets, however, the cost would be still less.
Besides cost saving, we have other reasons to use Batch. The freedom of stack is one of them. As long as you can package your requirements into a Dockerfile, and build that docker image, you can deploy it to Batch. This enables us to use a wide variety of technologies in our stack. We have jobs that are written using C/C++, Python, Rust, GoLang, Haskell, Java and other programming languages.
Freedom in the way that data is being processed is another reason. To process this amount of data, we have
introduced some file formats, one of them is opensource: <a href="https://github.com/traildb/traildb">TrailDB</a>. These
file formats do need special handling for creation and processing that we can handle using Batch. For comparison, Hadoop or Spark do enforce certain types and inputs.
The last item is deployment. The packaged code lives in a docker image, and deployment is as easy as pulling the image from Elastic Container Registry. Besides, since the image is being pulled from ECR to Batch nodes, the whole process ends up being pretty fast.
In a nutshell, these are our reasons to use Batch:
<ol>
<li>Cost Saving</li>
<li>Scale of data</li>
<li>Freedom of stack</li>
<li>Custom data processing formats</li>
<li>Ease of deployment</li>
</ol>
<h2 id="growth-of-batch-usage">Growth of Batch usage</h2>
Since batch was introduced in 2017, we have been rapidly adapting and using it for more and more reasons.
In the table below, you can see a comparison of Batch usage between 2017 and 2019:
<img src="/images/post_images/batch_growth.png" alt="Batch Growth" />
This shows how we grew about 10 times on each metric in only two years. How did this happen? First, we need to account for the growth of NextRoll during this time, but also those stats reveal a clear trend in the adoption of this technology across the company. Especially since teams are autonomous and they are free to use any tech they prefer to accomplish their goals, using Batch points how promising the results were. Additionally, it tells how batch can scale to our needs when we need to.
In the chart below, you can see the number of jobs launched on different dates.
<img src="/images/post_images/batch_date_count.png" alt="Jobs Count Growth" />
We are constantly improving our pipelines, removing the jobs that are not needed with recent refactors or improving the job run times, and that is why you sometimes see a decrease in the number of jobs. But generally, the trend is upward.
<h1 id="how-are-we-using-batch">How are we using Batch?</h1>
We did publish a <a href="http://tech.nextroll.com/blog/data/2018/08/08/running-jobs-with-aws-batch.html">blog post</a>
regarding how we are submitting jobs to the AWS Batch environment before. In this section, however, we are
going to have a more general view, and discuss the challenges on Batch as well.
<h2 id="data-diagram">Data Diagram</h2>
The data flow for Batch starts with Jenkins. After checking out the source code and building the Dockerfile, Jenkins builds the docker image and pushes it to ECR. ECR keeps all of the images that we need to run on Batch.
<img src="/images/post_images/batch_data_flow.png" alt="Batch data flow" />
We have some containers running on ECR which control and organize our jobs on ECR. The first one is the scheduler, which kicks off the jobs based on the time of day. Luigi acts as a dependency checker and ensures the dependency of the jobs was met. It also checks if the job was run before. If it did not run before, and all the dependencies are there, the job would be submitted to Batch. We also have our software running on Batch to monitor the jobs, called <a href="https://github.com/AdRoll/batchiepatchie">BatchiePatchie</a>. More about BatchiePatchie, below.
When the job kicks off on Batch, the batch environment provisions an instance pulls the docker image from ECR and runs the command line in the job on it.
<h2 id="sample-job-on-batch">Sample job on Batch</h2>
Now that we have seen how we submit jobs on Batch, let’s see how one of our sample jobs works on Batch. One of the use cases for us is to convert data from CSV format to <a href="https://github.com/traildb/traildb">TrailDB</a>. The job that we use for this purpose is called Baker. Baker converts data from one form to another and applies some filtering logic to it. It’s like a list comprehension but on a large scale.
<img src="/images/post_images/batch_baker_job.png" alt="Batch job - Baker" />
In this example, we have a bunch of CSV files that need to be converted. First, the planner job runs on a specific time of day. The planner reviews the CSV files and the contents. Based on the amount of data and split size, it turns them into several splits. The number of splits would be different on different days and hours. For each split, a baker job is launched on Batch. The Baker jobs perform their conversion and save their output on S3.
Next, the merge jobs run. These act as reduce functions and reduce data to a specific number of shards that are processed later on.
In this example, you see that the data determines the scale of processing. If, for instance, we have only one CSV, one baker and one merger are enough. However, given the amount of our data, we have hundreds of Bakers running every day, and this map-reduce job is running on AWS Batch reliably in our pipeline all the time.
<h2 id="purpose-of-the-jobs">Purpose of the jobs</h2>
Going back to what has been mentioned before, we have at least 80B predictions every day, which cause 80B events.
Each event is about 1KB, so the prediction events alone are 80TB of information. Jobs that are processing this much
data are good candidates to run on Batch.
<h3 id="attribution">Attribution</h3>
Every day, we are registering about 500,000 conversions. In marketing, a conversion is an event that the customer cares about the most, which mostly is Sales. So you can read it as 500,000 sales. Anyways, for each conversion event, we need to look back 30 days to find out the event that caused that conversion. That requires us to process 30 times 80B which would be more than a few petabytes of information. And this is critically needed to provide reports to our customers.
<h3 id="machine-learning">Machine Learning</h3>
We mentioned that our AI system needs to make tens of billions of predictions every day. The model that powers the
predictions needs to be refreshed with fresh data periodically. The training for that model also happens on Batch. Batch, in my opinion,
is a great tool for training machine learning models. In our case, we have a single case of batch processing that happens once and then is
used several times. So models can be trained on Batch and served on a tiny web server.
Generally speaking, Batch is a good use case for:
<ul>
<li>Periodic or nightly pipelines</li>
<li>Automatic scaling processes</li>
<li>Tasks that are flexible in the stack</li>
<li>Tasks that can benefit from the use of different instance types</li>
<li>Orchestrating instances to run tasks</li>
</ul>
On the topic of instance types, it should be noted that some of our
queues are configured to use “optimal” instance types, which means that they are provisioned with the tiniest instances possible. On the other hand, we have queues
that are using instances as big as x1.16xlarge to run heavy jobs. The freedom of choosing the right instance type for a job is
a great advantage for us.
In a few words, we can say Batch is great for orchestration. The platform enables you to provision your tasks the way you
like on the platform you like with a very short delay.
<h2 id="challenges">Challenges</h2>
Using Batch was not free of troubles for us. Just like any other technology, we had some challenges that we needed to
overcome. We feel it is best to keep the conversation honest and mention the drawbacks of this technology as well.
<h3 id="monitoring">Monitoring</h3>
We submit millions of jobs and keeping track of them is a big challenge for us. Just finding the job that caused trouble
or searching the jobs to find out the status of a situation when things are not going as planned is not possible using
the AWS Batch console. That made us develop our tooling around it.
<h3 id="out-of-instance-on-the-spot-market">Out of instance on the spot market</h3>
Some of our queues are using certain instance types. It happened before that the instance type is not available all of a sudden. It is possible that other customers of AWS are provisioning those instance types or that certain issues are happening across the data centers. The reason is usually not transparent to us. Whatever the cause is, we would then need to find other available instance types. That hunt is not easy, and we were only able to solve it using try and error.
The instance-type shortage does not happen often. Normally, we have cases like that once every three months or so. We
have disclosed the problem with the Spot team, and they said they are working on a product to resolve it.
<h3 id="disk-cleanup">Disk cleanup</h3>
Normally the life span of instances on Batch is very limited: The instances are provisioned, a docker
image is being deployed, a job would be launched, and the instance would be terminated right after. However, since we
send so many jobs to the queues, we have cases where instances would be alive for a long time serving tens of different.
kinds of jobs. If a job does not do efficiently clean up its temporary files, the next job may fail because of a disk shortage issue.
The way we have worked around this problem was to bake our own AMIs with logic to cleanup aggressively for
dangling docker images and unused volumes. The cleanup would make sure the current job has enough disk space to run.
This simple change has resolved the problem for us.
<h3 id="manual-work-for-mapreduce-jobs">Manual work for Map/Reduce jobs</h3>
There are some frameworks for big data processing such as Spark and Hadoop. For machine learning, there are
frameworks for training like SageMaker. When using those tools, you only need to write the map/reduce logic and the
framework handles the boilerplate code to scale as needed. On Batch, however, the framework is only handling
the orchestration for you. That can be an advantage or a disadvantage. In some applications, that can be considered an
advantage, since you can be in control of every detail. But, sometimes having to focus so much on the details can be counterproductive.
Some frameworks like Hadoop would provide a file system (i.e. HDFS) to process the big data. When using batch, you
can use S3 or Lustre to handle the shared storage.
<h3 id="managed-queues">Managed queues</h3>
Each queue on Batch can be Managed or Unmanaged. For unmanaged queues, you would need to handle the scaling yourself, which is very tricky. We do use Managed queues. For them, AWS handles the scaling of the underlying ECS cluster transparently. There is an issue, though. Batch sets the desired vCPUs for each queue, and in our experience, it takes a while for the queue to catch up with the provided load.
To solve this problem, we have added some logic to BatchiePatchie to set the minimum vCPUs required to run all active jobs. This way the queue handles the scaling much faster.
<h1 id="batchiepatchie">Batchiepatchie</h1>
Since we started using Batch in 2017, we had some issues with it. During HackWeek, our engineers decided to take a shot at making life easier for themselves. They started working on a project to report the jobs in a way that makes more sense for us. Lorenzo Hernandez came up with the idea, and Mikko Juola suggested the name Batchiepatchie. It is a funny name, but it provides a serious service.
Mikko spent another hack week to take Batchiepatchie to next level and made it opensource. It is now available on <a href="https://github.com/AdRoll/batchiepatchie">GitHub</a> to download. In this section, we are going to review the main functionalities of this software.
<h2 id="monitoring-1">Monitoring</h2>
We already mentioned that one of our challenges with Batch is monitoring. Since our data pipeline consists of several jobs that modify and transform data in several ways, monitoring is a big challenge.
The AWS Batch console provides some views. You can review the job queues and computation environment. Because of queue activity, it takes a while for the Batch dashboard to load, and when it loads, it can only show certain jobs, and in different views. Batchiepatchie, however, shows the granular status of the jobs in a single view. For instance, if you have submitted 30 jobs, 20 of them succeeded, 5 of them are running and 5 failed, you can see all of them in a consolidated place.
In the screenshot below, you can see both the AWS Batch console and batchiepatchie snapshot for the same job queue.
<table>
<tbody>
<tr>
<td><img src="/images/post_images/aws_batch_1.png" alt="Batch console 1" /></td>
<td><img src="/images/post_images/bp_1.png" alt="Batchiepatchie 1" /></td>
</tr>
</tbody>
</table>
<h2 id="logs">Logs</h2>
Batch only recently (as in 2019) has added a feature that allows us to review the logs of jobs. Using a job-id, Batch console would provide a link to Cloudwatch where logs would be available. As useful as that feature is, it has several shortcomings:
<ul>
<li>When scrolling up and down, you need to wait for logs to be loaded</li>
<li>In the log window, you do not have access to job details</li>
<li>Searching in the logs is not easy</li>
</ul>
Batchiepatchie has been written by developers for developers. It provides all the logs generated by the process in one view which can be scrolled and searched easily. The exceptions and errors are also easily accessible.
In the screenshots below, the views of the Batch console and Batchiepatchie logs view for comparison:
<table>
<tbody>
<tr>
<td><img src="/images/post_images/aws_batch_2.png" alt="Batch console 2" /></td>
<td><img src="/images/post_images/bp_2.png" alt="Batchiepatchie 2" /></td>
</tr>
</tbody>
</table>
<h2 id="search">Search</h2>
Jobs can be searched by job-id in the AWS Batch console. If you happen to have job-id, you could access the job details and logs through the console. However, to find the job-id of the jobs that were submitted before, you need to go through some digging. Most of the time, we know the job name or the command line of the job that we are interested to check. That’s when Batchipatchie’s search feature is most useful.
When you type a keyword in the search bar, Batchiepatchie would look through all the fields including job name and command line of the job to list the jobs that have been submitted before or that are running at the moment. Check the screenshots below for comparison.
<table>
<tbody>
<tr>
<td><img src="/images/post_images/aws_batch_3.png" alt="Batch console - 3" /></td>
<td><img src="/images/post_images/bp_3.png" alt="batchiepatchie - 3" /></td>
</tr>
</tbody>
</table>
<h2 id="holistic-view">Holistic view</h2>
In the recent HackWeek, in 2019, Joey has added a functionality to show the flow of jobs in queues. In the stats view, the number of jobs that have certain properties has been counted and portraited. This is very useful to have a glance of the queue status and see what is happening overall.
In the screenshot below, you can see the number of jobs that failed during the week. Just by looking at the chart, you can tell something was going on during October 16th. This could be helpful to catch silent failures or troubleshoot errors.
<img src="/images/post_images/bp_4.png" alt="Batchiepatchie - 4" />
<h2 id="cost-analysis">Cost Analysis</h2>
AWS Cost explorer would provide a lot of breakdowns for costs on AWS usage. By tagging different queues, you can query the cost for each queue separately. The cost of each job is not provided, however. To find out what is the most costly job in our day-to-day usage, we are using Batchiepatchie’s database.
Batchiepatchie stores all status data for jobs and instances on a PostgreSQL database. The data that is not available in the UI could be queried on the database itself. Using queries, we can estimate the dollar cost of each job.
A list of tables in that database has been listed below.
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>batchiepatchie=&gt; \d+
List of relations
Name | Type | Owner | Size
------------------------------+----------+----------------+-----------
activated_job_queues | table | batchiepatchie | 16 kB
compute_environment_event_log | table | batchiepatchie | 381 MB
goose_db_version | table | batchiepatchie | 40 kB
goose_db_version_id_seq | sequence | batchiepatchie | 8192 bytes
instance_event_log | table | batchiepatchie | 28 GB
instances | table | batchiepatchie | 1537 MB
job_status_events | table | batchiepatchie | 40 kB
job_summary_event_log | table | batchiepatchie | 296 MB
jobs | table | batchiepatchie | 25 GB
task_arns_to_instance_info | table | batchiepatchie | 2452 MB
(10 rows)
</code></pre></div></div>
If you use AWS Batch, you should check out Batchiepatchie! And if you use Batchiepatchie, please share your experience with us. We would appreciate any contribution to this repo and would love to hear from you.
</description>
</item>
<item>
<title>
Interning at NextRoll
</title>
<link>https://tech.nextroll.com/blog/culture/2019/10/15/interning-at-adroll.html</link>
<pubDate>Tue, 15 Oct 2019 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/culture/2019/10/15/interning-at-adroll</guid>
<description>Being a part of the team at NextRoll detailed through the eyes of an intern on the analytics team.
<code class="language-plaintext highlighter-rouge">7 minute read</code>
<hr />
‘Cali or Bust’. Anyone in the math faculty’s co-op program at the University of Waterloo can tell you what that means – wanting to land a sweet internship in California or nothing at all. I’m currently in my 4th co-op (out of 6), and since my first application period, I had become accustomed to the norm of dreaming about a term in California. Everyone else’s aspirations had somehow shaped my own and I started aiming towards achieving a job in the tech hub of the world.
One of the things that’s great about the program I’m in is the frequent switch between school terms and work terms. When handing in countless assignments become your life’s purpose and biggest immediate stressor, looking forward to 4 or 8 months of working and a lack of anxiety-inducing exams helps students like myself pull through the difficulties of academic life. Similarly, when the work routine and waking up every day to commute to a 9 to 5 job might start to feel mundane, the independence of a self-regulated school schedule starts to feel appealing again.
In May of this year, applications for the Fall co-op term began. I let myself be pickier than usual this term because I knew I didn’t want to solely apply to jobs based on their location, but also wanted to find job descriptions that resonated with my goals. Scattered throughout thousands of postings, were a few data driven roles that I was interested in. The posting for an analytics internship at a company called AdRoll (now NextRoll), felt like a potential good match. RTB? ‘What’s that?’ I wondered, looking it up to discover the interesting field of auction tech and real time bidding algorithms. ‘What’s a golden bagel?’ ‘Look at these cute culture creatures on their website.’ In particular, I liked the owl, which represented hiring great people and helping each other grow. Growth, on both individual and company levels, is hugely valued at NextRoll. As a student trying to jump into the workforce, encouragement and resources to expand my skillset are reserved as the most valuable things I can be provided with.
<img src="/images/post_images/owl.jpg" alt="Culture owl" />
So, yeah, NextRoll seemed like a company with interesting work, a diverse and engaging culture, and also happened to be in San Francisco. Application submitted.
Over the next few weeks of May into June, I juggled lectures and assignments and phone calls and technical challenges and interviews and info sessions. The more companies I interacted with, the more I knew I felt I would fit in best with NextRoll. The recruiters were all friendly and really cared about matchmaking the right interns to the right positions. They held a great social in Waterloo where I got to interact with employees from the team I had applied to as well as other teams. I began to stop caring about ‘Cali or Bust’ and start caring about focusing on doing good data science work and meeting inspiring people at this company.
The interviewing period came to an end in June and I ended up receiving an offer, made an excited phone call to my dad, and started planning the co-op term with NextRoll.
The first few weeks of the internship were full of training sessions for the NewRollers that joined NextRoll in September. The trainings introduced the new recruits to every team at the company, not just the one we would be working on, and these were great overviews to show us how the company functioned. We even got to have a discussion with the CEO, Toby Gabriner, which showed me that every employee at every level is seen as important. The diversity of the group I was in trainings with was obvious to me, faces of different ages and countries and experiences sitting next to me. Everyone was outgoing and intelligent and ready to contribute their talents towards growing this company. There were 4 other UW co-ops aside from me, which was cool because I haven’t closely worked with other UW students at my past 3 co-op jobs. These co-ops turned into some really great friends.
<img src="/images/post_images/interns-at-fair.JPG" alt="The interns" />
NextRoll has a great office in Mission District. It’s a sunny, open space concept with lots of good snacks, company dinners and – OMG – a bunch of cute dogs. The hours are flexible; as long as you get your work done, you are entitled to a fairly independent schedule.
<img src="/images/post_images/sf-office.jpg" alt="The office" />
My work involves answering questions or conducting analyses for the product or engineering teams. There is a lot of data to work with and a variety of tools at your disposal to get what you need to do done. Transitioning into new companies so often becomes a ritual, but setup and acclimating to different norms remains difficult in the beginning. Everyone on the analytics team I am on was kind enough to have a 1 on 1 chat with me to introduce themselves and the team from their perspective. Being a part of this collection of smart, accomplished, and yet down to earth people is inspiring. Every day, I have many questions, and rather than feel self-conscious about how much I feel I need to learn, I find myself being provided with encouragement and everlasting guidance. Experiencing this level of support motivates me to one day become a helpful and knowledgeable full-timer.
I value being at NextRoll because of not just the good work, but the engaging company events, the community service program, inclusive groups like Women in Tech, and of course the super dog-friendly office. In my few months of being here I’ve been lucky enough to visit a good range of other companies headquartered in the area, and I can say that not every company has all of these things that I now look for.
<img src="/images/post_images/nr-gives-back.jpg" alt="PHC" />
Working at a company in California shouldn’t be the goal.
Working at a company with values that align with yours should be the goal. Working at a company with a culture and job responsibilities that make you excited to go to work should be the goal. A company that emphasizes diversity, having resource groups and appreciates work-life balance should be the goal.
To anyone in the search for a job, be open minded in finding a match that makes you feel fulfilled regardless of location. Sure, there may be a higher density of such companies in the state of California, but I think they exist everywhere. All my internships have been in different cities, each one its own amazing opportunity and learning experience. Don’t just ‘Cali or Bust’. Find your NextRoll.
</description>
</item>
<item>
<title>
HyperLogLog in Python
</title>
<link>https://tech.nextroll.com/blog/dev/2019/10/01/hll-in-python.html</link>
<pubDate>Tue, 01 Oct 2019 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2019/10/01/hll-in-python</guid>
<description>We open-sourced a Python library for HLLs compatible with postgresql-hll.
<code class="language-plaintext highlighter-rouge">5 minute read</code>
<hr />
<h2 id="what-is-python-hll">What is python-hll?</h2>
We recently open-sourced <a href="https://github.com/AdRoll/python-hll">python-hll</a>, which is an implementation of
<a href="http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf">HyperLogLog</a> whose goal is to be
<a href="https://github.com/aggregateknowledge/hll-storage-spec">storage compatible</a>
with <a href="https://github.com/aggregateknowledge/java-hll">java-hll</a>, <a href="https://github.com/aggregateknowledge/js-hll">js-hll</a>,
and <a href="https://github.com/citusdata/postgresql-hll">postgresql-hll</a>. At NextRoll, we use these libraries to do fast
counting of unique values in PostgreSQL and Java, but we were dismayed that there was no Python library for it.
So we decided to make one.
<h2 id="what-are-hlls">What are HLLs?</h2>
In a <a href="http://tech.nextroll.com/blog/dev/2019/08/06/hll-in-postgresql.html">previous blog post</a>, we talked about
how we use HLLs in PostgreSQL at NextRoll. As a quick recap, HLLs let you quickly count unique values. You
basically throw a bunch of unique values into a blob called an HLL. Then you can ask the HLL how many unique
values it has:
<center>
<img alt="Adding values to an HLL" src="/images/post_images/one-hll.png" width="500" />
</center>
Not only that, but you can also union HLL blobs together then ask it how many unique values it has:
<center>
<img alt="Unioning two HLLs together" src="/images/post_images/two-hlls.png" width="500" />
</center>
Here’s how it works using our Python library:
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from python_hll.hll import HLL
import mmh3
hll1 = HLL(13, 5)
hll2 = HLL(13, 5)
hll1.add_raw(mmh3.hash('17'))
hll1.add_raw(mmh3.hash('17'))
hll1.add_raw(mmh3.hash('31'))
hll1.add_raw(mmh3.hash('5'))
cardinality = hll1.cardinality()
# Output: 3

hll2.add_raw(mmh3.hash('6'))
hll2.add_raw(mmh3.hash('6'))
hll2.add_raw(mmh3.hash('31'))
hll2.add_raw(mmh3.hash('5'))
cardinality = hll2.cardinality()
# Output: 3

hll1.union(hll2)
cardinality = hll1.cardinality()
# Output: 4
</code></pre></div></div>
And then you can export the HLL to a string that works with postgresql-hll:
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from python_hll.util import NumberUtil
bytes = hll1.to_bytes()
output = "\\x" + NumberUtil.to_hex(bytes, 0, len(bytes))
# \x128D7F00000000201E4B390000000027FA7CC000000000531A35E4000000006E6F0B04
</code></pre></div></div>
<h2 id="how-we-built-and-tested-it">How We Built and Tested It</h2>
We built python-hll as part of NextRoll’s June 2019 Hack Week, a time when engineers take some time
to build things that interest them personally. As we only had a week, we thought it would be
expedient to do a fairly literal translation/port of <a href="https://github.com/aggregateknowledge/java-hll">java-hll</a>
to Python.
To minimize changes, we decided to <a href="https://github.com/AdRoll/python-hll/blob/44349efa8ca35f3ce6a33260f5f270b0a655f921/python_hll/util.py#L107">internally represent</a>
bytes as Java-style bytes (-128 to 127) rather than true Python bytes (0 to 255). Issues we ran into
included: <a href="https://github.com/AdRoll/python-hll/blob/44349efa8ca35f3ce6a33260f5f270b0a655f921/python_hll/util.py#L86">how to translate Java’s <code class="language-plaintext highlighter-rouge">&gt;&gt;&gt;</code></a>,
into Python, <a href="https://github.com/AdRoll/python-hll/blob/44349efa8ca35f3ce6a33260f5f270b0a655f921/python_hll/util.py#L130">how to handle integer overflows from <code class="language-plaintext highlighter-rouge">&lt;&lt;</code></a>
identically to Java so the tests pass (actually we were quite surprised that so many integer overflows were happening in
the Java unit tests), and how to make sure the tests pass in both Python 2.7 and Python 3.
Fortunately, postgresql-hll has an extensive set of <a href="https://github.com/citusdata/postgresql-hll/tree/master/sql/data">test data</a>
with 22,614 data points that we were able to use to <a href="https://github.com/AdRoll/python-hll/tree/master/tests/data">test our library</a>.
In addition, we also ported the
<a href="https://github.com/aggregateknowledge/java-hll/tree/master/src/test/java/net/agkn/hll">unit tests</a>
from java-hll <a href="https://github.com/AdRoll/python-hll/tree/master/tests">to Python</a>.
Unfortunately one drawback to our python-hll library is that it is quite slow compared to java-hll.
For example, in Java, <a href="https://github.com/aggregateknowledge/java-hll/blob/master/src/test/java/net/agkn/hll/serialization/HLLSerializationTest.java">HLLSerializationTest</a>
takes 12 seconds to run, while in Python, the equivalent <a href="https://github.com/AdRoll/python-hll/blob/master/tests/test_hll_serialization.py">test_hll_serialization</a>
takes 1.5 hours to run - it’s about 400x slower. It should be okay if a few HLL operations are needed,
but for many HLL operations you may find that it could slow your program down. Test it and see.
<h2 id="acknowledgements">Acknowledgements</h2>
Thanks to the 7 engineers who translated code and tests from Java to Python for this project at NextRoll’s June 2019 Hack Week:
<ul>
<li><a href="https://www.linkedin.com/in/jonathanaquino">Jon Aquino</a></li>
<li><a href="https://www.linkedin.com/in/kushagra391/">Kushagra Verma</a></li>
<li><a href="https://www.linkedin.com/in/alexcleu/">Alex Leu</a></li>
<li><a href="https://www.linkedin.com/in/michaeltran10/">Michael Tran</a></li>
<li><a href="https://github.com/rodrigoadroll">Rodrigo Westrupp</a></li>
<li><a href="https://www.linkedin.com/in/subramaniansridharan/">Sridharan Subramanian</a></li>
<li><a href="https://www.linkedin.com/in/piyush-srivastava-5343a63/">Piyush Srivastava</a></li>
</ul>
</description>
</item>
<item>
<title>
Can Engineering principles solve sales problems?
</title>
<link>https://tech.nextroll.com/blog/culture/2019/09/17/engineering-principles.html</link>
<pubDate>Tue, 17 Sep 2019 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/culture/2019/09/17/engineering-principles</guid>
<description><code class="language-plaintext highlighter-rouge">5-7 minute read</code>
<hr />
<blockquote>
“Onboarding customers is taking too long!”
</blockquote>
<blockquote>
“Our customers are feeling frustrated!”
</blockquote>
<blockquote>
“Our sellers are missing their quotas!”
</blockquote>
<blockquote>
“Q4, our busiest season of the year, is right around the corner, and we need a solution ASAP!”
</blockquote>
All of the above complaints were funneled into the Solutions Engineering team at NextRoll. We were not providing support that met the standards expected from our team. In turn, our sales teams and our customers were suffering. If we did not solve some of these challenges quickly, we could’ve missed out on substantial revenue for Q4.
The Solutions Engineering (SE) team at NextRoll is responsible for the technical aspects of onboarding clients, debugging post launch platform issues, and building tools to automate manual tasks. At our core, we sit at the junction between sales and engineering, communicating problems bi-directionally. We represent the voice of the client within engineering teams, but we also provide engineering expertise for sales teams. When a bug impacting clients is discovered, the SE team conveys the impact and priority to the engineering team. When an SE is onboarding a client and is being pressured to rush, it is imperative that the SE advocates for what is best for the client, technically.
When we first started learning about the problems our stakeholders –the sales teams– were experiencing with using our services, my response was, “Oh! Of course the end-user doesn’t understand how to use our system!”. Patrick Mee, our EVP of Engineering, patiently waded through this initial defensiveness I felt about our team. He scheduled meetings with our stakeholders and continuously displayed real empathy to their concerns. Through these meetings, I was able to remove myself from the SE team and instead place myself in the end user’s perspective.
After accepting that our end users had concerns, I could actually begin to listen to their complaints. Even though the heart of these issues were people and processes, the solution naturally came to me from thinking through this from an engineering lens. My background is in systems engineering, and the first step in building any system is to write down the use cases. How would an end user use our services as a system?
The previous user story flowed like this:
<ol>
<li>Customer has a question</li>
<li>Customer reaches out to their account manager/self service chat support representative (Tier 1)</li>
<li>Account manager/chat rep reaches out to Support Engineers (Tier 2)</li>
<li>Support Engineer escalates to Solutions Engineer (Tier 3) where needed</li>
<li>Solutions engineer contacts engineering if bug is found</li>
<li>Engineering provides resolution - bug fixed/won’t fix, potential workarounds, and any associated timelines</li>
<li>Solutions Engineer provide this feedback back to customers via account managers</li>
</ol>
<h2 id="the-interfaces-are-the-weaknesses">The interfaces are the weaknesses</h2>
The above user story shows a minimum of seven handoffs between teams to solve one issue. If you’ve ever played telephone, then you know information is lost as it transfers between parties. That’s because in any system, the interfaces are the weaknesses. The less handoffs there are between systems, the less information is lost between components. Here we had both component and interface issues. We started thinking about which handoffs to eliminate to reduce information transfers. We thought, “let’s reduce turnaround times for customers, making our sellers and customers happy at the same time”. Win-win, right?
We first removed the barrier for support engineers to reach out to solutions engineers prior to reaching out to engineering teams. Due to some legacy baggage around not properly reaching out to the right channels, support engineers had been restricted from creating bug tickets for engineering as well as reaching out to those engineering teams in slack channels.
Now, the user story flows like this:
<ol>
<li>Customer has a question</li>
<li>Customer reaches out to their account manager/self service chat support representative</li>
<li>Account manager/chat rep reaches out to Support Engineers</li>
<li>Support engineer contacts engineering if bug is found</li>
<li>Engineering provides resolution - bug fixed/won’t fix, potential workarounds, and any associated timelines</li>
<li>Support Engineer provides this feedback back to customers via account managers</li>
</ol>
<h2 id="perform-qa-on-individual-components-prior-to-integrating-components">Perform QA on individual components prior to integrating components</h2>
Another bit of feedback we received was that Support Engineers were occasionally failing to provide sufficient information to the teams they were working with to onboarding clients. This led to multiple back and forths between SEs, Account Managers, and customers, introducing potential churn risks at a critical phase in the client relationship. We needed to ensure that communications from our team were thorough, accurate, and identified any needed data immediately. To ensure every onboarding started on the right path, we needed to improve the quality of written information provided on the initial response in the ticket.
We introduced code reviews. Code reviews, in engineering, are meant to provide a systematic way to improve code. In slack, we created peer review channels, pairing a support engineer with a solutions engineer. Solutions Engineers reviewed the onboarding instructions written by support engineers to ensure quality before they were sent to new clients. We noticed that the overall time to resolution was reduced significantly. By approaching our service as a system and performing Quality Assurance on the individual components prior to integration, we were able to have a smoother deploy to production.
<h2 id="focus-on-the-right-metrics">Focus on the right metrics</h2>
The support engineers were being held accountable to a 24 hour SLA(service level agreement). The support engineer needed to respond within 24 hours to a ticket, regardless of the quality of responses within the ticket. Unfortunately, time to response as an SLA was emphasized over other metrics, leading to lack of concern for other metrics such as number of iterations per ticket.
Frankly, our end users were not concerned about receiving an incomplete response back within 24 hours. We shifted the focus from “time to response” to reducing the number of iterations between the requester and the SE on the ticket. We found that the higher the number of iterations on a ticket, the more likely end users were frustrated by customer service.
<h2 id="gathering-feedback-post-changes">Gathering Feedback Post Changes</h2>
The feedback we received from the sales teams after we implemented the changes truly honored the amazing work our Solutions Engineering teams do. The Support Engineering and Solutions Engineering teams at AdRoll work incredibly hard to improve the quality of service we provide to our customers. Hearing positive feedback from the same teams we provide service to makes us want to celebrate a great release, just like the engineers.
<hr />
</description>
</item>
<item>
<title>
HyperLogLog in PostgreSQL Amazon Aurora RDS
</title>
<link>https://tech.nextroll.com/blog/dev/2019/08/06/hll-in-postgresql.html</link>
<pubDate>Tue, 06 Aug 2019 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2019/08/06/hll-in-postgresql</guid>
<description>We moved from HLLs stored in HBase to HLLs stored in Postgres with great results.
<code class="language-plaintext highlighter-rouge">5-10 minute read</code>
<hr />
<h2 id="what-is-hyperloglog-hll">What is HyperLogLog (HLL)?</h2>
HyperLogLog (HLL) is a useful and interesting probabilistic data structure used to count unique values in a given data set with good accuracy and speed. Normally, to count unique values accurately requires memory proportional to the number of unique values. This becomes a problem when working with large data sets. HyperLogLog solves this problem by allowing us to trade memory consumption for (tunable) precision, making it possible to estimate cardinalities larger than 1 billion with a standard error of 2% using only 1.5 kilobytes of memory. For more on how this works, you can refer to our past blog post <a href="http://tech.adroll.com/blog/data/2013/07/10/hll-minhash.html">here</a>.
At Adroll Group, we use this data structure in our reports to measure metrics like unique visitors, engaged visitors, etc. We had implemented HLL in-house and stored it in HBase. But HBase was causing us some headaches.
<h2 id="problems-with-hbase">Problems with HBase</h2>
While working with HBase we encountered two major challenges:
<ol>
<li>Defining the key in a way that supports a variety of queries</li>
<li>Having to build aggregation operations in a Thrift layer.</li>
</ol>
<h3 id="key-structure">Key structure</h3>
HBase being a key-value data store requires a key which uniquely identifies the record. This key is used to extract metrics and aggregation in reporting. It was always a challenge to define the key in a way which makes searching of records optimized for different parameters, especially compared to a relational database.
As an example, let’s say we are storing unique visitors for a customer with details about visitors every day.
For that we can create a table called Daily Unique Visitors with the following columns:
<ul>
<li>Date</li>
<li>Customer ID</li>
<li>Browser Type</li>
<li>Location (Country)</li>
<li>Unique Visitors (HLL)</li>
</ul>
Since the value we care about is only the last column and the others are its key, in an HBase Table we’ll need to define a key-structure using the first 4 columns concatenated by underscores. Something like: <code class="language-plaintext highlighter-rouge">Date_Customer-id_browser-type_location</code>
A key will then look like <code class="language-plaintext highlighter-rouge">2019-07-01_A_Chrome_US</code>, i.e. 1st July, for customer A, chrome browser, US location (Key) =&gt; Unique visitors (Value)
This works well when we need to scan it in the same order.
For example, to get the number of unique visitors in the month of July 2019, for a given customer, for the Chrome browser, for all locations.
But as soon as we have different search criteria - like getting the number of unique visitors in July, for Customer A, for all browsers, in the US - the key structure is no longer effective and requires more processing to discard irrelevant keys on the scan results.
<h3 id="thrift-layer">Thrift layer</h3>
For interaction with HBase from a non-Java platform, people usually use <a href="https://blog.cloudera.com/blog/2013/09/how-to-use-the-hbase-thrift-interface-part-1/">Thrift</a> since it provides a language-independent interface. To support our reporting needs we have to implement operations like aggregation, filtering, and ordering of HLL records efficiently in our Thrift layer. This adds to development, maintenance, and resource costs. We also encountered stability issues with Thrift which required frequent restarts.
In the earlier HBase table example: scanning, filtering, ordering, etc. is done in the Thrift layer.
Sometimes the date ranges are so big that Thrift makes lots of parallel HBase scan threads, causing resource crunches and frequent timeouts.
<h2 id="hll-in-postgresql-amazon-aurora-rds">HLL in PostgreSQL Amazon Aurora RDS</h2>
We started looking for alternatives and fortunately, Amazon Aurora RDS started supporting the <a href="https://github.com/citusdata/postgresql-hll">postgresql-hll</a> extension to provide a new HLL data type. Switching to a relational database system solves a lot of our problems. If we use RDS as a data store, then we can define our schema efficiently using indexing, etc., to fulfill all our reporting needs (no more single keys). Also, we don’t have to maintain anything for our aggregation, filtering and ordering needs as RDS has that built in (no more Thrift).
Going back to our daily unique visitors table example:
We can get results easily and efficiently using SQL:
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT hll_cardinality(hll_union_agg(unique_visitors))
FROM daily_unique_visitors
WHERE ts &gt;= '2019-07-01'
AND ts &lt; '2019-08-01'
AND customer_id = 'A'
AND location = 'US'
</code></pre></div></div>
Using various relational database optimization techniques like indexing, table partitioning etc. you can get results fast.
<h3 id="proof-of-concept">Proof-of-concept</h3>
To measure the improvements we did a proof-of-concept. We replicated 3 months of data in RDS and compared the HBase and RDS results.
<h4 id="query-performance-improvements">Query performance improvements</h4>
Long date range queries spanning 3 months used to take more than 60 seconds in HBase, but gave results in RDS in less than 10 seconds.
<h4 id="less-management-effort">Less management effort</h4>
HBase and the Thrift layer require maintenance such as, installation, monitoring, trouble shooting, development etc, which is already taken care of with Amazon Aurora since it’s a managed service.
After evaluation, we decided to move to RDS, but it had its own challenges and limitations as explained below.
<h2 id="challenges-in-migration">Challenges in migration</h2>
<ol>
<li>The biggest challenge we encountered was our old data. Our HBase HLLs were not compatible with the new RDS HLL format due to the different hashing and conversion mechanisms for storage between our HBase implementation and the RDS extension. Where possible we rebuilt the data using RDS HLLs from raw source data. Where we were unable to do that, we generated HBase HLLs simultaneously with RDS HLLs on a go-forward basis. The day when RDS data was available was called the “cutoff date”. Whenever we had to query a time period requiring old data, before the cutoff date, we used the HBase HLLs. Data for date ranges after the cutoff date used RDS. After some time, we were able to retire our HBase queries completely.</li>
<li>Limited library support in the programming languages we use. The current PostgreSQL extension has library support in <a href="https://github.com/aggregateknowledge/java-hll">Java</a> and <a href="https://github.com/aggregateknowledge/js-hll">Javascript</a> only. This works fine if you are using one of these languages in your technology stack, but otherwise, it will be a limitation. We use Python a lot. Hence, we had to put some effort to port these libraries to Python. We tackled that project in our most recent HackWeek.</li>
</ol>
<h2 id="python-hll-postgresql-hll-extension-python-library">Python-hll: PostgreSQL-hll extension Python library</h2>
We created a Python library to read, write, count and do operations like the union of PostgreSQL compatible HLLs. This is similar to the <a href="https://github.com/aggregateknowledge/java-hll">Java library</a>. We plan to open-source it soon – stay tuned.
<h2 id="conclusion">Conclusion</h2>
Migrating from HBase to PostgreSQL HLL on Amazon Aurora was a big win for us. We found the PostgreSQL HLL extension a lot better to store our HLL reporting metrics. Also, the benefits that stem from using an RDS with support for HLL, like Amazon Aurora, serve our reporting needs to retrieve and present this information efficiently. We thoroughly recommend that you check it out.
<hr />
</description>
</item>
<item>
<title>
AdRoll Group and The Erlang Ecosystem Foundation
</title>
<link>https://tech.nextroll.com/blog/culture/2019/07/31/erlang-ecosystem-foundation.html</link>
<pubDate>Wed, 31 Jul 2019 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/culture/2019/07/31/erlang-ecosystem-foundation</guid>
<description>We recently decided to sponsor the Erlang Ecosystem Foundation. We believe this is a very important initiative and we’re trying to contribute with it as much as we can. Learn more about it, the reasons behind our sponsorship, and the other ways in which we’re contributing in this article.
<code class="language-plaintext highlighter-rouge">5-7 minute read</code>
<hr />
<h1 id="whats-the-eef">What’s The EEF?</h1>
Paraphrasing from <a href="http://erlef.org/">their own website</a>…
<blockquote>
The Erlang Ecosystem Foundation is a new non-profit organization dedicated to furthering the state of the art for <a href="https://erlang.org/">Erlang</a>, <a href="https://elixir-lang.org/">Elixir</a>, <a href="http://lfe.io/">LFE</a>, and other technologies based on the BEAM. Their goal is to increase the adoption of this sophisticated platform among forward-thinking organizations. With member-supported Working Groups actively contributing to libraries, tools, and documentation used regularly by individuals and companies relying on the stability and versatility of the ecosystem, they actively invest in critical pieces of technical infrastructure to support their users in their efforts to build the next generation of advanced, reliable, realtime applications.
</blockquote>
The community behind Erlang, Elixir, etc. is not new and during its many years it has experienced multiple diverse phases and has grown with the help from many independent contributors. One of the goals behind the EEF is to unite these independent contributions, give them the support that they deserve and help everyone involved to get the best from them.
These are the very first days of the Foundation, but they already have many working groups covering a variety of very intersting ideas:
<ul>
<li>Building and Packaging: Their mission is to evolve the tools in the ecosystem related to building and deploying code, with a strong focus on interoperability between BEAM languages</li>
<li>Observability: Their goal is to evolve the tools in the ecosystem related to observability, such as metrics, distributed tracing and logging, with a strong focus on interoperability between BEAM languages.</li>
<li>Security: They will work on identifying security issues, and providing solutions, developing guidance, standards, technical mechanisms and documentation.</li>
<li>Education, Training &amp; adoption: The objective of this working group is to facilitate, evolve education and training and consolidate educational material(s) for all BEAM languages and the BEAM itself.</li>
<li>Documentation: They will work to improve the accessibility, interoperability and quality of the documentation across projects and languages in the Erlang Ecosystem.</li>
<li>Fellowship: This is a group created to formally nominate community members for a fellowship role according to the Erlang Ecosystem Foundation bylaws.</li>
<li>Sponsorship: This group function is to approve sponsors candidacy.</li>
<li>Marketing: To expand awareness of Erlang Ecosystem and participation in its community. To promote the Erlang Ecosystem Foundation and its activities, and to increase engagement in the foundation.</li>
</ul>
<h1 id="why-do-we-care">Why do we care?</h1>
As you might have seen in some previous entries in this blog, some of our core systems (like our <a href="/blog/web/2014/04/29/valentino-presents-adrolls-rtb-infrastructure.html">real-time bidding platform</a>) are fully powered by Erlang. And while we have a team of amazing developers, we don’t build everything from scratch. We invest deeply and depend heavily on the open-source libraries and tools that power up our infrastructure. Maintaining and improving those projects is a big challenge for their contributors, with many of them not being backed up by any company big or small. We believe that the EEF can help on the area a lot.
On a related subject, we know that these languages even when not esoteric are not as popular as many others. Keeping a reasonable offer of developers that are capable of and willing to satisfy the increasing demand is not easy. We believe that the foundation will have a profound impact on this area.
We are also always looking for our devs to grow professionally. With the help of the different working groups within the EEF, we believe that this will be much easier. The Foundation will allow our devs to learn new skills, use better tools, standarize their best practices, contribute to the community and more.
In general, we believe that the EEF will bring a very positive influx of energy to an already energetic and growing ecosystem of which we’re part of.
<h1 id="contributing-with-the-eef">Contributing with the EEF</h1>
AdRoll is a founding member of the EEF, we were there supporting it since its inception through the Erlang Industrial User Group.
<a href="https://twitter.com/miriampena">Miriam Pena</a>, staff engineer at AdRoll, helped define the bylaws and working groups, worked on the paperwork, marketing, presented the foundation in diverse places, and much more. Miriam is at this stage a board member for the EEF and she is also participating in the sponsorship, marketing, fellowship and education working groups.
Besides her, some of our most talented engineers are also members of various working groups. For instance, <a href="https://about.me/elbrujohalcon">Brujo Benavides</a> is a member of the <a href="https://erlef.org/education-training-adoption/">Education, Training &amp; Adoption</a> group.
The EEF has a fairly simple membership program that includes 3 options:
<ul>
<li>Basic Membership: You’re entitled to attend meetings but not to vote on any resolutions.</li>
<li>Annual Supporting Membership: Supporting Members may attend meetings and cast votes on resolutions. Your commitment also supports Foundation activities, such as Working Groups and community events.</li>
<li>Lifetime Supporting Membership: All the benefits of an Annual Supporting Membership at a reduced rate! This level demonstrates your alignment with our vision of the Erlang Ecosystem as the primary technology of the future.</li>
</ul>
You can become a member by signing in at <a href="https://members.erlef.org/join-us">their website</a>.
This initiative can be an amazing thing for all the people that love these languages and this wonderful community. But that will only happen if we all take part of it.
So, what are you waiting for? <a href="https://members.erlef.org/join-us">Become a member today!</a>
<hr />
</description>
</item>
<item>
<title>Series: Tech Women of AdRoll Group</title>
<link>https://tech.nextroll.com/blog/culture/2019/03/08/tech-women-of-adroll-group-part-4.html</link>
<pubDate>Fri, 08 Mar 2019 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/culture/2019/03/08/tech-women-of-adroll-group-part-4</guid>
<description>In celebration of International Women’s Day 2019 and National Women’s History Month, we’re continuing our tradition of highlighting Tech Women at AdRoll Group.
I love reading biographies, but not for the reason you might be thinking. You see, I love origin stories. No two people take the same path through life and it’s those different experiences that shape how we approach our work. That’s why I jumped at the chance to interview several members of Tech Women at AdRoll Group. I had so much fun learning more about my coworkers and am happy to share what I learned with you.
<h1 id="monica-lloyd">Monica Lloyd</h1>
Business Intelligence Engineer
<img src="/images/post_images/tech-women-monica-lloyd.jpg" alt="Monica Lloyd" />
From a young age, Monica wanted to be a journalist. She was inspired by her father, who had an impressive career as a combat cameraman in Vietnam and later worked for CBS News and 60 Minutes. After graduating from UC Berkeley with a degree in political science, she joined her father’s documentary film production company. She and her father made several films and participated in film festivals.
As her father was preparing to retire, Monica looked for new opportunities and challenges. By chance, she was connected with Aaron Bell, founder of AdRoll Group, who was then just building out the company. She was hired to help manage the expansion of the AdRoll Group office. In true startup fashion, Monica also worked the front desk in addition to her other responsibilities. As AdRoll grew, so did Monica’s role, evolving into larger project management duties for the sales team. Through her work with the sales team, she saw a need for and moved her career in the direction of analytical support. Today, Monica works on the Analytics team, providing insights that drive decision making throughout AdRoll Group.
Advice from Monica:
Monica encourages anyone interested in STEM careers to reach out and find a support system. She notes that she was able to find good mentors that went above and beyond their day jobs to provide guidance and support. Prior to that, Monica wasn’t aware of the variety of STEM careers available. Today, Monica participates in Tech Women at AdRoll Group to help others learn and grow in their tech careers.
<h1 id="farah-khan">Farah Khan</h1>
Senior Solutions Engineer
<img src="/images/post_images/tech-women-farah-khan.jpg" alt="Farah Khan" />
When she was a kid, Farah wanted to be President of the United States. She would often ask her friends if they’d vote for her (they said yes!). As she grew up, her interests turned toward a career at the FBI. At the University of Maryland, she earned a degree in criminology, with a minor in Arabic. But it wasn’t until she took an Oracle DBA class her senior year that she considered a STEM career.
That class on databases ended up being a key moment in her career. After writing her first SQL statement, she was hooked. After graduating from UMD, she worked as a QA engineer and loved the challenge of breaking software. Farah went on to earn a master’s degree in systems engineering. While working on her master’s degree, she also started working as a systems engineer at Dell EMC, leveraging her interpersonal skills with a systems thinking approach to onboard clients and explain the value of internal tech.
After Dell EMC, Farah started her own clothing line out of a tiny apartment in NYC, and knew she needed to reach more customers to help them find modest fashion wear. Not knowing anything about marketing, some googling led her to find AdRoll. She wanted to combine her SE background with newfound knowledge about small business marketing challenges, helping other clients on their journeys.
Farah now works as a senior solutions engineer at AdRoll Group. At AdRoll, she appreciates not only having mentors but also sponsors, those in power who have advocated for her and given her the chance to try new things. All ideas are entertained, and the resources to execute those ideas made available.
Farah says that solving challenging problems is the most interesting part of her STEM career. “These tough problems keep engineers humble,” she says, “and these problems can range from building scalable processes to serve customers more efficiently to making internal tools more accessible to end users.”
Advice from Farah:
To those interested in STEM careers as engineers, she recommends embracing the imposter syndrome: it means you’re trying new things and growing. A few suggestions from Farah: avoid “analysis paralysis,” go out and do things, and don’t be afraid to get started even if it’s small.
<h1 id="sadie-wilhelm">Sadie Wilhelm</h1>
Senior Software Engineer
<img src="/images/post_images/tech-women-sadie-wilhelm.jpg" alt="Sadie Wilhelm" />
When she was growing up, Sadie always wanted to be a math teacher. She loved math and was encouraged by her mother, a CPA, to pursue a STEM career. Along the way, she had fantastic teachers and not so fantastic teachers. She saw the difference a great math teacher could have on young minds. She went on to become a math teacher, then ran a tutoring company for several years.
Sadie loved working with her students but yearned for larger challenges. Her then boyfriend, now husband, encouraged her to try some online coding courses. It was an instant hit: Sadie knew she would figure out how to be an engineer. Sadie began by learning SQL and was eventually hired as part of the Business Intelligence team at AdRoll Group. Once on the team, she began to automate parts of her job with Python. Wanting to go further, Sadie took a three-month hiatus from work to attend the Hackbright Academy coding boot camp. After Hackbright, Sadie returned to AdRoll Group as an engineer for several years before leaving to try out another company. Nine months later, Sadie came back to work as an engineer on other systems.
As a working mother, Sadie has found her experience at AdRoll Group to be better than those of her peers at other companies. After returning from maternity leave, Sadie took advantage of flexible hours and was encouraged to leave early for daycare pickup. The message was, “We trust you,” Sadie says. She continues, “AdRoll Group trusted me to get my work done, and in turn, I trusted AdRoll Group to support me while I took care of my family and did my job well.”
Advice from Sadie:
Sadie’s favorite part of her engineering career is that she’s never bored. Working on a variety of projects using a plethora of new technologies keeps her engaged. For those interested in STEM careers, she suggests embracing failure instead of fearing it. Her mother encouraged her to study engineering in college, but the fear of failure kept her from pursuing it. Sadie faces failure every single day and says that it’s from failure that she learns, improves, and grows.
Want more profiles of Tech Women? Read our <a href="http://tech.adroll.com/blog/culture/2018/03/22/tech-women-of-adroll-group-part-3.html">previous entry</a>.
Interested in working with these and other talented women? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>
Spot The Discrepancies with Dialyzer for Erlang
</title>
<link>https://tech.nextroll.com/blog/dev/2019/02/19/erlang-dialyzer.html</link>
<pubDate>Tue, 19 Feb 2019 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2019/02/19/erlang-dialyzer</guid>
<description>Dialyzer is a great tool to validate Erlang code, but it might slow down your development process if devs are applying it to huge codebases constantly. Particularly if that code was never analyzed with it.
This article is our answer to the big question: How to start using dialyzer in a huge project where it was never applied before?
<code class="language-plaintext highlighter-rouge">10-15 minute read</code>
<hr />
Continuing with our series of articles about the usage of <a href="http://www.erlang.org/">Erlang/OTP</a> to build our <a href="/blog/web/2014/04/29/valentino-presents-adrolls-rtb-infrastructure.html">real-time bidding platform</a>, we would like to show you now how we added <a href="http://erlang.org/doc/man/dialyzer.html">Dialyzer</a> to our CI pipelines.
The main Erlang application for our Real-Time Bidding servers was created way before <a href="https://rebar3.org">rebar3</a> existed. Performing the task equivalent to <code class="language-plaintext highlighter-rouge">rebar3 dialyzer</code> was not easy and it was also very time consuming.
We recently started using <code class="language-plaintext highlighter-rouge">rebar3</code> to manage our projects and, as part of that process, we decided to finally deal with that bit of technical debt.
Of course, it was not as easy as just run dialyzer on the code and remove all the warnings. When we started, we had approximately 1800 warnings to deal with, between our main repo and its dependencies. So, we tackled them incrementally. Let me walk you through our process…
<h1 id="introduction">Introduction</h1>
Before we begin, let’s talk a little bit more about Dialyzer and why it’s so important to use it.
<h3 id="what-is-dialyzer">What is Dialyzer?</h3>
Dialyzer is a discrepancy analyzer for Erlang/Elixir code. It checks your applications to find discrepancies such as definite type errors, code that has become dead or unreachable because of programming error, and unnecessary tests, among other things.
<h3 id="how-to-run-dialyzer">How to run Dialyzer?</h3>
Dialyzer can be run from command line (it’s one of the several tools that are shipped with Erlang/OTP) but these days it’s far more common to run it with rebar3, i.e. <code class="language-plaintext highlighter-rouge">rebar3 dialyzer</code>.
<h3 id="why-you-should-use-dialyzer">Why you should use Dialyzer?</h3>
Dialyzer will not warn you about all your errors, but if dialyzer emits a warning about something in your code, you can be sure that there is a bug there. You’ll see more on that bug finding stuff in the paragraphs below.
And now… let’s go back to our story!
<h1 id="our-goal">Our Goal</h1>
<blockquote>
To be able to consistently reduce the number of discrepancies over time without altering our development speed.
</blockquote>
<h1 id="metrics">Metrics</h1>
With that goal in mind, we established a plan of attack and, since you can’t improve what you can’t measure, our first step was to instrument the number of dialyzer warnings so we could keep an eye on it and, hopefully, watch it go down to zero eventually.
We use <a href="https://www.datadoghq.com/">Datadog</a> for our real-time metrics. In what may be considered a severe misuse of this tool, we decided to write a simple bash script using the datadog agent to report the number of warnings found by dialyzer. The idea was to provide a nice way to visualize our progress relative to our goal stated above. But an important question popped up: how do we find the number of warnings?
<h4 id="the-warnings-file">The Warnings File</h4>
Luckily for us <code class="language-plaintext highlighter-rouge">rebar3 dialyzer</code> generates a file with the list of warnings, called <code class="language-plaintext highlighter-rouge">_build/${REBAR_PROFILE}/${OTP-VERSION}.dialyzer_warnings</code> that looks like this:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">/path/to/your_module.erl:371: The call your_other_module:your_function...
/path/to/your_module.erl:470: The call another_one:thing(Name::[any(),...</code></pre></figure>
So, this is what our instrumentation script looks like:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">#!/bin/bash
WARNINGS="code_quality.dialyzer.discrepancies:$(sed '/^$/d' _build/default/*.dialyzer_warnings | wc -l | tr -d '[:space:]')|g|#shell"
echo ${WARNINGS} | nc -4u -w0 localhost 8125</code></pre></figure>
Let’s read that backwards: the script reports the number of dialyzer discrepancies by echoing the contents of the environment variable <code class="language-plaintext highlighter-rouge">WARNINGS</code> to the datadog agent listening on port 8125 on our machine. The datadog agent in turn ships it over to datadog servers. The contents of the <code class="language-plaintext highlighter-rouge">WARNING</code> variable are simply the name of the metric (<code class="language-plaintext highlighter-rouge">code_quality.dialyzer.discrepancies</code>), following by the count.
The count is determined by reading the <code class="language-plaintext highlighter-rouge">.dialyzer_warnings</code> file, using <code class="language-plaintext highlighter-rouge">sed</code> to remove empty lines, piping that to wc to count lines and cleaning up wc’s output of spaces with <code class="language-plaintext highlighter-rouge">tr</code>.
<h4 id="the-number-of-warnings">The Number of Warnings</h4>
We added that to our Makefile target for dialyzer, as follows:
<figure class="highlight"><pre><code class="language-makefile" data-lang="makefile">dialyzer: ## Check the project with dialyzer
@$(REBAR3) dialyzer || ./scripts/instrument_dialyzer.sh</code></pre></figure>
But then, when we started watching that metric, we noticed something odd…
<img src="/images/post_images/dialyzer/datadog_spikes.png" alt="Datadog Spikes" />
The numbers seemed right in general, but what about those odd spikes every now and then? Turns out, they were generated when running dialyzer for the first time (i.e. on a clean clone of the project). That’s because, when <code class="language-plaintext highlighter-rouge">rebar3 dialyzer</code> runs for the first time for a project, it generates the <a href="http://erlang.org/doc/apps/dialyzer/dialyzer_chapter.html#the-persistent-lookup-table">persistent lookup table (PLT)</a> including the project dependencies and those dependencies generate some dialyzer warnings of their own. Those warnings are not generated again once you already have a plt, so the get the actual number of discrepancies we want for our main project, we need to run <code class="language-plaintext highlighter-rouge">rebar3 dialyzer</code> once the PLT is already generated. That’s why our Makefile target actually looks like the following.
<figure class="highlight"><pre><code class="language-makefile" data-lang="makefile">dialyzer: ## Check the project with dialyzer
@$(REBAR3) dialyzer || $(REBAR3) dialyzer || ./scripts/instrument_dialyzer.sh</code></pre></figure>
<h1 id="organizing-the-work">Organizing the Work</h1>
Once we had that metric in place, we now wanted to see it go down and to achieve that, our first idea was to write tickets to fix the warnings. But removing all 1800 warnings together was, of course, impossible. Can you imagine reviewing such a massive pull request?
That’s why, using the <code class="language-plaintext highlighter-rouge">dialyzer_warnings</code> file again, we decided to turn it into tickets. We use <a href="https://www.atlassian.com/software/jira">JIRA</a> to organize our work, and it comes with a handy <a href="https://marketplace.atlassian.com/apps/6398/jira-command-line-interface-cli">CLI</a> that we used to write an escript that groups warnings and creates a ticket for each module. I won’t paste the whole script here, but the meaty part is this:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">build_command({Mod, Msg}) -&gt;
binary_to_list(iolist_to_binary([
"jira create "
"--noedit "
"--override='components:boodah' "
"--override='summary: There are ",
integer_to_list(length(Msg)),
" dialyzer warnings on ", Mod, "' "
"--override='description: The warnings reported"
" at the time of this writing are:\n"
" {code:Erlang} \n", Msg, " {code}'"
])).
main([]) -&gt;
...
{ok, Output} = file:read_file(WarningFile),
Warnings =
group_by_mod(
lists:filtermap(
fun parse_line/1,
binary:split(Output, &lt;&lt;"\n"&gt;&gt;, [global, trim]
)
)
),
lists:foreach(
fun run_command/1,
lists:map(fun build_command/1, Warnings)),
io:format("DONE!~n").</code></pre></figure>
<h1 id="keeping-discrepancies-at-bay">Keeping Discrepancies at Bay</h1>
Of course, we didn’t stop the development process to put everyone to work on the tickets generated with that script.
Well, actually…
<h4 id="hackweek">HackWeek!</h4>
One of the things that all rollers enjoy working here are unquestionably the <a href="https://blog.adroll.com/news/adroll-named-sf-business-times-best-places-to-work-for-the-fourth-time">HackWeeks</a>. And during the last one of 2018, a team of 4 developers decided to remove as many dialyzer warnings as possible from our code.
And removing warnings… we did! From the original 1800 we went down to a whooping 300 without affecting performance nor functionality in the slightest!!
Oh, and in that process, we eliminated tons of dead code and fixed 11 bugs that were still undetected by our tests!!
But after that week, we wanted to ensure that we didn’t start adding new discrepancies as we moved on.
<h4 id="ci-additions">CI Additions</h4>
We didn’t want to require our developers to run dialyzer each time (although we strongly recommended it) since our original goal explicitely included not altering our development speed. That’s why we added dialyzer to our CI instead!
But, of course, we still had 300 warnings. We couldn’t require a clean run of dialyzer for each pull request. What we decided instead was to reject PRs if they included new dialyzer warnigns.
We use <a href="https://buildkite.com">Buildkite</a> for CI, so (using <code class="language-plaintext highlighter-rouge">*.dialyzer_warnings</code> once more) we added a new pipeline that generates a normalized warnings list each time anything is pushed to <code class="language-plaintext highlighter-rouge">master</code>:
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml">- label: 'Run dialyzer and upload the resulting list of warnings to s3'
command:
- make harsh_clean
- make dialyzer
- normalize_warnings.sh _build/default/*.dialyzer_warnings master.dialyzer_warnings
- aws s3 cp master.dialyzer_warnings s3://...</code></pre></figure>
And that <code class="language-plaintext highlighter-rouge">normalize_warnings.sh</code> script? It looks like this:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">#!/bin/bash
set -e
cp $1 pre.dw
sed -e "s|`pwd`||g" pre.dw | sed -e '/^$/d' | sort &gt; $2
rm pre.dw</code></pre></figure>
Basically, three steps:
<ol>
<li>Remove the current folder from the paths in the file</li>
<li>Remove empty lines</li>
<li>Sort them</li>
</ol>
Once we had that in place, we only needed to update our existing PR verification pipeline to generate, normalize and compare its list of warnings with the one from <code class="language-plaintext highlighter-rouge">master</code>. We did that by extending our already existing <code class="language-plaintext highlighter-rouge">make dialyzer</code> as follows and verifying that <code class="language-plaintext highlighter-rouge">new.dialyzer_warnings</code> is empty on Buildkite.
<figure class="highlight"><pre><code class="language-makefile" data-lang="makefile">dialyzerfast: ## Check the project with dialyzer
@$(REBAR3) dialyzer || $(REBAR3) dialyzer || ./scripts/instrument_dialyzer.sh
new_dialyzer_warnings: ## Verifies if there are new dialyzer warnings
normalize_warnings.sh _build/default/*.dialyzer_warnings branch.dialyzer_warnings
@aws s3 cp s3://.../master.dialyzer_warnings .
comm -2 -3 branch.dialyzer_warnings master.dialyzer_warnings &gt; new.dialyzer_warnings
dialyzer: ## Check the project with dialyzer and report new warnings
dialyzer: dialyzerfast new_dialyzer_warnings</code></pre></figure>
<h4 id="caveat">Caveat</h4>
Keen-eyed folks may notice that since we’re comparing files with line-numbers in them, we’re not exactly detecting just new warnings. If your changes alter the line numbers of lines with existing warnings, they will be reported as new ones.
When we noticed that we decided that it was fair. The rule was: If you modify a module with dialyzer warnings, you should fix them as part of your PR, too.
<h1 id="caching-plts">Caching PLTs</h1>
Now that we had added dialyzer to our CI we had not slowed down our development time when working on our computers but we had still slowed down our CI times considerably. So, to regain that lost speed, we needed to avoid recompiling the PLTs in each run. Luckily for us, <code class="language-plaintext highlighter-rouge">rebar3</code> places the PLTs in a very convenient and configurable location. So, we adjusted our buildkite pipelines like you can see below…
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml">...
- aws s3 sync s3://.../ . --exclude "*" --include "*plt" --no-follow-symlinks
- make dialyzer
- aws s3 sync . s3://.../ --exclude "*" --include "*plt" --no-follow-symlinks
...</code></pre></figure>
We basically synched our plts with s3 each time. And that was it! CI runs fast as usual, detects new warnings and we all keep improving our code constantly until we eventually get to 0 warnings.
Look, this is our progress last week…
<img src="/images/post_images/dialyzer/datadog_week.png" alt="Datadog Week" />
We went from 300 to 225 in just a week! And the most important part is that we effortlessly included dialyzer as part of our development process forever, thus increasing the quality of our code significantly.
I hope this story inspires you to do the same in your project and be as happy and as proud of your code as we are of ours :)
<hr />
Do you enjoy building high-quality large-scale systems? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>
AdRoll Group Loves the Grace Hopper Conference
</title>
<link>https://tech.nextroll.com/blog/culture/2019/02/14/grace-hopper-recap.html</link>
<pubDate>Thu, 14 Feb 2019 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/culture/2019/02/14/grace-hopper-recap</guid>
<description>Grace Hopper is happening on October 2-4 2019 and
the <a href="https://ghc.anitab.org/calendar/2019-grace-hopper-celebration/">Call for Participation, Abi Award nominations, and Scolarship applications are now open</a>!
Here is why we love Grace Hopper.
<code class="language-plaintext highlighter-rouge">10-15 minute read</code>
<hr />
In September 2018, sixteen Rollers joined around 20,000 attendees from all over the
world for Grace Hopper, the largest gathering of women in computing. Although women
make up more than half of the U.S. workforce, we make up less than 20 percent of the
tech jobs, which makes this conference extra magical to the women in attendance.
<img src="/images/post_images/grace_hopper_recap/booth.png" alt="Adroll Group recruiting table" />
As Julie Zhou, our Director of Product Management described it, “I found it inspiring to be amongst
so many ambitious and talented women in one place…many people have a picture in their mind
of what a tech conference looks like and that picture tends to have lots of young, white men there.”
<img src="/images/post_images/grace_hopper_recap/wanted.png" alt="Rubi, Farah, Julie and Andrew" />
Although there are plenty of opportunities to network, Grace Hopper is also an opportunity to learn.
There were ten technical tracks that covered topics all the way from product to Computer Systems
Engineering and Computer Science. Jessica Grist, Sr. Software Engineer at AdRoll Group, said she “really
liked that there was the option of a 20-minute quick introduction to a specific topic or a longer 1-hour
deep dive and that an in-depth technical talk could coexist with work-life balance talks. More than
anything, she loved the recruiting and interviewing efforts, focused on attracting and networking
with women in technology.”
<img src="/images/post_images/grace_hopper_recap/workshop.png" alt="Miriam in a workshop" />
In the conference’s spirit to get and keep women in the computing workforce, we all happily spent
countless hours chatting with hundreds of women interested in careers across data science,
engineering, and product. “It was inspiring to see so much female talent in one place…
Also special for me was that my mom was there as an engineering lead at her company (UBS
Financial Services)… Her presence gave me a better appreciation for the women from the previous
generation,” said Kelly Eng, Sr. Product Manager at AdRoll Group.
<img src="/images/post_images/grace_hopper_recap/team.png" alt="team2" />
We also appreciated the men that came to represent AdRoll with us! Patrick, our SVP of Engineering,
“was excited by all their enthusiasm and hoped he could make this a more welcoming industry.”
Attending a conference where men were the stark minority resulted in many of them empathizing
with our experience as women in most technology conferences.
<img src="/images/post_images/grace_hopper_recap/team2.png" alt="team2" />
Overall, the enthusiasm during our recruiting efforts by our entire team throughout the
conference was a sincere reflection of the commitment to diversity and inclusion that I
have happily seen continuously nurtured throughout my year here at AdRoll Group.
As if that wasn’t exciting enough, we successfully hired Kruti Chauhan, who shared,
“meeting with the Adroll Group Team, I learned about their support for women in their
engineering teams… it was indeed a celebration to bag an offer from AdRoll.”
I can’t wait to continue to see the fruition of our investment in diversity come thrive this year!
<hr />
Do you enjoy working on a company that cares about a diverse and inclusive world? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>
Managing DynamoDB Autoscaling with Lambda and Cloudwatch
</title>
<link>https://tech.nextroll.com/blog/dev/2019/02/05/dynamodb-managed-autoscaling.html</link>
<pubDate>Tue, 05 Feb 2019 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2019/02/05/dynamodb-managed-autoscaling</guid>
<description>Autoscaling was a great addition to DynamoDB and it lets you forget about assigned capacity. Here’s how we implemented our own algorithm to improve on this idea.
<code class="language-plaintext highlighter-rouge">10-15 minute read</code>
<hr />
DynamoDB is a NoSQL key-value database service provided by AWS. We use it at the core of many of our systems, since the consistent read latency is awesome for realtime applications. If you are not entirely familiar with it, it might be worth refreshing the basics with the <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html">AWS Documentation</a>
One of the first things you learn as a new DynamoDB user is that you have to manage your table capacity in order to keep both your needs and costs in check, which can be harder than it sounds.
DynamoDB tables and indexes offer 2 core metrics that you can use to achieve this: provisioned and consumed capacity.
Initially, the only way around this problem was to assign the capacity manually, based on experience and traffic. If your traffic varied, you ended up having some margin to absorb variations, which leads to wasted capacity.
Eventually, DynamoDB ended up introducing an autoscaling feature, which lets you set a relation of consumed to provisioned capacity, up to 70%. This means that if you set your table for 70% and you consume 7 units, you’ll get 10 provisioned units.
<h2 id="problems-we-faced-with-the-default-autoscaling-algorithm">Problems we faced with the default autoscaling algorithm</h2>
The default autoscaling algorithm provided by AWS works by setting up a series of alarms that trigger if the capacity is above the defined rate for more than 5 minutes (please visit <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html">the docs for autoscaling</a> if you are interested in the details).
<img src="/images/post_images/dynamodb_autoscaling/autoscaling_graph.png" alt="Stock Autoscaling Architecture" />
This poses a problem for an application with varying and bursty workload: the table will scale up only based on consumption, triggering these alarms time after time, until it reaches the desired level. Ideally, the table should scale based on the number of requests that we are making , not the number of requests that are successful.
Additionally, at the time of implementing this algorithm, the DynamoDB capacity could not be brought down automatically if the consumption was exactly zero, which can happen if you write to your table in batch instead of realtime, for example.
From the AWS docs:
`
Currently, Auto Scaling does not scale down your provisioned capacity if your table’s consumed capacity becomes zero.
As a workaround, you can send requests to the table until Auto Scaling scales down to the minimum capacity,
or change the policy to reduce the maximum provisioned capacity to be the same as the minimum provisioned capacity.
`
This meant that, when enabling autoscaling, tables that were read in realtime, but written to in batch, still needed manual intervention to bring the write capacity down after our jobs were done writing.
Gladly, the DynamoDB team has recently fixed this issue, so at the time of writing, tables now downscale on their own.
DynamoDB tables also have a hidden reserved burst capacity metric, which can be consumed to absorb traffic spikes (but it’s also at the disposal of DynamoDB for internal operations. More on that <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-design.html#bp-partition-key-throughput-bursting">here</a>).
Another interesting point that might bite users is that capacity decreases are an expensive operation for AWS, so they’re limited.
From the AWS docs:
`
For every table and global secondary index in an UpdateTable operation, you can decrease ReadCapacityUnits or WriteCapacityUnits (or both).
The new settings do not take effect until the UpdateTable operation is complete. A decrease is allowed up to four times any time per day.
A day is defined according to the GMT time zone. Additionally, if there was no decrease in the past hour, an additional decrease is allowed, effectively bringing the maximum number of decreases in a day to 27 times (4 decreases in the first hour, and 1 decrease for each of the subsequent 1-hour windows in a day)
`
The number of decreases cited in the documentation can be achieved under very special conditions, since you need to have 4 decreases in the first hour of the day plus one for each of the remaining hours, for a total of 4 (first hour) + 23 (1 hourly) = 27. As you can imagine, getting to this number is rare and not entirely efficient.
You can read more about these limits <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-decreasing-provisioned-throughput">here</a>
<h2 id="initial-approach">Initial approach</h2>
Our initial approach was to manage capacity as fixed steps for batch loads:
<ul>
<li>
Fixed step for read capacity
<img src="/images/post_images/dynamodb_autoscaling/autoscaling_initial_read.png" alt="Initial Read Approach" />
</li>
<li>
Fixed step for write capacity
<img src="/images/post_images/dynamodb_autoscaling/autoscaling_initial_write.png" alt="Initial Write Approach" />
</li>
</ul>
While this is better than keeping the capacity up the whole time, you’ll realize that a lot of the time the assigned capacity is greater than needed and that it doesn’t go down as fast as it should.
This led to an acceptable performance for the batch writes, but at the cost of limiting our options on when to run the jobs. As we started adding more jobs to the mix that either wrote to or read from these tables, we started to need this to be managed automatically.
<h2 id="helping-the-autoscaling-algorithm">Helping the autoscaling algorithm</h2>
We really wanted to use autoscaling for these tables, as the step increase was too wasteful.
We came up with an idea to add a lambda function that would run every 5 minutes to check if tables were on the “0 capacity consumed state” for at least a few minutes, and bring the capacity down if that was the case.
We were finally able to use the stock algorithm for these tables at this point, which led to a nice cost drop.
<img src="/images/post_images/dynamodb_autoscaling/autoscaling_manager_graph.png" alt="Autoscaling lambda Architecture" />
However, we still had the issue of tables rising capacity very slowly and not really paying attention to the amount of requests that were being rejected.
This caused some of our operations (for example a full daily scan, or millions of requests for a batch job) to take hours of EMR/EC2 time.
<h2 id="replacing-the-autoscaling-algorithm">Replacing the autoscaling algorithm</h2>
Finally, building upon the lambda idea, we decided to replace the autoscaling algorithm entirely.
We designed a new version of the autoscaling lambda that uses tags and consumed and provisioned metrics like the last one, but that adds a new type of metric: the throttled requests.
Under normal conditions the algorithm works exactly the same as the normal autoscaling algorithm.
When throttling is present, the response of the algorithm is to increase capacity more aggressively, at fixed steps (we plan to make this proportional soon!).
Additionally, this algorithm works differently in the case of increases and decreases, which plays a little bit better with the limited decreases during the day.
While 27 decreases seems like a lot, the stock algorithm will lower the capacity as soon as the threshold is met, which can happen several times over the course of a few hours.
<ul>
<li>Managed autoscaling for read capacity and how it relates to throttling</li>
</ul>
<img src="/images/post_images/dynamodb_autoscaling/autoscaling_read.png" alt="Managed autoscaling read" />
<img src="/images/post_images/dynamodb_autoscaling/autoscaling_read_throttling.png" alt="Managed autoscaling read throttling" />
<ul>
<li>Managed autoscaling for write capacity and how it relates to throttling</li>
</ul>
<img src="/images/post_images/dynamodb_autoscaling/autoscaling_write.png" alt="Managed autoscaling write" />
<img src="/images/post_images/dynamodb_autoscaling/autoscaling_write_throttling.png" alt="Managed autoscaling write throttling" />
Here is where we detected our costs for our batch tables dropping to around 30% of the initial cost.
<h2 id="example-algorithm">Example algorithm</h2>
The algorithm we are implementing for the autoscaling lambda is fairly simple and written in Python 3, using <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/index.html">boto3</a>.
First, we use the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/resourcegroupstaggingapi.html">tagging client</a> to figure out which tables we want to target:
<figure class="highlight"><pre><code class="language-python" data-lang="python">response = resource_groups_tagging_client.get_resources(
TagFilters=[{
'Key': 'autoscaling_manager_enabled', 'Values': ['true']
}],
ResourceTypeFilters=['dynamodb']
)
table_names = ([item['ResourceARN'].split('/')[1]
for item in response['ResourceTagMappingList']]
if response['ResourceTagMappingList'] else [])</code></pre></figure>
The tag system could be used to manage other parameters as well, for example max or min settings.
Then, we make sure we are not stepping on a table that is using the AWS algorithm instead, using the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/autoscaling.html">application autoscaling client</a>:
<figure class="highlight"><pre><code class="language-python" data-lang="python">response = autoscaling_client.describe_scaling_policies(
ServiceNamespace='dynamodb',
ResourceId='table/{}'.format(table_name))
is_autoscaling_enabled = bool(response['ScalingPolicies'])</code></pre></figure>
Once we have the list of tables, we retrieve their definitions and current setup from the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html">dynamo client</a>:
<figure class="highlight"><pre><code class="language-python" data-lang="python">table_definition_and_settings = dynamo_client.describe_table(
TableName=table_name)['Table']</code></pre></figure>
Up to this point, we already know the structure of the table and its currently provisioned capacity, so we use the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudwatch.html">cloudwatch client</a> to retrieve historic data from the past minutes about how the table is being used and how many requests have been throttled:
<figure class="highlight"><pre><code class="language-python" data-lang="python">end_date = datetime.datetime.utcnow().replace(second=0,
microsecond=0)
start_date = end_date - datetime.timedelta(minutes=5)
metric = cloudwatch_resource.Metric('AWS/DynamoDB',
metric_name)
dimensions = [{'Name': 'TableName', 'Value': self.table_name}, ]
response = metric.get_statistics(
Dimensions=dimensions,
StartTime=start_date,
EndTime=end_date,
Period=60,
Statistics=['Sum'],
)
data_points = {
calendar.timegm(data_point['Timestamp'].timetuple()): data_point
for data_point in response['Datapoints']}
metric_report = []
current_date = start_date
while current_date &lt; end_date:
current_timestamp = calendar.timegm(
current_date.timetuple())
sum_value = data_points.get(
current_timestamp, {}).get('Sum', 0)
metric_report.append({
'timestamp': current_timestamp,
'value': (sum_value / 60
if metric_name
in CAPACITY_METRICS else sum_value)
})
current_date += datetime.timedelta(minutes=1)</code></pre></figure>
Some metrics are averaged per minute while others are absolute values, it took some experimenting to figure out how to match them to what we see in the DynamoDB UI.
It’s also worth mentioning that we needed to fill the metrics that have a 0 value for consistency. There’s a chance that this is related to the issue AWS faced when decreasing the capacity for tables without consumption.
Finally, we run a simple check over the collected metrics to decide if we need to change the table capacity. Keep in mind this is a simplified example that operates only on read capacity for the table:
<figure class="highlight"><pre><code class="language-python" data-lang="python">currently_provisioned = current_settings['ReadCapacityUnits']
throttling_metrics = usage_metrics['ReadThrottleEvents']
average_throttling = (
sum(record['value'] for record in throttling_metrics) /
len(throttling_metrics))
consumed_metrics = usage_metrics['ConsumedReadCapacityUnits']
average_consumed = (
sum(record['value'] for record in consumed_metrics) /
len(consumed_metrics))
utilization = average_consumed / currently_provisioned # This is the target utilization defined by the stock algorithm
can_increase_capacity = currently_provisioned &lt; MAX_READ_CAPACITY # We set a cap to keep the cost in check
target_provisioning = int(average_consumed * (1 / self.TARGET_UTILIZATION)) # Our target utilization is 0.8, higher than the original max value
min_provisioning = 5 # Arbitrary minimum

if average_throttling and can_increase_capacity: # While throttling, we do quantized increases, with 2 fixed rates
 new_capacity_settings = (currently_provisioned +
(self.HIGH_THROTTLING_CAPACITY_INCREASE
if average_throttling &gt;= self.THROTTLING_THRESHOLD
else self.LOW_THROTTLING_CAPACITY_INCREASE))
elif currently_provisioned &lt; min_provisioning or not average_consumed:
new_capacity_settings = min_provisioning # If we are below min or nothing is using this table
elif utilization &gt; self.TARGET_UTILIZATION and can_increase_capacity:
new_capacity_settings = target_provisioning # This means we are a bit tight, so we increase to match 0.8
elif not average_throttling and utilization &lt; self.BUFFERED_TARGET_UTILIZATION:
new_capacity_settings = target_provisioning # This means we are wasting some capacity, so we decrease it

if (new_capacity_settings and
new_capacity_settings != currently_provisioned):
# Proceed to update the table with the new settings
else:
# Skip updating this table</code></pre></figure>
While this code might be a little long to read, it’s pretty straightforward: it works exactly like the stock DynamoDB algorithm but it will increase the capacity aggressively by a fixed amount when there’s throttling involved.
Please check this <a href="https://gist.github.com/alvarotuso/589ea02dfa5823328e0f1e356563b8a1">gist</a> for the complete implementation, including write capacity and GSIs support!
Keep in mind that the fixed steps for throttling suited our needs, but a proportional approach could be used with any table workload. Another improvement would be to support multi region tables (right now, most of our tables are in one region).
If you want to give it a try, you’ll have to create a <a href="https://docs.aws.amazon.com/lambda/latest/dg/welcome.html">lambda function</a> with the code from the gist, and set up a <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Scheduled-Rule.html">cloudwatch rule</a> to trigger the function periodically, every 5 minutes.
You should make sure that the role you use in the lambda has the following permissions enabled:
<ul>
<li>Read / Write permissions on the DynamoDB tables you want to test</li>
<li>Read permissions on Cloudwatch</li>
<li>Read permissions on Application Autoscaling</li>
<li>Read permissions on Resource Groups Tagging</li>
</ul>
Once you’ve set everything up, the only missing piece is adding the <code class="language-plaintext highlighter-rouge">autoscaling_manager_enabled=true</code> tag to your table.
You should be able to monitor the execution of your lambda and the status of the DynamoDB table through the AWS console.
<h2 id="about-on-demand-mode">About On-Demand mode</h2>
On-Demand mode is a new addition to AWS that might help ease the pain of managing table capacity. The premise is that the user is now able to pay per request on unknown loads.
While this sounds like a very good solution to the same problem, the pricing scheme is different and there’s currently no way to reserve this capacity.
<hr />
Do you enjoy building high-quality large-scale systems? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>Questioning Second-Price Auctions in Ad Tech</title>
<link>https://tech.nextroll.com/blog/data-science/2018/12/21/second-price-auctions.html</link>
<pubDate>Fri, 21 Dec 2018 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data-science/2018/12/21/second-price-auctions</guid>
<description><h1 id="introduction">Introduction</h1>
A primary responsibility of the data science team at the AdRoll Group is to determine what we should bid in real-time
auctions for ad inventory. Among other things, our bid price is dependent on the type of auction in which we are
participating.
Second-price auctions, where the highest bidder wins and pays the second highest bid price, are popular in ad tech
because they incentivize bidders, such as AdRoll, to bid their true value for the ad impression being auctioned
<a href="#footnotes">(1)</a>. First-price auctions, where the highest bidder wins and pays their bid, are also popular,
but they incentivize bidders to reduce their bids below their valuation. Exchanges communicate the auction type to bidders,
like us, in the bid request. From the perspective of ad exchanges, there exists a (shortsighted) incentive to make bidders
believe they are bidding into second price auctions, so they bid their true valuation, while charging them something more
than the second price.
In this post, we consider only auctions in which we are explicitly told the auction is second-price. We will examine how our
clearing prices compare to our bid prices in these second-price auctions, and we will observe some irregular patterns that
lead us to question whether these auctions really are second-price. We will then discuss the algorithmic defense against
misspecified auction mechanisms that we’ve developed at the AdRoll Group.
<h1 id="questioning-auction-mechanisms">Questioning Auction Mechanisms</h1>
In this section we will examine how our clearing prices compare to our bid prices in what are supposed to
be second-price auctions on four exchanges, each displaying a unique relationship between clearing prices and bid
prices. We will do this in two ways, and in both cases we will observe some questionable patterns. First, we will graphically
examine how clearing prices are related to the bid prices in second-price auctions. Second, we will compare the empirical
utility we gain from second-price auctions to theoretical calculations.
<h2 id="graphical-analysis-of-second-price-auctions">Graphical Analysis of Second-Price Auctions</h2>
In this subsection we will look at histograms of the ratios of our clearing price to our bid price, and scatter plots
of our clearing prices as a function of our bid prices.
<h4 id="exchange-a">Exchange A</h4>
We begin with an exchange that we believe runs true second-price auctions, for the most part. One reason we believe this is
that we regularly win impressions for a small fraction of our bid price, as you can see in the plots below. The only odd
feature in these plots is that we sometimes pay what we bid, but compared to other exchanges this effect is very mild.
<table>
<tbody>
<tr>
<td><img src="/images/post_images/second_price_utility/hist_exchange_A.png" alt="" /></td>
<td><img src="/images/post_images/second_price_utility/scatter_exchange_A.png" alt="" /></td>
</tr>
</tbody>
</table>
<h4 id="exchange-b">Exchange B</h4>
Next, we analyze an exchange whose second-price auctions we have reason to question. On this exchange we almost never win
impressions for a small fraction of our bid price. Most of the time we end up paying exactly what we bid. In fact, these
auctions look more like first-price auctions than second-price auctions.
<table>
<tbody>
<tr>
<td><img src="/images/post_images/second_price_utility/hist_exchange_B.png" alt="" /></td>
<td><img src="/images/post_images/second_price_utility/scatter_exchange_B.png" alt="" /></td>
</tr>
</tbody>
</table>
<h3 id="exchange-c">Exchange C</h3>
Here is another exchange whose second-price auctions are irregular <a href="#footnotes">(2)</a>. On this exchange, as
with the previous exchange, we see that we are often charged exactly what we bid. Moreover, we are often charged
roughly 90% of our bid price. How can the second-price in an auction so frequently be a function of our bid price?
<table>
<tbody>
<tr>
<td><img src="/images/post_images/second_price_utility/hist_exchange_C.png" alt="" /></td>
<td><img src="/images/post_images/second_price_utility/scatter_exchange_C.png" alt="" /></td>
</tr>
</tbody>
</table>
<h4 id="exchange-d">Exchange D</h4>
This exchange does not exhibit the same strange patterns as “Exchange B” or “Exchange C”. Nonetheless, we almost never win
impressions for a small fraction of our bid price. Put another way, the bottom right corner of the scatter plot below is
conspicuously empty. Can this really arise from second price auctions?
<table>
<tbody>
<tr>
<td><img src="/images/post_images/second_price_utility/hist_exchange_D.png" alt="" /></td>
<td><img src="/images/post_images/second_price_utility/scatter_exchange_D.png" alt="" /></td>
</tr>
</tbody>
</table>
<h2 id="analysis-of-utility-in-second-price-auctions">Analysis of Utility in Second-Price Auctions</h2>
In this subsection we will analyze second-price auctions by comparing the utility we realize empirically to the utility we
theoretically expect to realize. Assume we are in a second-price auction, and let $M$ be a random variable representing
the highest bid among all other auction participants with PDF $f_M$ and CDF $F_M$. The fact that we are in a second-price
auction implies our bid $b$ should be our valuation $v$. We will win the auction when $M &lt; v$, and the utility we realize
when we win is $v-M$. Putting this together, the expected utility in second-price auctions $U_{SP}$ can be written as
<script type="math/tex; mode=display">
%<![CDATA[
U_{SP} = E\left[\mathbb{1}_{\{M<v\}}\left(v-M\right)\right]
%]]>
</script>
Chris Evans, a Staff Data Science Engineer at the AdRoll Group, worked out a very nice result for this quantity. Following
the derivation of the tail-sum formula for expectation, we have <a href="#footnotes">(3)</a>:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{align*}
U_{SP} &= \int_0^v (v-m) \ f_M(m) \ dm && \text{Definition of expectation.} \\
&= \int_0^v \left( \int_0^\infty \mathbb{1}_{\{(v-m)>x\}}\ dx \right) f_M(m) \ dm && \text{Rewrite with indicator.}\\
&= \int_0^\infty \left( \int_0^v \mathbb{1}_{\{(v-m)>x\}}\ f_M(m) \ dm \right) dx && \text{Exchange order of integration.}\\
&= \int_0^\infty F_M(v-x)\ dx && \text{Definition of CDF.}\\
&= \int_0^v F_M(x)\ dx && \text{Change of variables.}
\end{align*}
%]]>
</script>
This result enables us to predict the utility we will realize in a second-price auction given $F_M(x)$, information about
how the bids of other bidders are distributed. Fortunately, we had already developed an accurate model to predict $F_M(x)$
for other applications, as we will see in the next section. Now that we are able to calculate the predicted utility we can
compare it to the empirically realized utility. Below we can see a plot of the average utility empirically realized for
different (binned) values of predicted utility.
<img src="/images/post_images/second_price_utility/utility_in_second_price_auctions.png" alt="" />
As this figure shows, this calculation predicts we will realize much more utility than we do for the three exchanges where
we saw irregular auction mechanisms (namely, B, C, and D). On the other hand, this calculation comes fairly close to correctly
predicting the utility we will realize in the case of “Exchange A”, where the auction mechanisms look truly second-price.
This plot is another reason to question the auction mechanisms as communicated by some exchanges, and further highlights our
need for an algorithmic response.
<h1 id="responding-to-auction-mechanisms">Responding to Auction Mechanisms</h1>
In this section we will discuss the algorithmic defenses we have built in response to misspecified auction mechanisms.
As background, we will first discuss bidding into auctions with well-specified auction mechanisms.
<h2 id="bidding-into-specified-auction-mechanisms">Bidding into Specified Auction Mechanisms</h2>
Ideally, we would know the exact auction mechanism into which we are bidding because it affects our optimal bid. With knowledge
of the true auction mechanism, we’d have an understanding of how the cost $C$ of the impression varies as a function of our
bid. For example, in a first-price auction, where we pay what we bid, $C(b) = b$. Given $C$ we can compute the expected utility
for our bid as
<script type="math/tex; mode=display">
%<![CDATA[
U(b) = E\left[ \mathbb{1}_{\{M<b\}}\left(v - C(b) \right)\right] \\
%]]>
</script>
Ryoma Canastra, a Data Science Engineer at the AdRoll Group, proved that to maximize the value we provide to customers,
each bid we submit should be chosen to maximize $U$ <a href="#footnotes">(4)</a>. For example, in a first-price auction
our bid $b_{FP}$ for an impression we valued at $v$ dollars would be <a href="#footnotes">(5)</a>
<script type="math/tex; mode=display">
%<![CDATA[
\begin{align*}
b_{FP} &= \text{argmax}_{b} \ E\left[ \mathbb{1}_{\{M<b\}}\left(v - b \right)\right] \\
&= \text{argmax}_{b} \ F_M(b) \left(v - b \right)
\end{align*}
%]]>
</script>
<h2 id="bidding-into-misspecified-auction-mechanisms">Bidding into Misspecified Auction Mechanisms</h2>
Without knowledge of the auction mechanism we cannot bid in the theoretically optimal way. Instead, we have to resort
to a more empirical approach. The approach we’ve settled on is quite simple. When we aren’t confident in the auction mechanism,
we predict it.
Specifically, we predict $p_{FP}$, the probability the auction will result in us paying something close to first-price.
To simplify things, we assume there are only first and second price auctions, and train a classifier to differentiate these
cases. The more confident we are in having to pay a large fraction of our bid price, the more we reduce our bid. In particular,
we bid
<script type="math/tex; mode=display">
%<![CDATA[
\begin{align*}
b &= p_{FP} \ b_{FP} + (1 - p_{FP}) \ v
\end{align*}
%]]>
</script>
This approach has a few nice properties. First, this approach reduces to optimal behavior when we are confident in the auction
mechanism. For example, when we are confident that we are in a first price auction ($p_{FP} = 1$) we will bid $b_{FP}$, which
is optimal. Similarly, when we are confident that we are in a second-price auction ($p_{FP} = 0$) we will bid $v$, which is
also optimal. Second, this approach learns and updates our bids as auction mechanisms change. Third, this approach is scalable,
requiring minimal human intervention. Most important, we like this approach because it has performed well in A/B tests, capturing
significantly more utility for us on behalf of our customers than submitting the optimal bid for the auction type passed by
exchanges.
<h1 id="conclusion">Conclusion</h1>
In the first section of this post we saw evidence that second-price auctions in ad tech may not always be purely second-price.
First, we saw irregularities in plots relating our clearing prices to our bid prices in what were supposed to be second-price
auctions. Next, we saw that the utility we realize in the questionable second-price auctions is much less than theory would
predict.
That said, in some cases there may be harmless explanations for the observations of this blog post. It’s possible that header
bidding or different bid densities in the different exchanges could cause qualitatively different clearing price vs. bid price
behavior. Or, perhaps we are miscommunicating with certain exchanges about the type of auctions in which we are participating.
Nonetheless, we have decided to defend against the possibility of misspecified auction mechanisms. In the second section of
this post we saw an approach for algorithmically defending against opaque auction mechanisms, one of the many ways the
data science team is working to deliver maximal value for our customers.
<h1 id="-footnotes"><a name="footnotes"></a> Footnotes</h1>
<ol>
<li>The <a href="https://en.wikipedia.org/wiki/Vickrey_auction">Vickrey auction</a> Wikipedia page explains why bidding your valuation
is optimal in a second-price auction.</li>
<li>We only consider a subset of second-price auctions for this exchange. These are all private marketplace deals understood to
be second-price auctions.</li>
<li>Section 1.2.1 of <a href="https://inst.eecs.berkeley.edu/~cs70/su16/static/su16/extra_note/sinho_cs_70_notes.pdf">these course notes</a>
has a proof of the discrete tail-sum formula.</li>
<li>The details are beyond the scope of this post.</li>
<li>This was the original reason we developed $F_M(x)$, which we used in the first section to predict the utility realized
in a second-price auction.</li>
</ol>
<hr />
</description>
</item>
<item>
<title>
$ marks the spot: saving a ton with EC2 Spot Fleet
</title>
<link>https://tech.nextroll.com/blog/dev/ops/2018/10/15/x-marks-the-spot.html</link>
<pubDate>Mon, 15 Oct 2018 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/ops/2018/10/15/x-marks-the-spot</guid>
<description><style type="text/css">/*<![CDATA[*/
/*]]>*/</style>
We reduced by over 60% the EC2 instance cost of our 1000+ node globally-distributed log-producing application by migrating it entirely to EC2 Spot Fleet. We discuss some of the issues we faced and also present the log data recovery mechanism which helped enable this migration.
<code class="language-plaintext highlighter-rouge">15-20 minute read</code>
<hr />
<h1 id="background">Background</h1>
AdRoll operates a globally-distributed <a href="/blog/dev/2018/01/08/quaff-that-potion-saving-millions-with-elixir-and-erlang.html#real-time-bidding">real-time bidding</a>
platform running on <a href="https://aws.amazon.com/ec2/">Amazon EC2</a>. This platform has historically used <a href="https://aws.amazon.com/ec2/autoscaling/">EC2
Auto Scaling</a> in order to scale a fleet of <a href="http://www.erlang.org/">Erlang/OTP</a>
application nodes according to load, typically exceeding 1000+ nodes (16000+ vCPUs)
during times of peak volume. This article describes how we substantially reduced our
EC2 costs by migrating this application entirely to <a href="https://aws.amazon.com/blogs/aws/amazon-ec2-spot-fleet-api-manage-thousands-of-instances-with-one-request/">EC2 Spot
Fleet</a>.
<h2 id="motivation">Motivation</h2>
A focus of the company has always been operational efficiency; money not spent is money
earned, and reducing operating costs by <code class="language-plaintext highlighter-rouge">$X</code> is equivalent to earning <code class="language-plaintext highlighter-rouge">$X / margin</code> in
additional revenue (plus the <a href="https://en.wikipedia.org/wiki/Time_value_of_money">time value of money</a> for
avoided outlays).
<h3 id="reserved-instances">Reserved Instances</h3>
Historically, reducing operating costs for this application has been achieved by
purchasing <a href="https://aws.amazon.com/ec2/pricing/reserved-instances/">Reserved Instances</a>: for a partial up-front
reservation, in exchange for an up-front fee per instance an hourly fee is billed per
instance-hour regardless of usage. For what had until now been our application’s
primary instance type, this would have worked out to savings of about 40% relative to
the on-demand price.
A problem with this approach becomes apparent when considering autoscaling behavior:
<img src="/images/post_images/spot-fleet/rtb-load-fluctuation.png" alt="RTB application load fluctuation image" />
This image represents the instance count of our real-time bidding application over about
a week-long period of time: the sinusoidal red line corresponds to instances running,
and the horizontal green line corresponds to the median instance count over the larger
window of time containing this week-long window.
How many instances should we reserve? Assuming future load will be similar to
historical load, we could simply reserve the median multiplied by the expected growth
factor; about half the time we’d have more instances than actually needed, and otherwise
we’d not have enough and would need to purchase on-demand capacity. This would give a
reasonable amount of savings, although reserving the median count isn’t necessarily
optimal (and isn’t in AdRoll’s case; we use a different strategy which is outside the
scope of this article).
If our future usage prediction turns out to be non-optimal, we’ll either wind up having
spent too much on reservations, or too much on on-demand capacity. Making reservations
for 1-year terms amplifies the cost of any misprediction since we can’t modify them
after purchasing (and the <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ri-market-general.html">reserved instance marketplace</a> is not
viable in our case).
<h3 id="spot-instances">Spot instances</h3>
<a href="https://aws.amazon.com/ec2/spot/">EC2 Spot Instances</a> present a relevant and compelling savings narrative:
bid on market-rate capacity in exchange for an often substantial discount:
<img src="/images/post_images/spot-fleet/spot-market-2.png" alt="EC2 spot market history (c3.4xlarge instance type in us-east-1
region)" />
If an application can be made to use spot instances, substantial savings can ensue. In
the above image, in mid-2018 the <code class="language-plaintext highlighter-rouge">c3.4xlarge</code> type in <code class="language-plaintext highlighter-rouge">us-east-1</code> had a median spot
market price of about <code class="language-plaintext highlighter-rouge">$0.29/hr</code> (<code class="language-plaintext highlighter-rouge">$0.26/hr</code> excluding outliers) – <code class="language-plaintext highlighter-rouge">34%</code> of the
on-demand price, and about <code class="language-plaintext highlighter-rouge">57%</code> of the effective reserved price of a partial up-front
reservation!
Unfortunately, these massive savings come with a massive caveat: a spot instance may be
evicted at any time based on changes in market demand (i.e., when another customer has
placed a higher bid during a period of high demand and low capacity in the same
availability zone).
This eviction behavior relates to two problems historically preventing us from using
spot instances in our application: loss of precious log data and the likelihood of
occasional market capacity exhaustion.
<blockquote>
Author’s note:
At the time this section was written, AWS had already changed its spot market behavior
by making evictions randomly instead of based on bid price during periods of decreased
instance availability. This means a high spot market bid no longer confers a higher
chance of avoiding eviction. Our application is unaffected by this.
</blockquote>
<h4 id="problem-precious-log-data">Problem: precious log data</h4>
Our real-time bidding application produces a large volume of log data on a daily basis:
about 40TB (compressed) of logs, market data, and data used to bill customers, all
written to local storage and periodically uploaded to S3.
Typical refrains here include “live with occasional data loss” or “don’t store valuable
data on a spot instance”. These are unacceptable strategies for our application; we
can’t afford to routinely lose portions of this dataset (and don’t want to introduce new
complexity, failure modes, or cost such as by adopting <a href="https://aws.amazon.com/kinesis/data-firehose/">Kinesis Firehose</a>
for log emission), so we can’t use spot instances without a reasonable guarantee that
logs won’t be lost if an instance is evicted on short (or no) notice.
<h4 id="problem-market-fluctuations-and-capacity-exhaustion">Problem: market fluctuations and capacity exhaustion</h4>
A second issue arises from the nature of the spot market: it’s possible for the market
price of an instance type to rise far beyond historical levels, exceeding the effective
reserved price and meeting (and at least historically exceeding) the on-demand price in
that availability zone (and possibly all zones in a region). It’s also possible for
capacity of an instance type to be completely exhausted in an availability zone,
preventing an application from running if dependent on a specific zone or instance type.
A customer in such a situation experiences all of the drawbacks of spot usage with none
of the benefits. We can’t use spot instances absent resistance to market price
fluctuations and capacity exhaustion.
<h2 id="a-viable-building-block-ec2-spot-fleet">A viable building block: EC2 Spot Fleet</h2>
<a href="https://aws.amazon.com/blogs/aws/amazon-ec2-spot-fleet-api-manage-thousands-of-instances-with-one-request/">EC2 Spot Fleet</a> appears to meet most of our requirements while
granting access to spot market savings. A spot fleet allows a user to deploy spot
instances without having to manage individual spot requests while also allowing use of
multiple different instance types at the same time! In particular:
<ol>
<li>
It supports <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet-automatic-scaling.html">autoscaling</a>. As a result, our
application could continue to scale appropriately according to load.
</li>
<li>
It supports <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html#spot-instance-weighting">multiple instance types</a> in the same logical
grouping (unlike autoscaling groups), with different spot bid prices for each type,
and different allocation strategies such as “lowest price” and “diversified”. As a
result, our application could gain resistance to market fluctuations and insufficient
market capacity potentially affecting multiple instance types and availability zones.
</li>
<li>
It supports both <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html#on-demand-in-spot">on-demand (reserved) and spot</a>
instances in the same fleet. As a result, we have the ability to migrate our
existing deployment from autoscaling groups entirely to spot fleet, simplifying
operations.
</li>
</ol>
Most of the time a spot instance will be given a <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html#spot-instance-termination-notices">two-minute
warning</a> before eviction; while the timing and even
delivery of that notice isn’t guaranteed, it at least gives us an opportunity to
gracefully stop an instance about to be evicted (though possibly without enough time to
upload all of its log data, which we address in the next section).
Considering the above, it looks like we could deploy our application to EC2 Spot Fleet
and treat it much the same way as we treat autoscaling groups, as long as we handle the
case of eviction without having enough time to upload logs.
<h1 id="implementation">Implementation</h1>
The devil is in the details…
<h2 id="ec2-spot-fleets-are-not-entirely-unlike-auto-scaling-groups">EC2 Spot Fleets are not entirely unlike Auto Scaling Groups</h2>
Up to now, we’ve been using a robust and featureful product to run our application:
<a href="https://aws.amazon.com/ec2/autoscaling/">Auto Scaling Groups</a>. ASGs include support for instance monitoring
and health checking, various means of scaling according to load, instance lifecycle
management, upgrading in-place, and discoverability &amp; management via naming and tagging
of resources.
Unfortunately, spot fleets lack feature parity with ASGs, including almost all of the
above. Because we wanted to migrate our existing application with as few externally- or
application-visible changes as possible, we wound up implementing our own solutions or
finding alternatives for all of the following in order to present a compatible
environment for running and managing our application.
<h3 id="name-tags">Name tags</h3>
<h4 id="problem">Problem</h4>
Unlike ASGs, spot fleets can’t be named or tagged. Instead, the user is presented with
opaque UUIDs for created spot fleets and this user-unfriendly console for managing them:
<img src="/images/post_images/spot-fleet/spot-fleet-console.png" alt="EC2 spot fleet console" />
<h4 id="solution">Solution</h4>
This doesn’t fit into our existing workflow, which allows custom scripts (and tools like
<a href="https://github.com/aws/aws-cli">awscli</a>) to locate and manage resources using names and tags. The spot
fleet console is also effectively useless for locating resources of interest unless a
request ID is already known.
We resolved this by introducing a small DynamoDB table mapping resource names to
attributes (which we would otherwise have set as tags) and spot fleet request ids, and
updating our tools and workflow to reference the table.
<h3 id="lifecycle-hooks">Lifecycle hooks</h3>
<h4 id="problem-1">Problem</h4>
Unlike ASGs, spot fleets don’t support <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/lifecycle-hooks.html">lifecycle hooks</a>. This
feature allows instances to be kept in pending states during startup and before
termination, allowing for example external resources required for operation to be
created before an application starts and to be cleaned up after it exits, and for work
currently in progress to be finished before termination (e.g., log uploads or active
data processing):
<img src="/images/post_images/spot-fleet/lifecycle_hooks.png" alt="EC2 lifecycle hooks flow chart" />
<h4 id="solution-1">Solution</h4>
This is a feature which could perhaps be approximated for the <code class="language-plaintext highlighter-rouge">Pending:*</code> states by an
agent running on each host and coordinating with an external resource management
component. There is no substitute for the <code class="language-plaintext highlighter-rouge">Terminating:*</code> states (e.g., allowing an
instance to upload its log data before proceeding with termination).
We resorted to simply abandoning our (limited) use of lifecycle hooks and moving the
associated responsibilities into the application’s instance setup process (<code class="language-plaintext highlighter-rouge">Pending:*</code>
states) and introducing a log recovery system as described later (<code class="language-plaintext highlighter-rouge">Terminating:*</code>
states).
<h3 id="health-checks">Health checks</h3>
<h4 id="problem-2">Problem</h4>
An instance in an autoscaling group can be either in the <code class="language-plaintext highlighter-rouge">healthy</code> or <code class="language-plaintext highlighter-rouge">unhealthy</code> state.
An <code class="language-plaintext highlighter-rouge">unhealthy</code> instance is terminated by its ASG and replaced, and an end user may set
this state (e.g., from a user-operated health checker). ELBs also <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-add-elb-healthcheck.html">nicely
integrate</a> with this feature: if an application exposes a
health endpoint (such as an HTTP URL), an attached ELB will set the instance as
<code class="language-plaintext highlighter-rouge">unhealthy</code> if that endpoint fails to return a successful response.
Unfortunately, spot fleet offers no user-controllable health status; an instance will
only be replaced if it fails the basic EC2 health check which has no knowledge of an
application’s health. As ELBs don’t control the basic health check result, the ELB
health check feature does not work with spot fleet instances.
<h4 id="solution-2">Solution</h4>
We implemented our own health checking component using the versatile <code class="language-plaintext highlighter-rouge">awscli</code> tool: we
run a script in a loop on a utility instance which obtains the health check
configuration from each ELB (which would affect ASG instances but not fleet instances)
and apply it ourselves to each attached spot fleet instance (obtained from the
naming/tagging table); fleet instances older than the grace period which fail the number
of checks configured in the ELB’s health check are simply terminated (as there is no
user-controllable health state).
<h3 id="no-in-place-upgrading">No in-place upgrading</h3>
<h4 id="problem-3">Problem</h4>
Spot fleets use <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-launch-templates.html">launch templates</a>, a generalization of the
<a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchConfiguration.html">launch configuration</a> concept used in ASGs. An ASG includes a
reference to its current launch configuration, which can be changed by the user without
recreating the ASG. After changing the ASG’s active launch configuration, instances
newly launched in the ASG adopt that new launch configuration; this allows for example
the userdata of instances in an ASG to be updated, instance types to be changed, block
device mappings to be updated, and so on.
Unfortunately, spot fleet has no equivalent behavior: a launch template is specified
once at spot fleet request time and can’t later be changed; nor can the launch template
version be changed for an existing spot fleet request (launch templates introduced a
“version” concept: a launch template may have multiple versions and a “default” version,
but for a spot fleet the version must be fixed in the creation request).
<h4 id="solution-3">Solution</h4>
None; we must recreate our spot fleets whenever changing launch template parameters such
as userdata, bid prices, or instance type weights and overrides.
<h3 id="unexpected-scaling-behavior-stuck-fleets">Unexpected scaling behavior: stuck fleets</h3>
<h4 id="problem-4">Problem</h4>
We use <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scaling-simple-step.html">step scaling</a> to scale our application according to load.
This works well; each of our application instances can report its load status using a
simple mechanism, and a scaling action can be taken according to the severity of the
aggregate value.
In this example image, a deployment could consist of four nodes: two reporting
“overload” and two reporting “stable”. In aggregate, we can say that our application is
50% overloaded and take an appropriate action (say, increasing our capacity by 25%):
<img src="/images/post_images/spot-fleet/overload.png" alt="load reporting mechanism" />
Unfortunately, this can interact poorly with how spot fleet scaling works: there are
circumstances in which a user fleet can indicate scaling is required, <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet-automatic-scaling.html">but for which the
spot fleet scaling service will take no action</a> (visible in
rounding behavior and scaling against fulfilled capacity instead of target capacity).
This is bad for our application: if we know a scaling action is required, we must always
then execute such an action (or risk being unable to serve traffic and thus not earning
money, or spending too much money due to having an excessive number of hosts).
<h4 id="solution-4">Solution</h4>
We opted for a solution not requiring code or instrumentation changes in our
application: a “fleet kicker”.
This component (a shell script executing <code class="language-plaintext highlighter-rouge">awscli</code> commands) runs in a loop on a utility
instance and monitors the cloudwatch alarm state of the scaling policies associated with
each of our fleets. If any alarm remains in the <code class="language-plaintext highlighter-rouge">ALARM</code> state for too long, the
component adjusts the target capacity according to the associated scaling policy and
scalable target definition.
<img src="/images/post_images/spot-fleet/fleet-kicker.png" alt="stuck fleet kicker effect" />
This image shows the cloudwatch metric value (multiplied by 100 to be compatible with
the spot fleet UI in the AWS console) for one of our application’s fleets (negative
values represent “underload”, or having too many instances relative to traffic volume).
The first arrow indicates the beginning of a period in which our fleet should have
scaled down, but did not. The second arrow indicates the point at which our “fleet
kicker” detected the stuck fleet and activated, forcing a decrement to the fleet’s
target capacity.
With this component, our fleets avoid becoming stuck and can scale up and down as we’d
expect them to, and we can treat them conceptually as if they were ASGs.
[Note: AWS has indicated a possible alternate solution being switching to <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet-target-tracking.html">target
tracking</a> scaling policies, which could be useful if compatible
with an application. We have not yet validated use of target tracking policies with our
application.]
<h2 id="spot-savior-a-log-data-recovery-system">Spot savior: a log data recovery system</h2>
We have so far described our solutions to problems arising from attempting to treat spot
fleets conceptually the same way as autoscaling groups. Our application also now makes
use of many instance types in order to tolerate market fluctuations and makes
appropriate bids for each type thanks to <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html#spot-instance-weighting">spot fleet’s override and weighting
mechanic</a>. There’s one critical piece left: what to do
about inevitable spot market evictions.
<h3 id="much-ado-about-eviction">Much ado about eviction</h3>
Our application stages log data to a volume attached to each instance. We periodically
convey this data to S3 during normal operation, and until now also at instance
termination time (previously relying on lifecycle hooks for ASG instances, which have no
spot fleet analog):
<img src="/images/post_images/spot-fleet/app-instance-store.png" alt="instance using ephemeral instance store volumes" />
To make a long story short, we can’t guarantee that the two-minute eviction warning will
be sufficient for our application when running on a spot instance:
<ul>
<li>the warning might not actually be delivered with two minutes to spare, or at all;</li>
<li>the warning could be misinterpreted due to a bug;</li>
<li>the instance might have staged a disproportionate amount of log data which can’t all
be uploaded to S3 in two minutes;</li>
<li>adverse network or service conditions may exist, preventing timely or successful
uploading to S3;</li>
<li>the application might fail to gracefully stop in a timely manner, leaving insufficient
time for log uploading;</li>
</ul>
and so on.
Having staged our log data, the only guarantee we can make is: our writes were
acknowledged by a filesystem on some volume. What to do?
<h3 id="ebs-to-the-rescue">EBS to the rescue!</h3>
It turns out that <a href="https://aws.amazon.com/ebs/">Amazon EBS</a>, a foundational network block storage
product, allows block storage volumes to survive beyond the termination of the host to
which they are attached.
During normal operation, we can treat an EBS volume largely the same way as an ephemeral
instance store volume and stage our (append-only) logs, periodically uploading as usual:
<img src="/images/post_images/spot-fleet/app-ebs.png" alt="instance using EBS volumes" />
But we can also <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#preserving-volumes-on-termination">configure the volume to survive
independently</a> as an orphaned volume if its instance is
terminated:
<img src="/images/post_images/spot-fleet/app-orphaned-ebs.png" alt="orphaned EBS volume" />
This gives us an opportunity to recover logs from evicted spot instances without having
to worry about whether the two-minute warning was enough (or having to introduce a new
application logging mechanism). We created a generic log recovery pipeline external to
our application to do just that.
To recover logs from a volume, we need to execute a sequence of steps; each with a set
of pre- and post-conditions, an implementation, and a position relative to other steps.
For example: in order to copy a log file from a volume, we need to have first mounted
the volume; to have mounted a volume, we need it (or a copy of it) to exist in the same
availability zone as the current instance; to make a copy of a volume, we need to have
created a snapshot of it (possibly using one which already exists); and so on.
There’s a ready-made tool for clearly expressing solutions to this kind of problem:
<a href="https://luigi.readthedocs.io/">Luigi</a>.
<h3 id="luigi-for-connecting-a-series-of-tubes">Luigi: for connecting a series of tubes…</h3>
<img src="/images/post_images/spot-fleet/luigi-logo.png" alt="luigi logo" />
Luigi has been featured several times on AdRoll’s tech
blog<a href="/blog/data/2015/09/22/data-pipelines-docker.html">[1]</a><a href="/blog/data/2015/10/15/luigi.html">[2]</a><a href="/blog/data/2018/08/08/running-jobs-with-aws-batch.html">[3]</a>: it’s a useful tool for
orchestrating jobs consisting of large, dynamic, and explicit hierarchies of
parameterized tasks which can depend on other tasks, any of which may fail (making any
dependent tasks also fail). Even though much smaller than a typical Luigi job, this is
a good fit for our log recovery scenario:
<ul>
<li>recovering logs from an EBS volume is a batch process and insensitive to small delays
and retries occurring over a period of several minutes;</li>
<li>we need to recover an unknown number of logs from each volume;</li>
<li>at most a single worker should be attempting recovery of a volume at any time;</li>
<li>jobs are repeatable and may be incremental: each log file of interest on a volume has
a 1-1 correspondence with an object on S3 (thus, we could dynamically create a Luigi
task per log file and retry a failing overall job repeatedly until success);</li>
<li>each recovery job has many steps, any of which may fail due to external reasons (e.g.,
service outages, API throttling/failures in excess of retries, unexpected worker host
termination, etc).</li>
</ul>
[Aside: <a href="https://aws.amazon.com/step-functions/">AWS Step Functions</a> could also be a good fit for this
scenario if a step function Lambda task could mount and access the contents of an EBS
volume. Using a non-Lambda Activity worker might work, but we decided the overhead of
managing a separate group of workers integrated with Step Functions would outweigh any
benefit versus simply using Luigi.]
<h4 id="example-luigi-volume-recovery">Example: Luigi volume recovery</h4>
Here’s a simplified version of our Luigi-based recovery pipeline implementation:
<figure class="highlight"><pre><code class="language-python" data-lang="python">class RecoverLogs(luigi.wrapperTask):
volume = luigi.Parameter()
...
def requires(self):
return [UploadLogs(volume=self.volume, ...),
Cleanup(volume=self.volume, ...)]
class UploadLogs(luigi.Task):
volume = luigi.Parameter()
...
def output(self):
return luigi.contrib.s3.S3Target(
's3://bucket/%(volume)s/_SUCCESS' % dict(volume=self.volume))
def requires(self):
return {'mount': MountVolume(...)}
def run(self):
results = yield {path: RecoverOneFile(path)
for path in self.find_logs(
self.input()['mount'].read())}
self.output().write(
json.dumps({path: r.read()
for (path, r) in results.items()}))
class MountVolume(luigi.Task):
volume = luigi.Parameter()
...
def requires(self):
return {'attach': AttachAZLocalVolume(...)}
def run(self):
device = self.input()['attach'].read()
mountpoint = self.calculate_mountpoint(self.volume, device)
self.run_command(['mount', device, mountpoint])
self.output().write(mountpoint)
def complete(self):
return self.is_mounted(self.volume, self.output().read())
def on_failure(self, exc):
self.detach_and_unmount()
...
class Cleanup(luigi.Task):
volume = luigi.Parameter()
...
def requires(self):
return [UploadLogs(...), UnmountDetachVolume(...)]
def run(self):
...
yield DeleteVolume(...)</code></pre></figure>
In Luigi, a job is run by starting with the goal: <code class="language-plaintext highlighter-rouge">luigi.build([RecoverLogs(volume_id,
...)])</code>. Luigi builds a graph of task dependencies and runs each runnable task which
hasn’t yet completed (each of which can dynamically introduce additional subtasks),
starting from the leaves (tasks with no unmet requirements).
The end result is well-modularized and easily-testable code much like what we’d
eventually arrive at after improving a non-Luigi version (but we just skipped to the end
by using Luigi from the start). Here’s what a successful recovery of logs from an EBS
volume looks like in our system:
<img src="/images/post_images/spot-fleet/spot-savior-recovery.png" alt="luigi volume recovery pipeline result" />
Green vertices denote successful tasks; edges express dependencies. Any failing task
results in the whole job failing and being retried; but only the subtasks which haven’t
yet succeeded are re-run.
<h4 id="putting-it-all-together">Putting it all together</h4>
In practice Luigi has been a good fit for this scenario. Here’s how we put it all
together:
<ol>
<li>First, when running on a spot instance our application records log volume information
in a new DynamoDB table and configures its log volume(s) as <code class="language-plaintext highlighter-rouge">delete-on-terminate=False</code>,
allowing them to become orphans following instance termination (such as due to a spot
eviction or spot fleet scale-down event):
<img src="/images/post_images/spot-fleet/recovery-pipeline-1.png" alt="recovery pipeline: use of log volumes table" />
This allows us to keep track of volumes which could actually have received any
interesting application logs.</li>
<li>Second, a recovery component executing on a utility instance periodically scans this
volume information table:
<img src="/images/post_images/spot-fleet/recovery-pipeline-2.png" alt="recovery pipeline: dispatcher activity" />
Any volume in the table in the <code class="language-plaintext highlighter-rouge">available</code> state is eligible for recovery, and a
recovery message is dispatched to a regional queue.</li>
<li>Finally, an EC2 recovery worker (running in an ASG which scales according to the size
of the queue) polls its regional queue for recovery messages:
<img src="/images/post_images/spot-fleet/recovery-pipeline-3.png" alt="recovery pipeline: luigi pipeline activation" />
Having obtained a message, it executes the Luigi recovery pipeline for each, resulting
in a volume’s logs being recovered and the volume being deleted only upon success.
Naturally these workers also use spot instances!</li>
</ol>
With a real workload, this setup can recover logs from about 650 EBS volumes per hour
(in each region), largely limited by EC2 API throttling. Our implementation always
makes a copy of each volume before recovery to allow filesystem journals to be replayed
without modifying the original volume.
<h2 id="conclusion">Conclusion</h2>
We modified our real-time bidding application to run on over twenty different EC2
instance types, and we’re currently using more than ten of those types in the spot
fleets we created to replace our original autoscaling groups, giving a substantial
amount of resistance to spot market fluctuations.
To solve the problem of precious log data being lost on terminated spot instances, we
implemented a Luigi-based log recovery pipeline to recover those logs from the EBS
volumes now used by our application and surviving instance termination.
As a result we’ve been able to realize substantial (~60%) savings; the effective cost of
a compute unit is lower in the spot market relative to even the effective reserved
instance price, and we now also no longer need to make any up-front reservations!
Uncovering and overcoming the lack of feature parity between autoscaling groups and spot
fleets was time-consuming, but overall the benefits far outweigh the cost (and the
savings compound over time).
<hr />
</description>
</item>
<item>
<title>
Removing Erlang dead code with Xref
</title>
<link>https://tech.nextroll.com/blog/dev/2018/10/09/remove-erlang-dead-code-xref.html</link>
<pubDate>Tue, 09 Oct 2018 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2018/10/09/remove-erlang-dead-code-xref</guid>
<description>Dead code (as in functions that are not used anywhere) tends to pile up in big projects if you leave them unattended. Using one of Xref’s most underrated features, you will be able to detect and remove what you do not need anymore.
<code class="language-plaintext highlighter-rouge">10-15 minute read</code>
<hr />
We have already written several articles in this blog explaining how we use <a href="http://www.erlang.org/">Erlang/OTP</a> extensively to build our <a href="/blog/web/2014/04/29/valentino-presents-adrolls-rtb-infrastructure.html">real-time bidding platform</a> servers, among other things.
These systems are big and they have been around for a long time by now. Just like any big old system, they contain some pieces of code that are not used anymore. To be clear: they are not broken, they are even properly covered by tests and all, but they’re not in use in production.
In Erlang these pieces of dead code manifest themselves as unused functions. To be precise: they are unused *exports* since unused but not-exported functions are detected at compile time.
Finding unused exports in a big system can be tough. Luckily, Erlang/OTP already gives us a tool to do just that: <a href="http://erlang.org/doc/man/xref.html">Xref</a>.
<blockquote>
Xref is a cross reference tool that can be used for finding dependencies between functions, modules, applications and releases.
</blockquote>
If you manage your projects with <a href="https://rebar3.org">rebar3</a>, you can use Xref by simply running the following command:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ rebar3 xref</code></pre></figure>
If you had not specified anything about Xref in your rebar.config, that will check your entire project and perform all possible checks. Since, for big projects, the list of warnings that generates tends to be long, people usually have something like this in their configuration:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{xref_checks, [
undefined_function_calls,
locals_not_used,
deprecated_function_calls
]}.</code></pre></figure>
In other words, with that list the report will only include:
<ul>
<li>calls to functions that don’t exist (<code class="language-plaintext highlighter-rouge">undefined_function_calls</code>).</li>
<li>unused not-exported functions (<code class="language-plaintext highlighter-rouge">locals_not_used</code>).</li>
<li>calls to deprecated function (<code class="language-plaintext highlighter-rouge">deprecated_function_calls</code>).</li>
</ul>
You can find the full list of available checks in <a href="https://www.rebar3.org/docs/configuration#section-xref">rebar3 docs</a>, but I want you to notice that since the last 2 are also detected by the compiler (if you have the proper warnings enabled) the only effective check that’s being performed is <code class="language-plaintext highlighter-rouge">undefined_function_calls</code>. It’s a fine check to run, but it won’t help us with our original dead code issue.
Let’s take a look at the checks we’re not performing then. In general, <code class="language-plaintext highlighter-rouge">undefined_functions</code> will report the same results as <code class="language-plaintext highlighter-rouge">undefined_function_calls</code> but without the reference to the actual function call (Not very useful). Same goes for <code class="language-plaintext highlighter-rouge">deprecated_functions</code> and <code class="language-plaintext highlighter-rouge">deprecated_function_calls</code>. But then, we have <code class="language-plaintext highlighter-rouge">exports_not_used</code> which is exactly the check we’re looking for.
Adding <code class="language-plaintext highlighter-rouge">exports_not_used</code> to our list of checks will emit a warning for each function that we have exported but not used anywhere. It’s amazing.
But then why is nobody using it?🤔
There are a few caveats when using <code class="language-plaintext highlighter-rouge">exports_not_used</code>. I’ll list them now and I’ll tell you how to solve or at least work around them.
<h2 id="dynamically-evaluated-functions">Dynamically Evaluated Functions</h2>
Xref will report a warning for each function that is exported for which it could not find where it’s used in the code. But, the fact that Xref couldn’t find it, doesn’t mean that a place where the function is actually used doesn’t actually exist.
For instance, Xref can’t deal with dynamic function calls but they’re perfectly valid.
So, let’s say you have a module that looks like the following one (don’t ask me why):
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">-module(sample).
-exports([some_function/1, some_other_function/1]).
some_function(M) -&gt;
M:some_other_function(an_argument).
some_other_function(X) -&gt; {called, X}.</code></pre></figure>
And some other module where you call <code class="language-plaintext highlighter-rouge">sample:some_function(sample)</code>. Xref will not be smart enough to detect that <code class="language-plaintext highlighter-rouge">sample:some_other_function/1</code> is actually used, because it’s only used through dynamic evaluation. And the one above is just one way of performing dynamic evaluation. You can see some others here:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">% Classic dynamic evaluation
Module:Function(Argument, Argument2),
% Using erlang:apply/3
erlang:apply(Module, Function, Arguments),
% Using spawn[_link]/3
erlang:spawn(Module, Function, Arguments),
% Using timer:tc/3
timer:tc(Module, Function, Arguments),
% In a supervisor spec
{ChildName, {Module, Function, Arguments}, permanent, 5000, worker, dynamic},</code></pre></figure>
Side note: if you add <code class="language-plaintext highlighter-rouge">{xref_warnings, true}.</code> to your rebar.config file, Xref will at least print out warnings for these dynamic calls that it could not parse. Like this one:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">sample: 1 unresolved call</code></pre></figure>
In any case, as soon as your system becomes slightly bigger than a prototype, you’ll start having this unused exports everywhere. But don’t panic, there is actually a way to avoid those warnings, and it comes with some extra benefits. You can use <code class="language-plaintext highlighter-rouge">ignore_xref</code>.
<code class="language-plaintext highlighter-rouge">-ignore_xref</code> is an attribute you can add to your modules to prevent Xref from emitting warnings about certain functions. It looks like this:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">-module(sample).
-exports([some_function/1, some_other_function/1]).
%% This function should be dynamically invoked through sample:some_function/1
-ignore_xref([{?MODULE, some_other_function, 1}]).
...</code></pre></figure>
Now, if you go check the Xref docs from OTP, you won’t find any mention to it. That’s because it’s not an official attribute.
<code class="language-plaintext highlighter-rouge">ignore_xref</code> is an undocumented feature of rebar3 xref (and also <a href="https://hex.pm/packages/xref_runner">xref_runner</a>). It’s an attribute you can add to your modules where you can list all the functions you don’t want Xref to complain about. The syntax is as follows:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">-ignore_xref([{module(), function(), arity()} | {module(), function()}]).</code></pre></figure>
Using this, you can effectively remove all warnings related to functions that are exported so they can be dynamically evaluated. As a bonus, as you can see in our example above, you can also use this place in the code to add some documentation stating where these functions are expected to be used.
<h2 id="dynamically-generated-code">Dynamically Generated Code</h2>
You might not be a fan of dynamically generated code, but sometimes there is no way around it.
For instance, we use protocol buffers in several places and that lead us to use <a href="https://hex.pm/packages/gpb">gpb</a> and its rebar3 plugin. When gpb is writing a module, it doesn’t know how it will be used so it has no way to tell if certain functions should not be exported/are not needed. That means when we run Xref we get warnings about all the unused exports generated by it.
How to avoid those warnings? We can’t use <code class="language-plaintext highlighter-rouge">-ignore_xref</code> since we’re not writing those modules. Turns out, there is another way. We can use the dyslexically named <code class="language-plaintext highlighter-rouge">xref_ignores</code> attribute in rebar.config. It basically allows you to have a global list of functions to ignore everywhere. It looks like this:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{xref_ignores, [
{my_gpb_generated_module, some_function, 1},
{my_gpb_generated_module, some_other_function, 0},
{my_gpb_generated_module, a_function_with_various_arities},
...
]}.</code></pre></figure>
There is no way yet to ignore all the functions in a module, but I already wrote <a href="https://github.com/erlang/rebar3/issues/1905">a ticket</a> requesting it. Maybe you can tackle that as a <a href="https://hacktoberfest.digitalocean.com/">#hacktoberfest</a> project?
<h2 id="functions-exported-to-be-used-externally">Functions Exported to be used Externally</h2>
What if you have some functions that are only exported because you use them in the shell when you log in remotely to your servers in production? What if they’re used by external scripts that execute <code class="language-plaintext highlighter-rouge">rpc</code> calls into your nodes or stuff like that?
Well, in that case, I encourage you to use <code class="language-plaintext highlighter-rouge">-ignore_xref</code> and add a proper comment there stating how/when/where those functions are expected to be used. It will pay off in the future, I promise.
<h2 id="functions-exported-just-for-tests">Functions Exported just for Tests</h2>
A different situation I’ve seen sometimes (particularly when people work <a href="https://www.goodreads.com/book/show/44919.Working_Effectively_with_Legacy_Code">with Legacy Code</a>) are functions that are exported so they can be used in tests. The idea is to either mock them or to have access to some internal logic that should otherwise be hidden to the system in production.
First of all, if you’re using <a href="http://erlang.org/doc/man/eunit.html">eunit</a> you don’t need to export them. You can use not-exported functions in your tests.
Now, if you use common test or other frameworks that require tests to be written outside the module under test, that’s a different story. I think it’s important to consider that exporting functions just to use them in tests is something that’s undesirable in general since…
<blockquote>
<ul>
<li>If your functions are not exported and unused they are detected by the compiler, as we stated before, allowing you to find bugs much earlier.</li>
<li>If you are adding functions that should not be available in production and/or doing stuff that should not be done in production, then your test is not simulating the real scenario accurately which may lead to test passing with code that still doesn’t work as expected.</li>
</ul>
</blockquote>
Still, sometimes there is just no way around it: You need to mock some stuff that is otherwise invisible to the external world, you need to verify some data that’s only exposed in very complex formats or detect side effects that are really hard to capture. In those scenarios, once again, <code class="language-plaintext highlighter-rouge">ignore_xref</code> and a neat comment are wonderful tools to avoid surprises and frustration for future developers that, finding an unused function, decide to remove it.
<h2 id="library-facades">Library Facades</h2>
Finally, there is one other scenario where you do need to export functions that you don’t actually use within your application: When your application is a library (i.e. when you’re building an app to be used as a dependency in other systems). In that case, some functions constitute the facade of your app and they’re not to be used by it. They’re exposed so your users can invoke them in their apps.
Those functions will all be reported as unused exports and it’s not nice to have to write <code class="language-plaintext highlighter-rouge">ignore_xref</code> / <code class="language-plaintext highlighter-rouge">xref_ignores</code> for all of them. But what is really nice is to cover them in tests. And if you do that, you have a way to avoid the warnings and actually only generate warnings for functions that are exported, unused and not tested. You can run xref as…
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ rebar3 as test xref</code></pre></figure>
Using the test profile, rebar3 will include all your test modules into the analysis and, since your facade functions will be used there, it will not warn you about them.
<h2 id="conclusion">Conclusion</h2>
While Xref is a powerful tool, it needs some tweaks to extract its full potential.
First of all, you have to use the right checks. Our recommended list is…
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">{xref_checks, [
undefined_function_calls,
exports_not_used
]}.</code></pre></figure>
Then you have to use <code class="language-plaintext highlighter-rouge">-ignore_xref</code> attributes and <code class="language-plaintext highlighter-rouge">xref_ignores</code> configuration param appropriately to identify all the functions that are intentionally exported and unused. If you’re writing a library, you should also consider your tests in the analysis using <code class="language-plaintext highlighter-rouge">rebar3 as test xref</code>.
With all that in place, you should expect to have 0 warnings reported and therefore you can be sure that you don’t have any dead code in your project.
Well… actually… You don’t have any dead functions (unused exports). Which is a lot, but you can still have dead code in the form of unused function clauses, unused case clauses, etc… Xref will not detect those problems.
For that, you need a much more powerful tool: <a href="http://erlang.org/doc/man/dialyzer.html">dialyzer</a>. We won’t cover it in this article, but stay tuned…
<hr />
Do you enjoy building high-quality large-scale systems? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>Running large batch processing pipelines on AWS Batch</title>
<link>https://tech.nextroll.com/blog/data/2018/08/08/running-jobs-with-aws-batch.html</link>
<pubDate>Wed, 08 Aug 2018 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data/2018/08/08/running-jobs-with-aws-batch</guid>
<description>The attribution team at AdRoll computes metrics out of petabytes of data every
night. This is accomplished using a batch processing pipeline that submits jobs
to AWS Batch. In this blog post we discuss how this is organized and
orchestrated with Luigi. We announce Batchiepatchie, a job monitoring tool for
AWS Batch. Batchiepatchie is an improvement over Amazon’s own monitoring
solution for AWS Batch and it has saved us countless hours of engineer time.
<code class="language-plaintext highlighter-rouge">10-15 minute read</code>
<hr />
To start off, <a href="https://github.com/AdRoll/batchiepatchie">we have open sourced Batchiepatchie that is discussed in this
blog post</a>. Read on to understand
what this piece of technology is!
Before we delve into technicals of batch processing, I want to establish some
real world context. Our batch pipeline was created for the purpose of doing
something called attribution in Internet advertising. What is attribution, in
adtech sense? To put it very simply, it is the problem of figuring out which
advertising events deserve credit when people purchase things. For
example, if we show an advertisement to a person about a T-shirt and one day
later they buy said T-shirt, we can say that advertisement impression deserves
some credit for making that purchase happen. Attribution is used to understand
how some advertising campaign is working out.
The problem of how much credit should be given and in which circumstances and how
to report on it is a complicated topic and is a story for another time. This
post is about the batch job infrastructure that can compute metrics around this
problem by making use of vast amounts of data.
To compute attribution metrics, we need to be able to look at the history of
all event trails of each user we track. An event trail contains all the
relevant information of each user for attribution purposes, such as when we
displayed advertisement from certain advertising campaigns and when purchases
were made on a customer’s website. From this information, we can compute how
much credit each advertising campaign deserves.
At AdRoll’s scale, this is a non-trivial amount of data. All the attribution
metrics must be computed on a daily basis. Even a big EC2 instance cannot process
all of it in a reasonable time; just downloading all the required data in
compressed format would take too long with a single box. Thus, we must
distribute the problem to many computers. This is where AWS Batch comes in.
<h4 id="aws-batch">AWS Batch</h4>
So what is AWS Batch anyway? You can read the <a href="https://aws.amazon.com/batch/">official description at Amazon’s
website</a> but for our purposes it can be described as follows.
AWS Batch is a system that you submit jobs into. AWS Batch runs these jobs on
EC2 instances. AWS Batch scales up a bunch of instances as needed so that the
jobs can run. Once all the jobs are done, the instances are put down. This way,
you will pay for instances only when you actually have some jobs running. These days
<a href="https://aws.amazon.com/blogs/aws/new-per-second-billing-for-ec2-instances-and-ebs-volumes/">AWS has per-second billing of EC2 instances</a>
, which makes fast scaling a critical feature in terms of minimizing your AWS bill.
Another way to describe this process is that you give AWS Batch a Docker image URL, some
command line arguments and CPU and memory requirements and AWS Batch will
figure out how to run your job in some way.
AWS Batch is a relatively simple way to distribute large amounts of batch jobs
onto lots of EC2 instances and in a way that you only pay when you actually
have jobs running.
<h3 id="example-of-a-batch-job-pipeline">Example of a batch job pipeline</h3>
So, how do you use the AWS Batch system to split off your massive batch
processing so that bunch of boxes will run your job instead of just one box? I
thought one of the most illustrative ways to describe how our system works is
to show a representative example of actual technologies we use to do this.
The most important technologies involved with this process is Amazon S3, Docker
and <a href="https://github.com/spotify/luigi">Luigi</a> and our internal (<a href="https://github.com/AdRoll/batchiepatchie/">but now open
sourced!</a>) AWS Batch dashboard
called “Batchiepatchie”.
<h4 id="luigi">Luigi</h4>
At AdRoll’s attribution team we use a Python library called Luigi to orchestrate
tasks. <a href="/blog/data/2015/10/15/luigi.html">We have talked about Luigi before in our blog</a> so
you can read our older blog post for some more background on that topic; most
of it is still relevant for this Luigi section, although we use a slightly
different set of technologies than we did back in 2015.
Let’s say that hypothetically (or not so hypothetically, this example is a
simplified version of a real-world job) we want to compute the last click
credit on advertising campaigns using
<a href="http://traildb.io">TrailDBs</a> as our source of data.
We have TrailDB files in S3 in an organized directory structure, where each day
has its own set of TrailDBs. For example, we have a type of attribution TrailDB
that contains data relevant for attribution purposes.
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ aws s3 ls s3://example-bucket/traildbs/attributiondb/2018-07-26/
2018-07-27 18:24:03 attributiondb-0.tdb
2018-07-27 18:24:39 attributiondb-1.tdb
2018-07-27 18:25:07 attributiondb-2.tdb
2018-07-27 18:23:38 attributiondb-3.tdb
2018-07-27 18:18:41 attributiondb-4.tdb
2018-07-27 18:55:22 attributiondb-5.tdb
2018-07-27 18:36:03 attributiondb-6.tdb
2018-07-27 18:10:10 attributiondb-7.tdb
</code></pre></div></div>
In the above example, we have 8 files. These files contain all data for that
particular day; in this example, that day is 2018-07-26.
This naturally distributes to at least 8 workers: just submit one batch job per
file. One worker processes one file.
We will look at how to do this with some Luigi, S3 and AWS Batch.
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code># lastclick.py
import luigi.s3
import luigi
class AttributionLastClickJob(luigi.WrapperTask):
date = luigi.DateParameter()
s3_input_prefix = luigi.Parameter()
s3_output_prefix = luigi.Parameter()
shards = luigi.IntParameter()
def requires(self):
sub_jobs = []
for shard in range(self.shards):
sub_jobs.append(AttributionLastClickShard(
date=self.date,
s3_input_prefix=self.s3_input_prefix,
s3_output_prefix=self.s3_output_prefix,
shard=shard))
yield sub_jobs
</code></pre></div></div>
Let us first go through what is happening in this file. We have four arguments
to our task, date, input prefix, output prefix and shards. These
specify the date we are processing, an S3 input prefix where our input data is
located, an S3 output prefix where we should put output data and the number of
shards we have (in this example, shards is 8).
For this example, we could have, <code class="language-plaintext highlighter-rouge">date=2018-07-26</code>,
<code class="language-plaintext highlighter-rouge">s3_input_prefix=s3://example-bucket/traildb/attributiondb</code>,
<code class="language-plaintext highlighter-rouge">s3_output_prefix=s3://example-bucket/attribution-results</code> and <code class="language-plaintext highlighter-rouge">shards=8</code>.
This particular Luigi task is a wrapper task which means it does not run jobs
by itself, it only requires other tasks and those other tasks do the actual
work. <code class="language-plaintext highlighter-rouge">AttributionLastClickJob</code> wants to distribute the 8 input shards into 8
batch jobs. We collect each job shard into <code class="language-plaintext highlighter-rouge">sub_jobs</code> list. When we do <code class="language-plaintext highlighter-rouge">yield
sub_jobs</code>, Luigi will concurrently run all the jobs listed in <code class="language-plaintext highlighter-rouge">sub_jobs</code>
list.
The task class above refers to something called <code class="language-plaintext highlighter-rouge">AttributionLastClickShard</code>. Here is the source code for that:
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code># lastclick.py - continued
import os
import pybatch
class AttributionLastClickShard(luigi.Task):
date = luigi.DateParameter()
s3_input_prefix = luigi.Parameter()
s3_output_prefix = luigi.Parameter()
shard = luigi.IntParameter()
def output(self):
return luigi.s3.S3Target(
os.path.join(self.s3_output_prefix,
self.date.strftime("%Y-%m-%d"),
"output.{}.sqlite3".format(self.shard)))
def run(self):
cmdline = [
"do_lastclick_attribution.py",
"--input-prefix", self.s3_input_prefix,
"--output-prefix", self.s3_output_prefix,
"--shard", self.shard
]
pybatch.run_on_awsbatch(container='attribution:1.0',
cmdline=cmdline,
cpus=16,
memory=110000,
jobqueue='attribution-job-queue')
if __name__ == '__main__':
luigi.run()
</code></pre></div></div>
This is where submitting the job to AWS Batch happens. In the <code class="language-plaintext highlighter-rouge">output()</code>
method, we define what result this job will ultimately create; in this example,
it is an SQLite file in S3.
In <code class="language-plaintext highlighter-rouge">run()</code> method, we define a command line that runs a Python script
<code class="language-plaintext highlighter-rouge">do_lastclick_attribution.py</code>. This command line is passed to AWS Batch and it
describes the command line arguments we pass to a Docker container when AWS
Batch runs it.
The call <code class="language-plaintext highlighter-rouge">pybatch.run_on_awsbatch</code> actually submits the job to AWS Batch. This
is an AdRoll-internal job submitting function that knows about AdRoll
infrastructure and simplifies the submission process. In your application, dear
reader, you would likely use <a href="https://boto3.readthedocs.io/en/latest/reference/services/batch.html">boto Python libraries to do
this</a>.
This function will only return when the job has completed successfully. If the
job fails, then <code class="language-plaintext highlighter-rouge">run_on_awsbatch</code> will throw an exception.
Running this job is not difficult. Just invoke the script with some arguments:
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ chmod +x lastclick.py
$ ./lastclick.py --date 2018-07-26 \
--s3-input-prefix s3://example-bucket/traildbs/attributiondbs \
--s3-output-prefix s3://example-bucket/attributions \
--shards 8 \
--workers 8
</code></pre></div></div>
This will invoke our distributed job, with Luigi handling concurrent submission to AWS Batch.
Most of our real batch jobs have been set up in this manner. There are some additional complexities
with our real systems I did not elaborate on in this example, such as:
<ul>
<li>
We have an automatic crontab generator that will set up a scheduling box
based on Luigi task files in a git repository. It will (try to) run all
jobs every 5 minutes. We rely on Luigi to stop duplicate jobs from running and stopping
any jobs that have already completed.
</li>
<li>
All jobs have been designed to be idempotent. If some job fails then we can safely resubmit it.
</li>
<li>
AWS Batch wants you to specify something called “job definitions”. If you
have worked with Amazon ECS, then these are AWS Batch equivalents to ECS
Task Definitions. The internal function <code class="language-plaintext highlighter-rouge">pybatch.run_on_awsbatch</code> creates job definitions
on the fly for given arguments.
</li>
<li>
Almost all batch jobs we have use S3 as the data store; inputs are pulled
from S3 and pushed to some other location in S3. All data is immutable. In
some cases outputs are pushed to a PostgreSQL database instead.
<a href="/blog/data/2016/05/24/traildb-open-sourced.html">We’ve had some blog posts related to immutable data before in our blog.</a>
</li>
<li>
The actual amount of data we have is much more than just one day with 8
shard files ;) We also have hundreds of different batch jobs, using a
diverse set of technologies.
</li>
</ul>
<h4 id="batchiepatchie">Batchiepatchie</h4>
Once jobs have been submitted, they will eventually run and (hopefully) will do
their job, but sometimes batch jobs will fail. Perhaps someone introduced a bug
in the code or the EC2 instance running the jobs failed for some reason. Maybe
some job became really slow because of quirks in the data.
To investigate and debug issues like this, you need monitoring.
Unfortunately, we feel the AWS Batch console in AWS Management Console leaves a
lot to be desired, especially if you really scale up your batch job use.
<ul>
<li>
It is difficult to get a holistic view of all jobs in the system. The UI in the management console
will not show all jobs neatly in a single view.
</li>
<li>
The management console forgets about jobs after some time.
</li>
<li>
Searching for jobs based on image name or command line arguments is hard.
In fact, searching for any particular job at all is hard.
</li>
<li>
Logs for any batch job is in CloudWatch logs; you have to painstakingly navigate
around the management console just to see job logs.
</li>
</ul>
If we only had a few jobs per day it would not be that bad. However, AdRoll
submits tens of thousands of batch jobs per day through AWS Batch. It was very clear we
needed a much better monitoring solution as we started to scale up.
And this is why we created
<a href="https://github.com/AdRoll/batchiepatchie">Batchiepatchie</a>. Batchiepatchie is a
monitoring tool for AWS Batch.
<img alt="Batchiepatchie screenshot" src="/images/post_images/batchiepatchie_screenshot.png" />
So what does Batchiepatchie do? Pedantically speaking, not much more than AWS
Management Console. It just makes certain use cases much faster.
<ul>
<li>
Batchiepatchie has a little search box that takes freeform text to find
jobs. We spent some time making this feature fast even with millions of
jobs in job history. This is by far the most important feature in
Batchiepatchie.
</li>
<li>
Batchiepatchie does not forget about jobs. It does not forget about your mistakes either. Ever.
</li>
<li>
Clicking on a job gives you an instant view of its standard output and error.
</li>
<li>
Terminating jobs is easy; select them on the dashboard and kill terminate job.
</li>
<li>
Batchiepatchie figures out IP address of the instances where batch jobs
run, making it easier to SSH inside a worker to monitor a job in more
detail when necessary.
</li>
<li>
It is easy to link a job page to a coworker.
</li>
<li>
Batchiepatchie implements timeouts for AWS Batch, something that is not
supported out of box. This requires some special logic from the code that
submits the batch jobs.
</li>
</ul>
When you have tens of thousands of jobs per day, these features can
(figuratively) save your life. Finding some specific failed job out of
thousands of succeeded jobs is much easier than trying to find it with AWS
Batch’s own user interface. Batchiepatchie became so useful at AdRoll that I
often see my coworkers refer to AWS Batch as “Batchiepatchie” even though our
monitoring tool is just, you know, a monitoring tool.
Batchiepatchie has some more features than listed here; I encourage you to look
inside the git repository and read through its documentation if you are
interested.
<h4 id="concluding-notes">Concluding notes</h4>
AdRoll is having a good experience with AWS Batch. Submitting thousands of jobs
per day is not a problem in itself; AWS Batch scales up quickly but Amazon’s
own monitoring solution is not great, which is why we decided to create a
custom monitoring tool designed to be great at finding specific jobs.
As we said, we use Luigi to organize our batch pipeline. AWS Batch does support
a simple form of jobs depending on other jobs but we almost always depend on
Luigi to handle our dependencies for us instead.
Compared to some other tools like Hadoop or Spark, AWS Batch is Docker-only.
This lets you use some pretty exotic technologies inside your batch jobs if you
wish to do so. We have Python-written jobs, C and C++ jobs, R jobs, Java jobs
and even some Rust and Haskell jobs, sometimes mixed together in one Docker
container. You can also select which EC2 instance types you want and get huge
boxes to do your computation. If you also use spot instances, you can get
these boxes cheap. At AdRoll, we even modified our system images to allow us to
memory map more files in a batch job than the Linux kernel allows by default for
some truly intensive batch jobs that memory map lots and lots of files.
If you need to build a batch processing pipeline and you like Docker a lot, AWS
Batch could be for you. It is a cost effective and a flexible batch job system.
We have saved a lot of engineering time by having a reliable batch job system
that can be used with a diverse set of technologies. Maybe you can save some of
your time as well!
<h4 id="links">Links</h4>
<ul>
<li><a href="https://aws.amazon.com/batch/">AWS Batch</a></li>
<li><a href="https://github.com/spotify/luigi">Luigi</a></li>
<li><a href="https://github.com/adroll/batchiepatchie">Batchiepatchie</a></li>
<li><a href="http://traildb.io">TrailDB</a></li>
</ul>
<hr />
Is your goal in life to build the ultimate batch job pipeline? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>If you build it, they will come.</title>
<link>https://tech.nextroll.com/blog/dev/2018/07/25/python-meetup-2018.html</link>
<pubDate>Wed, 25 Jul 2018 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2018/07/25/python-meetup-2018</guid>
<description>When the leadership team decided to move AdRoll HQ to the spacious new location in the Mission,
one has to wonder whether they had events such as last Wednesday’s evening gathering in mind.
A crowd of 100+ Python enthusiast filled the all hands area to share in some delicious catered
food and drinks as well as a bit of Python knowledge to boot.
The gathering was part of <a href="https://www.meetup.com/sfpython/">SF Python Groups monthly Project Night</a> event
which is held every 3rd Wednesday of the month.
<img src="/images/post_images/sfpython-patrick.jpg" alt="sfpythonpatrick" />
The guests started trickling in around 6:00 pm just as the freshly baked Empanadas were
being served. The food and ice cold beverages were provided by AdRoll as gracious hosts.
The proceedings were kicked off by <a href="https://www.meetup.com/sfpython/members/6993348/">Grace Law</a>, who’s one of the original founders of the
<a href="https://www.meetup.com/sfpython/">SF Python meetup group</a>. Grace introduced <a href="https://www.linkedin.com/in/patrickmee/">Patrick Mee</a>,
Head of Engineering at AdRoll,
who welcomed the growing crowd that had been networking while enjoying the delightful
food and drinks. Patrick highlighted the tech stacks used by the engineering teams at
AdRoll along with the reliance on Python as the glue to our various infrastructures.
The tutorials for the evening along with the AdRoll mentors were also introduced.
Next came the lightning round where the floor was opened up to the audience for announcements.
Whether one was seeking a new job opportunity or looking for mentorship and collaborators for
a private project, this was an opportunity for individuals from the crowd to come up and
introduce themselves to the gathering. The AdRoll HR team also had a presence with a booth
filled with cool AdRoll swag to give away.
After the announcements had concluded, the guests dispersed to various parts of the all
hands area to attend the tutorials for the evening. Some decided to just hang around to
work on their projects and get help from the mentors. In order to accommodate the large
group of people interested in attending the two AdRoll employee led tutorials, our amazing
IT team made available the training rooms where everyone could comfortably sit.
<a href="https://www.linkedin.com/in/brianandrewweiser/">Brian Weiser</a> lead an engaging and informative tutorial on server-side rendering of react.
This tutorial was geared for attendees with intermediate level competency of Python.
Brian’s demo walked the audience step by step on how to convert a todo application
written with React and a Flask backend to support server-side rendering.
<img src="/images/post_images/sfpython-brian.jpg" alt="sfpythonbrian" />
For those in the audience that were new to Python, <a href="https://www.linkedin.com/in/aasif-shabbir-84a08643/">I lead</a> an introductory tutorial on Python
and GraphQL. In the tutorial, I guided the audience on how to use the AdRoll GraphQL
Reporting API endpoints to write their very first GraphQL query and execute it.
<img src="/images/post_images/sfpython-aasif.jpg" alt="sfpythonaasif" />
On a personal level, I felt quite honored to be given an opportunity to host a tutorial for
the SF Python meetup event. It’s been 4 years, though it felt like it was just yesterday,
that I was in the audience attending my first Python meet up with the SF Python meetup group.
Being able to share the knowledge I’ve gained over the years with the crowd was an amazing
feeling and I’m truly grateful for AdRoll for being generous in providing a venue for
the <a href="https://www.meetup.com/sfpython/events/fmtsvnyxkbxb/">Project Night event</a>: but more importantly a place for people of varying skills
and knowledge to come together and share in ideas and learn from each other.
Are you interested in working with Python and joining in on all the fun? Come <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>
TRecs: Declutter Recommendations by going Real-time
</title>
<link>https://tech.nextroll.com/blog/adtech/2018/06/26/trecs-api.html</link>
<pubDate>Tue, 26 Jun 2018 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/adtech/2018/06/26/trecs-api</guid>
<description>As a performance-oriented growth platform, AdRoll is always looking for ways to improve our users’ ad experience. Our Dynamic ads are the highest performing ad type, but some of the underlying systems were becoming increasingly hard to scale and change. In this post, we will talk about how moving those systems to a real-time service made our product more nimble, scalable and easy to A/B test.
<code class="language-plaintext highlighter-rouge">15-20 minute read</code>
<hr />
<h1 id="high-level-problem">High Level Problem</h1>
Our Dynamic Ads product is designed to show a user ads with specific, recommended products based on their browsing history. Each advertiser gives us a list of all of its products beforehand so that we can generate a “product catalog” for the advertiser. Using this product catalog and <a href="http://tech.adroll.com/blog/adroll/2015/05/04/aws-summit-keynote.html">AdRoll’s raw logs</a>, we use algorithms to generate both per product and per advertiser recommendations daily via Amazon’s EMR service. Each ad is not associated to a single algorithm but to a combination of algorithms and weights called the product stream.
So, in order to serve relevant recommendations to our users based on their browsing history, we had an offline processing job that precomputed recommendations for each permutation of every product stream, for all the products in our every advertiser’s product catalog daily (and advertisers can have millions of products). These pre-generated recommendations were stored in a dynamodb table and served via an akamai fronted endpoint. 
While this process worked for awhile, we ran into both scaling and ad performance issues:
<ol>
<li>Only 1% of the 600 million (3TB of data) precomputed recommendations stored in DynamoDB table were being used on a daily basis.</li>
<li>The Javascript that loads at ad render time was responsible for getting recommendations from the DynamoDB table for every product that the user had interacted with. Because of the strict response time constraints, if the recommendations didn’t return in time and there weren’t enough products to animate the ad, a failsafe (static) ad was shown. Obviously, our customers who expect to see dynamic ads were not pleased that a percentage of their traffic was seeing static ads.</li>
<li>Our product recommendation A/B testing framework tests variations at a product stream level. Every time a product stream was created, new recommendations were generated for every product in our system for that product stream. Thus, every new experiment almost duplicated the entire data set, doubling Dynamodb writes everyday, thereby increasing costs and also deteriorating the table’s performance.</li>
</ol>
Given these challenges, we determined that a real-time system that would generate only the needed recommendations for each displayed ad made the most sense. We had to build a multi-region, light weight, highly available, easily scalable service with fast response time (under 50ms) and a robust fallback mechanism.
In order to accomplish this task, we needed to do the following:
<ul>
<li>Centralize all the recommendation metadata.</li>
<li>Replicate underlying data to all the regions to support a multi-region service.</li>
<li>Translate the offline processing job into a real-time API.</li>
<li>Deploy the new service in multiple regions.</li>
<li>Implement an effective load balancing solution.</li>
<li>Come up with a rollout plan to smoothly switch the production traffic to the new system and shutdown the old system.</li>
</ul>
<h1 id="design">Design</h1>
This diagram shows real-time recommendation system in one aws region.
<img src="/images/post_images/trecs/trecs_system.png" alt="recommendation system" />
<h3 id="data-sources">Data Sources</h3>
Our real-time recommendation service a.k.a TRecs has a stateless retrieval API that fetches information from the underlying data sources in real-time to generate recommendations. In this section, we will go over the the data sources needed for building recommendations and the modifications we did to each of them to make this data easily consumable by trecs.
<ol>
<li>
Recommendation metadata 
We have several EMR jobs in our system which are responsible for generating product recommendations for each of our recommendation algorithms. For legacy reasons, these EMR jobs wrote data to varied data sources like postgres, hbase. In order for the real-time API to work, we needed to aggregate all the algorithm data with a standardized scoring method in a low latency data store. 
We were looking for a fast, scalable, consistent and cost effective data store. Elastic cache and Dynamodb were our top contenders. Given the recommendation metadata is generated by batch jobs, the writes were expected to be non-uniform. Since dynamodb costs depends on throughput rather than storage, using dynamodb with autoscaling was a more cost effective option than elastic cache. Thus we chose dynamodb as our data store. 
We updated each EMR job to write the data to a new DynamoDB table where the key would be advertiser_id/algorithm/source_product_id and the value would be the recommended products, represented as a comma separated string of product_id’s with the first product listed being the most relevant recommendation. We leveraged the dynamodb’s BatchWrite interface to make the writes as fast and efficient as possible.
</li>
<li>
Product metadata 
Our product data pipeline already parsed our advertiser’s product catalogs and stored the product metadata in a Dynamodb table. Since the nature of writes for this datasource was also non-uniform, fetching data from dynamodb (with autoscaling enabled) was a high performant and cost effective option for TRecs. So, we didn’t have to tweak this data source.
</li>
<li>
Cookie Interactions data 
Our Ad servers record cookie interactions in every aws region in a Dynamodb table. Thus, this data was readily available for use.
</li>
<li>
Advertiser metadata 
TRecs needs advertiser meta information in real-time to filter out stale &amp; blacklisted products. This data is present in RDS(postgres) and as you can imagine, it doesn’t update as frequently. So, we created a task that runs every quarter of an hour and dumps the relevant data from postgres onto a file in S3. This file is then loaded into the TRecs server memory once every 15 mins. In order to fetch this file with minimum latency from all TRecs regions, we configured Akamai to front the S3 bucket.
</li>
</ol>
<h3 id="data-replication">Data Replication</h3>
Before TRecs service came into existence, most of our dynamic ad infrastructure (pipelines and data sources) was geo-located only in us-west AWS region and relied on Akamai’s CDN to serve product recommendations to our ads globally. In order to achieve fast response time with multi-region real-time recommendations service, the underlying data had to be available in all the aws regions where TRecs service was going to be hosted. Thus, we had to replicate recommendation metadata and product metadata to three other aws regions. Since Dynamodb was the underlying data store, we had to solve Dynamodb cross region replication problem.
Replicating Dynamodb data is a two part process:
<ol>
<li>
One time table copy 
We leveraged <a href="https://aws.amazon.com/about-aws/whats-new/2013/09/12/announcing-dynamodb-cross-region-copy-feature-in-aws-data-pipeline/">AWS Data Pipeline’s DynamoDB Cross-Region Copy</a> feature for doing the one time table copy. It uses s3 as the intermittent storage to copy the data. Launching the EMR job in the same region as the destination table and increasing the (read/write) provisioned capacities on both source and destination tables enabled us to complete the one time table copy process faster.
</li>
<li>
Setting up real-time updates 
Setting up real-time updates was non-trivial. Common idea was to apply live DynamoDB stream records from the source table to the destination table to get real-time updates. We had two options to leverage Dynamodb streams, either use an in-house developed solution or use <a href="https://github.com/awslabs/dynamodb-cross-region-library">AWSLabs Dynamodb cross region library</a>. The in-house solution needed a few tweaks to be usable but the aws solution almost worked out of the box for us, so we decided to go with that.
While this system worked well with small test traffic, we saw a lot of write throttles on the destination tables when we rolled <a href="https://github.com/awslabs/dynamodb-cross-region-library">AWSLabs Dynamodb cross region library</a> to production. Thereby ironically failing to replicate real-time updates. It turns out that Dynamodb streams don’t copy data at a uniform rate. Since both product metadata and recommendation metatadata source tables have bursty traffic and rely on autoscaling, the destination tables weren’t able to scale at the same rate as the source table causing records to be dropped.
Alas, we had to scratch the <a href="https://github.com/awslabs/dynamodb-cross-region-library">AWSLabs Dynamodb cross region library</a> solution and modified our product pipelines/EMR jobs instead to write to Dynamodb tables in all the regions. Since our offline systems only exist in us-west region, we had to scale them up to absorb additional network latencies that were incurred while writing to Dynamodb tables in three other AWS regions.
AWS recently announced <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GlobalTables.html">global tables</a> to solve for Dynamodb data replication problem. We are waiting for global tables to be available in the Asia Pacific regions to try it.
</li>
</ol>
<h3 id="real-time-recommendation-service-trecs">Real-time recommendation service (TRecs)</h3>
Our Real-time Recommendation service is built using <a href="http://www.dropwizard.io/1.2.2/docs/">Dropwizard</a>, a Java framework for RESTful web services. It runs within a docker container deployed on a standalone AWS ECS cluster. We use terraform templates to launch the underlying AWS infrastructure. It is deployed using jenkins2. We use datadog for montoring and logentries to collect logs. TRecs is integrated with JMX to collect jvm and jetty stats for monitoring and optimizing our environments.
<h4 id="the-trecs-api">The TRecs API</h4>
It’s a HTTP GET API which is called for all dynamic ads from Javascript at ad render time to fetch recommendations for a given user (cookie). The API does the following in real-time to generate recommendations:
<ol>
<li>Fetches cookie data 
The API gets a list of cookie interacted products from the cookie metadata table.</li>
<li>Fetches recommendation metadata 
As mentioned earlier, every dynamic ad is associated with a product stream. Each product stream has multiple recommendation sources associated with it and each recommendation source has a weight. The weight is used to determine how we should weigh various sources with respect to each other. Both per-product recommendations and per-advertiser recommendations are fetched from recommendation metadata and a confidence (or score) is assigned to each recommendation.</li>
<li>Processes recommendation metadata 
It’s very likely that the same product gets recommended by multiple sources, thus we need to “blend” those recommendations to come up with a normalized recommendation. Recommendations are then sorted on confidences and deduped.</li>
<li>Fetches product metadata 
For each of the recommended products, product metadata is fetched from the Dynamodb table.</li>
<li>Filters products 
We run a final check to filter out stale and blacklisted products before returning the list of recommendations to the dynamic ad.</li>
</ol>
To reduce network latency, the API leverages Dynamodb’s <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html">BatchGetItem</a> on every step of the way. Since each Dynamodb item is only a few bytes, a single batch call can fetch up to 100 items. This vastly improved our API’s response time.
<h4 id="load-balancer">Load balancer</h4>
TRecs API is fronted by an endpoint in the adserver. Since cookie values are only accessible from the same domain as they are set on, to get the adroll cookie, we had to route the dynamic ad traffic to TRecs via the adserver. 
To scale the service horizontally, we were looking for a fast,reliable and cost effective load balancer for TRecs. It was mostly between choosing AWS ALB service or an in-house developed load balancing erl library. While AWS ALB worked well, we realized that AWS not only charges for keeping the ALB running but also charges for per unit traffic served by it. Since the recommendation service will experience high throughput, AWS ALB would have been a very pricy option. 
Given, we were already forwarding the traffic to TRecs via the Erlang web server, it was fairly easy to integrate the in-house load balancing library in the same endpoint.
<h4 id="fallback-mechanism">Fallback mechanism</h4>
Coming from a system where all the recommendations were precomputed, Dynamic ads never experienced missing recommendations due to system outages. To ensure the dynamic ad performance remains unaffected during request failures / system outages, It was very important to have a robust fallback mechanism with reasonable response times. We deployed a “slim” version of the precomputed recommendations offline job to generate only per advertiser recommendations and store them in S3. We use a Akamai fronted s3 endpoint to fetch these recommendations to save us from replicating them across AWS regions while allowing faster response times.
<h1 id="rollout-plan">Rollout plan</h1>
We did the following to ensure a smooth switch to the new recommendation system -
<ul>
<li>Performed multiple load tests to make sure the TRecs service lives up to the throughput and response time SLA’s.</li>
<li>Turned on data replication to replicate underlying data sources to three other AWS regions</li>
<li>Deployed tasks to dump the needed data in S3</li>
<li>To rollout the traffic to our new recommendation system incrementally, we leveraged our recommendation A/B testing framework.</li>
<li>While scaling up the traffic to TRecs, we used presto to query our <a href="http://tech.adroll.com/blog/adroll/2015/05/04/aws-summit-keynote.html">raw logs</a> to validate the ad recommendation performance.</li>
<li>Refactored the javasccript code that runs at ad render time to use TRecs as the primary recommendation service</li>
<li>Migrated other systems at adroll to use TRecs</li>
<li>Sunset the old recommendation system</li>
</ul>
<h1 id="performance">Performance</h1>
We are very excited to share the improvements of our Real-time system
Fast response time 
TRecs API has a median response time of ~30ms and can support over 1 million requests per host per hour
<img src="/images/post_images/trecs/trecs_stats.png" alt="API response time" />
Easy to scale 
It is deployed as a ECS service which is easy to scale horizontally
Better product recommendations 
The recommendations shown in a dynamic ad is a combination of user interacted product recommendations and top products for that advertiser. Since TRecs filters stale products in real-time, our dynamic ads saw a 10% increase in user interacted recommendations. Showing more relevant recommendations leads to better ad performance.
No additional costs to A/B test new recommendation algorithms 
We don’t have to precompute recommendations to test new algorithms. This has unlocked the opportunity to test and rollout new algorithms which will help improve dynamic ad recommendations
Fewer network calls to fetch recommendations 
The dynamic Ad has to make one network call to fetch all the recommendations, making the ads respond faster
Easy to troubleshoot 
Previously the recommendation functionality was split between the offline recommendation job and the Javascript ad render component, making it difficult to troubleshoot recommendations served for a given advertiser, cookie combination. But with TRecs, debugging such issues has become a lot easier as all the functionality is wrapped in a single API.
It has been fun working on this project. We hope this post is helpful for people who are looking to transform their offline systems to real-time services using AWS services. This project is a good illustration of the type of work we do on the Dynamic Ads team. If this piques your interest <a href="https://www.adroll.com/about/careers/">Roll with Us</a>
</description>
</item>
<item>
<title>
Data Driven Women… and Allies!
</title>
<link>https://tech.nextroll.com/blog/culture/2018/05/17/ddw.html</link>
<pubDate>Thu, 17 May 2018 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/culture/2018/05/17/ddw</guid>
<description>Recently I attended the <a href="https://www.meetup.com/Data-Driven-Women/events/249711184/">Data Driven Women Meetup event</a> hosted by AdRoll Group, and felt inspired to share this blog post,
from a man’s perspective. I heard of the event through my work (AdRoll Group) and after reading up on the event details,
I realized four of my women colleagues were the featured speakers. It was at that time I remember thinking,
“I should really go to support them.”
Admittedly, I wasn’t even sure if men were allowed to attend, thinking it might just be a women only thing… wrong! I
reached out to one of my colleagues just to make sure and she quickly debunked my ill conceived thought process by
letting me know that men were not only invited, but encouraged to attend.
The event started out like most meetups with music, drinks, snacks, and networking. There was a very inviting feel to
the room, with couches and tables set up to create a comfortable environment where people could sit, talk and listen
to the featured speakers. The space filled up quickly and I noticed that I was not the only man in room; there were
also quite a few of our male coworkers in attendance. During the
networking portion of the evening, there were working
sessions where attendees could get tips on crafting their resumes, interviewing, and other career advice from a wide
range of women (and men) in varying levels of seniority from different companies. These sesions were sponsored by
<a href="http://albertslist.org/">Alberts List</a>, <a href="http://cybersn.com">cybersn</a> and <a href="https://hirepool.io/">hirepool</a>.
The featured speakers for the evening included a few of the female leaders on the Product and Engineering teams here
at AdRoll Group:
<h3 id="kelly-eng-senior-product-manager">Kelly Eng, Senior Product Manager</h3>
<img alt="Kelly" src="/images/post_images/post_ddw/Kelly_Eng.png" style="max-width: 300px" />
<h3 id="julie-zhou-director-of-product">Julie Zhou, Director of Product</h3>
<img alt="Julie" src="/images/post_images/post_ddw/Julie.png" style="max-width: 300px" />
<h3 id="jessica-grist-senior-software-engineer">Jessica Grist, Senior Software Engineer</h3>
<img alt="Jessica" src="/images/post_images/post_ddw/Jessica.jpg" style="max-width: 300px" />
<h3 id="miriam-pena-staff-engineer">Miriam Pena, Staff Engineer</h3>
<img alt="Miriam" src="/images/post_images/post_ddw/MiriamPena.jpg" style="max-width: 300px" />
Throughout the evening each speaker shared her own unique story and experience of how she broke into her career and current role, amidst what is still today a male-dominated industry. As each of my coworkers told their story, it helped me to understand what motivated them to pursue their own career in tech, the approaches and paths they each took to navigate the course, as well as the personal and political challenges they faced along the way. Although each experience and path was different, I noticed a few themes emerge.
1.)Overcoming fear- Whether it was fear of changing careers later in life, stepping into a lead role for the first time, applying for a position with non-traditional experience, or even public speaking, it was inspiring to hear each speaker talk openly about their own fears, which as a society we often consider to be vulnerability or weakness. It’s only after we expose our fears for what they are that we can then start to make strides towards overcoming them to become stronger and more confident.
2.)The importance of aligning yourself with the right mentors and/or allies - This could come in the shape of an all female coding bootcamp, a strength training coach, a male coworker willing to push you, or retracing the steps of early engineering languages to find that strong females were a big part of the early history of tech.
3.)The importance of using your voice to help other women - Not only women currently or soon to be in the job market, but also the young women of the world who are dreaming about what they want to be when they grow up. We all know the power and encouragement that comes with being able to self-identify with someone who is accomplished in a particular field of interest. It’s infectious! Point being, we need more female leaders in tech to help show the future generations of ambitious women that tech is a career they, too, can pursue.
I’ve always had a deep rooted respect and support for women, but after attending this event I have gained a newfound appreciation for not only my coworkers, but also the need for further support from male allies and industry leaders to make sure the tech industry is a place where people from all backgrounds feel welcome.
You can watch the recording here:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/wQsNWxoS92s" frameborder="0" allowfullscreen=""></iframe></div></figure>
Are you interested in working with a team of enthusiastic developers in a remote-friendly globally distributed team? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>
BuzzConf BA 2018: A Conference for Developers by Developers
</title>
<link>https://tech.nextroll.com/blog/dev/2018/05/08/buzzconf-2018.html</link>
<pubDate>Tue, 08 May 2018 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2018/05/08/buzzconf-2018</guid>
<description>Highlights from the BuzzConf BA 2018 conference. April 26th, 2018
<hr />
<h1 id="buzzwords-as-the-motivation-for-a-conference">Buzzwords as the Motivation for a Conference</h1>
As google promptly help us define, buzzword is…
<blockquote>
a word or phrase, often an item of jargon, that is fashionable at a particular time or in a particular context.
</blockquote>
In the world of software, these words come and go all the time. While they’re on the spot, everbody talks about them.
That means the amount of information spread with everyone talking about them is minimal. You get lost in so many articles, videos, tweets, etc. and it’s hard for anybody to really understand the subject and learn about it.
That’s where <a href="http://buzzconf.org/">BuzzConf</a> comes in. The organizers (including <a href="https://twitter.com/iraunkortasuna">Iñaki Garay</a> and <a href="http://twitter.com/folano_">Facundo Olano</a> from our Supply Engineer Team) created a conference that, in one day, covered most of the buzz-word-worthy topics that are around today. We had talks about Space, Cryptocurrencies, Diversity &amp; Inclusion, Functional Programming, Machine Learning, Video Games, AI, Distributed Systems, Rust, Elixir and others. Each talk was delivered by someone that is actually an expert in the field and they all were everything but superficial.
<h2 id="the-minimalistic-conference">The Minimalistic Conference</h2>
BuzzConf was a one-track conference, with 12 talks in quick succession. There ware no fancy introductions, no conference t-shirts, barely one or two banners with sponsor logos, no contests, no social media promotion, etc. The organizers specifically avoided all the superfluous stuff that abounds in other venues to focus on the most important parts:
<ul>
<li>Great speakers, checked!</li>
<li>High quality content, checked!</li>
<li>Space and time for socializing and networking, checked!</li>
<li>Cheap tickets so that anybody can join, checked!</li>
<li>As much learning as you can fit in a day, checked checked checked!</li>
</ul>
For a conference with BuzzWords as its main subject, I think they did an amazing job trimming away all the noise and providing as much signal as they could.
<h2 id="highlights">Highlights</h2>
From the 12 talks, I’ll briefly describe my favorites below. I encourage you to check the rest of them on <a href="https://github.com/lambdaclass/buzzconf/tree/master/slides">BuzzConf’s GitHub Repo</a>.
<h3 id="cryptocurrencies--blockchain">Cryptocurrencies / BlockChain</h3>
We had two talks on the subject of crypto currencies, blockchain and decentralized apps.
I particularly enjoyed the talk by <a href="http://twitter.com/smpalladino">Santiago Palladino</a> (from <a href="http://zpl.in">Zeppelin</a>) in which he presented many applications of blockchain mechanisms (with examples from Ethereum) in a very objective way, providing pros and cons and paths to move forward. He managed to pack a lot of knowledge in a relatively short time and left us all wondering what else is out there for us to get our hands on.
<a href="https://github.com/lambdaclass/buzzconf/blob/master/slides/02.%20Santiago%20Palladino/BuzzConf%202018%20-%20Crypto%20beyond%20currencies%20%5BShared%5D.pdf"><img src="/images/post_images/buzzconf-thumbnail-santi.png" alt="santi-thumbnail" /></a>
<h3 id="diversity--inclusion">Diversity &amp; Inclusion</h3>
<a href="www.dianamaffia.com.ar">Diana Maffía</a> gave a very enthusiastic talk about the role of women in science both in Argentina and the world, highlighting the work that <a href="http://www.ragcyt.org.ar/">ragcyt</a> (Argentinian Network about Gender, Science and Technology) has been doing over the last decades (they started in 1999). One of their goals is to objectively understand the current situation and challenges that women face when attempting to pursue a career in science (particularly in Argentina) and provide advice and guidance for everyone trying to build a more inclusive world.
<a href="https://github.com/lambdaclass/buzzconf/blob/master/slides/04.%20Diana%20Maff%C3%ADa/buzz%20conf-%20abril%202018%20%5BModo%20de%20compatibilidad%5D.pdf"><img src="/images/post_images/buzzconf-thumbnail-diana.png" alt="diana-thumbnail" /></a>
<h3 id="video-games">Video Games</h3>
Even at a conference focused on new and emerging topics, we still had time to talk about the usual suspects. We had a talk about Linux Kernel and also an unexpectedly great talk about video games. In this talk, <a href="https://twitter.com/reduzio">Juan Linietsky</a> presented his framework: <a href="https://godotengine.org/">Godot Engine</a>. Godot is a very advanced video game framework that allows you to develop 2D and 3D games with a lot of flexibility. It’s focused on teamwork and, the most important part: It’s 100% open-source!.
<a href="https://github.com/lambdaclass/buzzconf/blob/master/slides/08.%20Juan%20Linietsky/Godot%20Engine%20-%20Introducci%C3%B3n%20Informativa.pdf"><img src="/images/post_images/buzzconf-thumbnail-juan.png" alt="juan-thumbnail" /></a>
<h3 id="property-based-testing">Property-Based Testing</h3>
With examples in <a href="https://elixir-lang.org/">Elixir</a>, but a presentation that spoke to everyone regarding of their favorite language, <a href="https://twitter.com/whatyouhide">Andrea Leopardi</a> introduced the audience to and promoted the usage of <a href="https://elixir-lang.org/blog/2017/10/31/stream-data-property-based-testing-and-data-generation-for-elixir/">Property-Based Testing</a>. This is not a new concept for some of us, but the amount of work in this area in the last few years is astonishing. In his talk, Andrea presented the basics of this technique and also showed many of the latest developments and future plans for Elixir in particular and property based testing in general. It was inspiring.
<script async="" class="speakerdeck-embed" data-id="2be1ee1eb1c449da997ac515622bcbde" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
<h3 id="distributed-systems">Distributed Systems</h3>
After the last break, everyone was ready to enjoy the closing keynotes. And we were not dissappointed! In the first one, <a href="http://christophermeiklejohn.com">Christopher Meiklejohn</a> gave us a crash course into distributed systems and showed some of the most important challenges of this area today. Then he presented some proposed solutions to those challenges, involving tools like <a href="https://github.com/lasp-lang/partisan">Partisan</a> and his work on CRDTs. It was a very interesting talk that left us with a bright vision of the future of distributed systems, particularly those written in <a href="http://www.erlang.org">Erlang</a>.
<a href="https://github.com/lambdaclass/buzzconf/blob/master/slides/11.%20Christopher%20Meiklejohn/Keynote%20-%20Buzzconf.pptx?raw=true"><img src="/images/post_images/buzzconf-thumbnail-chris.png" alt="chris-thumbnail" /></a>
<h3 id="rust">Rust</h3>
Finally, the last talk of the day was delivered by the always amazing <a href="steveklabnik.com">Steve Klabnik</a>. Of course, his talk was about <a href="https://www.rust-lang.org/">Rust</a>, but it was certainly not your usual introductory hello-world talk. While showcasing the main features of the language he created, he also showed some advanced concepts that left many of us looking for excuses to use this fantastic language in some of our upcoming projects.
<a href="https://github.com/lambdaclass/buzzconf/tree/master/slides/12.%20Steve%20Klabnik"><img src="/images/post_images/buzzconf-thumbnail-steve.png" alt="steve-thumbnail" /></a>
<h2 id="last-thoughts">Last thoughts</h2>
For people in remote parts of the southern hemisphere, away from the places were most of the greatest conferences are organized every year, being able to attend a conference with high quality speakers giving great talks on relevant subjects is something unheard of.
Some of us, we have the fortune of working remotely for a company like AdRoll that allows us to travel up north and enjoy conferences like <a href="/blog/dev/2018/04/12/codebeam-sf-2018.html">CodeBEAM</a>. Yet everybody here would love to have more conferences and events in our home towns. That’s why we decided to take matters into our own hands and organize our own events, either <a href="http://www.erlang-factory.com/eflba2017">in person</a> or <a href="http://spawnfest.github.io">online</a>. This movement is growing and the community is growing with it. With accessible conferences like BuzzConf that give you a lot of great content for a more than fair price, everybody wins!
Are you interested in working with a team of enthusiastic developers in a remote-friendly globally distributed team? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>
Making 1M Click Predictions per Second using AWS
</title>
<link>https://tech.nextroll.com/blog/data-science/2018/04/26/just-binary-classifier.html</link>
<pubDate>Thu, 26 Apr 2018 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data-science/2018/04/26/just-binary-classifier</guid>
<description>Click prediction may be a simple binary classification problem, but it requires a robust system architecture to
function in production at scale. At AdRoll, we leverage the AWS ecosystem along with a suite of third party tools to build the predictors that
power our pricing engine, BidIQ.
This is a tour of the production pipelines and monitoring systems that keep BidIQ running.
<hr />
<h1 id="introduction">Introduction</h1>
AdRoll participates on <a href="https://en.wikipedia.org/w/index.php?title=Real-time_bidding&amp;oldid=836020656">Real Time Bidding (RTB)</a> exchanges to display ads for our advertisers. When a user visits a website,
a real-time auction takes place while the page loads. AdRoll submits an ad and a bid price, and if our bid wins, our ad is displayed.
In more detail, AdRoll has a number of machines we call “bidders,” which are integrated to the various ad exchanges which run RTB auctions.
These exchanges send “bid requests” to our bidders. For each bid request, our bidders determine a list of ads we might show and
for each ad, our bidders query <a href="https://blog.adroll.com/product/reintroducing-bidiq-the-intelligence-that-powers-adroll-campaigns">BidIQ</a>, our pricing engine, to determine what price we are willing to bid for it. Then the bidders take the ad with the highest bid price and send it to the exchange.
<img src="/images/post_images/just_binary_classifier/auction.png" alt="Auction Flow" />
The Data Science Engineering team works on <a href="https://blog.adroll.com/product/reintroducing-bidiq-the-intelligence-that-powers-adroll-campaigns">BidIQ</a> and the loose coupling between the bidders and BidIQ gives us complete flexibility in how we price each ad
and what infrastructure we use. BidIQ determines the bid price for an ad based on the advertiser’s campaign goals and a series of models
that predict the probabilities that the ad will be viewable, that the user will click our ad, and that the user
will go on to take an “action” (e.g. buy something), among others.
In this blog post I’ll focus on BidIQ’s click predictor, which we built first and have used the longest. When I tell people familiar with machine learning
about it, they are often skeptical that there is much to do: “Isn’t that just a binary classifier?”
Indeed click prediction is a classic binary classification problem.
We can build a dataset with one line for each impression (displayed advertisement), labeled 1 or 0 according to whether it was
clicked or not, along with a list of features (time of day, geographic location, etc.) for that impression. Then for a fresh impression, our problem is to predict
how likely it is to be 1 based on its features.
<img src="/images/post_images/just_binary_classifier/dataset.png" alt="Dataset" />
Now standard machine learning approaches apply. We can split the dataset into a train and test set, choose a subset of features, train a logistic
regression on the train set, evaluate it on the test set, and repeat with different subsets of features until we find the subset
which minimizes the average log-loss on the test set. Then, after training a model with the selected subset of features
over the whole set of data, we can use it to make click predictions for future ads.
Were this a textbook exercise, we’d be done, yet we are just getting started. This is where the “Engineering” in Data Science Engineering comes in.
For our production system we need to:
<ul>
<li>Automate everything: training set generation, model training, model deployment, and updates</li>
<li>Update models frequently, and with zero downtime; we need to price ads 24/7</li>
<li>Handle a throughput of one million queries per second, responding to each with less than 20 milliseconds of latency</li>
<li>Facilitate model improvements and feature discovery by our engineers in parallel</li>
<li>Allow multiple live A/B tests of different models</li>
<li>Monitor our production pipeline carefully as well as the inputs and outputs of the model</li>
</ul>
To build our system architecture satisfying these requirements, we use the AWS ecosystem heavily as well as a variety
of third party and open source tools. A list of these tools and their descriptions is in the <a href="#appendix">Appendix</a>.
<h1 id="training-the-model">Training The Model</h1>
There are three steps in our training pipeline:
<ol>
<li>Generate the training dataset</li>
<li>Train a model against the training dataset</li>
<li>Wrap the model with serving code in a docker image for easy deployment</li>
</ol>
We handle these steps with <a href="https://www.docker.com/">Docker</a>, <a href="https://aws.amazon.com/batch/">AWS Batch</a>, and <a href="https://github.com/spotify/luigi">Luigi</a>.
<img src="/images/post_images/just_binary_classifier/pipeline.png" alt="Pipeline" />
Our starting point is the impression and click logs our bidders generate and store on <a href="https://aws.amazon.com/s3/">S3</a>.
For step one, we’ve built a Docker image which takes a time range, downloads
the impression and click logs in that time range, joins them, and uploads the resultant dataset to S3.
To generate the model, we’ve built a Docker image which takes a time range
and model configuration, downloads the training dataset in that time range, trains a model against the dataset, and uploads it to S3.
Finally, we download the model, combine it
with serving code in a Docker image, and upload it to our <a href="https://aws.amazon.com/ecr/">ECR</a> repository. This final image can then be easily deployed to any <a href="https://aws.amazon.com/ec2/">EC2</a> instance where it can price queries on a
specified port.
These tasks are placed in a Luigi pipeline and kicked off via AWS Batch when the upstream requirements are satisfied. This ensures that
freshly generated logs are joined and incorporated into our most recent model with minimal delay and computing resources.
<h1 id="deploying-and-updating-the-model">Deploying and Updating the Model</h1>
<h2 id="model-deployment">Model Deployment</h2>
AdRoll displays ads to users all over the world. Given the tight latency restrictions of RTB, we’ve placed our bidders all over the world as well.
More concretely, our bidders run on EC2 instances in multiple AWS regions. Thus, in order to keep the latency between the
bidders and BidIQ under 20ms, we also need to deploy BidIQ servers in each AWS region. Furthermore, in each region we may need BidIQ servers
for multiple versions of the model (e.g. for A/B tests).
Architecturally, we handle this by having one <a href="https://aws.amazon.com/autoscaling/">Auto Scaling Group (ASG)</a> per (BidIQ Version, AWS Region) pair. Then, by changing the size of the ASG, we
can trivially scale up or down the throughput of ad auctions that BidIQ can price. For example, we can scale down BidIQ versions which are only tested on a small percentage
of traffic or scale up heavily trafficked AWS regions.
We use <a href="https://www.terraform.io/">Terraform</a> to manage the deployment of the BidIQ ASGs and store the generated <code class="language-plaintext highlighter-rouge">.tfstate</code> files on S3, sharded by BidIQ version and AWS region. In this way, any engineer
can easily deploy or modify the ASG for a given (BidIQ Version, AWS Region) with a simple <code class="language-plaintext highlighter-rouge">terraform apply</code>.
<h2 id="service-discovery">Service Discovery</h2>
So now we have BidIQ ASGs serving the model, and bidder instances which need to query the model. How can the bidders know which host and port
to query to reach a given BidIQ version?
For this we use a <a href="https://aws.amazon.com/dynamodb/">DynamoDB</a> table called <code class="language-plaintext highlighter-rouge">Services</code>. We have one such table per AWS region, and each BidIQ instance in that region
announces itself on that table. The BidIQ instance periodically heartbeats an entry to the table consisting of:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HOST, PORT, BIDIQ_VERSION, EXPIRY_TS
</code></pre></div></div>
BidIQ upholds the contract:
<ol>
<li>Each <code class="language-plaintext highlighter-rouge">(HOST, PORT)</code> can be queried for that <code class="language-plaintext highlighter-rouge">BIDIQ_VERSION</code> until the current time exceeds <code class="language-plaintext highlighter-rouge">EXPIRY_TS</code>.</li>
<li>For each <code class="language-plaintext highlighter-rouge">BIDIQ_VERSION</code> there will be sufficient <code class="language-plaintext highlighter-rouge">(HOST, PORT)</code> tuples to meet throughput.
To guarantee this, BidIQ ensures both:
<ul>
<li>For each <code class="language-plaintext highlighter-rouge">HOST</code>, at least one <code class="language-plaintext highlighter-rouge">PORT</code> is available</li>
<li>We have sufficiently many <code class="language-plaintext highlighter-rouge">HOST</code>s for the given <code class="language-plaintext highlighter-rouge">BIDIQ_VERSION</code></li>
</ul>
</li>
</ol>
<h2 id="model-updates">Model Updates</h2>
Our pipeline is constantly pushing updates to the model as more recent data come in. We need to swap in these fresh models
while upholding the contract above.
To do this, each instance in the BidIQ ASG uses the following logic, orchestrated by a Python script:
<ul>
<li>Every five seconds, query the locally running model as a health check. If this health check fails, restart the model.</li>
<li>Every minute, check for an updated model. If found:
<ol>
<li>Download the new model and run it on a new port</li>
<li>Start heartbeating the new port to the <code class="language-plaintext highlighter-rouge">Services</code> table</li>
<li>Stop heartbeating the old port to the <code class="language-plaintext highlighter-rouge">Services</code> table</li>
<li>After a grace period for draining, shut down the model on the old port</li>
</ol>
</li>
</ul>
This logic is simple, allows for smooth model updates with no down time, and requires no coordination between individual instances in a BidIQ ASG. And
our bidders don’t need to know anything about the internals of BidIQ. They simply look up a <code class="language-plaintext highlighter-rouge">(HOST, PORT)</code> in the <code class="language-plaintext highlighter-rouge">Services</code> table for a given <code class="language-plaintext highlighter-rouge">BIDIQ_VERSION</code> and
query it to get a bid price.
<h1 id="improving-the-model">Improving the Model</h1>
I’ve explained how we handle model updates for a given version of BidIQ, but how do we improve the model to make new versions of BidIQ?
Model improvement is broken into two phases, Backtesting and live A/B testing.
<h2 id="backtesting">Backtesting</h2>
For backtesting new models, we have a simple web application, “Juxtaposer”. A user can define an experiment via a YAML configuration consisting of which versions of our code to use,
what features to use in the model, which train and test sets to use, and even the Docker image to run the experiment in.
We then kick off this dockerized experiment via AWS Batch.
When finished, it reports metrics back to the web app, where we can compare them side-by-side (hence the name “Juxtaposer”). With Docker and AWS Batch it is trivial to kick off
an arbitrarily large number of experiments in parallel, and our engineers can focus on improving the model without worrying about infrastructure.
<figure>
<img src="/images/post_images/just_binary_classifier/juxtaposer.png" alt="Juxtaposer experiment comparison" />
<figcaption>Juxtaposer comparison of two experiments. Some metrics have been redacted</figcaption>
</figure>
For feature engineering, we typically look at standard binary classification metrics (average log-loss, AUC, etc.) on the test set, whereas for changes to our code, we look out for
regressions—backtesting gives us a dry run of how the model will behave in production.
<h2 id="live-ab-testing">Live A/B Testing</h2>
Once we have a new model that looks good in backtesting, we A/B test it live. For this we use <a href="/blog/rtb/2016/04/29/collider.html">Collider</a>. In a nutshell, Collider instructs
the bidders to apply custom logic to a fixed percentage of live traffic. For our purposes, we simply tell the bidders to use a specified <code class="language-plaintext highlighter-rouge">BIDIQ_VERSION</code> rather than the
production version.
For A/B testing, in addition to binary classification metrics, we look at broader ad-tech metrics such as our observed cost-per-click (CPC) or cost-per-action (CPA). If the test version of BidIQ
looks good we roll it out as the new production version.
<h1 id="monitoring-our-production-model">Monitoring our Production Model</h1>
With the great power of BidIQ (pricing one million ads per second, each within 20ms, 24/7) comes great responsibility: BidIQ is an automated system which controls how we spend money so it
is imperative that we monitor it closely, especially as much of the data it ingests comes from ad exchanges, which may introduce changes outside of our control.
To start, we use <a href="https://www.datadoghq.com/">Datadog</a> for real-time monitoring. Each BidIQ server outputs simple metrics in UDP packets, using the <a href="https://github.com/etsy/statsd">StatsD</a> protocol, which are picked up by
a local Datadog agent, aggregated, and displayed in a real-time dashboard. In this way, we can see our live average predictions and average bids across a number of dimensions
and send a <a href="https://www.pagerduty.com/">Pager Duty</a> alert if something suddenly changes.
<figure>
<img src="/images/post_images/just_binary_classifier/datadog.png" alt="Datadog stacked area chart" />
<figcaption>Realtime stacked area chart via Datatog of queries/sec by AWS region</figcaption>
</figure>
Next, our bidders log the bid prices and click predictions that BidIQ returns in our impression and click logs on S3. Using <a href="https://prestodb.io/">Presto</a>, we can
easily query these logs with SQL. With some simple SQL queries we can check, on various subsets of traffic, whether the ratio of expected clicks (sum of click-probabilities over impressions)
to actual clicks is close to one, as it should be. With slightly more complicated queries we can build <a href="http://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html">calibration curves</a> to verify that our predictions
have been accurate.
<figure>
<img src="/images/post_images/just_binary_classifier/calibration.png" alt="Calibration chart" />
<figcaption>Calibration chart -- points on the dotted line are perfect.</figcaption>
</figure>
Next, we expose our joined training logs to Presto so we can directly query the data BidIQ is ingesting. We also have a script that taps our live BidIQ servers (using <code class="language-plaintext highlighter-rouge">tcpdump</code>)
to see exactly what data our bidders are sending them. These have proven invaluable in investigating changes in upstream data.
Next, most of our tasks deliver email messages on failure, and our task to train the model runs additional sanity checks to ensure that parameters such as average prediction over
the test set are within reasonable bounds. We also have a catch-all monitoring repository, “Night’s Watch,” which contains a series of Python scripts which send us warning
emails if anything is off (e.g. a BidIQ model hasn’t been updated recently).
Finally, each of our code repos has a thorough set of unit and integration tests.
<h1 id="conclusion">Conclusion</h1>
Hopefully the above gives you a sense of the architecture and infrastructure involved in bringing even a simple binary classifier into stable production at scale.
And that’s just our high-level architecture. With our predictors being so critical to our business, we’ve written custom code to train and serve our models in a
low-level language, <a href="/blog/data/2014/11/17/d-is-for-data-science.html">D</a>. We’ve optimized our D code for our use cases: we’ve disabled garbage collection (violates our latency requirement), removed any allocations on the heap,
rewritten performance bottlenecks using <a href="https://en.wikipedia.org/w/index.php?title=Intrinsic_function&amp;oldid=827077730">intrinsics</a>, added logic to price a batch of ads at a time with multithreading, and more. And on the algorithm side, we now use an extension of logistic regression, <a href="/blog/data-science/2015/08/25/factorization-machines.html">factorization machines</a>.
So far we’ve only discussed our click predictor. We have a number of other predictors with their own quirks as well as an ever-evolving set of logic to choose a bid price in the face
of evolving auction dynamics, notably the recent rise in <a href="http://adprofs.co/beginners-guide-to-header-bidding/">header bidding</a>.
The best thing about working on Data Science Engineering at AdRoll is that you work at the intersection of so many interesting fields: math, statistics, computer science,
machine learning, computer networking, economics, game theory, and even web development. Most of our projects overlap with a number of these fields, and our team has a diverse group of engineers with backgrounds
across these fields so there is always something new to learn and someone to learn from. We are regularly looking for new engineers so if these areas interest you, <a href="https://www.adroll.com/about/careers/">let us know</a>!
<h1 id="-appendix"><a name="appendix"></a> Appendix</h1>
List of AWS tools we use
<ul>
<li><a href="https://aws.amazon.com/autoscaling/">ASG</a> – scalable group of EC2 instances</li>
<li><a href="https://aws.amazon.com/batch/">Batch</a> – easy way to run a docker image</li>
<li><a href="https://aws.amazon.com/dynamodb/">DynamoDB</a> – low latency NoSQL database</li>
<li><a href="https://aws.amazon.com/ec2/">EC2</a> – on-demand machines to run our code</li>
<li><a href="https://aws.amazon.com/ecr/">ECR</a> – repo for docker images</li>
<li><a href="https://aws.amazon.com/s3/">S3</a> – cloud storage for flat files</li>
</ul>
List of third party / open source tools we use
<ul>
<li><a href="https://www.datadoghq.com/">Datadog</a> – reporting for live metrics</li>
<li><a href="https://www.docker.com/">Docker</a> – simple containerization</li>
<li><a href="https://github.com/spotify/luigi">Luigi</a> – simple pipeline logic to kick off jobs once upstream ones finish</li>
<li><a href="https://www.pagerduty.com/">Pager Duty</a> – alarm system to alert engineers when something goes wrong</li>
<li><a href="https://prestodb.io/">Presto</a> – fast SQL query engine which can also query large csv files on S3</li>
<li><a href="https://www.terraform.io/">Terraform</a> – simple declaritive language to create and modify AWS infrastructure</li>
</ul>
<hr />
Do you enjoy designing and deploying machine learning models at scale? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>
CodeBeam SF 2018: Erlang and Elixir in production
</title>
<link>https://tech.nextroll.com/blog/dev/2018/04/12/codebeam-sf-2018.html</link>
<pubDate>Thu, 12 Apr 2018 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2018/04/12/codebeam-sf-2018</guid>
<description>Highlights from the CodeBeam SF 2018 conference, March 15-16 2018
<hr />
<h2 id="once-was-erlang-factory">Once was Erlang Factory</h2>
The name has changed - yearly in fact, for the past few years - but this conference is always a destination for the <a href="http://www.erlang.org">Erlang</a> faithful. The premise has been broadened since the <a href="http://www.erlang-factory.com/past_conferences">Erlang Factory</a> days of yore to welcome other <a href="http://www.erlang-factory.com/upload/presentations/708/HitchhikersTouroftheBEAM.pdf">BEAM</a> based languages, the most prominent of those being <a href="https://elixir-lang.org">Elixir</a> of course, but worthy mentions here are <a href="http://lfe.io">LFE (Lisp Flavored Erlang)</a> as well as the more experimental efforts happening: <a href="https://github.com/rvirding/erlog">Erlog</a>, <a href="http://clojerl.org">Clojerl</a>, <a href="https://github.com/rvirding/luerl">Luerl</a>, <a href="http://efene.org">Efene</a>, and <a href="https://github.com/alpaca-lang/alpaca">Alpaca</a>. I find the big tent to be a really great thing for the Erlang ecosystem in general. It’s good to be reminded that one of the lasting beauties of Erlang is the reusability and relevance of <a href="http://www.erlang-factory.com/upload/presentations/708/HitchhikersTouroftheBEAM.pdf">BEAM</a> itself to the networked world we work in.
The subplot this year is a good one: 2018 marks the 20th year since Ericsson first released Erlang as open source! This has had much positive influence on the lifetime of this language and BEAM.
<h2 id="highlights">Highlights</h2>
The two-day conference featured 50 speakers in all, with two keynotes each day. The organizers, <a href="https://www.codesync.global">Code Sync</a>, have merged the long-running <a href="http://www.erlang-factory.com/past_conferences">Erlang Factory</a> event series under their umbrella in the past year, alongside the <a href="https://www.codesync.global/conferences/#CodeElixir">Code Elixir</a> and <a href="https://www.codesync.global/conferences/#CodeMesh">Code Mesh</a> series. This seems to have worked out well. The Code Beam conference certainly went very smoothly - the scheduling made sense, presentation technology worked properly, and the volunteers were knowlegeable and helpful. I should note the integration of the <a href="https://whova.com">Whova</a> app for navigating around between talks, the darned thing actually worked really well.
The conference was organized into six <a href="https://www.codesync.global/conferences/code-beam-sf-2018/#Themes">themes</a>: introductory tutorials and overviews of the Erlang ecosystem; the state of the BEAM toolset for production environments. deployment, maintenance, monitoring, and testing; case studies in high reliablity system building; in-depth BEAM topics, including compiler implementation issues, language extensions, and active areas of research; frameworks including Phoenix, RabbitMQ, Nerves, MongooseIM and others; and last but far from least distribution, concurrency, multicore and functional programming in the Erlang ecosystem.
The following list of highlights is fairly randomly chosen across those tracks, as there were quite a few contenders. This is just a sampling of talks that struck me as particularly memorable - a number of solid tutorials and more general reflections are skipped here. I highly recommend paging through the full set of recorded talks on <a href="https://www.youtube.com/results?search_query=Code+BEAM+SF+2018">on youtube</a>.
<h2 id="openerlang-party">#OpenErlang Party</h2>
The organizers celebrated 20 years of open source Erlang by throwing <a href="https://www.meetup.com/CodeBEAMSF/events/248435574">a very nice bash</a> at <a href="https://www.galvanize.com">Galvanize</a> the first evening. <a href="https://www.codesync.global/speaker/miriam-pena">Miriam</a> worked the room:
<h5 id="miriam-and-robert-virding">Miriam and Robert Virding</h5>
<img src="/images/post_images/miriamandrobert.jpeg" alt="miriam-and-robert" />
<h5 id="miriam-and-irina-guberman">Miriam and Irina Guberman</h5>
<img src="/images/post_images/miriamandirina.jpeg" alt="miriam-and-irina" />
<h5 id="miriam-and-andrew-thompson">Miriam and Andrew Thompson</h5>
<img src="/images/post_images/miriamandandrew.jpeg" alt="miriam-and-andrew" />
<h5 id="miriam-and-lennart-öhman">Miriam and Lennart Öhman</h5>
<img src="/images/post_images/miriamandLennart.jpeg" alt="miriam-and-Lennart" />
<h2 id="adroll-content">AdRoll Content</h2>
I was most excited about the representation by AdRoll among the speakers, unsurprisingly! Three members of the RTB team spoke, with an eye-opening historical keynote about the significant but infrequently-mentioned contributions of women to the development of <a href="http://www.erlang.org">Erlang</a> and <a href="http://learnyousomeerlang.com/what-is-otp">OTP (Open Telecom Platform)</a> from our own <a href="https://www.codesync.global/speaker/miriam-pena">Miriam Pena</a>:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/j6wbuV8pMx8" frameborder="0" allowfullscreen=""></iframe></div></figure>
Also on the first day, our own <a href="https://www.codesync.global/speaker/brujo-benavides">Brujo Benavides</a> delivered a solid rationale for the practice of using opaque (module-scoped) data types in <a href="http://www.erlang.org">Erlang</a> code, for maintainability and readbility reasons:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/hxLh0qFX9oI" frameborder="0" allowfullscreen=""></iframe></div></figure>
And of course <a href="https://www.codesync.global/speaker/mike-watters">Mike Watters</a> described his recent rewrite of the RTB profile cache system using <a href="https://elixir-lang.org">Elixir</a> and the <a href="https://hexdocs.pm/flow/Flow.html">Flow framework</a>, which increased peak throughput by 2X on that critical system (described in his blog post <a href="/blog/dev/2018/01/08/quaff-that-potion-saving-millions-with-elixir-and-erlang.html">here</a> as well):
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/CvjVpGktLC0" frameborder="0" allowfullscreen=""></iframe></div></figure>
<h2 id="the-usual-suspects">The usual suspects</h2>
An Erlang convocation is not really complete without a keynote from <a href="https://www.codesync.global/speaker/joe-armstrong">Joe Armstrong</a> needless to say. We were treated to a customary eclectic dilation on topics arcane, perhaps best described as a paleobiological view across the brief decades since computer science became a thing… not unlike a peek at the <a href="https://en.wikipedia.org/wiki/Burgess_Shale">Burgess Shale</a>, but for computers:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/-I_jE0l7sYQ" frameborder="0" allowfullscreen=""></iframe></div></figure>
<a href="https://www.codesync.global/speaker/robert-virding">Robert Virding</a> always has something fascinating to say and he joined with <a href="https://www.codesync.global/speaker/mariano-guerra">Mariano Guerra</a> to present a survey of some language implementation projects based on <a href="http://www.erlang-factory.com/upload/presentations/708/HitchhikersTouroftheBEAM.pdf">BEAM</a>, along the way describing the contortions big or small that these involved:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/lkAbwmn5Rv8" frameborder="0" allowfullscreen=""></iframe></div></figure>
Naturally I have to mention fabulous <a href="https://www.codesync.global/speaker/fred-hebert">Fred Hebert</a>, scribe and scholar responsible for must-have references <a href="http://learnyousomeerlang.com">Learn You Some Erlang For Great Good</a> and <a href="http://www.erlang-in-anger.com">Erlang in Anger</a>, who talked about the importance of using supervisor trees to prepare for and gracefully handle unexpected complex failures in distributed systems:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/W0BR_tWZChQ" frameborder="0" allowfullscreen=""></iframe></div></figure>
My introduction to property based testing came via <a href="https://www.codesync.global/speaker/kostis-sagonas1">Kostis Sagonas</a> in his talk about <a href="http://proper.softlab.ntua.gr">PropEr</a>, <a href="http://concuerror.com">Concuerror</a> and using optimal partial order reduction to find outlier bugs in your code without waiting 48 days for naive exhaustive tests to complete:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/4-vJeCmkCZE" frameborder="0" allowfullscreen=""></iframe></div></figure>
<a href="https://www.codesync.global/speaker/scott-lystig-fritchie56">Scott Lystig Fritchie</a> talked about why he isn’t using Erlang at <a href="https://www.wallaroolabs.com">Wallaroo Labs</a> - the answer is <a href="https://www.ponylang.org">Pony</a> and it’s complicated:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/uv-3ptTD8hg" frameborder="0" allowfullscreen=""></iframe></div></figure>
<a href="https://www.codesync.global/speaker/raimo-niskanen">Raimo Niskanen</a> walked us through the gen_statem OTP behaviour class, introduced in OTP 19.0, which simplifies the API surface of the original gen_fsm behaviour while at the same time adding a significant amount of extra functionality and ease of use:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/f_jl6MR3kXQ" frameborder="0" allowfullscreen=""></iframe></div></figure>
<h2 id="file-under">File under…</h2>
<a href="https://www.codesync.global/speaker/andrew-thompson54">Andrew Thompson</a> described a IoT-to-cloud communications scheme implemented largely in Erlang (in software and firmware) on <a href="https://www.helium.com">Helium’s</a> low-power bidirectional 802.15.4 based radio module, the <a href="https://www.helium.com/products/atom-prototyping-module">Atom</a> - which operates using a variation of blockchain architecture based on ‘proof of coverage’ rather than the unwieldy ‘proof of work’ most of us think of when talking about blockchains:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/QnHYWNfvV2c" frameborder="0" allowfullscreen=""></iframe></div></figure>
<a href="https://www.codesync.global/speaker/irina-guberman">Irina Guberman</a> gave an in-depth look at the decisions her team made while implementing distributed mutable counters for metrics:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/peKLuq-tdQo" frameborder="0" allowfullscreen=""></iframe></div></figure>
<a href="https://www.codesync.global/speaker/emma-cunningham53">Emma Cunningham</a>, who was a formal semanticist before joining the engineering world, began a review of the mathematical and philosophical origins of type theory with the memorable sentence, “I’ve been a functional programming advocate since before I became a software engineer” - she concludes with a compelling argument to just use Dialyzer for all the things:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/yZO6FkkdJEg" frameborder="0" allowfullscreen=""></iframe></div></figure>
An overview of <a href="https://www.amqp.org">AMQP</a> and <a href="https://www.rabbitmq.com">RabbitMQ</a>, the poster child for Erlang implementations in the wild, was given by <a href="https://www.codesync.global/speaker/brett-cameron">Brett Cameron</a>:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/wN0jqAHmqXc" frameborder="0" allowfullscreen=""></iframe></div></figure>
Lastly I include a fun talk given by <a href="https://www.codesync.global/speaker/simon-thompson">Simon Thompson</a> that describes a very functional approach to implementing lazy evaluation - it’s a great idea for an interview problem too:
<figure><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/uh7qYf8rC00" frameborder="0" allowfullscreen=""></iframe></div></figure>
<h2 id="lightning-talks">Lightning talks</h2>
<a href="https://www.codesync.global/speaker/brujo-benavides">Brujo Benavides</a> talked about <a href="https://spawnfest.github.io">SpawnFest</a> (mostly… okay he also proselytized for ROK style leading commas in your source, but we love him anyway):
<iframe src="//slides.com/elbrujohalcon/deck-3/embed" scrolling="no" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
<a href="http://joedevivo.com">Joe DeVivo</a> talked about a composable modeling scheme for 3D printing with Elixir.
<a href="https://www.codesync.global/speaker/mariano-guerra">Mariano Guerra</a> talked about many things, including <a href="http://efene.org">Efene</a> and a call for polyglot ADTs over BEAM.
<a href="https://www.codesync.global/speaker/erik-stenman">Erik Stenman</a> talked about using <a href="http://erlang.org/doc/man/HiPE_app.html">HiPE</a> inspection tools to peer at Erlang’s innards.
<a href="https://www.codesync.global/speaker/fred-hebert">Fred Hebert</a> talked about what we like and what we hate about Erlang… plus he did it upside down on his laptop:
<img src="/images/post_images/fredupsidedown.jpeg" alt="fred-upside-down" />
<h2 id="updates-from-the-mothership">Updates from the mothership</h2>
Naturally we were treated to the latest state of the union announcements from the Erlang and Elixir support teams:
<h2 id="update-otp-210">Update: OTP 21.0</h2>
<a href="https://www.codesync.global/speaker/raimo-niskanen">Raimo Niskanen</a> talked about the <a href="https://youtu.be/hHhm0bfdj-4">changes coming up in OTP</a>:
OTP 21.0 is due out June 20 2018 - we’re looking forward to full stack traces from exceptions, optimized small maps, the introduction of the <code class="language-plaintext highlighter-rouge">logger</code> module, and I personally am delighted to see <code class="language-plaintext highlighter-rouge">lists:search(List, fun/1)</code> added to the <code class="language-plaintext highlighter-rouge">lists</code> module! Don’t judge me.
Interesting mentions in the “Longer term plans” segment of Raimo’s summary: more compiler optimizations using Single Static Assignment representation, profiling tool improvements, Erlang Language Server Protocol to get support for Erlang in common editors, oh and the unicorn was spotted - sorry, the JIT compiler is approaching HiPE performance now…
<h2 id="update-elixir-16">Update: Elixir 1.6</h2>
<a href="https://www.codesync.global/speaker/james-fish">James Fish</a> walked us through the <a href="https://youtu.be/l4ISqxmZQtE">significant developments in Elixir 1.6</a>:
The Elixir team is focusing on productivity, maintainability, reliability… Version 1.6 features a <code class="language-plaintext highlighter-rouge">go fmt</code> style code formatting module, which is good for all the reasons that a stable and universal source code format is good. It is callable from code and from the command line. The AST has been surfaced with an API and some utilities that will allow term structure of code to be inspected easily. <code class="language-plaintext highlighter-rouge">require/1,2</code> has been added to notify one’s compiler and one’s self about [in particular] macro dependency relations. Two new attributes, <code class="language-plaintext highlighter-rouge">@deprecated</code> and <code class="language-plaintext highlighter-rouge">@since</code>, mark whether a function or macro is deprecated and when that happened. <code class="language-plaintext highlighter-rouge">defguard/defguardp</code> generate macros suitable for use in guard expressions, enhancing readbility. <code class="language-plaintext highlighter-rouge">DynamicSupervisor</code> is a new behaviour, or more precisely a reformed behaviour, and I quote James here: “A replacement for <code class="language-plaintext highlighter-rouge">simple_one_for_one</code>, but actually simple”
Plans for the subsequent major release at the end of 2018 (1.7) include a <code class="language-plaintext highlighter-rouge">mix</code> which will generate releases, a new PBT tool called <code class="language-plaintext highlighter-rouge">StreamData</code> which does data generation and simple PBT, better Dialyzer support, and greatly improved documentation.
<h2 id="last-thoughts">Last thoughts</h2>
As usual I’m invigorated by attending this conference. The breadth and depth of work being done using BEAM is remarkable - IoT/Edge netorks/embedded applications, language-agnostic testing frameworks, high performance low-latency distributed computation… Erlang and its progeny have passed through a number of major paradigm shifts in this industry and seem to always stay relevant, somehow offering something new even as the tech stack grows broader and deeper.
It is always a good feeling to be present at a tech conference when you have a horse in the race - and in this case Adroll very much does, being prominent Erlang proponents and users. It seems self-evident that showing up and participating is a vital part of selling our technology and engineering culture to our peers, and that can only be a good thing, if for no other reason than helping attract motivated and informed people to our team!
Are you interested in working with Erlang and Elixir in a high-performance globally distributed environment? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>
Writing Elixir stubs for better testing
</title>
<link>https://tech.nextroll.com/blog/dev/2018/03/28/elixir-stubs-for-tests.html</link>
<pubDate>Wed, 28 Mar 2018 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2018/03/28/elixir-stubs-for-tests</guid>
<description>In this post we’re going to see a handy technique to create and use stubs
for your Elixir projects, leading to better tests, more maintainable code, and
a lot of fun while using ETS, match specs, and macros.
<code class="language-plaintext highlighter-rouge">15-20 minute read</code>
<hr />
<h1 id="lets-jump-right-in-untestable-code">Let’s jump right in: Untestable code</h1>
As an example, let’s say you have a module that uses some dependency to query
DynamoDB (although note that this could be any other type of
external service), and your code looks like this:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule MyApp.UsesAws do
def do_something_with_dynamodb(table, query) do
result = DynamoDependency.get_item(table, query)
other_stuff(result)
end
end</code></pre></figure>
As you can see we don’t stand a chance to test this code without doing a real
request to DynamoDB, because our code is tightly coupled to a specific
implementation.
Experienced erlang devs would say “mock it”! Let’s dig a bit deeper into that option.
<h2 id="stubs-or-mocks">Stubs or mocks?</h2>
<a href="https://en.wikipedia.org/wiki/Method_stub">A stub</a> is a piece of code that implements a
contract, pre-programmed to return specific responses so you can test how your code
behaves when handling those responses.
<a href="https://en.wikipedia.org/wiki/Mock_object">A mock</a>, is similar, but also includes expectations (like “make sure
this function is called N times with these args”, etc). You will usually verify
all these expectations at the end of your test case.
<h2 id="fine-use-meck">Fine, use meck…</h2>
… or rather don’t. <a href="https://github.com/eproxus/meck">Meck</a> is a mocking
library for Erlang. It is sometimes used for stubbing rather than mocking.
It has a few downsides, as <a href="https://twitter.com/elbrujohalcon">Brujo</a> mentions in
<a href="/blog/dev/2018/02/27/erlang-speed-tests.html">a previous post</a>, including:
<ul>
<li>Overhead to setup, tear down, and reset the mocked modules</li>
<li>Difficulty running tests concurrently.</li>
<li>Complex dependencies which increase as we mock more tests.</li>
</ul>
With these issues in mind, we should consider injecting our dependencies, and
using stubs rather than mocking everything we need.
<h2 id="making-the-code-more-testable">Making the code more testable</h2>
With only a small change, and by using <a href="https://elixir-lang.org/getting-started/module-attributes.html">module attributes</a>
we can really improve the initial situation and end up with this code:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule MyApp.UsesAws do
@dynamodb_backend Application.get_env(
:my_app, :dynamodb_backend, DynamoDefaultImplementation
)
def do_something(table, query) do
result = @dynamodb_backend.get_item(table, query)
:expected_value == result
end
end</code></pre></figure>
And that’s how we’ve decoupled our code from a specific implementation. Now we
can “inject” a stub and avoid using meck altogether.
This was actually discussed by José Valim in <a href="http://blog.plataformatec.com.br/2015/10/mocks-and-explicit-contracts/">a 2015 post</a>.
<h2 id="injecting-the-stub">Injecting the stub</h2>
In this case, the way would be to define a <code class="language-plaintext highlighter-rouge">config.exs</code> file like this:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">use Mix.Config
# Our config goes here...
import_config "#{Mix.env()}.exs"</code></pre></figure>
And then a <code class="language-plaintext highlighter-rouge">test.exs</code> file like this:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">use Mix.Config
config :my_app,
dynamodb_backend: MyApp.Test.Stub.Dynamo</code></pre></figure>
Then, when running <code class="language-plaintext highlighter-rouge">mix test</code> the right module name will be compiled and used
in our code (which is our stub).
<h2 id="sample-stub-and-test">Sample stub and test</h2>
Let’s create the stub for this dependency. We’re going to put it in <code class="language-plaintext highlighter-rouge">test/lib/stub/dynamo.ex</code>.
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule MyApp.Test.Stub.Dynamo do
def get_item("this_table", "this_query") do
:expected_value
end
def get_item(_, _) do
:default_value
end
end</code></pre></figure>
The test is pretty straightforward:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">test "the code should do this" do
assert MyApp.UsesAws.do_something("this_table", "this_query")
end</code></pre></figure>
Our code is completely decoupled from the dependency and we can go ahead and
test every branch of it without issuing any requests to external actors.
<h3 id="add-our-stub-to-the-compile-path">Add our stub to the compile path</h3>
In our <code class="language-plaintext highlighter-rouge">mix.exs</code> file, we should add a <a href="https://hexdocs.pm/mix/Mix.Tasks.Compile.Elixir.html#module-configuration">configuration for the compiler</a>
adding the path to where our stub modules are located (for example, <code class="language-plaintext highlighter-rouge">test/lib</code>):
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule MyApp.Mixfile do
use Mix.Project
require Logger
def project do
[
app: :my_app,
# A lot of stuff here...
elixirc_paths:
if Mix.env() == :test do
["lib", "test/lib"]
else
["lib"]
end
]
end</code></pre></figure>
So are we done yet? Nah, not even close!
<h3 id="a-good-first-step-still-a-long-way-to-go">A good first step, still a long way to go</h3>
The proposed solution still has a few downsides:
1.- We have to add a function clause to our stub for every combination (more
or less) of arguments to return the needed value.
2.- How do we match anything for some arguments but specific values for others?
Let’s see how ETS and match specifications can help us with these issues.
<h1 id="first-improvement-dynamic-matching-of-arguments">First improvement: Dynamic matching of arguments</h1>
Let’s change our stub to support “dynamic configuration”. For every stubbed
function, we’re going to “record” the needed information in an <a href="http://erlang.org/doc/man/ets.html">ETS table</a>
and then for every call we’re going to try to find the matching function name
and arguments from there.
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule MyApp.Test.Stub.Dynamo do
@ets :dynamo_stub
# We call this one from our tests to setup a specific return
# value for a given function/args. "value" can be a function
# or any other term. If it's a function, it will be executed
# with the given args and its return value will be returned
# to the caller.
def respond_to(fun_name, list_of_args, value) do
table = @ets
true = :ets.insert(table, { {fun_name, list_args}, value})
end
# A stubbed function.
def get_item(key, query) do
respond! :get_item, [key, query]
end
# The "magic" happens here. The right combination of function
# name/arguments will be looked up in the ETS. An error is
# raised if none is found. If a function is found as the
# value, it will be executed and its return value will be
# returned to the caller. Otherwise the plain value found
# will be returned.
defp respond!(fun_name, args) do
table = @ets
case :ets.lookup(table, {fun_name, args}) do
[] -&gt; raise RuntimeError,
"Didn't find a response for #{__MODULE__}:#{fun_name} " &lt;&gt;
"with #{inspect args}"
[{_fun_and_args, value}] -&gt; if is_function(value) do
:erlang.apply(value, args)
else
value
end
end
end</code></pre></figure>
Then the test code becomes:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">test "the code should do this" do
MyApp.Test.Stub.Dynamo.respond_to(
:do_something, ["this_table", "this_query"], true
)
assert MyApp.UsesAws.do_something("this_table", "this_query")
end</code></pre></figure>
And let’s not forget that we need to create the <code class="language-plaintext highlighter-rouge">:dynamo_stub</code> ETS table, which
can be done in <code class="language-plaintext highlighter-rouge">test/test_helper.exs</code>:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">:dynamo_stub = :ets.new(:dynamo_stub, [:public])
ExUnit.start()</code></pre></figure>
Now there’s no need to create different function clauses per stubbed function. But
wait, there’s more.
<h1 id="second-improvement-matching-for-any-argument">Second improvement: Matching for any argument</h1>
So there’s (at least) one thing remaining. How could we match for any argument? I.e:
let’s say that we’d like our test to be like:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">test "the code should do this" do
# Reply with the same value no matter what the table name is
MyApp.Test.Stub.Dynamo.respond_to(
:do_something, [:_, "this_query"], true
)
assert MyApp.UsesAws.do_something("any other table", "this_query")
end</code></pre></figure>
Since we’re already using ETS we can look at a useful erlang feature called
“match specifications”.
<h2 id="what-are-match-specifications">What are match specifications?</h2>
To quote <a href="http://erlang.org/doc/apps/erts/match_spec.html">Match Specifications in erlang.org</a>:
<blockquote>
A “match specification” (match_spec) is an Erlang term describing
a small “program” that tries to match something. It can be used
to either control tracing with erlang:trace_pattern/3 or to search
for objects in an ETS table with for example ets:select/2.
</blockquote>
The main idea is to use <a href="http://erlang.org/doc/apps/erts/match_spec.html">Match Specifications</a> to
match the called function arguments and return the right value.
Match specifications are really complex, and we’re only going to use them to match the special
<code class="language-plaintext highlighter-rouge">:_</code> symbol as a wildcard meaning any argument.
For this we’re going to use <a href="http://erlang.org/doc/man/ets.html#test_ms-2">ets:test_ms/2</a>,
it accepts a list of match specifications and tests it against a specific term.
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">[{match_pattern(), [term()], [term()]}]</code></pre></figure>
Described in <a href="http://erlang.org/doc/man/ets.html#select-2">ets:select/2</a>
<blockquote>
This means that the match specification is always a list of one
or more tuples (of arity 3).
The first element of the tuple is to be a pattern as described
in match/2.
The second element of the tuple is to be a list of 0 or more
guard tests.
The third element of the tuple is to be a list containing a
description of the value to return. In almost all normal cases,
the list contains exactly one term that fully describes the value
to return for each object.
The return value is constructed using the “match variables” bound
in MatchHead or using the special match variables ‘$_’ (the whole
matching object).
</blockquote>
<h2 id="testing-match-specifications">Testing match specifications</h2>
Let’s give it a try in the Elixir console. If we want to stub a function with
3 arguments, we could use match specs like this:
<h3 id="matching-by-exact-argument-values">Matching by exact argument values</h3>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">:ets.test_ms(
{:arg1, :arg2, :arg3},
[{ {:arg1, :arg2, :arg3}, [], [:"$_"]}]
)
{:ok, {:arg1, :arg2, :arg3}}</code></pre></figure>
Meaning that we’d like to match a 3-element tuple with exactly these arguments. No
guards applied, and the result in case of a match will be the complete tuple.
<h3 id="matching-for-any-argument">Matching for any argument</h3>
Let’s say we’d like to only match arguments 1 and 3 and we don’t care about argument 2,
we could do the following:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">iex(2)&gt; :ets.test_ms(
{:arg1, :arg2, :arg3},
[{ {:arg1, :_, :arg3}, [], [:"$_"]}]
)
{:ok, {:arg1, :arg2, :arg3}}</code></pre></figure>
<h3 id="no-arguments-match">No arguments match</h3>
Cool. Now let’s try the case where we don’t match one or more args:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">iex(3)&gt; :ets.test_ms(
{:arg1, :arg2, :arg3},
[{ {:arg1, :arg4, :arg3}, [], [:"$_"]}]
)
{:ok, false}</code></pre></figure>
No results, so this works rather well for our use case, actually.
<h3 id="updating-our-stub-to-use-match-specs">Updating our stub to use match specs</h3>
Now that we’ve learned how to use match specs for this, let’s change our stub
module.
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule MyApp.Test.Stub.Dynamo do
@ets :dynamo_stub
def respond_to(fun_name, list_of_args, value) do
table = @ets
key = fun_name
# We're converting the list of arguments into a tuple because
# match specs need to match on tuple elements. We also add the
# returned value as the first elements so we can easily strip
# it later before matching.
element = List.to_tuple([value | list_of_args])
true = :ets.insert(table, {key, element})
:ok
end
# A stubbed function.
def get_item(key, query) do
respond! :get_item, [key, query]
end
defp respond!(fun_name, list_of_args) do
table = @ets
candidate_matches = :ets.tab2list(table)
tuple_to_test = List.to_tuple(list_of_args)
# Iterate through our ets and :ets.test_ms/2 each candidate
# against this call. Note: stubs with exactly the same
# arguments CAN AND WILL step on each other.
matches = :ets.foldl(fn({^fun_name, query}, acc) -&gt;
# Convert to a list so we can extract the value to return
# and match only on arguments.
case Tuple.to_list(query) do
[value | match_args] -&gt;
# Back to tuple and trying to match
query_tuple = List.to_tuple(match_args)
match_spec = {query_tuple, [], [:"$_"]}
case :ets.test_ms(tuple_to_test, [match_spec]) do
# No results, continue.
{:ok, false} -&gt; acc
{:ok, []} -&gt; acc
# Match found, save it and continue.
{:ok, element} -&gt; [{match_args, value} | acc]
end
_ -&gt; acc
end
# Skip if this stub is intended for other function names
(_, acc) -&gt; acc
end,
[],
table
)
return!(fun_name, args, matches)
end
defp return!(fun_name, args, matches) do
case matches do
# Return the first match only
[{_args, value} | _] -&gt; if is_function(value) do
:erlang.apply(value, args)
else
value
end
result -&gt;
raise RuntimeError,
"Didn't find a response for #{__MODULE__}:#{fun_name} " &lt;&gt;
"with #{inspect args}"
end
end
end</code></pre></figure>
This is so much better. We don’t need to add function clauses with fixed
arguments and return values in our stub module. Also, we can match on some
but not all arguments as needed. But there’s still something else we can do.
<h1 id="third-improvement-return-the-best-match-less-generic-match">Third improvement: Return the best match (less generic match)</h1>
What if a test setups our stub to return a value when called with the args
<code class="language-plaintext highlighter-rouge">:arg1, :arg2, :arg3</code>, while at the same another one sets it to return a
different value for <code class="language-plaintext highlighter-rouge">:arg1, :arg2, :_</code>?
We have a conflict there, we won’t know in advance which value is the right
one to return to the caller since both configurations will match.
In cases like this, we’d like to always consistently return the result from the
stub that best matches (i.e: a specific argument value should be preferred over
a generic match like <code class="language-plaintext highlighter-rouge">:_</code>).
An easy solution would be to sort all the matches from the match specs and
use the one that has fewer <code class="language-plaintext highlighter-rouge">:_</code> in it.
So one more time we go to our stub module:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule MyApp.Test.Stub.Dynamo do
# ...
defp respond!(fun_name, list_of_args) do
matches = # .. everything is still the same, BUT now we do:
sorted_matches = sort_by_less_generic(matches)
return!(fun_name, list_of_args, sorted_matches)
end
defp sort_by_less_generic(matches) do
# Pick the one with less :"_" in the args (should be
# the more specific match)
sorted_matches =
Enum.sort(
matches,
fn({a_args, _a_value}, {b_args, _b_value}) -&gt;
al = a_args
bl = b_args
count_a = Enum.count(al, fn e -&gt; e === :_ end)
count_b = Enum.count(bl, fn e -&gt; e === :_ end)
count_a &gt; count_b
end)
end
end</code></pre></figure>
<h1 id="fourth-improvement-generic-base-stub-module-as-a-macro">Fourth improvement: Generic base stub module as a macro</h1>
As a “last” improvement (at least in terms of this post) we can wrap up our
work by creating a generic stub module that can be reused in different mocks
for our tests. We can do that by embedding our code inside a <a href="https://elixir-lang.org/getting-started/meta/macros.html">macro</a>
that can then be <a href="https://elixir-lang.org/getting-started/alias-require-and-import.html#use">used</a> in the different stubs:
<h2 id="create-an-ets-table-per-stub">Create an ETS table per stub</h2>
Since we’re going to have multiple module stubs we need an ETS table
for each one of them, let’s set them up in<code class="language-plaintext highlighter-rouge">test/test_helper.exs</code>:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">stubs = [
dynamodb_backend: MyApp.Test.Stub.Dynamo,
another_one: MyApp.Test.Stub.Another,
# ...
]
# Create the tables, saving the table names in our
# application environment so the stubs can get them
# later, when needed.
_ =
for {name, mod} &lt;- stubs do
ets_table = mod
Application.put_env(
:my_app,
ets_table,
:ets.new(ets_table, [
:public,
# We need a duplicate bag here because we're
# going to use the function name as a key
# (instead of fun_name + args).
:duplicate_bag,
{:read_concurrency, true},
{:write_concurrency, true}
])
)
# Save the ets table name in the application
# environment so the stub module used can pick
# it up from there.
Application.put_env(:my_app, name, mod)
end</code></pre></figure>
<h2 id="move-the-code-into-a-generic-stub-module">Move the code into a generic stub module</h2>
The final code is pretty much the same, except that now we get the ETS table
from the application environment:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule MyApp.Test.Stub.Base do
defmacro __using__(_opts) do
quote location: :keep do
def respond_to(fun_name, list_of_args, value) do
table = get_table()
# ..same as before...
end
defp respond!(fun_name, args) do
table = get_table()
# ..same as before...
end
defp sort_by_less_generic(matches) do
# ..same as before...
end
defp return!(fun_name, args, matches) do
# ..same as before...
end
defp get_table() do
Application.get_env(:my_app, __MODULE__)
end
end
end
end</code></pre></figure>
Now our original stub will look like this:
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule MyApp.Test.Stub.Dynamo do
@moduledoc false
use MyApp.Test.Stub.Base
def get_item(key, query) do
respond! :get_item, [key, query]
end
end</code></pre></figure>
And that’s pretty much it. Hopefully this will help you write more and better
tests for your Elixir/Erlang code. Until next time :)
<h1 id="conclusion">Conclusion</h1>
As a final word, it’s important to notice how much we gained by just investing
20 minutes more on refactoring our code to make it more testable, and to allow
us to write better tests. By writing a few stubs and helpers, this allowed us to
avoid mocking global modules, and in the end this will be really worth our while,
our tests will run faster, will be smaller, and less complex.
<hr />
Do you enjoy building high-quality large-scale systems? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>Series: Tech Women of AdRoll Group</title>
<link>https://tech.nextroll.com/blog/culture/2018/03/22/tech-women-of-adroll-group-part-3.html</link>
<pubDate>Thu, 22 Mar 2018 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/culture/2018/03/22/tech-women-of-adroll-group-part-3</guid>
<description>In the spirit of <a href="https://womenshistorymonth.gov/">National Women’s History Month</a>, we are publishing our third volume of our <a href="https://blog.adroll.com/tag/tech-women-of-adroll/">Tech Women of AdRoll Group series</a> to celebrate and honor women across AdRoll Group (BI, Engineering, Product Management), and to acknowledge their diversity of backgrounds, viewpoints, and experiences. We had Nitasha Syed, creator and writer of the blog series Women of Stem, sit down with the women of AdRoll Group to better understand their backgrounds and what led them to careers in STEM. These women are engineers, accountants, humanitarians, rock climbers and even tattoo artists, who all found their way into STEM careers. Read below to discover their unique and inspirational stories.
Also, please <a href="https://www.eventbrite.com/e/how-to-vocalize-your-achievements-a-free-workshop-for-underrepresented-women-in-tech-tickets-44244068199">join us</a> next Tuesday, March 27th @5:30pm for a free workshop we are hosting in partnership with <a href="https://techqueria.org/">Techqueria</a> on “How to Vocalize Your Achievements: A Free Workshop for Underrepresented Women in Tech” led by the amazing Career Success Coach Maria Eleanora.
<img alt="Joyce" src="/images/post_images/tech-women-joyce.jpg" style="max-width: 375px" />
<h2 id="joyce-huynh">Joyce Huynh</h2>
Sr. Product Manager, Reporting
“I wasn’t sure what I wanted to be when I grew up, but my family wanted me to be a doctor. When I was in high school I took chemistry and mathematics and found both subjects very interesting. I decided to study chemical engineering, and got my bachelors from Berkeley and my Ph.D. from Caltech. During my Ph.D. studies, I discovered working in a lab was too isolating and not something I wanted to pursue as a career.
Switching out of science made finding my first job challenging. Despite having to switch fields, my STEM education background enabled me to get my first job as a Data Analyst for a consulting company. After being in consulting for a little over a year, I discovered I wanted more ownership and direct impact from my work. During this time, I learned about and became interested in the Product Management role and decided to make a switch. It’s because of my STEM background, I was able to explore different roles. A STEM education trains you to think critically and focuses on solving problems, which are valuable skills to have regardless of the role.
My mom has always been a source of inspiration for me, and pushed me to do my best in everything. I would like to advise parents to make sure they encourage their kids to be whatever they want to be. If your kids have an interest in STEM, help them learn how to think through problems from a young age. When I was younger, we would go to the grocery store and I would calculate the $/ounce and compare it to other options in the store to determine whether we were getting the best deal. Problem solving opportunities are all around us and you just need to push your children to explore them.”
<img alt="Prathibha" src="/images/post_images/tech-women-prathibha.jpg" style="max-width: 375px" />
<h2 id="prathibha-deshikachar">Prathibha Deshikachar</h2>
Director, Software Engineering
“A career in science and math was a natural progression from high school, as I grew up surrounded by engineers and scientists. My dad was a math professor at an engineering college, so math made its way to our dining room conversations. I never thought of it as something special or something I was not allowed to seek. It just seemed very natural at that time. When I entered engineering school, I realized that there were fewer girls. The gender disparity only seemed much larger after I started working. I loved the challenges of solving a problem and the sense of accomplishment thereafter. The foundation to this was laid when I was very young. I was always taught to believe in my abilities and constantly challenge myself. This is important for parents today. There are many ways to solve a problem and kids should be encouraged to explore it on their own. It is ok to be frustrated and failure should be considered a stepping stone to success. A career in STEM takes a lot of perseverance, so it is necessary to lay that foundation when kids are young.”
<hr />
Thanks for reading! If you would like to learn more about our tech culture and tech team, check us out at <a href="http://tech.adroll.com">tech.adroll.com</a>. If you would like to learn about all current open roles by location, check out our <a href="http://www.adroll.com/about/careers/open-positions">job board</a>.
</description>
</item>
<item>
<title>
Testing at warp speed: Why you should care about your test speed
</title>
<link>https://tech.nextroll.com/blog/dev/2018/02/27/erlang-speed-tests.html</link>
<pubDate>Tue, 27 Feb 2018 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2018/02/27/erlang-speed-tests</guid>
<description>We reduced our test run times by more than 50%, thus speeding up development and deployment considerably with the application of some generic tips that you can use in your system as well.
<code class="language-plaintext highlighter-rouge">15-20 minute read</code>
<hr />
As you can read in <a href="/blog/dev/2018/01/08/quaff-that-potion-saving-millions-with-elixir-and-erlang.html">our previous blog post</a>, at AdRoll we use <a href="http://www.erlang.org/">Erlang/OTP</a> extensively. In particular, we use it to build our <a href="/blog/web/2014/04/29/valentino-presents-adrolls-rtb-infrastructure.html">real-time bidding platform</a>.
These systems are central to our business and that’s why we’re very rigorous about writing tests and making sure they all run smoothly in every pull request and before every deploy.
We have a fairly complex test structure which has proven useful in keeping our system maintainable. But for these tests to be truly helpful they need to run quickly because we run them all the time. Balancing test completeness and test speed is not an easy task.
In our effort to keep that balance, we have discovered quite a few things. This article describes the lessons we have learned ensuring our tests run fast enough to allow developers to work at top speed without compromising the quality of our software.
<h1 id="why-do-you-need-to-keep-your-tests-running-fast">Why do you need to keep your tests running fast?</h1>
It’s not uncommon to find projects where a single test run could last well over 10 minutes. Since those test runs tend to grow steadily over time and the systems they’re testing are also huge, even when everybody knows their tests are slow, nobody notices how slow they are until it becomes critical or until a new developer comes along and has to wait forever for tests to complete for the first time. But test runs lasting more than a few minutes have serious consequences…
<h2 id="on-development">On Development</h2>
How many test rounds do you need to develop a new feature or fix a bug? If you practice <a href="https://en.wikipedia.org/wiki/Test-driven_development">Test Driven Development</a> you need at least three (the initial red one, the first green one after implementing your code and the second green one after refactoring). But even if you don’t practice TDD, you will probably still want to test the code you write before submitting a pull request.
Three test runs, several minutes each, means a lot of time spent just waiting for tests to run. And more often than not while you wait for tests to run you can’t do much else. Even if you start working on something else in parallel while the tests run, <a href="https://www.linkedin.com/pulse/context-switching-developers-paul-graham/">context switching</a> is generally not good for developers. Besides, your new task will most likely also require running tests.
On top of that, if you have configured your <a href="https://www.thoughtworks.com/continuous-integration">Continuous Integration</a> tool as we have (we run a full round of tests for each pull request and for each branch merged into master), then merging even the smallest PR would take at least two full test runs – sometimes many more depending on how many PRs are open at the same time and how fast or parallelizable your CI is.
These scenarios lead to two opposite approaches to dealing with long-running tests:
<ol>
<li>Being overly careful with your code modifications. Not touching anything that is not 100% related to what you’re implementing for fear of making a passing test fail. Behaving like this slows down your refactoring process considerably, affecting the overall code quality of the system.</li>
<li>Including many unrelated changes in a single PR. The goal being to run CI tests just once and avoid waiting for many PRs to have green lights. This affects your ability to properly review, understand, and revert your changes if needed.</li>
</ol>
Code reviews are also affected since reviewers will be more wary of requesting changes if those are not explicit merge-blockers. That’s because a change request might require the author to spend another hour running tests on their machine and then CI spending the same amount of time running the tests again. Reducing code reviews to just the most blatant problems affects the culture of your team and also the overall coherence of your code, drastically reducing maintainability.
<h2 id="on-deployment">On Deployment</h2>
We have a general policy of deploying often so that in case a deployment impacts our system performance we can detect the source of the issue quickly. The idea is that since we don’t spend too much time between deploys, we are not introducing too many changes and therefore detecting the one that’s causing problems is easier. You might find yourself in a similar situation.
But before each deploy, you want to make sure what you’re deploying actually works as expected. To achieve that, you have to include at least one full test run in your deployment process. In other words, to deploy something you would at least need time for a full test run.
That process being slow can also make people overly cautious with deploys, and reduce the available timeframe for them since nobody would want to deploy anything during the last three or four hours of their day. What if the deploy goes south and you have to redeploy what you had before (i.e. rollback)? What if you have to deploy a patch? That may very well require multiple hours of work because you will need time to run tests on your computer, then CI on the corresponding PR, then your build system for the deploy. And that’s assuming there are no comments or failing tests in between and that you can find and fix the bug almost immediately.
In a nutshell, it’s not good to have long test runs. Keeping your test run times in check is very important for your system and your team. With that in mind, let me show you what we’ve learned so far…
<h1 id="testing-adrolls-rtb-servers">Testing AdRoll’s RTB servers</h1>
In our system, we have multiple levels of testing:
<ul>
<li>we write unit tests for our modules, with <a href="http://erlang.org/doc/apps/eunit/">eUnit</a>.</li>
<li>we write integration tests with <a href="http://erlang.org/doc/apps/common_test/">Common Test</a>.</li>
<li>we have our own tool to run black-box tests: we run client simulations against one of our servers, which are also included in a <a href="http://erlang.org/doc/apps/common_test/">Common Test</a> suite.</li>
</ul>
For most of our integration tests we need to mock one or more underlying pieces (particularly those that manage connections with databases, download files from the internet or connect with external services) and for that, we use <a href="https://hex.pm/packages/meck">meck</a>.
A full test run would run all of the aforementioned tests, starting from the unit tests, all the way through the integration ones and finally the black box suites. For this article, we’ll forget about unit tests (which are, predictably, the fastest ones anyway) and we’ll focus on those we run through Common Test.
<h1 id="benchmarking">Benchmarking</h1>
When trying to improve test speed, the first step is always benchmarking your test runs to find which tests are taking longest. Luckily, Common Test is a very complete and flexible framework that allows us to define hooks. Using <a href="http://erlang.org/doc/man/ct_hooks.html">ct_hooks</a>, we implemented our own hook module. Actually, we implemented two of them: One for regular test runs and one for benchmarking. The code within the benchmarking module looks like…
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">%%% @doc Common Test Hook module for benchmarking.
-module(timer_cth).
-export([init/2]).
-export([pre_init_per_suite/3]).
-export([post_init_per_suite/4]).
%% ...and all the other required callbacks

-record(state, {
time_init_suite = os:timestamp() :: pos_integer(),
time_end_suite = os:timestamp() :: pos_integer(),
time_init_testcase = os:timestamp() :: pos_integer(),
time_end_testcase = os:timestamp() :: pos_integer(),
...
}).
init(_Id, _Opts) -&gt; {ok, #state{}}.
%% @doc Called before init_per_suite is called.
pre_init_per_suite(Suite, Config, State) -&gt;
{
Config,
State#state{
suite = Suite,
time_init_suite = os:timestamp()
}
}.
%% @doc Called after init_per_suite.
post_init_per_suite(Suite, _Config, Return, State) -&gt;
io:format(
"CT-TIME | ~8.3f | ~p:init_per_suite~n",
[time_diff(State#state.time_init_suite), Suite]
),
{Return, State}.
time_diff(Init) -&gt;
timer:now_diff(os:timestamp(), Init) / 1000000.</code></pre></figure>
With functions as simple as the ones shown above, you can measure (and print) how much time is spent in the different stages of each test suite and test case. And if you use easily greppable strings (in this case <code class="language-plaintext highlighter-rouge">CT-TIME</code>) you can pipe your test run into grep and extract just the info you need for future analysis.
<blockquote>
<h5 id="lesson-learned">Lesson Learned</h5>
Use <code class="language-plaintext highlighter-rouge">ct_hooks</code> to benchmark your tests and remember you can have multiple <code class="language-plaintext highlighter-rouge">ct_hooks</code> modules and use the benchmarking one only when you’re interested in the performance of your tests.
</blockquote>
<h1 id="mocking">Mocking</h1>
When you write integration tests, it’s not uncommon to require a fair share of mocking. In Erlang, that’s usually accomplished using <a href="https://hex.pm/packages/meck">meck</a>. Meck is a very flexible mocking library that allows you to temporarily replace existing modules with a new implementation for some or all of their functions. The usual lifecycle of a mock built with Meck looks something like this…
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">...
% initialize your mock
meck:new(your_module, [passthrough]),
meck:expect(
your_module,
a_function,
fun(Arg) -&gt; return_something_trivial(Arg) end
),
% run your test using the mock...

% maybe check stuff with meck, for instance...
1 = meck:num_calls(your_module, a_function, [some_expected_arg]),
% destroy the mock
meck:unload(your_module),
...</code></pre></figure>
But if you write exactly that code in your test cases you may run into trouble. If your test fails before calling <code class="language-plaintext highlighter-rouge">meck:unload/1</code>, that module will remain mocked and that may affect other test cases. Depending on how you have configured your test suites, Common Test sometimes boots up different processes for different tests and sometimes it reuses the same process. By default, Meck links mocks to their creating processes, so if your process crashes or stops the mock is removed. But, if Common Test uses the same process to run the next test case, the module will remain mocked and that’s not good.
That’s why you usually want to put mocking initialization and unloading outside your main test case function and you can do that with Common Test:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">init_per_testcase(the_testcase, Config) -&gt;
% initialize your mock
 meck:new(your_module, [passthrough]),
Config.
the_testcase(Config) -&gt;
meck:expect(
your_module,
a_function,
fun(Arg) -&gt; return_something_trivial(Arg) end
),
% run your test using the mock...

% maybe check stuff with meck, for instance...
1 = meck:num_calls(your_module, a_function, [some_expected_arg]),
ok.
end_per_testcase(the_testcase, Config) -&gt;
% destroy the mock
 meck:unload(your_module),
Config.</code></pre></figure>
This is also very handy if you want to mock the same module in all your tests. You just need to replace <code class="language-plaintext highlighter-rouge">init_per_testcase</code> and <code class="language-plaintext highlighter-rouge">end_per_testcase</code> function clause heads with something like <code class="language-plaintext highlighter-rouge">init_per_testcase(_TestCase, Config)</code> et voilà!
But… What about performance? Now that you are mocking the same modules over and over again, it is worth asking yourself if unloading and recreating them between each pair of test cases is affecting the overall time of your test runs or not.
Turns out, it is. Meck dynamically compiles a new module each time you create a mock and then it boots up a new <a href="http://erlang.org/doc/design_principles/gen_server_concepts.html"><code class="language-plaintext highlighter-rouge">gen_server</code></a> for it (to track the history of calls to its functions, among other things). Compiling Erlang code is something that consumes several milliseconds per module (maybe even seconds, if your modules are long enough).
Well then, we can just create the mocks at the beginning of our suite and unload them at the end. But if we do so, how do we avoid the issue mentioned above (namely, that modules/functions remain mocked between test cases). This is the solution we are currently using:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">%% We initialize our mocks for the whole suite here.
%% Notice the no_link: we want mocks to stay with us until the end of the
%% suite, regardless of how many processes common_test wants to use.
init_per_suite(Config) -&gt;
meck:new([module1, module2], [passthrough, no_link]).
%% We don't do much here except for those functions
%% that we want to always mock in the same way (e.g. DB connections).
init_per_testcase(_TestCase, Config) -&gt;
meck:expect(module1, function1, 1, result1),
Config.
%% Our test cases remain the same
the_testcase(Config) -&gt;
meck:expect(
your_module,
a_function,
fun(Arg) -&gt; return_something_trivial(Arg) end
),
% run your test using the mock...

% maybe check stuff with meck, for instance...
1 = meck:num_calls(your_module, a_function, [some_expected_arg]),
ok.
%% We remove all expects here so that modules, even when mocked,
%% are basically _clean_ (i.e. they work as if they weren't mocked at all)
%% That's thanks to the passthrough option we provided on init_per_suite
end_per_testcase(_TestCase_, Config) -&gt;
Modules = [module1, module2],
meck:reset(Modules),
[meck:delete(Module, Fun, Arity, false)
|| {Module, Fun, Arity} &lt;- meck:expects(Modules, true)],
Config.
%% we do destroy all mocks here
end_per_suite(Config) -&gt;
meck:unload([module1, module2]).</code></pre></figure>
A few things to notice:
<ul>
<li>Meck doesn’t have a <code class="language-plaintext highlighter-rouge">delete_all</code> function. That’s why we have a list comprehension in <code class="language-plaintext highlighter-rouge">end_per_testcase</code>.</li>
<li>We need to clean up module statistics as well as the expects; that’s why we’re calling <code class="language-plaintext highlighter-rouge">meck:reset/1</code>.</li>
<li>Each call to <code class="language-plaintext highlighter-rouge">meck:delete/4</code> is not cheap, but it’s way cheaper than <code class="language-plaintext highlighter-rouge">meck:new/2</code>. We only want to call it for the functions that we are actually mocking, that’s why we use <code class="language-plaintext highlighter-rouge">meck:expects(Modules, true)</code> instead of <code class="language-plaintext highlighter-rouge">meck:expects(Modules)</code> (which would include the passthrough functions) or <code class="language-plaintext highlighter-rouge">Module:module_info(exports)</code> (which would include all the functions in the module).</li>
</ul>
<blockquote>
<h5 id="lesson-learned-1">Lesson Learned</h5>
If you’re going to reuse mocks on all your tests, create them on <code class="language-plaintext highlighter-rouge">init_per_suite</code>, delete the expectations on <code class="language-plaintext highlighter-rouge">end_per_testcase</code> and destroy them on <code class="language-plaintext highlighter-rouge">end_per_suite</code>.
</blockquote>
<h1 id="ct-comments">CT Comments</h1>
Common Test comes with more than a few little known secrets. One of my favorites is <a href="http://erlang.org/doc/man/ct.html#comment-1"><code class="language-plaintext highlighter-rouge">ct:comment/1,2</code></a> and <code class="language-plaintext highlighter-rouge">{comment, ""}</code>. They let you add some context to your tests that end up in the generated HTML summary of your test runs. This lets you understand where a test failed at a glance.
For instance, consider this test:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">failing_test(_Config) -&gt;
ct:comment("Dividing by zero using div should not fail"),
infinity = 1 div 0,
ct:comment("Dividing by zero should not fail"),
infinity = 1 / 0,
{comment, ""}.</code></pre></figure>
That test will obviously fail and thanks to <code class="language-plaintext highlighter-rouge">ct:comment/1</code> you will get something like this in your test report:
<img src="/images/post_images/erlang_tests/ct_comment.png" alt="CT Comment in Test Report" />
It’s a really nice feature but, as you might be already guessing at this point, it has some performance issues. <code class="language-plaintext highlighter-rouge">ct:comment/1,2</code> in and on themselves are not really that bad; they take 100ms or so. But if you write tests like the following one…
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">failing_test(_Config) -&gt;
lists:foreach(
fun(I) -&gt;
ct:comment("Dividing by ~p using div should not fail", [I]),
infinity = 1 div I,
ct:comment("Dividing by ~p should not fail", [I]),
infinity = 1 / I
end, lists:seq(100, 0, -1)),
{comment, ""}.</code></pre></figure>
In cases like the one above each successful run of the test uses <code class="language-plaintext highlighter-rouge">ct:comment</code> 200 times and that certainly adds up.
<blockquote>
<h5 id="lesson-learned-2">Lesson Learned</h5>
If you’re going to use <code class="language-plaintext highlighter-rouge">ct:comment/1,2</code> make sure you’re not using it within a recursive function/loop.
</blockquote>
<h1 id="parallelization">Parallelization</h1>
Common Test lets you choose how to run your tests with <a href="http://erlang.org/doc/man/common_test.html#Module:groups-0"><code class="language-plaintext highlighter-rouge">groups/0</code></a>. You can run tests multiple times, in sequential order, in parallel, etc. But sometimes, for instance with our black-box tests, you can’t just run all the tests in parallel to speed up the process as a whole. In our case, each of our tests is booting up a different instance of our server (with different configuration parameters) and hitting it with our client simulator to verify that it produces the responses we expect. In theory, we could run these tests in parallel, booting up as many servers as we need, etc. But in practice, that requires a lot of isolation-related effort that is just not worth it.
Nevertheless, it’s worth considering that the parallelism that Common Test gives you is certainly not the only one you can take advantage of. Check how our tests looked when we realized this:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">init_per_testcase(Exchange, Config) -&gt;
Folder = local_folder(Exchange),
download_samples_from_s3(Exchange, Folder),
boot_up_server(Exchange),
Config.
foo(_Config) -&gt;
% run client tests against the server
 ...
ok.
bar(_Config) -&gt;
% run client tests against the server
 ...
ok.
end_per_testcase(Exchange, Config) -&gt;
Folder = local_folder(Exchange),
delete_folder(Folder),
tear_down_server(Exchange),
Config.</code></pre></figure>
As you can see, we were performing all these operations (i.e. download from s3, server start, test run, folder removal, server stop) sequentially for each testcase. Also, the file download from s3 was one of the most expensive operations in that suite. So, we refactored that code into something like this…
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">init_per_suite(Config) -&gt;
download_all_samples(),
Config.
init_per_testcase(Exchange, Config) -&gt;
boot_up_server(Exchange),
Config.
foo(_Config) -&gt;
% run client tests against the server
 ...
ok.
bar(_Config) -&gt;
% run client tests against the server
 ...
ok.
end_per_testcase(Exchange, Config) -&gt;
tear_down_server(Exchange),
Config.
end_per_suite(Config) -&gt;
delete_files().</code></pre></figure>
And we made sure that <code class="language-plaintext highlighter-rouge">download_all_samples/0</code> downloaded as many files from s3 as it could in parallel, spawning several erlang processes and waiting for a <code class="language-plaintext highlighter-rouge">done</code> signal from them.
<blockquote>
<h5 id="lesson-learned-3">Lesson Learned</h5>
You can use <code class="language-plaintext highlighter-rouge">groups/0</code> in your suites to parallelize tests, but even if you can’t parallelize whole test cases, you can still use pure Erlang to simultaneously run some parts of them.
</blockquote>
<h1 id="what-about-you">What About You?</h1>
We found all these things in our constant effort to have quick tests and an agile development methodology. We’re sure we’re not alone here, so why don’t you let us know how you speed up your tests in the comments below?
<hr />
Do you enjoy building high-quality large-scale systems? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>
Quaff that potion: saving $millions with Elixir and Erlang
</title>
<link>https://tech.nextroll.com/blog/dev/2018/01/08/quaff-that-potion-saving-millions-with-elixir-and-erlang.html</link>
<pubDate>Mon, 08 Jan 2018 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2018/01/08/quaff-that-potion-saving-millions-with-elixir-and-erlang</guid>
<description><style type="text/css">/*<![CDATA[*/
.highlight code.erlang { background-color: #315050; }
.highlight code.erlang .c { color: #FED633; }
.highlight code.erlang .n { color: #CBBA77; }
.highlight code.erlang .err { color: #CBBA77; background-color: #315050; }
.highlight code.erlang .nl { color: yellow; }
.highlight code.erlang .nv { color: #10D010; }
.highlight code.erlang .o { color: #CBBA77; }
.highlight code.erlang .p { color: #CBBA77; }
.highlight code.erlang .s { color: #FD9226; }
.highlight code.erlang .ni { color: #18E7AE; }
.highlight code.erlang .nn { color: #CBBA77; }
.highlight code.erlang .kn { color: #4DFFFF; }
.highlight code.erlang .k { color: #4DFFFF; }
.highlight code.erlang .ow { color: #2DFFFE; }
.highlight code.erlang .nb { color: #29F89D; }
.highlight code.erlang .nf { color: #29F89D; }
.highlight code.erlang .bp { color: #FEA07E; }
.highlight code.erlang .mi { color: #CBBA77; }
/*]]>*/</style>
We slashed our DynamoDB costs by over 75% using Kinesis, DynamoDB streams, and
Erlang/OTP (and now Elixir) to implement a global cache warming system. We present that
system and two new open-source libraries for processing Kinesis and DynamoDB streams in
a similar way using Elixir and Erlang.
<code class="language-plaintext highlighter-rouge">15-20 minute read</code>
<hr />
AdRoll uses <a href="http://www.erlang.org/">Erlang/OTP</a> as the basis for several internal products, including a
<a href="/blog/web/2014/04/29/valentino-presents-adrolls-rtb-infrastructure.html">real-time bidding platform</a> running on <a href="https://aws.amazon.com/ec2/">Amazon EC2</a>.
Erlang/OTP is the king of robust highly-concurrent soft real-time systems such as these.
This article describes how we substantially reduced the cost of an element of our
real-time bidding platform (its DynamoDB usage) by implementing a global cache warmer
using <a href="https://aws.amazon.com/kinesis/">Kinesis</a> and <a href="http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html">DynamoDB streams</a>, written in
Erlang. We later doubled the performance of that component by adapting it to use the
<a href="https://hexdocs.pm/flow/Flow.html">Flow</a> framework from the authors of <a href="https://elixir-lang.org/">Elixir</a>.
We also present two new open-source libraries for doing this kind of processing using
Elixir and/or Erlang.
<h1 id="background">Background</h1>
<h2 id="real-time-bidding">Real-time bidding</h2>
When a publisher (such as a website or mobile app author) wants to monetize its
inventory, it can sell ad space to buyers. This can be done using a variety of means;
one of these is a programmatic auction conducted on behalf of the publisher by an
advertising exchange (clearinghouse) with which the publisher is integrated, such as
Google’s <a href="https://www.doubleclickbygoogle.com/">DoubleClick Ad Exchange</a> or <a href="https://www.taboola.com/">Taboola</a>.
For most inventory, these programmatic auctions have multiple interested buyers. Much
of the time, each buyer is notified by each exchange with which it is integrated
whenever an opportunity to buy ad space occurs. AdRoll currently participates in well
over one million programmatic auctions per second for a substantial part of each day.
In addition to supporting this high request volume, buyers must also respond quickly
(typically in less than 100ms, hence “real-time”): these auctions may be occurring while
a resource (web page, video, etc) is in the process of loading. A response may include
a bid price, which is the maximum price that buyer is willing to pay for that particular
ad space shown to a specific user. Barring any special circumstances, the highest bidder
wins the auction and may display an ad in the space which was won, paying an amount
determined by the dynamics of the auction (often <a href="https://en.wikipedia.org/wiki/Vickrey_auction">second-price</a>).
<h2 id="ad-targeting">Ad targeting</h2>
In order to bid in an auction for ad space, a buyer needs to be able to determine how
much it values a particular impression. AdRoll is able to value impressions for
specific users by combining <a href="/blog/data-science/2015/12/08/data-science_event_processing.html">large-scale machine
learning</a> with the keeping of profiles for each of
the billions of users (generally: cookies) we know about.
For web and mobile app inventory, we maintain for a period of time a set of targetable
“segments” for each user who has visited an AdRoll customer’s site and who has not opted
out of tracking. A segment could be something like “has visited more than N pages of
the site”, “has placed an item in the shopping cart”, “has looked at brown backpacks”,
“has purchased an item”, and so on. A user’s profile is the set of these segments along
with timestamps and certain other information <a href="/blog/ops/2013/10/02/dynamodb-replication.html">stored in
DynamoDB</a>.
The <a href="https://help.adroll.com/hc/en-us/articles/212939358">retargeting</a> product allows an AdRoll customer to show
ads to users who have previously visited their site, along with other restrictions. For
example, a customer could show a “10% off brown backpacks” promotion on Thursdays and
Fridays between 3 and 6 PM Pacific time to users in San Francisco who previously added
them to a shopping cart but did not purchase, plus a general “5% off” promotion to
similar users anywhere in California but excluding San Francisco.
AdRoll’s <a href="https://help.adroll.com/hc/en-us/articles/213429627">upper-funnel</a> product allows customers to show ads
to new “act-alike” users–users who are likely to act in similar ways as existing
desirable (converting) users, but who would not otherwise be reachable via
retargeting–similarly across the web and on social media such as Facebook.
When an opportunity to buy ad space presents itself, associated with that opportunity is
an identifier for the user who would see our customer’s ad if we won the auction. Using
that identifier, we can look up the user’s profile data and find the ads which are
relevant, and then with advanced machine learning models price in real time a set of
possible bids using those ads.
<h2 id="profile-data-system-overview">Profile data system overview</h2>
As described above, DynamoDB is the effective source of truth for our advertising
profile data; this works very well.
Unfortunately, it’s also very expensive: in order to meet our low-latency requirement,
we need to replicate our profile data globally in each AWS region where we operate (and
due to some design constraints we could not limit replication to a subset of profile
data). Under this scheme our DynamoDB costs, dominated by write capacity costs, are
multiplied by the number of regions we use.
<h3 id="original-design">Original design</h3>
To access this profile data from our bidding system we initially implemented a
straightforward setup like the following, replicated globally. From our application’s
perspective, this works like a read-through cache:
<img src="/images/post_images/quaff_elixir/before.png" alt="historical system overview image; image made with draw.io" />
In this diagram, boxes on the left represent user profile updates which occur constantly
throughout the day. The updates are written to various DynamoDB tables in every region
by separate data processing pipelines. When a bidder instance seeks to obtain a user’s
profile, it checks its local caches before reading from DynamoDB. If recent cached data
is found, it is used; otherwise, the instance reads from DynamoDB and then caches the
data for next time.
As described below, we eventually created a similar system allowing the removal of most
usage of DynamoDB from all but one region, cutting our DynamoDB costs by a very large
degree.
<h3 id="new-design">New design</h3>
While DynamoDB is our bidding system’s source of truth for profile data, we found that
we don’t actually need to read it most of the time: most data for users of interest can
be made to reside in cache by increasing the cached item TTL. Of course, doing that
alone would lead to stale data and inaccurate user targeting and bid pricing. If we
could have long-lived cache data for most users which is also kept up to date, we
wouldn’t need to consult DynamoDB most of the time.
We devised a solution to support just that: a set of large(r) caches exists in each
region, and each region constantly receives updates–up to 500,000/sec depending on
time-of-day–made on a set of master tables in a single region. Kinesis and DynamoDB
streams were the natural choices for this workload:
<h4 id="dynamodb-stream-aggregator">DynamoDB stream aggregator</h4>
<img src="/images/post_images/quaff_elixir/aggregator.png" alt="DynamoDB stream aggregator image" />
Here, a data-agnostic aggregator component observes updates as they occur on the set of
relevant DynamoDB streams. It batches these updates together and replicates them to
Kinesis streams in all target regions which formerly had the same DynamoDB tables as the
master region.
<h4 id="kinesis-stream-processor">Kinesis stream processor</h4>
<img src="/images/post_images/quaff_elixir/writer.png" alt="Cache writer image" />
In each target region operates a Kinesis stream reader (and cache writer) component
which updates long-lived items in a set of caches. Both it and our real-time bidding
system fall back to reading from tables in the master region (a relatively slow but rare
operation) when data is missing from the cache or is incomplete, writing it back to the
local cache using a <code class="language-plaintext highlighter-rouge">CAS</code> operation to ensure consistency.
This allowed us to keep the same conceptual design for the multiple involved systems
while still realizing substantial savings: we can tolerate a slower source of truth if
most data is accurately cached and long-lived, which is now the case.
<h1 id="implementation">Implementation</h1>
Chief among our concerns in designing this system were correctness (if we have bad data,
we can’t make good bidding decisions), robustness (if the system is down, we’ll have bad
or missing data), and scalability (we need to handle an arbitrary volume of data).
Secondarily, speed was also a consideration. After a successful (but not very scalable)
proof-of-concept which used Python, we elected to implement the complete solution using
Erlang, building on our expertise in this area while also being a good fit in general
for this type of problem.
<h2 id="on-scalability">On scalability</h2>
The essence of scalability is being able to efficiently apply a repeatable formula to an
arbitrary load. In the realm of software systems, Erlang/OTP makes this easy: with a
set of robust building blocks (OTP), systems comprised of shared-almost-nothing
sequential blocking processes can easily be created–and more importantly their behavior
can be easily understood. Process independence enables arbitrary horizontal scaling.
Contrast for example with a deferred/promises or asynchronous callbacks model more
common in other environments with more cumbersome concurrency, or shared-state threads
which bring their own issues.
<h2 id="on-robustness">On robustness</h2>
Erlang/OTP promotes robust system designs. “Let it crash” and supervision trees makes
for simple-to-understand control flow and process behavior. A process is always part of
a greater whole, and will “do stuff until it can’t”, then crash &amp; burn knowing it’s part
of a system which was designed from the ground up with appropriate failure modes. If
appropriate, such failing processes may be restarted by a supervisor somewhere in the
system to try again.
<h2 id="on-correctness">On correctness</h2>
Erlang’s syntax and design promotes the succinct and functional expression of programs
and avoids unnecessary defensive programming. Less code written means less code to be
read and understood later, and fewer places for bugs to hide.
<h2 id="but-enough-about-erlang-enterjava">But enough about Erlang; enter…Java?</h2>
To ensure the correct ordered processing of data and abstract away the handling of
details like work distribution and instance failure, we elected to use the existing
Java-based <a href="https://github.com/awslabs/amazon-kinesis-client">Kinesis Client Library</a> provided by Amazon. The KCL provides a
JSON-based <a href="https://github.com/awslabs/amazon-kinesis-client/blob/73ac2c0e25a25776cbc88f2c685223fb049e6757/src/main/java/com/amazonaws/services/kinesis/multilang/package-info.java">language-agnostic interface</a> (MultiLangDaemon) which allows
Kinesis processing (and DynamoDB streams processing using an adapter) to be done in any
language and environment without writing any Java code.
Unfortunately, the KCL/MultiLangDaemon expects to be the driver of any resulting
processing system: it emits events to which the processor reacts and launches one
processor executable per owned stream shard to effect that processing. While this makes
for very simple processing code, we need to efficiently process up to thousands of
shards across multiple streams, and launching thousands of BEAM nodes to do that would
be wasteful (and partly defeat the purpose of using Erlang). We also wanted to manage
the KCL processes themselves.
<h3 id="erlang-multilangdaemon-interface">Erlang MultiLangDaemon interface</h3>
Erlang systems are pretty good at being network servers, so we turned this problem
around by creating an adapter which launches a MultiLangDaemon whose worker subprocesses
become lightweight network clients of our Erlang node:
<img src="/images/post_images/quaff_elixir/beam-mld.png" alt="BEAM-MLD concept image" />
For each stream being processed, our Erlang node launches a MultiLangDaemon subprocess
as a port program. Each such subprocess is configured to launch a <code class="language-plaintext highlighter-rouge">netcat</code>-like program
(<code class="language-plaintext highlighter-rouge">socat</code>) as a shard processing program, which simply maps <code class="language-plaintext highlighter-rouge">stdin</code>/<code class="language-plaintext highlighter-rouge">stdout</code> back to the
port on which the node is listening for that MLD’s stream.
Thus, each owned shard is mapped to an independent Erlang process (a <code class="language-plaintext highlighter-rouge">gen_statem</code> state
machine) in a single node, and each such process is only concerned with handling a
single stream record at a time (and deciding when to periodically checkpoint). The
result looks like this:
<img src="/images/post_images/quaff_elixir/erlang-mld-workers.png" alt="erlang-based processing diagram" />
This gives most of the benefits of using the KCL and results in a processing application
which is similar to how Java-based KCL processing applications work (multiple threads of
execution within a single VM), but without requiring the user to write any Java code.
It allows for very simple shard processors like this minimal example:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">-module(noisy_worker).
-behavior(erlmld_worker).
-export([initialize/3,
ready/1,
process_record/2,
shutdown/2]).
-include_lib("erlmld/include/erlmld.hrl").
-record(state, {shard_id, count = 0}).
initialize(_Opaque, ShardId, ISN) -&gt;
State = #state{shard_id = ShardId},
io:format("~p initialized for shard ~p at ~p~n",
[self(), ShardId, ISN]),
{ok, State}.
ready(State) -&gt;
{ok, State}.
process_record(#state{shard_id = ShardId, count = Count} = State,
#stream_record{sequence_number = SN} = Record) -&gt;
io:format("~p (~p) got record ~p~n", [ShardId, Count, Record]),
case Count &gt;= 10 of
true -&gt;
{ok, State#state{count = 0},
#checkpoint{sequence_number = SN}};
false -&gt;
{ok, State#state{count = Count + 1}}
end.
shutdown(#state{shard_id = ShardId, count = Count}, Reason) -&gt;
io:format("~p (~p) shutting down, reason: ~p~n",
[ShardId, Count, Reason]),
case Reason of
terminate -&gt;
{ok, #checkpoint{}};
_ -&gt;
ok
end.</code></pre></figure>
<h3 id="new-open-source-library-erlmld">New open-source library: erlmld</h3>
We created an <a href="https://github.com/AdRoll/erlmld">open-source Erlang library for Kinesis and DynamoDB streams processing
(erlmld)</a> using MultiLangDaemon.
It also includes support for compressed KPL-style record aggregation and an additional
behavior which turned out to be a common pattern in our own usage of the library:
accumulating batches of records and flushing them periodically or when “full”. This
allows even simpler shard processor definitions:
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">-module(noisy_flusher).
-behavior(erlmld_flusher).
-export([init/2,
add_record/3,
flush/2]).
-include_lib("erlmld/include/erlmld.hrl").
-record(state, {shard_id, buf = []}).
init(ShardId, _Opaque) -&gt;
#state{shard_id = ShardId}.
add_record(#state{buf = Buf}, _, _) when length(Buf) &gt;= 10 -&gt;
{error, full};
add_record(#state{buf = Buf} = State, Record, Token) -&gt;
{ok, State#state{buf = [{Token, Record} | Buf]}}.
flush(#state{shard_id = ShardId, buf = Buf} = State, _Kind) -&gt;
{ProcessedTokens, Records} = lists:unzip(Buf),
io:format("~p processing batch: ~p~n", [ShardId, Records]),
timer:sleep(1000),
NState = State#state{buf = []},
{ok, NState, ProcessedTokens}.</code></pre></figure>
Here, a processor builds a batch of records to be processed. A backpressure mechanism
exists to ensure records are supplied only as fast as the processor can handle: if a
batch is full, the processor is instructed to flush (i.e., perform a batch of work),
which can take an arbitrary amount of time.
After flushing, the processor returns the opaque tokens corresponding to each
successfully-processed record, which allows the system to periodically checkpoint only
up to the highest contiguous completed record on the stream. If a worker dies or a new
one comes online and steals a lease, some work might be repeated but none will be lost.
<h2 id="beyond-good-and-erlang">Beyond Good and Erlang</h2>
The arrangement described above is scalable and works well, but the
process-per-owned-shard model means our concurrency is limited by the number of stream
shards. If we have a processing system with 16 cores and we’re processing a stream with
8 shards, at least half of our computing resources are wasted unless we take special
measures which make our simple processing code be not so simple.
For our use case, it turns out we can scale our processing independently of stream
shards while preserving approximate (ordered by cookie) in-order processing and simple
processing code, which is especially helpful for I/O-bound tasks like ours.
<h3 id="new-open-source-library-exmld">New open-source library: exmld</h3>
We created an <a href="https://github.com/AdRoll/exmld">open-source Elixir library for Kinesis and DynamoDB streams processing
(exmld)</a> which builds on our Erlang library <a href="https://github.com/AdRoll/erlmld">erlmld</a>. It
can be used in both Elixir and Erlang-based processing systems.
It makes use of the <a href="https://hexdocs.pm/flow/Flow.html">Flow</a> framework to set up a MapReduce-style processing
pipeline inside BEAM: instead of having a set of worker processes which each handle a
single shard, data from all shards owned by the current node feeds into a pipeline which
sends data downstream to a set of reducers of arbitrary number. Each reducer
consistently handles a partition of the key space (i.e., record keys are hashed and
distributed among reducers, and the same reducer will always receive records having the
same key in approximate order).
Here’s what we had before with the Erlang-based system:
<img src="/images/post_images/quaff_elixir/erlang-mld-workers.png" alt="erlang-based processing diagram" />
And here’s what we now have with the Elixir Flow-based processing pipeline:
<img src="/images/post_images/quaff_elixir/elixir-mld-pipeline.png" alt="elixir flow-based pipeline diagram" />
Here, using the Flow framework, our shard workers no longer directly process records.
Instead, they feed records into a set of mappers which extract zero or more items to be
processed from each. Each of these items has an associated key which is used to
distribute them among a set of reducers, which do the actual processing work. As work
is completed, the workers which originated each record are notified so they can
checkpoint and make progress on the shard they own.
This configuration allows re-use of any existing processing code with minor
modifications, but with greatly increased concurrency which is unrelated to the number
of stream shards owned by the current node (we can use an arbitrary number of mappers
and reducers). It helps us because we have an I/O bound task where processing can occur
independently for each cookie which is seen; ordering of events is generally preserved
for single cookies within a single worker (but not between different cookies).
<h4 id="elixirerlang-sample-processing-code">Elixir+Erlang sample processing code</h4>
Here’s a complete example of using Erlang processing code with this new Elixir library.
The “disposition” concept is used to keep track of the status of event processing to
support stream checkpointing and provide backpressure; see <a href="https://github.com/AdRoll/exmld">exmld</a> docs
for details. In a real application, an actual supervision tree would be used. The same
organization of code would also apply to an Elixir processing application using this
library.
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">-module(exmld_example).
-export([start/0]).
-record(state, {count = 0}).
-include_lib("erlmld/include/erlmld.hrl").
%% this will result in calls to process_event/2 for items extracted from
%% records seen on `StreamName` in `StreamRegion`. run
%% erlmld/priv/download.sh before running this to download needed JARs.
start() -&gt;
StreamRegion = "us-east-1",
%% this affects kcl state table name:
 AppName = "your-erlang-kcl-application",
StreamName = "your-stream-name",
{ok, Stage} = 'Elixir.Exmld.KinesisStage':start_link([]),
FlowSpec = #{stages =&gt; [Stage],
extract_items_fn =&gt; fun extract_items/1,
%% use first extracted item element as a reducer
 %% partition key:
 partition_key =&gt; {elem, 0},
state0 =&gt; fun () -&gt; #state{} end,
process_fn =&gt; fun process_event/2,
flow_opts =&gt; []},
{ok, FlowWorker} = 'Elixir.Exmld':start_link(FlowSpec),
ProducerConfig = #{record_processor =&gt; erlmld_batch_processor,
record_processor_data =&gt;
#{flusher_mod =&gt; 'Elixir.Exmld.KinesisWorker',
flusher_mod_data =&gt; [{stages, [Stage]}],
flush_interval_ms =&gt; 10000,
checkpoint_interval_ms =&gt; 60000,
watchdog_timeout_ms =&gt; 600000,
description =&gt; {StreamName, StreamRegion},
on_checkpoint =&gt; fun on_checkpoint/2},
kcl_appname =&gt; AppName,
stream_name =&gt; StreamName,
stream_region =&gt; StreamRegion,
stream_type =&gt; kinesis,
%% these would normally be set via app env:
 listen_ip =&gt; loopback,
listen_port =&gt; 0,
app_suffix =&gt; undefined,
initial_position =&gt; &lt;&lt;"LATEST"&gt;&gt;,
idle_time =&gt; 1000,
metrics_level =&gt; &lt;&lt;"SUMMARY"&gt;&gt;,
metrics_dimensions =&gt; &lt;&lt;"Operation"&gt;&gt;,
failover_time =&gt; 10000,
ignore_unexpected_child_shards =&gt; false,
worker_id =&gt; undefined,
max_records =&gt; 1000,
max_lease_theft =&gt; 1,
shard_sync_time =&gt; 60000},
{ok, ProducerSup} = erlmld_sup:start_link(ProducerConfig),
{ok, [Stage, FlowWorker, ProducerSup]}.
on_checkpoint(Description, ShardId) -&gt;
io:format("~p checkpointed on ~p~n", [Description, ShardId]).
%% this function should normally return a list of items to be processed
%% by process_event/2. if records are using the KPL-style aggregation
%% supported by erlmld, this can just return single-element lists. due
%% to the definition of 'partition_key' above, this should return
%% 2-tuples where the first element is a partition key to use in
%% distributing work among reducers. this example just uses the
%% record's partition key.
extract_items(#{'__struct__' := 'Elixir.Exmld.KinesisStage.Event',
stage := Stage,
worker := Worker,
event :=
#{'__struct__' := 'Elixir.Exmld.KinesisWorker.Datum',
stream_record := R}}) -&gt;
Value = case R of
{heartbeat, X} -&gt;
{X, heartbeat};
#stream_record{partition_key = K} -&gt;
{K, {Stage, Worker, R}}
end,
[Value].
%% this function is used as a Flow reducer. it accepts an extracted
%% item and a reducer state, and returns an updated state after doing
%% any needed processing.
process_event({Key, Value}, #state{count = C} = State) -&gt;
case Value of
{Stage, Worker, #stream_record{sequence_number = SN}} -&gt;
%% notify upstream stage of disposition of processing so the
 %% originating worker can make progress and checkpoint. in a
 %% real application this would be done in batches after work is
 %% completed, and the status would vary based on processing
 %% outcome.
 WorkerMap =
#{Worker =&gt;
[#{'__struct__' =&gt; 'Elixir.Exmld.KinesisWorker.Disposition',
sequence_number =&gt; SN,
status =&gt; ok}]},
ok = 'Elixir.Exmld.KinesisStage':disposition(Stage, WorkerMap),
%% do some work and return updated state:
io:format("~p processing ~p (seen: ~p)~n", [self(), Key, C]),
timer:sleep(100),
State#state{count = C + 1};
heartbeat -&gt;
%% heartbeats are used to prevent stalls when a worker is
 %% waiting for processing to happen and dispositions to be
 %% returned.
io:format("~p ignoring heartbeat event~n", [self()]),
State
end.</code></pre></figure>
<h3 id="impact">Impact</h3>
Using this new arrangement with our existing Erlang code, we doubled the performance of
our system. Here’s a snapshot of what happened to our stream delay metric after we
deployed an Elixir Flow-using version of our application following a service
degradation:
<img src="/images/post_images/quaff_elixir/elixir-improvement.png" alt="Erlang-&gt;Elixir improvement image" />
Here, the Erlang-only system had fallen behind and entered an unhealthy “far behind tip
of stream” state. It had scaled up to the maximum size and was recovering, but not very
quickly. The Elixir-using version was deployed to gauge performance during recovery;
stream processing delay fell off a cliff, and this Elixir-using version has been
running since.
<h1 id="conclusion">Conclusion</h1>
Using Erlang and now Elixir, we were able to implement a scalable system which cut our
DynamoDB costs by over 75%. Using some Elixir glue code[2] enabling use of the
<a href="https://hexdocs.pm/flow/Flow.html">Flow</a> framework, we doubled the performance of our previously Erlang-only
system with only minor modifications. We also open-sourced[1][2] the code we’re using
to process high-volume Kinesis and DynamoDB streams data in Erlang and Elixir.
I enjoyed working on this system, and Elixir is now firmly in our toolbox for use in the
future!
[1]: <a href="https://github.com/AdRoll/erlmld">https://github.com/AdRoll/erlmld</a> 
[2]: <a href="https://github.com/AdRoll/exmld">https://github.com/AdRoll/exmld</a>
Do you enjoy working with large-scale systems? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>How to Run a Front-End Infrastructure Team</title>
<link>https://tech.nextroll.com/blog/frontend/2017/08/29/how-to-run-a-front-end-infrastructure-team.html</link>
<pubDate>Tue, 29 Aug 2017 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/frontend/2017/08/29/how-to-run-a-front-end-infrastructure-team</guid>
<description>Over the past years, AdRoll has grown from a humble startup built around a
single feature to a global marketing platform with a <a href="https://www.adroll.com/product">diverse suite of
products</a>. Along with the growth of the company,
we have put a lot of work into building a solid infrastructure for user
interface development. In this post, we talk about the human aspects of
front-end projects that are shared between multiple engineering teams.
<figure class="full">
<img src="/images/post_images/frontend-infra-layers.jpg" alt="A screenshot of an AdRoll dashboard with analytics, i18n, CSS and UI components visualized as layers" />
<figcaption>Every web application at AdRoll relies on multiple layers of front-end infrastructure</figcaption>
</figure>
Our current front-end infrastructure consists of <a href="/blog/frontend/2015/11/05/rollup-shared-ui-components.html">UI
components</a>, a <a href="https://medium.com/adroll-design/the-journey-to-design-consistency-ee26bef2fd88">UX
pattern
library</a>
and various JavaScript packages for internationalization (i18n), analytics and
A/B testing. We consider all these projects “internally open sourced”. This
means we encourage our engineers to use contributions to these projects as a way
of sharing their knowledge across team boundaries. Ultimately, we believe
<a href="https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar">leveraging a wider range of ideas makes software more
robust</a>.
If one of our vertical product teams wants to propose a change to the
infrastructure, they are expected to gather feedback from other teams that might
be affected by it. In the simplest case, this could mean making a style in a UI
component customizable through a property. In more advanced cases, this could
mean building an entirely new shared library (such as an API client) to abstract
out complexity from other engineering teams.
While this model allows each team to build the exact solution they need, it’s
also important to have people whose sole purpose is to think about how
everything fits together. Without proper ownership and oversight, shared
infrastructure tends to become a complex mess of patches that feels more like a
bottleneck than a helpful tool.
<h2 id="the-frontend-core-team">The Frontend Core Team</h2>
Since 2015 each new product and internal tool we have built has been implemented
as a <a href="https://en.wikipedia.org/wiki/Microservices">microservice</a>. This loosely
coupled architecture has let us iterate on new functionality faster but it has
also fragmented our approach to UI development. Initially, a group of senior
engineers was able to build a foundation for our current front-end
infrastructure, but it was still unclear who should act as the first point of
contact for prioritizing tasks and triaging issues.
In 2016 we formalized our approach to maintaining front-end infrastructure by
founding a team we call Frontend Core. Today, this team works closely with our
front-end engineers, UI designers and product leaders to make sure our web
applications are built consistently across all teams. Most importantly, the
Frontend Core team oversees and maintains all of our collaborative front-end
projects.
The rest of this post covers the UI development guidelines set forth by our
Frontend Core team in five sections. Each section explains our general approach
to the topic and provides a list of practical tips for adopting similar
practices in other organizations:
<ul>
<li><a href="#remove-bottlenecks-for-iteration">Remove Bottlenecks for Iteration</a></li>
<li><a href="#communicate-across-team-boundaries">Communicate Across Team Boundaries</a></li>
<li><a href="#look-for-common-problems-and-solutions">Look for Common Problems and Solutions</a></li>
<li><a href="#lower-the-barrier-to-success">Lower the Barrier to Success</a></li>
<li><a href="#keep-on-experimenting">Keep on Experimenting</a></li>
</ul>
<h2 id="remove-bottlenecks-for-iteration">Remove Bottlenecks for Iteration</h2>
As infrastructure projects become more complex and more teams depend on them, it
becomes harder to make changes without breaking existing functionality. With
this in mind, we are constantly streamlining the development workflow around our
front-end infrastructure. Our ultimate goal is to make our shared projects a joy
to work with so that more people will adopt them and keep making improvements.
Since the beginning of 2016, we have had 47 internal contributors make changes
to our front-end infrastructure, resulting in over 500 pull requests. Each
change has gone through code review and a suite of automated tests to prevent
unintended side-effects. Our Frontend Core team keeps a close eye on open pull
requests and makes sure they get reviewed as quickly as possible.
To measure improvements in the process, we also gather statistics on pull
request lifetime over time. The chart below shows a typical bimodal distribution
for code reviews: Most changes take about a week to review but there’s a subset
of quick changes (such as production patches) that are deployed within an hour
of opening the pull request.
<figure>
<img src="/images/post_images/frontend-infra-pr-lifetime.png" alt="A bar chart of pull request lifetime" />
<figcaption>To get charts like this for your GitHub project, we recommend checking out <a href="https://github.com/pgjones/push-pull" class="no-wrap">this open source tool</a></figcaption>
</figure>
Here’s how you can remove common bottlenecks in your UI development process:
<ul>
<li>Make simple changes fast and complex ones possible. Production issues can
often be mitigated with UI changes so make sure patches can be reviewed and
released as quickly as possible. Encourage talking through large changes
before the actual programming work.</li>
<li>Consider bundling similar JavaScript packages in a monorepo. A single code
repository allows packages to share build tools and test scripts with less
overhead compared to completely separate projects. See
<a href="https://github.com/babel/babel/blob/19c4dd2d8c99b909b02ccca27de7b0b32eb416ea/doc/design/monorepo.md">Babel</a>
for a popular example and <a href="https://github.com/lerna/lerna">Lerna</a> for a
pre-existing toolkit.</li>
<li>Use <a href="https://help.github.com/articles/creating-a-pull-request-template-for-your-repository/">pull request
templates</a>
and <a href="https://help.github.com/articles/about-codeowners/">CODEOWNERS files</a> for
GitHub projects. PR templates should include a checklist for commonly
overlooked aspects of UI development such as reusability, translatability and
style consistency. Use code reviews as a way of transferring knowledge between
UI teams.</li>
<li>Set up staging deployments for feature branches. Having a live demo means
the reviewer doesn’t always have to run the code locally.
<a href="https://storybook.js.org/">Storybook</a> is an excellent tool for building demos
for React components.</li>
<li>Don’t over-optimize. Wondering how much time you should spend improving
common tasks? <a href="https://xkcd.com/1205/">XKCD has an answer</a>.</li>
</ul>
<h2 id="communicate-across-team-boundaries">Communicate Across Team Boundaries</h2>
Internal open source projects rely heavily on communication for announcing new
features and breaking changes. One of the most important roles for an
infrastructure team is to share context and act as a relay between contributors.
At AdRoll, the Frontend Core team holds a meeting every two weeks with engineers
and designers from all of our product teams. In the meeting, teams share updates
on any major front-end changes they are working on, even if they don’t directly
affect our infrastructure. A recurring meeting like this has proven to be very
useful for identifying surprising connections between teams.
When a team mentions they are working on a new feature, another team might
mention that they already have a solution for it. In other cases, we have
identified pain points in the infrastructure that everyone feels but nobody has
tackled yet. These discussions often result in new tickets in our Frontend Core
backlog or updates to the development process.
Conversations around front-end infrastructure can be challenging because UI
development involves so many different specialties. Designers, engineers and
product managers often use different words for the same things depending on
their background (e.g. “dropdown” vs. “pull-down menu” vs. “select input”). It
can also be difficult for a single developer to get a high level overview of all
the resources available to them (e.g. global color variables and CSS classes).
To establish a common conceptual language and help with discoverability, we
built a <a href="http://ux.adroll.com/">UX pattern library</a> and released it as a
resource both inside and outside the company. To complement the pattern library,
we also have an internal site that lists all of our UI components and lets
developers compare their functionality over multiple versions (see video below).
These sites allow everyone in the company to get a complete picture of our
front-end infrastructure and use explicit URLs when referring to specific
UI elements.
<figure>
<div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/enZuX4RuSB8" frameborder="0" allowfullscreen=""></iframe></div>
<figcaption>Our internal site for demoing shared UI components, built using the <a href="https://github.com/awslabs/aws-js-s3-explorer" class="no-wrap">AWS JavaScript S3 Explorer</a></figcaption>
</figure>
Here are some ways you can improve communication around infrastructure:
<ul>
<li>Set up a dedicated discussion channel for front-end infrastructure. Public
Slack channels or email lists work well for this purpose. If necessary, set up
a separate support channel for urgent issues.</li>
<li>Communicate visually whenever possible. Screenshots, screen captures and
live demos are worth a thousand chat messages.</li>
<li>Use <a href="http://semver.org/">Semantic Versioning</a> when publishing shared
packages. Proper versioning helps build trust between engineering teams.
Thinking in terms of breaking changes, new features and bug fixes is a helpful
framework for updates that affect user-facing functionality.</li>
<li>Help teams through breaking changes. In an internal open source project,
you have a direct line of communication to every user. Do your best to make
breaking changes less painful by supporting the teams working on upgrading
components.</li>
<li>Be open to feedback and suggestions. The infrastructure is your product
and engineers are your users. Encourage new engineers to question technical
choices to surface new ideas.</li>
</ul>
<h2 id="look-for-common-problems-and-solutions">Look for Common Problems and Solutions</h2>
Even with open and active channels of communication, pain points aren’t always
surfaced in an explicit way. When one of your coworkers faces an issue, they
might not be aware that it’s a common problem for others. A good maintainer of
front-end infrastructure is able to distill the chatter around UI development
into clear problem statements and work on solutions for the most common ones.
Here are some typical messages and their implications from an infrastructure
point of view:
<blockquote>
<ul>
<li>I wish we had…</li>
<li>Since we don’t have X, we can’t…</li>
<li>I’ve been meaning to build…</li>
<li>Do we have support for X yet?</li>
</ul>
Messages like these can be a sign that the infrastructure doesn’t support some
use cases. Reach out to the person or team to understand the context and see
if they could become a contributor.
<ul>
<li>I’m seeing this error when…</li>
<li>How do you run…</li>
<li>I couldn’t figure out X</li>
<li>Do you know what I’m doing wrong?</li>
</ul>
When people mention errors or ask for help, it can either be a real bug or a
sign that the project isn’t documented well enough. Help the person resolve
the issue and see how you can update the instructions in your project
afterwards.
<ul>
<li>It was taking forever so I…</li>
<li>X is too simple so I had to…</li>
<li>I couldn’t use that option so…</li>
<li>This is hacky but…</li>
</ul>
Software engineers take pride in working around issues. Take note when people
mention hacky solutions and improve the infrastructure accordingly. Official
solutions are easier to maintain over time than application specific hacks.
<ul>
<li>Oh, I forgot to run the tests again</li>
<li>Joe, could you deploy this?</li>
<li>I’m waiting for Ted to…</li>
<li>Do you think that’s ready to merge?</li>
</ul>
Procedural discussion around internal open source projects can be a sign that
the development process isn’t as streamlined as it could be. Consider
automating tests and train more people to act as maintainers.
</blockquote>
Over time, maintainers should automate the day-to-day tasks of their projects as
much as they can so they can focus on long term improvements. The tasks that
can’t be automated should be well documented so they are easily repeatable. For
example, when reviewing changes to our shared UI components, we always look out
for changes in global namespaces – CSS classes, global
<a href="http://sass-lang.com/">SASS</a> variables and <a href="http://redux.js.org/">Redux</a>
actions.
Documentation and guidance can be automated as well: Instead of referring people
to static documents like wiki pages, design a development workflow that gives
users enough instructions so they can solve problems on their own. <a href="https://facebook.github.io/react/blog/2016/07/11/introducing-reacts-error-code-system.html">React’s
error code
system</a>
is a great example of self-documenting software: The library itself warns
developers if they are doing something wrong and points them to a helpful
documentation page.
One of the most effective ways of keeping your infrastructure stable is
abstracting out parts of the codebase that most contributors don’t need to
modify. This means thinking of your development workflow a series of <a href="https://en.wikipedia.org/wiki/Black_box">black
boxes</a>. Just like you don’t typically
modify the source code of your code editor or web browser, the contributors in
infrastructure projects shouldn’t need to modify the build process or the test
harness.
For us, one such improvement was automating the publish process for our shared
UI components. Previously publishing a new version of a component required
multiple manual steps and only a few senior engineers had the credentials
required to run them. The results also varied slightly depending on each
publisher’s local environment. Once we moved the entire process to a
<a href="https://jenkins.io/\">Jenkins</a> job, we could publish a component just by
picking its name from a list and entering a version number. This has allowed
more contributors to publish changes on their own and reduced the workload on
our Frontend Core team.
<figure>
<img src="/images/post_images/frontend-infra-jenkins-publish.png" alt="A screenshot from Slack showing notifications from Jenkins and GitHub" />
<figcaption>Slack notifications from the Jenkins job that publishes our shared UI components</figcaption>
</figure>
Here are some of our tips for building self-documenting projects:
<ul>
<li>Use type checking to enforce assumptions in your code. Our shared UI
components make heavy use of <a href="https://facebook.github.io/react/docs/typechecking-with-proptypes.html">React
PropTypes</a>.
We’re also experimenting with <a href="https://www.typescriptlang.org/">TypeScript</a>.</li>
<li>Test and lint your code extensively. Make sure tests and linters are run
automatically against every proposed change. Linters like
<a href="https://eslint.org/">ESLint</a> and
<a href="https://github.com/brigade/scss-lint">scss-lint</a> can even help improve the
runtime performance of your applications.</li>
<li>Log instructions and next steps in command-line tools. For example, when
starting a development server, make sure the local URL is shown in the console
output.</li>
<li>Send notifications from build systems. Contributors should be notified if
they break a build or a deployment. Maintainers should be cc’d so they can
spot recurring issues.</li>
<li>Write actionable error messages. Don’t expect everyone to be familiar with
the latest front-end tools. Error pages in internal services should give users
ways to debug the issue.</li>
</ul>
<h2 id="lower-the-barrier-to-success">Lower the Barrier to Success</h2>
Success in UI development doesn’t just mean giving end users access to new
functionality, it means that new features have to seamlessly blend in with
everything else in the product. Bad practices in UI development directly result
in a bad user experience. This puts a lot of pressure on front-end engineers and
forces them to think of shared infrastructure earlier in a company’s lifetime
compared to their back-end oriented colleagues.
When an engineer builds something on top of existing infrastructure, they should
feel like they’re standing on the shoulders of giants – like they can achieve
something they didn’t know they could. Front-end infrastructure should enable
every engineer to build beautiful, user-friendly features that can be tested,
monitored and deployed easily.
In order to remain useful, infrastructure has to grow with the company. In our
experience, the best way to guarantee that a project survives over time is to
focus on developer engagement. This is why we believe the open source process is
such a good match with software infrastructure: When people know they had a part
in building a project, they are more likely to care about it (also known as the
<a href="https://en.wikipedia.org/wiki/IKEA_effect">IKEA effect</a>).
<figure>
<img src="/images/post_images/frontend-infra-ikea-instructions.jpg" alt="An excerpt of IKEA's assembly instructions" />
<figcaption>IKEA helps people succeed with their famous assembly instructions</figcaption>
</figure>
Here’s how you can make developers engaged in collaborative projects:
<ul>
<li>Be transparent about decision making. Your fellow engineers should know
why something is built the way it is and why you chose a certain technology
over another.</li>
<li>Help people learn more about the infrastructure and how it is used.
Encourage people to read design documents and source code. Open up access to
usage metrics and site analytics.</li>
<li>Make sure maintainers follow the same guidelines as everyone else. Nobody
should be above code reviews or testing guidelines. When exceptions have to be
made, be open about the reasons behind them.</li>
<li>Highlight achievements on a personal level. Celebrate wins from engineers
who make their first front-end contributions. Let people show their own work
in meetings through presentations and screencasts.</li>
<li>Let others lead and take responsibility. Having contributors you can trust
is necessary for scaling the development process. Delegate the ownership of
specialized UI components to the teams that built them.</li>
</ul>
<h2 id="keep-on-experimenting">Keep on Experimenting</h2>
Almost everything we have learned about front-end infrastructure has been a
result of controlled experimentation. We’re constantly trying out new promising
technologies and improvements in our development process. Whenever we identify
different approaches to the same problem, we try out the most promising ones and
review the results afterwards.
It is good to remember that even the best infrastructure comes with some
overhead. Every piece of shared code will eventually break or become outdated.
An important part of the experimentation mindset is questioning the choices you
have made and knowing when standardization is not the right approach.
<figure>
<img src="/images/post_images/frontend-infra-xkcd-1319-adapted.png" alt="XKCD comic #1319 with 'automation' changed to 'infrastructure'" />
<figcaption>Adapted from <a href="https://xkcd.com/1319/">XKCD #1319 "Automation"</a></figcaption>
</figure>
Some of our ongoing experiments in front-end infrastructure are:
<ul>
<li>Replacing Browserify and Gulp with Webpack as a standard front-end build
process</li>
<li>Integrating <a href="http://redux.js.org/">Redux</a> actions into our shared React
components</li>
<li>Extending our <a href="http://ux.adroll.com/">public style guide</a></li>
<li>Unifying usage analytics, exception tracking and A/B testing using a custom
JavaScript library</li>
<li>Standardizing integration testing with Enzyme (as
<a href="https://speakerdeck.com/jessicagrist/redux-apps-with-enzyme">presented</a> at
the ReactJS SF Meetup on July 13th)</li>
<li>Building a <a href="http://yeoman.io/">Yeoman</a> generator for bootstrapping web
applications</li>
<li>Using <a href="https://storybook.js.org/">Storybook</a> for the live demos in our shared
UI component library</li>
</ul>
<hr />
Thanks for reading! We hope this post is helpful for people who are looking to
improve their UI development processes. If you would like to hear more about a
specific topic or share your experiences on maintaining front-end
infrastructure, don’t hesitate to post a comment below.
PS: <a href="https://www.adroll.com/about/careers">We are hiring</a> both remote and local
engineers!
</description>
</item>
<item>
<title>TrailDB 0.6 Released</title>
<link>https://tech.nextroll.com/blog/data/2017/05/15/traildb-0.6.html</link>
<pubDate>Mon, 15 May 2017 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data/2017/05/15/traildb-0.6</guid>
<description>Today, almost a year since <a href="http://tech.adroll.com/blog/data/2016/05/24/traildb-open-sourced.html">the initial TrailDB open-source release</a>,
we are happy to announce the next major version of TrailDB, 0.6. The
release is long overdue: this release is packed with major new features
(and some minor bug fixes) that have been driven both by increasing
internal use of TrailDB at AdRoll, as well as requests and contributions
by the community.
You can download and install the latest version by following
the <a href="http://traildb.io/docs/getting_started/">getting started</a> guide.
<h3 id="traildb-in-action">TrailDB in Action</h3>
First, let’s start with some highlights about the usage of TrailDB.
At AdRoll, the amount of data stored in TrailDBs has been growing
exponentially. It took two years to reach the first trillion events
stored. Now we are routinely creating new TrailDBs storing trillions
of events every week. Besides the scale, TrailDBs are powering even
more critical services at AdRoll, including various machine learning
pipelines, large-scale data analysis, and our new <a href="https://www.adroll.com/resources/guides-and-reports/past-present-future-of-attribution">attribution engine</a>.
We want TrailDB to be a small, battle-hardened piece of
software which works with different languages and environments without
too much trouble. We have been preaching this gospel at a number of
events. For instance, see our presentation at <a href="https://www.youtube.com/watch?v=qK43uLbnFtg">an SF Data Mining meetup</a> or
<a href="https://www.youtube.com/watch?v=-oPFxSwn0lM">a TrailDB tutorial at the PyData SF 2016 conference</a>:
<iframe src="//slides.com/villetuulos/traildb-tutorial-pydata-2016/embed" width="576" height="420" scrolling="no" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
Most important, we are excited to see TrailDB gaining traction
outside AdRoll. Thank you for all the raised issues, suggestions, and
contributions, such as the nascent language bindings for <a href="https://github.com/poynt/traildb-node">NodeJS</a>. Recently, we adopted community-contributed <a href="https://github.com/traildb/traildb-rust">Rust bindings</a>
under the official TrailDB organization in GitHub. If you have any
feedback or ideas about TrailDB, get in touch on <a href="https://gitter.im/traildb/traildb">the TrailDB Gitter channel</a>.
<h3 id="major-new-features">Major New Features</h3>
You can see the full list of changes since the 0.5 release in <a href="https://github.com/traildb/traildb/blob/master/CHANGELOG.md">the
changelog</a>. Here are some highlights:
<h4 id="trck---a-highly-optimized-query-language-for-traildb">Trck - a highly optimized query language for TrailDB</h4>
When we made the initial open-source release, we hinted, “We have
a number of tools built on top of TrailDB which make computing various
user-level metrics easier. We are planning to open-source some of
these tools in the future.” This finally happened in March, when we
open-sourced <a href="http://tech.adroll.com/blog/data-science/2017/03/21/trck.html"><code class="language-plaintext highlighter-rouge">trck</code>, a highly optimized query language for TrailDB</a>.
In brief, if you have ever had a need to run queries on user-level funnels
like “how many users have first done A, then B within T seconds”, <code class="language-plaintext highlighter-rouge">trck</code> is
the query language for you. Almost all
systems powered by TrailDB at AdRoll use <code class="language-plaintext highlighter-rouge">trck</code> in one form or another.
It is hard to overstate its usefulness when your data is shaped like
TrailDB. To learn more You should read a separate <a href="http://tech.adroll.com/blog/data-science/2017/03/21/trck.html">blog post about
<code class="language-plaintext highlighter-rouge">trck</code></a>
and its <a href="https://github.com/traildb/trck/blob/master/README.md">documentation</a>.
<h4 id="filters-views-and-multi-cursors">Filters, Views, and Multi-Cursors</h4>
Most new features in this release are related to creating <a href="http://traildb.io/docs/technical_overview/#return-a-subset-of-events-with-event-filters">event
filters</a>
that select a subset of features or trails from TrailDB. You
can filter events by defining <a href="http://traildb.io/docs/api/#filter-events">a boolean expression over fields</a>,
including <a href="http://traildb.io/docs/api/#tdb_event_filter_add_time_range">timestamps</a>. See <a href="http://slides.com/villetuulos/traildb-tutorial-pydata-2016#/47">the PyData presentation for examples of filters</a>.
You can set filters to cover the whole TrailDB with <a href="http://traildb.io/docs/api/#tdb_set_opt">tdb_set_opt</a> which, in effect, creates <a href="http://traildb.io/docs/technical_overview/#return-a-subset-of-events-with-event-filters">a view over the TrailDB</a>
that can be <a href="http://traildb.io/docs/technical_overview/#create-traildb-extracts-materialized-views">materialized</a>. You can also set the filters to cover only
individual trails, allowing fine-grained <a href="http://traildb.io/docs/technical_overview/#whitelist-or-blacklist-trails-a-view-over-a-subset-of-trails">whitelisting and blacklisting
of trails</a>. Or you can <a href="http://traildb.io/docs/api/#tdb_cursor_set_event_filter">attach filters only to an individual cursor</a>.
<a href="http://traildb.io/docs/api/#join-trails-with-multi-cursors">Multi-cursors</a> allow iterating over separate trails (users) as if they
were a single trail. Multi-cursors even work across multiple separate
TrailDBs. This feature was motivated by the fact that multiple UUIDs
may correspond to the same logical user and we want to query all events
related to the user, even if they were stored as separate trails.
<h4 id="command-line-tool-improvements">Command line tool improvements</h4>
The <code class="language-plaintext highlighter-rouge">tdb</code> command line tool has improved in this release. Here are
the highlights:
<ul>
<li>
Specify an event filter with the <code class="language-plaintext highlighter-rouge">--filter</code> flag. You can speed
up filtered queries significantly by creating an index with
<code class="language-plaintext highlighter-rouge">tdb index</code>.
</li>
<li>
New <code class="language-plaintext highlighter-rouge">tdb merge</code> command for merging TrailDBs, even
when they have mismatching sets of fields.
</li>
<li>
You can return a subset of trails with the <code class="language-plaintext highlighter-rouge">--uuids</code> filter.
</li>
</ul>
Last but not least, the <code class="language-plaintext highlighter-rouge">tdb</code> functionality is now automatically tested
by Travis for every pull request.
<h3 id="experimental-features">Experimental Features</h3>
We want to keep the core TrailDB very stable and robust. At the same
time, it is fun and beneficial to experiment with new directions which
might find their way to the core eventually.
A good example of this is a feature that allows you to <a href="http://tech.adroll.com/blog/data/2016/11/29/traildb-mmap-s3.html">query TrailDBs
directly from Amazon S3 without downloading them locally</a>. This is
possible thanks to a relatively new feature in the Linux kernel,
user-space page fault handling, which allows us to download only
parts of TrailDB on demand with minimal changes to the TrailDB
codebase. This feature can reduce query latencies significantly if
your application needs to access only a subset of trails, events, or
fields.
Another experimental feature is <a href="https://github.com/traildb/reel">Reel, an AWK-like query language for
TrailDB</a>. As mentioned above, <code class="language-plaintext highlighter-rouge">trck</code> is our trusted workhorse for
expressing user-level queries. Reel was motivated by a particularly
complex query that needed to be executed over a trillion events. Although
it is not quite as mature as <code class="language-plaintext highlighter-rouge">trck</code>, you can easily embed and extend it
for your own use cases.
<h3 id="future-roadmap">Future Roadmap</h3>
TrailDB 0.6 is a robust data backend for applications that need to
execute complex computation over discrete events over time. As we
have emphasized before, we take the stability of the C API, ABI, and
especially the on-disk format very seriously. You can use the 0.6
release to read TrailDBs created with any previous version of the
software. This should hold true for any future version of TrailDB as
well.
Our next big focus after this release is to optimize creation of
TrailDBs. As mentioned above, AdRoll creates multi-trillion event
TrailDBs weekly. Even when using spot instances on AWS, it costs
thousands of dollars to create these files based on raw log files. There
are some easy optimizations that are targeted for the next 0.7 release
which should drastically lower the cost of creating massive TrailDBs.
Meanwhile, we hope that you enjoy the 0.6 release. If you have any
questions, comments, or contributions, you can reach us at
<a href="https://gitter.im/traildb/traildb">the TrailDB Gitter channel</a>.
</description>
</item>
<item>
<title>trck: querying discrete time series with state machines</title>
<link>https://tech.nextroll.com/blog/data-science/2017/03/21/trck.html</link>
<pubDate>Tue, 21 Mar 2017 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data-science/2017/03/21/trck</guid>
<description>Last year we released <a href="http://tech.adroll.com/blog/data/2016/05/24/traildb-open-sourced.html">TrailDB</a>, a library we use extensively at AdRoll to efficiently store event data. Today we’re open sourcing <a href="https://github.com/traildb/trck">trck</a>, a query engine complementary to TrailDB that we use to analyze trillions of discrete events in TrailDBs every day.
As you may remember, the TrailDB data schema is very simple:
<img src="/images/post_images/trck_traildb_datamodel.png" alt="traildb_datamodel" />
Every user (or “trail”) has a sequence of timestamped events associated with it. Every event has a number of fields, and the set of fields is fixed per TrailDB.
For example, a dataset of Wikipedia edits may contain a user’s IP address and page title as fields:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"># tdb dump -j -i wikipedia-history-small.tdb | head
{"uuid": "86f7f07096e5a9f01509b2b2e9380000", "time": "1152738820", "user": "", "ip": "81.152.155.52", "title": "ZZZap!"}
{"uuid": "86f7f07096e5a9f01509b2b2e9380000", "time": "1152739029", "user": "", "ip": "81.152.155.52", "title": "ZZZap!"}
{"uuid": "86f7f07096e5a9f01509b2b2e9380000", "time": "1152739044", "user": "", "ip": "81.152.155.52", "title": "ZZZap!"}
{"uuid": "86f7f07096e5a9f01509b2b2e9380000", "time": "1152739092", "user": "", "ip": "81.152.155.52", "title": "ZZZap!"}
{"uuid": "86f7f07096e5a9f01509b2b2e9380000", "time": "1152739159", "user": "", "ip": "81.152.155.52", "title": "ZZZap!"}
{"uuid": "86f7f07096e5a9f01509b2b2e9380000", "time": "1152739280", "user": "", "ip": "81.152.155.52", "title": "ZZZap!"}
{"uuid": "2e0774e93410d504bc1720b8ad3d0000", "time": "1159025033", "user": "", "ip": "219.110.177.197", "title": "Ip dip"}
{"uuid": "237589722f980f576dd156a29d440000", "time": "1109300460", "user": "", "ip": "134.53.178.231", "title": "Joss sticks"}
{"uuid": "c0ced428660ed821de06446f10720000", "time": "1294656212", "user": "", "ip": "180.193.58.138", "title": "Indian Rebellion of 1857"}
{"uuid": "221aeaa8b065796bc568e026c1b60000", "time": "1273111183", "user": "", "ip": "70.111.4.54", "title": "Dwight-Englewood School"}</code></pre></figure>
Right now, if you want to query these data you can use <a href="https://github.com/traildb/traildb-python">Python</a> or <a href="https://github.com/traildb/traildb-r">R bindings</a> to TrailDB. So, if we want to count the number of user sessions in a TrailDB we could do this in Python:
<figure class="highlight"><pre><code class="language-python" data-lang="python">import sys
import traildb
if __name__ == '__main__':
db = traildb.TrailDB(sys.argv[1])
sessions = 0
for cookie, trail in db.trails():
last_timestamp = 0
for event in trail:
if event.time &gt;= last_timestamp + 30:
sessions += 1
last_timestamp = event.time
print sessions</code></pre></figure>
Here, a user session is defined as a series of events no more than 30 minutes apart. If we run this script on a sample database from the <a href="http://traildb.io/docs/tutorial/">TrailDB tutorial</a>, it processes <a href="http://traildb.io/data/wikipedia-history-small.tdb">wikipedia-history-small.tdb</a> in about 100 seconds on my laptop.
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ time python example.py wikipedia-history-small.tdb
1774765
real 1m40.645s
user 1m31.032s
sys 0m2.963s</code></pre></figure>
It doesn’t sound like a lot, but that database is pretty tiny: it contains 6.5M events and 450K trails, and takes about 100MB on disk. We routinely use multi-gigabyte TrailDBs containing billions and trillions of events.
At large scale, Python starts to struggle to process all the data quickly enough. And it is not so much the Python interpreter itself; if you use <a href="https://pypy.org/">PyPy</a>, or even a completely different language, there is substantial overhead that comes just from decoding and marshalling event data since it is so granular.
That made us realize that it would be nice to have a way to query TrailDBs in some way that can bypass all that marshalling. After all, many queries are not so complicated to require a full-blown Turing complete language to express them. Conceptually, something like <code class="language-plaintext highlighter-rouge">grep</code>, <code class="language-plaintext highlighter-rouge">awk</code>, or a regular expression engine would do, except it has to work on structured events with fields and have a concept of time.
<h4 id="trck-regular-expressions-for-traildbs">trck: “regular expressions” for TrailDBs</h4>
One thing to note is that many queries on event-based data have much in common. Usually you want to keep some kind of minimal state per trail (that’d be <code class="language-plaintext highlighter-rouge">last_timestamp</code> in the above example) and then depending on what events you see down the line, you take some kind of action, such as incrementing a session counter.
That sounds very similar to regular expressions, or, speaking more generally, finite state machines. For example, the above script can be expressed as a state machine:
<img src="/images/post_images/trck_statem.png" alt="statem" />
We start at the <code class="language-plaintext highlighter-rouge">start</code> state. Then, when we see an event in the trail, we transfer to the <code class="language-plaintext highlighter-rouge">sess</code> state and then on any following event, we keep coming back to <code class="language-plaintext highlighter-rouge">sess</code>. If we sit in <code class="language-plaintext highlighter-rouge">sess</code> for 30 minutes and nothing happens, we transfer back to <code class="language-plaintext highlighter-rouge">start</code> (and increment session counter).
And that’s what our small query language called <code class="language-plaintext highlighter-rouge">trck</code> does: you can express your query as a state machine, with some actions, like incrementing a counter, attached to edges.
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang">start -&gt;
receive
* -&gt; sess
sess -&gt;
receive
* -&gt; sess
after 30m -&gt; yield $count, start</code></pre></figure>
Then we can easily compile this to C code that works with TrailDB directly. If we save the above snippet as <code class="language-plaintext highlighter-rouge">wikipedia_example.tr</code>, call the <code class="language-plaintext highlighter-rouge">trck</code> compiler to produce a binary, and then run it on <code class="language-plaintext highlighter-rouge">wikipedia-history-small.tdb</code>:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ ./bin/trck -c wikipedia_example.tr
Compiling wikipedia_example3.tr
Produced binary in matcher-traildb in 0.54 seconds with clang-omp[openmp]
$ time ./matcher-traildb wikipedia-history-small.tdb 2&gt;/dev/null
{ "$count": 1774765 }
real 0m0.813s
user 0m2.819s
sys 0m0.179s</code></pre></figure>
Note that the evaluation of a state machine can be also trivially parallelized, as there are almost no data dependencies between state machines for different users. The <code class="language-plaintext highlighter-rouge">trck</code> compiler employs <a href="http://www.openmp.org/">OpenMP</a> for that.
As a result, this example produces the same result as the Python script, except it runs on <a href="http://traildb.io/data/wikipedia-history-small.tdb">wikipedia-history-small.tdb</a> in about 800ms, or 100x faster. And if you run it on a much larger dataset <a href="http://traildb.io/data/wikipedia-history.tdb">wikipedia-history.tdb</a> that contains the full edit log of Wikipedia for fifteen years (5GB, 600M events), it takes about 90 seconds on my laptop and scales very well with the number of cores. On a reasonably sized 16-core EC2 instance, the same query runs in 40 seconds.
There is a lot more you can do with <code class="language-plaintext highlighter-rouge">trck</code>: for example, in addition to counters it supports set and multiset types. There are also nested timeouts on states and some interesting higher-level optimizations that allow it to skip parts of trails that have no chance of altering the state. Check out the <a href="https://github.com/traildb/trck">README.md</a> in the repository.
<h4 id="implementation">Implementation</h4>
The <code class="language-plaintext highlighter-rouge">trck</code> compiler front end is implemented in Python using <a href="http://www.dabeaz.com/ply/">PLY</a>. It takes the state machine and compiles it to C code, that is then linked with the <code class="language-plaintext highlighter-rouge">trck</code> runtime library (written in C99), producing a static binary. It is tested on Linux and on OS X. It is now available under the MIT license on <a href="https://github.com/traildb/trck">GitHub</a> .
If you have questions, you’re welcome to join our <a href="gitter.im/traildb/traildb">TrailDB Gitter channel</a>.
</description>
</item>
<item>
<title>Thompson Sampling and Bayesian Factorization Machines</title>
<link>https://tech.nextroll.com/blog/data-science/2017/03/06/thompson-sampling-bayesian-factorization-machines.html</link>
<pubDate>Mon, 06 Mar 2017 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data-science/2017/03/06/thompson-sampling-bayesian-factorization-machines</guid>
<description>On the Data Science Engineering team, we are constantly working to improve the
machine learning systems powering AdRoll’s products. We have recently started
investigating Thompson sampling and Bayesian Factorization Machines as a way to
ensure efficient exploration of the ad marketplace. In this post, I will
illustrate why such exploration is necessary, and then dive into some of the
math and algorithms required to make it work. Credit for inspiring this project
goes to the paper <a href="http://research.cs.rutgers.edu/~lihong/pub/Chapelle12Empirical.pdf">“An Empirical Evaluation of Thompson Sampling,”</a>
by Olivier Chapelle and Lihong Li. We worked through a derivation of their
approach, and then extended it to work with our <a href="http://tech.adroll.com/blog/data-science/2015/08/25/factorization-machines.html">Factorization Machines</a>
model.
<h2 id="motivation">Motivation</h2>
One of the central problems we solve at AdRoll is deciding which ad to show to
a given user browsing a given page on the Web. We have to make these decisions
in an environment that is constantly changing, as new ads get added to our
system, old ads get retired, and users appear, disappear, or change their
browsing behaviors. In mathematical terms, we say that the distribution we are
trying to model is non-stationary.
This non-stationary regime means that there is always some uncertainty about
which decision is optimal, and each sequential decision gives us additional
information that we can use to make better decisions in the future. In other
words, we have a classical <a href="https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node7.html">exploration vs. exploitation tradeoff</a>
from reinforcement learning.
To illustrate this, consider a simple example where we have two ads to choose
from: Ad 1 has a predicted conversion probability of 0.123, and we have shown
this ad many times before, so we are fairly certain in our estimate. Ad 2 has a
predicted conversion probability of 0.111, but we’ve only shown it a few times,
so its true value could be a bit higher or a bit lower. We have two options:
<ul>
<li>Exploit what we already know, and choose Ad 1 because it has a higher
predicted conversion probability.</li>
<li>Explore and choose Ad 2, in order to get more knowledge about it and
improve our decisions in the future.</li>
</ul>
After sufficient exploration, the conversion probability of Ad 2 could turn out
to be 0.100, in which case Ad 1 is much better and the exploration did not pay
off. On the other hand, Ad 2’s probability could turn out to be 0.130, in which
case it is better than Ad 1, and we wouldn’t have discovered this without
taking a chance on Ad 2. Exploration allows us to make better-informed
decisions going forward, but there is no way to tell ahead of time whether we
will find a better ad this way.
It is clear that some amount of exploration is necessary, otherwise our system
would always show old ads known to perform well, and never show any new ads
that our customers upload. On the other hand, it is also clear that excessive
exploration is wasteful: once we have done enough exploration to determine that
Ad 1 is much better than Ad 2, there is no need to continue showing Ad 2
anymore.
One naïve explore/exploit strategy is to always explore on X% of traffic, by
showing a random ad, and always exploit on the remaining (100 - X)% of traffic,
by showing the best-known ad. This <a href="https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node16.html">epsilon-greedy strategy</a> is
simple to implement, but it is hard to tell if it performs too much or too
little exploration. Ideally, we would like a system that adjusts the amount of
exploration dynamically, based on how uncertain it is about the value of each
decision.
Now that we understand the problem intuitively, it’s time to formalize it and
describe our solution.
<h2 id="contextual-bandits">Contextual Bandits</h2>
Our problem can be formulated as a <a href="https://en.wikipedia.org/wiki/Multi-armed_bandit#Contextual_Bandit">contextual multi-armed bandit</a>,
where we are playing a sequential decision game. At each iteration, we get a
context vector $x$, which is a feature vector containing all the relevant
information about the user and the page they are visiting. We choose an action
$a$ from a set of available actions $A$, which in our case is the set of
ads to show. Some time later, we receive a reward $r$, which is 1 if the user
converted, and 0 otherwise. Our goal is to choose actions in a way that
maximizes cumulative reward.
One obvious way to approach action selection is to build a predictive model of
the reward given a context and an action: $P(r|x,a)$. We can use our existing
<a href="http://tech.adroll.com/blog/data-science/2015/08/25/factorization-machines.html">Factorization Machines</a> (FM) model to do this, by just merging the context
features $x$ and the ad features $a$ into a single feature vector. At each
iteration, we would “score” each action $a$ by computing $P(r|x,a)$ using
our model, and then we’d have to decide whether to explore or exploit.
<h2 id="thompson-sampling">Thompson Sampling</h2>
It turns out that by taking a Bayesian approach, we can solve our
explore/exploit dilemma in a very elegant way. Consider our model from before,
$P(r|x,a)$. This model is parameterized by some set of parameters $\theta$,
for example, the set of weights in logistic regression or FM. In a non-Bayesian
setting, we have some optimization algorithm that finds a maximum-likelihood
estimate for $\theta$, which is equivalent to minimizing logistic loss on the
training set. In a Bayesian setting, we place a prior $P(\theta)$ on the
parameters, and use Bayes’ Rule to express the posterior after observing some
training data:
<script type="math/tex; mode=display">
%<![CDATA[
P(\theta|x_{1:N},a_{1:N},r_{1:N}) \propto P(\theta) \prod_{i=1}^N P(r_i|x_i,a_i,\theta)
%]]>
</script>
We could use this to obtain a maximum a posteriori estimate of $P(r|x,a)$ by
integrating over the unobserved $\theta$, but that is not our goal. Instead,
we will use the simple but powerful idea of <a href="https://en.wikipedia.org/wiki/Thompson_sampling">Thompson sampling</a> to
solve our explore/exploit dilemma. This consists of two steps:
<ol>
<li>Sample a set of parameters according to the posterior after seeing the
training data: $\theta' \sim P(\theta|x_{1:N},a_{1:N},r_{1:N})$.</li>
<li>Choose the action that is optimal with respect to the sampled set of
parameters: $\hat{a} = \mathop{\arg\max}_a P(r|x,a,\theta')$.</li>
</ol>
For an intuitive understanding of Thompson sampling, consider what would happen
if we had very little training data. The posterior over parameters would be
broad, and the $\theta'$ samples in each iteration would have high variance.
Thus, there would be a lot of randomness in the chosen actions $\hat{a}$,
which is exactly the definition of exploration. As we collect more training
data, the posterior over parameters becomes more peaked, and both the samples
of $\theta'$ and the chosen actions $\hat{a}$ become more stable, which is
exactly the definition of exploitation. To summarize, the system starts out
with a lot of exploration, and automatically adjusts towards more exploitation
as it become more certain in its belief about the parameters.
Notice that in our introductory example, we talked about exploration at the ad
level: Ad 1 had a high conversion rate with high certainty, while Ad 2 had a
lower conversion rate but with a lot of uncertainty. Thompson sampling is a
little more subtle, because it performs exploration at the parameter level;
it will automatically explore to decrease uncertainty about the parameters in
the model. This is an advantage in our application, since we use the <a href="https://en.wikipedia.org/wiki/Feature_hashing">hashing
trick</a> to keep our parameter space fixed, while ads appear and
disappear all the time.
Now that we understand how Thompson sampling works, let’s see how to represent
and update the posterior over parameters in the case of logistic regression and
factorization machines.
<h2 id="common-notation">Common Notation</h2>
We have a training set of $N$ examples $(\mathbf{x}_i, y_i)$, where the
feature vector is a $D$-dimensional vector: $\mathbf{x}_i \in \mathbb{R}^D$
and the label is binary: $y_i \in \{0, 1\}$. We will use $i$ to index
examples whenever possible.
For a linear model (traditional logistic regression), the parameters $\theta$
are just a weight vector $\mathbf{w} \in \mathbb{R}^D$. We will use $j$ to
index into the weight vector whenever possible.
For a factorization machines (FM) model, the parameters $\theta$ are a vector
$\mathbf{w} \in \mathbb{R}^D$ as above, and also a matrix \(\mathbf{V} \in
\mathbb{R}^{D \times K}\), where $K$ is the number of factors. We will use
$j$ to index into $\mathbf{w}$ and $jk$ to index into $\mathbf{V}$
whenever possible (we omit the comma when we have multiple indices).
We define $\mu_i$ to be the probability of a positive label for example
$i$:
<script type="math/tex; mode=display">
%<![CDATA[
\mu_i ~=~ p(y_i=1 | \mathbf{x}_i) ~=~ \frac{1}{1 + \exp(-F(\mathbf{x}_i))}
%]]>
</script>
where $F$ is a kernel, or model equation. For the linear case, $F$ is
just a dot product of the weights with the features, and for factorization
machines, $F$ captures second-order interactions:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{align}
F_\text{linear}(\mathbf{x}_i) &= \sum_{j=1}^D{ w_j x_{ij} } \\
F_\text{FM}(\mathbf{x}_i) &=
\sum_{j=1}^D{ w_j x_{ij} } +
\sum_{a=1}^D{ \sum_{b=a+1}^D{ \langle \mathbf{V}_a, \mathbf{V}_b \rangle x_{ia} x_{ib} } }
\end{align}
%]]>
</script>
(For simplicity, we omit the bias term in each model.)
In both cases, the log-likelihood of the training set is given by:
<script type="math/tex; mode=display">
%<![CDATA[
\text{loglik}(\theta) = \sum_{i=1}^N{( y_i \ln \mu_i + (1-y_i) \ln (1-\mu_i) )}
%]]>
</script>
where the $\mu_i$ terms use either $F_\text{linear}$ or $F_\text{FM}$
depending on the model.
<h2 id="the-laplace-approximation">The Laplace Approximation</h2>
For a Bayesian treatment of logistic regression, we want to put a prior on the
parameters $\theta$ and compute the posterior given observed data. We choose
a simple prior where the parameters are independent, and each parameter is
drawn from a normal distribution with its own mean $m$ and precision $s$:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{align}
w_j &\sim \mathcal{N}(m_j, s_j^{-1}) \\
v_{jk} &\sim \mathcal{N}(m_{jk}, s_{jk}^{-1})
\end{align}
%]]>
</script>
Using Bayes’ Rule, the posterior is proportional to the prior and likelihood:
<script type="math/tex; mode=display">
%<![CDATA[
p(\mathbf{w} | \mathbf{x}_{1..N}, y_{1..N}) ~\propto~
p(\mathbf{w}) \cdot p(y_{1..N} | \mathbf{w}, \mathbf{x}_{1..N})
%]]>
</script>
Unfortunately, this posterior is not a normal distribution. But we can
approximate it with a normal distribution by using the Laplace approximation
(see section 4.4 in Bishop’s <a href="https://www.amazon.com/Pattern-Recognition-Learning-Information-Statistics/dp/0387310738">“Pattern Recognition and Machine
Learning”</a> textbook). That way, the approximated posterior will have
the same form as the prior, allowing us to iteratively update our belief about
the parameters.
The Laplace approximation is a technique for approximating an arbitrary
distribution $p(z) = c^{-1} f(z)$ (where $f$ is an arbitrary function and
$c$ is a normalization term) with a normal distribution \(q(z) =
\mathcal{N}(z_0, A^{-1})\) where $z_0$ is a mode of $f(z)$, and $A$ is
the negative second derivative of $\ln f(z)$, evaluated at $z_0$:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{align}
\frac{\mathrm{d} f(z)}{\mathrm{d} z} &\Bigr|_{z=z_0} = 0 \\
A = \frac{-\mathrm{d}^2 \ln f(z)}{\mathrm{d} z^2} &\Bigr|_{z=z_0}
\end{align}
%]]>
</script>
The first step is to find a mode of the posterior distribution. We can do this
by finding a maximum a posteriori (MAP) estimate of the parameters. This is
equivalent to maximizing the log-posterior:
<script type="math/tex; mode=display">
%<![CDATA[
\text{logpost}(\theta) = \ln p(\theta) + \text{loglik}(\theta)
%]]>
</script>
The second step is to find the $A$ term, which is the negative second
derivative of $\text{logpost}(\theta)$ with respect to each parameter. (We
keep things simple by forcing our parameters to be independent.)
Before making these calculations concrete for the linear and FM kernels, we
show a general result that will greatly simplify those calculations. Consider
the partial derivative of $\text{loglik}(\theta)$ with respect to some
variable $z$:
<script type="math/tex; mode=display">
%<![CDATA[
\frac{\partial}{\partial z} \text{loglik}(\theta) ~=~
\sum_{i=1}^N{
\left( \frac{y_i}{\mu_i} - \frac{1-y_i}{1-\mu_i} \right)
\frac{\partial \mu_i}{\partial F(\mathbf{x}_i)}
\frac{\partial F(\mathbf{x}_i)}{\partial z}
}
%]]>
</script>
Note that $\mu_i$ is a logistic function of $F(\mathbf{x}_i)$, with the
useful property that \(\partial \mu_i / \partial F(\mathbf{x}_i) = \mu_i (1 -
\mu_i)\). Therefore our derivative simplifies to:
<script type="math/tex; mode=display">
%<![CDATA[
\frac{\partial}{\partial z} \text{loglik}(\theta) ~=~
\sum_{i=1}^N{ (y_i - \mu_i) \frac{\partial F(\mathbf{x}_i)}{\partial z} }
%]]>
</script>
We now take the second derivative:
<script type="math/tex; mode=display">
%<![CDATA[
\frac{\partial^2}{\partial z^2} \text{loglik}(\theta) ~=~
\sum_{i=1}^N{\left(
- \mu_i (1 - \mu_i) \left(\frac{\partial F(\mathbf{x}_i)}{\partial z}\right)^2
+ (y_i - \mu_i) \frac{\partial^2 F(\mathbf{x}_i)}{\partial z^2}
\right)}
%]]>
</script>
To make this concrete for the linear and FM kernels, we only need to evaluate
the terms $\frac{\partial F(\mathbf{x}_i)}{\partial z}$ and
$\frac{\partial^2 F(\mathbf{x}_i)}{\partial z^2}$ for the relevant model
equation $F$ and parameter $z$, and plug them into the equation above.
<h2 id="bayesian-logistic-regression">Bayesian Logistic Regression</h2>
For logistic regression with the traditional linear kernel, the log-posterior
is:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{align}
\text{logpost}(\mathbf{w}) &~=~ \ln p(\mathbf{w}) + \text{loglik}(\mathbf{w}) \nonumber \\
&~=~
\text{const}
- \frac{1}{2} \sum_{j=1}^D{ s_j (w_j - m_j)^2 }
+ \sum_{i=1}^N{( y_i \ln \mu_i + (1-y_i) \ln (1-\mu_i) )}
\tag{1}
\end{align}
%]]>
</script>
With a bit of work, the second derivative of the log-posterior turns out to be:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{align}
\label{a-lr}
A_j = - \frac{\partial^2}{\partial w_j^2} \text{logpost}(\mathbf{w})
&~=~
- \frac{\partial^2}{\partial w_j^2} \ln p(\mathbf{w})
- \frac{\partial^2}{\partial w_j^2} \text{loglik}(\mathbf{w}) \nonumber \\
&~=~
s_j + \sum_{i=1}^N{ \mu_i (1 - \mu_i) x_{ij}^2 }
\tag{2}
\end{align}
%]]>
</script>
We now have everything we need to express the Bayesian Logistic Regression
algorithm. It maintains the following invariant: after processing $t$ batches,
the weights are distributed as $w_j \sim \mathcal{N}(m_j^{(t)}, 1/s_j^{(t)})$,
which is an approximation of the true posterior after all the data observed so
far. In the algorithm below, we update the means $m_j$ and precisions $s_j$ in
place, so we omit the superscripts $(t)$.
<hr />
Algorithm: Bayesian Logistic Regression
<ul>
<li>Initialize the prior on each weight $w_j$ with $m_j = 0$, $s_j = \lambda$.</li>
<li>For each new batch of training data $(\mathbf{x}_i, y_i)$ for $i = 1 .. N$:
<ul>
<li>Find $\hat{\mathbf{w}}$ maximizing equation (1) by numerical optimization.</li>
<li>Compute $A_j$ for each weight according to equation (2).</li>
<li>Update the weight distribution: $m_j \gets \hat{w}_j$ and $s_j \gets A_j$.</li>
</ul>
</li>
</ul>
<hr />
Our priors have a natural interpretation. By starting with the prior
$w_j \sim \mathcal{N}(0, \lambda^{-1})$, our objective function in equation
(1) is exactly the same as for (non-Bayesian) Logistic Regression with an L2
regularizer $\lambda$.
The algorithm above matches algorithm 3 derived by Chapelle and Li,
confirming that we have derived the Laplace approximation correctly. The only
difference is one of notation: our labels are in $\{0, 1\}$ while their labels
are in $\{-1, 1\}$, so their objective function has a slightly different form.
<h2 id="bayesian-factorization-machines">Bayesian Factorization Machines</h2>
For logistic regression with the FM kernel, the log-posterior is:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{align}
\text{logpost}(\mathbf{w}, \mathbf{V}) &~=~ \ln p(\mathbf{w}, \mathbf{V}) + \text{loglik}(\mathbf{w}, \mathbf{V}) \nonumber \\
&~=~
\text{const}
- \frac{1}{2} \sum_{j=1}^D{ s_j (w_j - m_j)^2 }
- \frac{1}{2} \sum_{j=1}^D \sum_{k=1}^K{ s_{jk} (v_{jk} - m_{jk})^2 } \nonumber \\
&\phantom{~=~ const}
+ \sum_{i=1}^N{( y_i \ln \mu_i + (1-y_i) \ln (1-\mu_i) )}
\tag{3}
\end{align}
%]]>
</script>
Since our parameters are the weights $w_j$ and the $\mathbf{V}$ matrix
entries $v_{jk}$, we need second derivatives with respect to both of these in
order to use the Laplace approximation.
The second derivative of the log-posterior w.r.t. $w_j$ has the same form as
for the linear kernel, except that the $\mu_i$ terms now use the FM kernel:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{align}
A_j = - \frac{\partial^2}{\partial w_j^2} \text{logpost}(\mathbf{w}, \mathbf{V})
&~=~
- \frac{\partial^2}{\partial w_j^2} \ln p(\mathbf{w}, \mathbf{V})
- \frac{\partial^2}{\partial w_j^2} \text{loglik}(\mathbf{w}, \mathbf{V}) \nonumber \\
&~=~
s_j + \sum_{i=1}^N{ \mu_i (1 - \mu_i) x_{ij}^2 }
\tag{4}
\end{align}
%]]>
</script>
The second derivative of the log-posterior w.r.t. $v_{jk}$ turns out to be a
bit more complicated, but still easily computed with a single additional pass
through the data:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{align}
A_{jk} = - \frac{\partial^2}{\partial v_{jk}^2} \text{logpost}(\mathbf{w}, \mathbf{V})
&~=~
- \frac{\partial^2}{\partial v_{jk}^2} \ln p(\mathbf{w}, \mathbf{V})
- \frac{\partial^2}{\partial v_{jk}^2} \text{loglik}(\mathbf{w}, \mathbf{V}) \nonumber \\
&~=~
s_{jk} + \sum_{i=1}^N{ \mu_i (1 - \mu_i) \left(
\frac{\partial F_\text{FM}(\mathbf{x}_i)}{\partial v_{jk}} \right)^2 }
\tag{5}
\end{align}
%]]>
</script>
where
<script type="math/tex; mode=display">
%<![CDATA[
\begin{equation}
\frac{\partial}{\partial v_{jk}} F_\text{FM}(\mathbf{x}_i) ~=~
\left( \sum_{j'=1}^D{ v_{j'k} x_{ij'} } \right) x_{ij} - v_{jk} x_{ij}^2
\end{equation}
%]]>
</script>
We show the Bayesian Factorization Machines algorithm below. It maintains the
following invariant: after processing $t$ batches, the weights are
distributed as $w_j \sim \mathcal{N}(m_j^{(t)}, 1/s_j^{(t)})$ and the
elements of $\mathbf{V}$ are distributed as \(v_{jk} \sim
\mathcal{N}(m_{jk}^{(t)}, 1/s_{jk}^{(t)})\), which is an approximation of the
true posterior after all the data observed so far. In the algorithm below, we
update the means $m_j$, $m_{jk}$ and precisions $s_j$, $s_{jk}$ in
place, so we omit the superscripts $(t)$.
<hr />
Algorithm: Bayesian Factorization Machines
<ul>
<li>Initialize the prior on each weight $w_j$ with $m_j = 0$, $s_j = \lambda_\mathbf{w}$.</li>
<li>Initialize the prior on each $\mathbf{V}$ element $v_{jk}$ with $m_{jk} = 0$, $s_{jk} = \lambda_\mathbf{V}$.</li>
<li>For each new batch of training data $(\mathbf{x}_i, y_i)$ for $i = 1 .. N$:
<ul>
<li>Find $\hat{\mathbf{w}}$, $\hat{\mathbf{V}}$ maximizing equation (3) by numerical optimization.</li>
<li>Compute $A_j$ for each weight according to equation (4).</li>
<li>Compute $A_{jk}$ for each element of $\mathbf{V}$ according to equation (5).</li>
<li>Update the weight distribution: $m_j \gets \hat{w}_j$ and $s_j \gets A_j$.</li>
<li>Update the $\mathbf{V}$ matrix distribution: $m_{jk} \gets \hat{V}_{jk}$ and $s_{jk} \gets A_{jk}$.</li>
</ul>
</li>
</ul>
<hr />
As for the linear kernel, our priors have a natural interpretation. By
starting with the priors $w_j \sim \mathcal{N}(0, \lambda_\mathbf{w}^{-1})$
and $v_{jk} \sim \mathcal{N}(0, \lambda_\mathbf{V}^{-1})$, our objective
function in equation (3) is exactly the same as for (non-Bayesian)
Factorization Machines with L2 regularizers $\lambda_\mathbf{w}$ on
$\mathbf{w}$ and $\lambda_\mathbf{V}$ on $\mathbf{V}$.
<h2 id="final-words">Final Words</h2>
We have implemented Thompson sampling and Bayesian Factorization Machines, and
we are testing these methods to efficiently navigate between exploration and
exploitation. We can fine-tune the desired amount of exploration by scaling the
linear precisions $s_j$ and FM precisions $s_{jk}$ by two respective
constants. This might be useful, for example, if we change how much training
data we use or how many free parameters are in our model, and we want to
preserve a certain amount of exploration. Ultimately, these techniques help us
ensure that we are using our customers’ budgets in the most efficient way
possible.
I’d like to return to our motivating example and show how Thompson sampling
works in such a scenario. We evaluate two candidate ads, showing their
non-Thompson conversion probability in red, and a histogram of 10,000
Thompson-sampled conversion probabilities in blue.
<img src="/images/post_images/thompson-query-1.png" alt="hll-perf" />
<img src="/images/post_images/thompson-query-2.png" alt="hll-perf" />
The predicted conversion rate is 0.120 for the first ad and 0.080 for the
second ad. So a system with no exploration would always pick the first ad.
But the Thompson samples show that there is a lot more uncertainty about the
conversion rate of the second ad. (The standard deviation is 0.045 for the
first ad and 0.093 for the second ad.) This means that Thompson sampling will
pick the second ad some fraction of the time, thus performing exploration. Once
the variances shrink enough that the choice of best ad is unambiguous, Thompson
sampling will automatically start exploiting that ad.
This has been a really fun project, and the combination of math and engineering
involved is a good illustration of the type of work that we do on the Data
Science Engineering team. If this piques your interest, check out our <a href="https://www.adroll.com/about/careers">open
positions</a>!
</description>
</item>
<item>
<title>Querying Data in Amazon S3 Directly with User-Space Page Fault Handling</title>
<link>https://tech.nextroll.com/blog/data/2016/11/29/traildb-mmap-s3.html</link>
<pubDate>Tue, 29 Nov 2016 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data/2016/11/29/traildb-mmap-s3</guid>
<description>tl;dr AdRoll’s real-time bidding system produces tens of billions
events daily. We store these events in <a href="http://traildb.io">TrailDBs</a>,
a fast embedded database that <a href="http://tech.adroll.com/blog/data/2016/05/24/traildb-open-sourced.html">we recently
open-sourced</a>.
By leveraging <a href="https://lwn.net/Articles/636226/">a cool new feature in the Linux kernel</a>,
you can now query TrailDBs directly in <a href="https://aws.amazon.com/s3/">Amazon S3</a> without
having to store the database locally, like here:
<script type="text/javascript" src="https://asciinema.org/a/2v7egtu98dxx60224iqqcfhx6.js" id="asciicast-2v7egtu98dxx60224iqqcfhx6" async="" data-font-size="80%"></script>
<h2 id="tens-of-trillions-of-events-in-amazon-s3">Tens of Trillions of Events in Amazon S3</h2>
We have been very happy with <a href="http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html">our data processing
pipeline</a>,
built on top of Docker, Luigi, Spot Instances, and Amazon
S3. It allows us to process hundreds of terabytes of data stored in TrailDBs
cost-efficiently with low operational overhead.
A typical job at AdRoll downloads a set of TrailDB shards from S3,
processes them, and pushes results back to S3. This model works fine
for batch jobs that need to query a large amount of data, similar to a
full table scan in RDBMS. Over time, our TrailDB shards have grown in
size (more rows) and they include a wider variety of information (more
columns). Consequently, an increasing number of queries need to access
only a subset of rows and columns.
Having to download gigabytes of unneeded data to find the proverbial
needle in a haystack, expressed as a highly selective query, is quite
expensive. A more efficient solution would download only the specific
bytes that matter for the query. Traditional data warehouses address
this issue by carefully managing both storage and query optimization.
In contrast, our TrailDB-on-AWS architecture unbundles storage from
querying for ease of scalability and operation, similar to many other
big data systems.
However, we can still leverage a traditional database technique to solve
the problem, namely <a href="http://stackoverflow.com/questions/4401910/mysql-what-is-a-page">page management</a>.
Instead of downloading a full shard, we can fetch only the pages that are
required to execute the query.
<h2 id="user-space-page-fault-handling">User-Space Page Fault Handling</h2>
Cache and page managers are <a href="https://madusudanan.com/blog/understanding-postgres-caching-in-depth">very non-trivial subsystems of modern
databases</a>.
The topic is an inherently complex one but it is
further complicated by <a href="https://lwn.net/Articles/580542/">the complex interplay between the
kernel and the database</a>.
After all, the kernel has <a href="https://www.kernel.org/doc/gorman/pdf/understand.pdf">a very non-trivial subsystem of its
own</a> for page
management that doesn’t always work in perfect harmony
with its database counterpart.
It is a valid question whether the database even needs a virtual memory
system of its own, instead of just letting <a href="http://varnish-cache.org/docs/trunk/phk/notes.html">the kernel take care
of the job</a>.
In practice, it is not easy to make the kernel-managed
paging perform as well as a custom subsystem given how
much <a href="http://oldblog.antirez.com/post/what-is-wrong-with-2006-programming.html">the database knows about the data and its access
patterns</a>
in contrast to the kernel.
Enter a very promising new feature in the Linux kernel, <a href="https://lwn.net/Articles/636226/">user-space page
fault handling</a>. This feature allows
a user-space application, such as the TrailDB library, to handle page
faults, that is, a certain piece of data being missing from the process’
address space, as it best sees fit. In particular, this feature allows
us to fetch page-size (4KB) chunks of data from S3 with practically no
impact to the existing codebase.
While user-space page fault handling might not be flexible enough
yet to replace Postgres’ venerable page management subsystem, it
allowed us to implement this key piece of functionality in TrailDB
in less than 2000 lines of code while working harmoniously
with the kernel. As demonstrated by this case, it is extremely
powerful to be able to embed <a href="https://en.wikipedia.org/wiki/Separation_of_mechanism_and_policy">application-specific policies in kernel
mechanisms</a>
that take care of heavy lifting such as page fault handling, and, in other
cases, <a href="https://github.com/libfuse/libfuse">file systems</a>,
and <a href="http://lwn.net/Articles/603983">packet processing</a>.
The API for user-space page fault handling is surprisingly
straightforward. If you are curious, you can try
<code class="language-plaintext highlighter-rouge">userfaultfd()</code> yourself by following this <a href="http://noahdesu.github.io/2016/10/10/userfaultfd-hello-world.html">hello world
tutorial</a>.
<h2 id="traildb-external-data-architecture">TrailDB External Data Architecture</h2>
The solution implemented in TrailDB is fully generic and allows data to
be fetched from any source, not just Amazon S3. The block servers that
handle interfacing with the data sources are pluggable and implemented
outside the TrailDB library, so they can be conveniently implemented in
any programming language.
Interfacing with S3 is implemented by a block server written in Go,
<a href="https://github.com/tuulos/traildb-s3-server">traildb-s3-server</a>. In the
diagram below, this is the “server process.” The TrailDB application,
“client process,” communicates with the server using simple requests sent
over TCP.
The purple box is an unmodified application using the TrailDB library.
The blue boxes are responsible for paging and caching. The yellow box is
an external data source, in our case Amazon S3.
Let’s walk through a page fault situation, that is, what happens when TrailDB
tries to access a piece of external data like <code class="language-plaintext highlighter-rouge">s3://traildb.io/data/wikipedia-small.tdb</code>
that is managed by the user-space page fault handler:
<img src="/images/post_images/s3mmap-arch.png" style="display: block; width: 100%; margin-left: auto; margin-right: auto" />
<h4 id="1-page-fault-generated">1. Page fault generated</h4>
If <code class="language-plaintext highlighter-rouge">tdb_open()</code> is called with a non-local URL, e.g. <code class="language-plaintext highlighter-rouge">s3://</code>, memory
regions that are normally memory-mapped with a file are mapped as
<code class="language-plaintext highlighter-rouge">MAP_ANONYMOUS</code> and the region is registered to the user-space page
fault handler.
When the region is accessed by TrailDB as usual, a page fault is
generated which blocks the app. Instead of the page fault being handled
by the kernel, it sends a message to the <code class="language-plaintext highlighter-rouge">userfaultfd</code> file descriptor
indicating what address was requested.
source <a href="https://github.com/tuulos/traildb/blob/userfault/src/tdb_external.c">tdb_external.c</a>
<h4 id="2-user-space-page-fault-handler-triggered">2. User-space page fault handler triggered</h4>
When <code class="language-plaintext highlighter-rouge">tdb_open()</code> was called with a non-local URL, also a separate thread
was launched to handle page faults. The thread uses <code class="language-plaintext highlighter-rouge">poll()</code> to wait for
incoming messages in the <code class="language-plaintext highlighter-rouge">userfaultfd</code> file descriptor.
When a message is received, the virtual address requested is translated
to an offset in the TrailDB file. Next, the handler checks if the offset
corresponds to the latest block received from the server. If it does,
we already have the required data which can be copied back to the app.
source <a href="https://github.com/tuulos/traildb/blob/userfault/src/tdb_external_pagefault.c">tdb_external_pagefault.c</a>
If the offset doesn’t resolve to the latest block, we must request a
new block from the block server. A simple message is prepared which
contains the URL of the TrailDB, an offset, and the minimum number of
bytes requested, typically 4KB. This message is then sent over a TCP
connection to the server.
source <a href="https://github.com/tuulos/traildb/blob/userfault/src/tdb_external_comm.c">tdb_external_comm.c</a>
<h4 id="3-block-server-handles-the-request">3. Block server handles the request</h4>
The block server is expected to return bytes at the requested offset.
For efficiency reasons, our
<a href="https://github.com/tuulos/traildb-s3-server">traildb-s3-server</a> caches
blocks received from S3 locally. When a request is received, first we
need to check if the block exists in the local cache, possibly as a
sub-range of a previous cached larger block.
source <a href="https://github.com/tuulos/traildb-s3-server/blob/master/blockcache.go">blockcache.go</a>.
If the block is found, the server sends a response that includes a
local path and an offset where the client can find the requested block.
Currently we assume that both the client and the server share a common
filesystem.
source <a href="https://github.com/tuulos/traildb-s3-server/blob/master/s3tdb.go">s3tdb.go</a>.
<h4 id="4-block-server-fetches-a-block-from-s3">4. Block server fetches a block from S3</h4>
If the block is not found, we must fetch it from its original source,
in this case from Amazon S3. This is accomplished using the standard
<a href="https://aws.amazon.com/sdk-for-go/">AWS SDK for Go</a>, which, besides
downloading data, handles authentication transparently, in our case
using an instance-specific IAM role.
S3 supports standard HTTP range requests, which allows us to request
an exact range of bytes from S3. Since there is quite a high constant
overhead and a monetary price associated to each GET request in S3, it
makes sense to request a larger block instead of the single 4KB
page that was originally requested by the client. You can read more
about the effect of the block size below.
After a successful download, the block is added to the cache, and a response
is sent to the client as in Step 3.
source <a href="https://github.com/tuulos/traildb-s3-server/blob/master/s3tdb.go">s3tdb.go</a>.
<h4 id="5-the-requested-block-is-memory-mapped-by-the-client">5. The requested block is memory-mapped by the client</h4>
The page fault handler receives a path and an offset to the requested
block from the server. The block is memory mapped by the handler.
source <a href="https://github.com/tuulos/traildb/blob/userfault/src/tdb_external_pagefault.c">tdb_external_pagefault.c</a>
<h4 id="6-the-requested-page-is-delivered">6. The requested page is delivered</h4>
The page fault handler thread uses <code class="language-plaintext highlighter-rouge">UDDFIO_COPY</code> to copy the requested
page back to the app’s address space atomically. Once this call
finishes, the app thread unblocks and it proceeds to process data as
usual.
source <a href="https://github.com/tuulos/traildb/blob/userfault/src/tdb_external_pagefault.c">tdb_external_pagefault.c</a>
<h2 id="performance">Performance</h2>
As described above, handling a single page fault may require up to six
steps which involve a separate thread, a TCP connection, a server,
and an HTTPS request to S3. Is this going to be too slow for realistic
workloads?
Below we have benchmarked the S3-direct solution versus the baseline approach
that involves downloading the full TrailDB file first to a local disk. The
test data is <a href="http://traildb.io/data/wikipedia-history-small.tdb">a 100MB snapshot of Wikipedia edit history</a>, as seen in the <a href="http://traildb.io/docs/tutorial/">TrailDB tutorial</a>.
This TrailDB contains 6.3M trails. The benchmarks were performed on an <code class="language-plaintext highlighter-rouge">r3.8xlarge</code>
EC2 instance in the same region where the data is located in S3.
<h4 id="sequential-access">Sequential Access</h4>
First, consider a simple case where you want to access the first X trails
in the TrailDB. In the chart below the green line shows the constant
cost of the baseline: it takes about 1.6 seconds to download the full
TrailDB. After downloading it, there is practically no cost for accessing
the first 0-2000 trails.
<img src="/images/post_images/s3mmap-sequential.png" style="display: block; width: 100%; margin-left: auto; margin-right: auto" />
The orange line shows the cost for performing the same operation using
S3 directly with a 16MB block size. Accessing the first 0-2000 trails
involves downloading a single 16MB block, which takes about 500ms.
The blue line shows the cost for using S3 directly with a 16KB block
size. The small block size yields the lowest latency to the first
results: You can get results for the first 0-10 trails in about 100ms,
which is a 16x speedup compared to the baseline! However, the more
trails are processed the more small blocks need to be fetched. The
overhead adds up so that at about 200 trails it is cheaper to use
the baseline approach instead.
<h4 id="random-access">Random Access</h4>
It is expected that sequential access favors large block sizes. However,
the original needle in a haystack question is more about random access.
Like above, the baseline approach first downloads the full TrailDB and
then access X trails randomly. Again, the cost to download dominates the
total cost since a small 100MB TrailDB fits in memory easily.
<img src="/images/post_images/s3mmap-random.png" style="display: block; width: 100%; margin-left: auto; margin-right: auto" />
The break-even point is about 20-30 trails. If you need to access
fewer trails than this, it is faster to use S3 directly. Note that the
break-even point heavily depends on the size of the TrailDB. With many
of our production TrailDBs that can exceed a terabyte in size, the
break-even point is much higher.
Another thing to note above is how close the orange 16MB line is to the
baseline. The overhead of S3 direct is about 10-15%.
<h4 id="overhead-of-user-space-page-fault-handling">Overhead of User-Space Page Fault Handling</h4>
If we extended the X axis above to cover all the 6.3M trails, all blocks
of the test TrailDB would end up being cached locally. After this, there
is no need to download anything from S3 and the difference between the
S3 direct approach and the baseline should equal to just the overhead of
user-space page fault handling.
As mentioned in Step 2 above, if the latest block received contains
the data we need, we can skip the following Steps 2-5 which involve
a context switch to another process and sending a message over TCP,
amongst other things. Hence, the fewer blocks we need to request, the
less overhead there is.
We evaluate this overhead of the fully cached case as a function of the
block size versus the blue baseline. The operation performed is a full scan over
all the trails:
<img src="/images/post_images/s3mmap-overhead.png" style="display: block; width: 100%; margin-left: auto; margin-right: auto" />
With a small block size, 4-16KB, practically every page fault involves
all the steps, except S3 download in this case. With a larger block
size, the overhead reduces to 10-15% what we saw above. Interestingly,
when the block size approaches the size of the full DB, S3 direct
becomes faster than the baseline. This is caused by the extra overhead
of having to launch a separate process to download the data in the
baseline case.
Overall, user-space page fault handling has a surprisingly low overhead
compared to page fault handling in the kernel. The 10-15% overhead shown
above is caused by our particular implementation, not <code class="language-plaintext highlighter-rouge">userfaultfd()</code>
per se. This could be optimized further if needed.
<h4 id="s3-bandwidth">S3 Bandwidth</h4>
Of course, fetching data from S3 is by far the most expensive part
of the process. We want to evaluate if our way of downloading data from
S3 performs well.
We leverage the standard <a href="https://aws.amazon.com/sdk-for-go/">AWS SDK for
Go</a> for downloading,
which should be reasonably performant. It is widely known
that <a href="http://blog.zachbjornson.com/2015/12/29/cloud-storage-perfo
rmance.html">aggregate S3 bandwidth increases with more concurrent
downloaders</a>. The current implementation treats each <code class="language-plaintext highlighter-rouge">tdb</code> handle as
a separate thread both on the client and the server side. Hence, you
should be able to increase bandwidth by either opening multiple <code class="language-plaintext highlighter-rouge">tdb</code>
handles to a single TrailDB and sharding it by row, or by handling
multiple TrailDBs in parallel.
The benchmark supports this hypothesis:
<img src="/images/post_images/s3mmap-bandwidth.png" style="display: block; width: 100%; margin-left: auto; margin-right: auto" />
You can nearly saturate the S3 bandwidth on a single instance with
about 20 parallel handles. Based on our previous findings, the maximum
bandwidth achievable between a single EC2 instance and S3 is about
4Gbit/sec. In the above case, we use a single SSD as a backing store so
the 3.5Gbit/sec peak may be bottlenecked by the SATA interface. In other
words, using S3 directly can provide you the same bandwidth as a local
SSD.
<h2 id="conclusion">Conclusion</h2>
The benchmarks show that using S3 directly with user-space page fault
handling is an excellent solution to our original problem with highly
selective queries. We were positively surprised that the overhead
of this solution is low even for the full sequential
scan case, which suggests that all workloads could benefit from this
solution.
Finding a needle in a haystack is a perfect exemplar for using S3
directly. It turns out this approach has a number of other benefits as
well:
<ul>
<li>Infinitely lower time to first results, depending on the TrailDB size.
This is especially useful for ad hoc queries, which can be interrupted
if the first results are not satisfactory.</li>
<li>Makes it possible to process TrailDBs that are larger than local disk
space. This is a potential killer feature, since this allows you to
use instance types with little or no local disk space which in turn
means more opportunities for cost optimization.</li>
<li>The separate block server further clarifies the separation between
the storage and the query layer, opening up interesting new use cases.
The block server can be developed quickly outside of the core
TrailDB codebase.</li>
</ul>
Future work includes dynamic block sizes, smarter prefetching and eviction
policies for the block server, and support for new data sources besides
Amazon S3. Contributions are welcome!
<h4 id="give-it-a-try">Give it a try!</h4>
Although this work is not merged to the TrailDB master branch as of writing
of this blog article, you should be able to give it a try easily:
<ol>
<li>
Make sure that your kernel supports <code class="language-plaintext highlighter-rouge">userfaultfd</code>. You need Linux kernel
4.3 or newer with <code class="language-plaintext highlighter-rouge">USERFAULTFD</code> enabled. With many distributions, you can
check this by running <code class="language-plaintext highlighter-rouge">grep USERFAULTFD /boot/config*</code>
which should return <code class="language-plaintext highlighter-rouge">CONFIG_USERFAULTFD=y</code> if the feature is enabled in
your kernel. For instance, <a href="https://cloud-images.ubuntu.com/locator/ec2/">Ubuntu Yakkety Yak (16.10)</a>
works out of the box.
</li>
<li>
Git clone the <code class="language-plaintext highlighter-rouge">userfault</code> branch of TrailDB:
<code class="language-plaintext highlighter-rouge">git clone -b userfault https://github.com/tuulos/traildb</code>
and follow the <a href="http://traildb.io/docs/getting_started/#install-on-linux">installation instructions</a>.
</li>
<li>
Git clone the <code class="language-plaintext highlighter-rouge">traildb-s3-serve</code>:
<code class="language-plaintext highlighter-rouge">git clone https://github.com/tuulos/traildb-s3-server</code>
and run <code class="language-plaintext highlighter-rouge">go build</code> as usual.
</li>
</ol>
After this, you can start <code class="language-plaintext highlighter-rouge">./traildb-s3-server</code> and use e.g. the <code class="language-plaintext highlighter-rouge">tdb</code> command line
tool to access data directly from S3:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tdb dump -i s3://my-bucket/my.tdb
</code></pre></div></div>
If you have any trouble with installation or you have any other questions or feedback,
please join us at <a href="https://gitter.im/traildb/traildb">our TrailDB Gitter channel</a>.
</description>
</item>
<item>
<title>How to Create a Style Guide: Start with a UI Framework</title>
<link>https://tech.nextroll.com/blog/product/2016/07/29/how-to-create-a-style-guide.html</link>
<pubDate>Fri, 29 Jul 2016 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/product/2016/07/29/how-to-create-a-style-guide</guid>
<description>Originally published on Medium on <a href="https://medium.com/adroll-design/how-to-create-a-style-guide-start-with-a-ui-framework-2d7ea456fa33">July 13, 2016</a>.
Update May 16th, 2017: Our style guide is now published at <a href="http://ux.adroll.com">ux.adroll.com</a>. You can read more about the latest updates <a href="https://medium.com/adroll-design/the-journey-to-design-consistency-ee26bef2fd88">on our UX blog</a>.
<h2 id="qa-with-adrolls-ux-designer-on-why-we-did-it-and-what-we-learned">Q&amp;A with AdRoll’s UX Designer on why we did it and what we learned.</h2>
<blockquote>
Hi! I’m Arya Srinivasan, a UX Researcher at AdRoll. I sat down with Mason Lee, a UX Designer working on AdRoll’s native ads API product, to talk about his work developing AdRoll’s style guide.
</blockquote>
<h4 id="to-kick-things-off-why-did-you-start-the-style-guide-project-what-was-the-problem-that-needed-solving">To kick things off, why did you start the style guide project? What was the problem that needed solving?</h4>
Mason: The problem was design inconsistency, both across products and within a single product. For example, a button that should look the same everywhere but actually varies in color, font weight, and border style.
<center>
<img alt="Messy Buttons" src="/images/post_images/ux-blog-post-old-buttons.png" />

Buttons look different in individual designers’ mocks and our applications.

</center>
AdRoll’s rapid growth meant that we were focused on speed. We’re now a larger company with multiple products, so as a designer I believe it’s important for us to emphasize consistency in how we present our products: through the design.
To focus on design, we first needed to fix existing inconsistencies. UX designers here typically focus on one or two products, so in order to get our team to think about the design across all products, I set up a weekly “UI Smackdown” meeting to discuss UI guidelines.
In each meeting, we looked at design inconsistencies to decide on a single design. After a few meetings, designers still asked me about the correct color or padding, etc. We needed a central document with all of the answers, so I built our UI Framework in Sketch as a resource for designers. Whenever we realize there’s a missing component or want to include a new component, we discuss it and add it to the master UI Framework file.
<h4 id="you-mentioned-you-want-adroll-to-be-a-design-centric-companywhat-do-you-mean-by-that">You mentioned you want AdRoll to be a design centric company — what do you mean by that?</h4>
Mason: I want AdRoll to put design at the forefront so that it’s a competitive differentiator — recognized by customers as a well-designed product that really solves their needs.
<h4 id="what-were-your-immediate-goals-for-the-style-guide-do-you-have-a-longer-term-vision-for-this-project">What were your immediate goals for the style guide? Do you have a longer term vision for this project?</h4>
Mason: My short-term goal is to have design consistency between designers by standardizing our UI components. I want designers to speak the same language when talking design. For example, in a modal or dropdown, engineers build based on how the designer suggests. If the design elements are different between designers, engineers are going to make the same element different ways.
My mid-term goal is to have this style defined in our code in “RollUp,” AdRoll’s internal UI component library. If we have a predefined style sheet, all our engineers need to do is copy it. Designers and engineers can speak the same language.
<h4 id="did-you-run-into-any-problems-while-creating-the-style-guide-how-did-you-solve-them">Did you run into any problems while creating the style guide? How did you solve them?</h4>
Mason: One of the biggest hurdles was getting the buy-in from people across product teams. To get everyone involved, I set up a meeting with a clear list of agenda items to cover. I presented design inconsistencies, such as varying dropdown menus between products. Providing visual evidence triggers conversations, and in the end, people care about their product and want consistency.
Another challenge was deciding on the rules. When talking about standardizing a component, it should be applicable anywhere, in every context. You have to think of every edge case. The component has to be flexible, but at the same time feature-complete enough so it’s easily usable, consumable, and applicable.
<center>
<img alt="Geotargeting UI" src="/images/post_images/geo-gif.gif" />

Here’s an example of our style guide’s flexibility. Our initial decision for the padding in this geotargeting dropdown was too big, so we revised the style guide to account for this use case.

</center>
<center>
<img alt="Geotargeting UI Before and After" src="/images/post_images/geo-ui-before-after.png" />

Before (left), After (right)

</center>
I actually want to call out one more challenge! Naming can be difficult. As I said before, I want designers and engineers to speak the same language, but this needs to be done carefully. For something as simple as a dropdown, we actually have several variations (one with checkboxes, another with checkboxes and a text block, and another is a standard dropdown menu). How do we name three different dropdowns so there’s a universal understanding of which is which?
<center>
<img alt="AdRoll Dropdown Variations" src="/images/post_images/dropdowns.png" />
</center>
The semantics are challenging; our naming needs to makes sense. We used some cool collaboration tools to gain a consensus when we’re deciding on a name. For example, <a href="https://wake.com/">Wake</a> helped us collect all open questions and issues, discuss solutions, monitor the UI Smackdown decisions, and continue the conversation with the larger product team through an integration with Slack.
<center>
<img alt="Wake and Slack" src="/images/post_images/wake-slack.png" />

How we used Wake to discuss UI inconsistencies and collaborate on component rules.

</center>
<h4 id="is-there-anything-unique-about-adrolls-ui-that-you-had-to-consider-when-creating-the-style-guide">Is there anything unique about AdRoll’s UI, that you had to consider when creating the style guide?</h4>
Mason: Our dashboard is very data-heavy. In addition, the campaign creation flow gives advertisers a bunch of levers to pull. In order to meet the needs of less-experienced advertisers, we aim to have effective default settings. In our products, the components have complex functions but look simple and are easy to use.
<h4 id="are-there-some-things-you-wish-you-knew-when-you-started-creating-the-style-guide">Are there some things you wish you knew when you started creating the style guide?</h4>
Mason: I wish I had a deeper understanding of how all our products work from the start. For instance, we share how our respective product’s work in our weekly design critique meeting, so I know how SendRoll (our email retargeting product) works on the surface, but I don’t have the deep knowledge of SendRoll that it’s designer does. I think that nuanced understanding of the product definitely helps when working on the style guide, because I then have a better understanding of all of the potential use cases.
<h4 id="so-whats-the-best-way-to-achieve-that-common-understanding-of-a-designers-process-and-their-product">So what’s the best way to achieve that common understanding of a designer’s process and their product?</h4>
Mason: Even though we’re really focused on shipping our products, we do a good job of sharing our design process in our weekly design critique meeting. I think we can be better about closing the loop after each meeting — how did the designer incorporate the feedback from the meeting? Once the product is shipped and used by our advertisers, we could also share how advertisers are using the products based on the insights from analytics.
<h4 id="were-there-any-resources-you-referred-to-while-working-on-this-project">Were there any resources you referred to while working on this project?</h4>
Mason: I read Atomic Design by Brad Frost, researched online, and talked to other UX designers at MeetUps. If you think that a particular company practices good design, then it’s pretty likely they have talked about their style guide somewhere online.
<h4 id="whats-the-status-of-our-style-guide">What’s the status of our style guide?</h4>
Mason: I’ve captured and revisited all the UI elements that we use in our different products and grouped them into foundations, components, patterns, and pages, which will serve as a source of truth for our UI designs.
You can check out the foundation and component UI elements on <a href="https://dribbble.com/shots/2833155-AdRoll-UI-Framework">Dribbble</a>. If you’re familiar with Atomic Design, I grouped the “atom” and “molecule” levels into what I call “components.” For example, combining the form title and the input makes it easy for other designers to easily copy a completed form field.
<h3 id="thanks-for-reading">Thanks for reading!</h3>
Look out for these upcoming topics as we develop our style guide:
<ul>
<li>How a UI Framework simplifies collaboration</li>
<li>Developing new stylesheets based on the UI Framework</li>
<li>How to build a Style Guide website</li>
<li>The journey to finding our Voice and Tone</li>
</ul>
</description>
</item>
<item>
<title>Netflix and The Art of Product Evolution</title>
<link>https://tech.nextroll.com/blog/product/2016/07/05/netflix-art-product-evolution.html</link>
<pubDate>Tue, 05 Jul 2016 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/product/2016/07/05/netflix-art-product-evolution</guid>
<description>There is a <a href="http://www.nytimes.com/2016/06/19/magazine/can-netflix-survive-in-the-new-world-it-created.html?_r=0">terrific article</a> in last week’s Sunday Times outlining Netflix’s role in changing the way we consume media. The central question of the piece is whether Netflix will dominate the world it has created. Nimble competitors like Amazon, Hulu, and the major studios (once threatened by streaming) are each leveraging unique assets to disrupt legacy media models.
From the article, we at AdRoll gleaned a broader framework for how products can evolve gracefully. Much as Netflix evolved from sending DVDs in the mail to streaming syndicated shows to producing original content, AdRoll is poised to transition from retargeting ads to bigger marketing solutions. With that leap comes a fair amount of risk. What next-generation products should we choose? How do we keep our most passionate customers engaged? How much do we continue to invest in the core? Here are five learnings from the Netflix story:
<ol>
<li>
Innovate incrementally. The Netflix of 1998—sending DVDs in the mail—was not in a position to start negotiating content deals. From a technical standpoint, Netflix lacked the major distribution networks. From a branding standpoint, Netflix was still analogous to Blockbuster. Streaming was the intermediate step that allowed Netflix to transform their brand and build the requisite technology infrastructure to take on the studios.
In a similar manner, AdRoll is a trusted name in retargeting. Building on our customer needs and their data assets led us to <a href="https://www.adroll.com/product/prospecting">Prospecting</a> first and <a href="https://www.adroll.com/product/sendroll">SendRoll</a> second. Billboards or TV commercials shouldn’t be next; that’s too far a leap. But web page optimization, CRM, mobile app retargeting, and marketing vendor management are not so far-fetched. With products that align to these areas, AdRoll can leverage our strengths while building new assets, such as a <a href="https://blog.adroll.com/product/adrolls-cross-device-functionality-unifies-fragmented-customer-journeys">cross-device graph</a>. We can evolve our brand beyond display advertising. Most important, we can take our customers along with us slowly into areas that make intuitive sense for them.
</li>
<li>
Understand your customers. Investing $100M in House of Cards was a data-based decision for Netflix. The show aligned nicely to the interests of Netflix viewers—political dramas, Kevin Spacey, and David Fincher’s directorial work. It was a calculated leap.
Analogously, AdRoll customers are keenly interested in attribution and ROI, optimizing their web pages, and managing their marketing vendors better. Our product portfolio iteration should adhere to these themes. We don’t need to build advanced functionality to gauge customer feedback. An MVP offering is enough to learn from our users what resonates most.
</li>
<li>
Reallocate resources asymmetrically. Here’s the tricky thing about those red envelopes in the mail (yes, Netflix still offers them): the once-core DVD business for Netflix is shedding subscribers quickly. But from a contribution margin perspective (contribution to profit), DVD rentals are a cash cow for Netflix. DVDs help them fund the streaming business where profit margins are tiny, eroded by content bidding and large upfront fees.
Our situation at AdRoll may be the reverse. Retargeting, prospecting, and email represent huge markets in which we still have plenty of room to grow. Six million customers advertise on Google Search; we only speak to 25,000 of those customers today. Hence, our resources will continue to focus on growing these product lines. AdRoll should continue to treat innovations as mini-startups within the company, arming them with more materials as their proofs of concept become stronger.
</li>
<li>
Hiccups are OK if you confront them quickly. When Netflix divorced their streaming and DVD services in 2011—making the two products more costly than before—there was an uproar. The stock dropped 45% in three weeks, so the CEO apologized and reversed course. Good will bounced back, and the stock is up 134% this year.
As AdRoll’s product suite becomes more complex, we will confront very real questions about how to differentiate and price the component offerings. Experiments may help us here to gauge market reaction before wider deployment. Intuitively, we should probably price and package our offering so that customers enjoy economies of scale. (The more products they buy with us, the cheaper each component becomes—basically the opposite of what Netflix did.) But we may not get everything right here initially, and we think that’s OK. Customer passion is a sign that they care.
</li>
<li>
People trump everything else. By now the Netflix <a href="http://www.slideshare.net/reed2001/culture-1798664/61-Two_Types_of_Necessary_Rules1">culture deck</a> has become famous. The company screens actively against jerks, pays high for “fully-formed adults,” and demands high performance.
The good news is culture is paramount at AdRoll too. Like Netflix, we look for autonomous, responsible folks and throw tremendous responsibility their way. We believe the next SendRoll will start as the seed of an idea from a Roller’s customer conversation or market research. That seed germinates in her brain and becomes an experiment. Later, her data look promising so she rallies a team, a sprint, a launch, an iteration. That only happens when our Roller is so passionate that she drives the idea to fruition and relentlessly improves it. No lazy bones here. People are AdRoll’s biggest product superpower.
</li>
</ol>
</description>
</item>
<item>
<title>AdRoll and Diversity in Tech: A Q&A with Truc Nguyen and Jessica Grist</title>
<link>https://tech.nextroll.com/blog/culture/2016/06/20/code-screening-questions.html</link>
<pubDate>Mon, 20 Jun 2016 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/culture/2016/06/20/code-screening-questions</guid>
<description><h2 id="introduction">Introduction</h2>
<a href="http://www.codedocumentary.com">CODE: Debugging the Gender Gap</a> is a documentary that exposes the dearth of American female and minority software engineers and explores the reasons for these gaps. CODE raises the question: “What would society gain from having more women and minorities code?”
At AdRoll, we see the value and gains from having more women and minorities coding, which is why we screened this documentary at our headquarters as part of our wider Diversity and Inclusion initiatives. We followed the screening with a panel of female engineering managers—as well as some of our very own Rollers—to discuss not only how the gender gap came to be, but what we can do as individuals and as a collective to debug it.
To reproduce some of what we discussed, we conducted a Q&amp;A with <a href="https://twitter.com/truc_">Truc Nguyen</a> and <a href="https://twitter.com/thehackstress">Jessica Grist</a>, two of our Rollers who were panel speakers. Truc is a UX Designer on the SendRoll team; Jessica is a Software Engineer on the Retargeting team.
Following the Q&amp;A, you can read a bit more about some of our key learnings and takeaways for AdRoll.
<h2 id="qa-with-our-adroll-mini-panel">Q&amp;A With Our AdRoll Mini-Panel</h2>
<h4 id="can-you-tell-us-more-about-your-experience-as-a-female-in-tech">Can you tell us more about your experience as a female in tech?</h4>
Truc: I discovered CS in a roundabout way through the social sciences first. Like others who discovered CS later in life, I think I had a narrow understanding of what it was. After taking the intro class, I loved how it introduced me to new ways of deconstructing and solving problems. I hear a lot of horror stories, but in my experience, I never felt like I was explicitly discriminated against as a female. Like Jessica, one way it’s affected me is that I delayed or dismissed considering it as a field of study until later in undergrad. I never thought it was for me because I bought into the perception of CS as a hardcore field you can only do if you’ve been coding since you were twelve and taking apart computers. Luckily, I had a female section leader and a handful of role models (both male and female) who encouraged me to keep pursuing it. As I progressed further, I did notice the gender ratio swaying in one direction.
Jessica: Refreshingly, it’s been pretty positive! I started off in a supportive and woman-centered environment at <a href="https://hackbrightacademy.com/">Hackbright Academy</a>, where I was surrounded by incredible female role models, both the established engineers who taught the classes and the other women in my cohort. While my first job searches post-Hackbright were stressful, it seemed to be about par for the course from what I heard about other bootcamp grads’ experiences. At AdRoll, I’ve never felt marginalized or discriminated against because of my gender. I’ve also attended dozens of meetups in San Francisco and never felt unwelcome. However, the problem of underrepresentation of minorities in tech has had a huge impact on my professional life. If I had seen more female role models in tech growing up and in my college years, I probably would have become an engineer much sooner, rather than starting my professional life as a teacher.
<h3 id="what-can-i-do-as-an-individual-to-support-underrepresented-people-in-tech">What can I do as an individual to support underrepresented people in tech?</h3>
Truc: The best thing you can do is treat them as you would any other friend or colleague who is worthy of your respect. Offer your encouragement, support, and praise because you admire them for their good work and their traits, not because they “need it more.” If I never bring up my underrepresented status, don’t mention it unless it’s relevant to the conversation. Constantly reminding people they’re marginalized in tech could make them more self-conscious. If the issue comes up, be receptive and listen with an open mind and heart. And if you’re not marginalized, you can lend your voice to the conversation and raise awareness the next time the issue comes up.
Jessica: Treat them no differently than your other colleagues because in a professional setting, they aren’t different. They’re engineers (or designers, etc.) and so are you, and so is everyone else on your team. If you see anyone treating them differently just because they’re underrepresented, speak up about it. If they want to speak about their experiences as underrepresented people, listen. If they don’t, then don’t push them for it. Don’t expect them to be advocates for every other marginalized person in tech, but don’t be surprised if they want to be that either.
<h3 id="what-can-my-company-do-to-support-underrepresented-people-in-tech">What can my company do to support underrepresented people in tech?</h3>
Truc: It’s two-fold.
On the recruiting side, when you get a high volume of applications to comb over, you start to apply filters such as prestigious past experience, schooling, and referrals just to cut down on what you can review. I challenge companies to take the chance and effort to source candidates from different backgrounds and communities including non-traditional candidates. As the CS field can be perceived as male, nerdy, and insular from the outside, the way a company presents itself externally can affect the types of candidates it attracts. Qualified people may self-select or rule themselves out based on the company image or job descriptions that project unconscious bias. As an engineer, it’s really interesting that AdRoll recruiting is so willing to try out a data-driven approach to hiring and to use tools like <a href="https://textio.com">textio.com</a>, which uses natural language processing to make sure your recruiting communications aren’t a turnoff for candidates.
Once you’ve started to build a diverse workforce, it’s about creating an inclusive culture where people feel comfortable voicing their honest opinions without judgment. There’s no real playbook for that process; it has to happen somewhat naturally. As a leader, set a good example by showing that D&amp;I is an important issue on your radar. Encourage town halls or breakout sessions where people can learn more and talk openly. This is something we have found to be effective at AdRoll and has been a catalyst for getting initiatives off the ground and into practice.
Jessica: It all starts with recruiting. Seek out recruiting events sponsored by organizations for minority engineers (or designers, or whatever your job req is for). Be open to non-traditional backgrounds. If you strictly limit your engineering workforce to people with four-year CS degrees from prestigious universities, your company is not fully supporting the inclusion of underrepresented minorities in tech. There’s no need to lower your skill standards; just don’t assume that people who haven’t gone to the “right” universities can’t be good engineers. Beyond hiring, get your leadership involved in D&amp;I initiatives—and not just leaders from gender and ethnic minorities. Don’t assume that your female leaders, or your leaders from ethnic minorities, will want to be involved in D&amp;I, though encourage them if they do.
<h4 id="how-do-you-address-people-who-think-that-diversity-in-tech-isnt-an-issue">How do you address people who think that diversity in tech isn’t an issue?</h4>
Truc: I’ve heard some dissenting opinions: focusing on “women in tech” marginalizes women in tech; women aren’t interested in STEM fields, so we shouldn’t push them; and that the gender gap no longer exists. I challenge them to try to view the issue through a different lens. Though we’ve made great strides towards equity in the workplace over the past twenty, fifty, etc. years, there’s still a long way to go. As a member of an underrepresented group, it can sometimes feel like you’re a foreigner in another country going into a field, company, or organization where you don’t see much diversity. This lack of diversity in tech goes deeper than race, gender, or visible traits and is still a statistical truth. I try to appeal with <a href="http://www.scientificamerican.com/article/how-diversity-makes-us-smarter/">data</a>, <a href="http://web.mit.edu/cortiz/www/Diversity/Jayne%20and%20Dipboye%202004.pdf">figures</a>, and <a href="http://www.mckinsey.com/business-functions/organization/our-insights/why-diversity-matters">research</a> that show creating a more inclusive workplace benefits everyone.
Jessica: This is a really tough one because when you bring up diversity in tech, and then are told that it’s not an issue, that puts you in the position of having to defend and prove your stance. Having to explain your own marginalization and exclusion is especially frustrating for members of marginalized groups. To be honest, I’m not sure of the best way to change these minds.
<h4 id="what-are-the-best-ways-to-raise-awareness-and-combat-apathy">What are the best ways to raise awareness and combat apathy?</h4>
Truc: Heh, having attended many D&amp;I events it can feel like we’re preaching to the choir. I honestly believe most people have good intentions and recognize that D&amp;I is an issue but aren’t sure what actions they can take. If you’re not a manager or you don’t identify as underrepresented, it’s not clear how it relates to you as an individual. Being an ally to an underrepresented minority doesn’t necessarily have to take a lot of work. You don’t have to vocally advocate on behalf of a D&amp;I movement (that’s asking a lot of anyone), but it does take conscious awareness. It means small things, like checking your bias, being more empathetic by trying to imagine what another’s experience is like, or just listening.
Jessica: The only answer I have to this is persistence. Apathy is an enormous part of the problem, perhaps even bigger than active opposition to D&amp;I. The majority of people in tech will acknowledge that diversity in tech is a problem, but not that it’s their problem. But if it isn’t their problem, then whose is it? Underrepresented minorities already struggle enough to break into tech; it’s not automatically their responsibility to increase awareness of diversity too. So have your leaders talk about D&amp;I. Have fun D&amp;I events that will appeal to everyone. Make it a consistent and persistent part of your company’s culture.
<h2 id="what-is-adroll-doing-to-address-the-gender-gap">What is AdRoll doing to address the gender gap?</h2>
We don’t have all the answers at AdRoll, but people are making the right efforts and having open conversations. Some of the key takeaways we’ve gotten from our CODE screening are to:
<ul>
<li>Be aware of unconscious bias as individuals and as a company. We’ve rolled out unconscious bias training globally to generate more awareness about our own biases.</li>
<li>Build a network of allies (colleagues, managers, etc.) to help advocate or raise issues for underrepresented groups in your company.</li>
<li>Be action-oriented: speaking up when you see something you disagree with, working with teams, managers, and recruiting to make a difference through events, training, and facilitated discussions.</li>
<li>Be engaged, not only internally, but also externally. Partner with other companies and organizations and learn from each other, as well as open up Diversity and Inclusion work to an external audience.</li>
</ul>
We thoroughly enjoyed opening up this conversation to a broader audience at AdRoll and look forward to future events that continue the discussion and drive momentum forward. If you want to contribute to this discussion, then be sure to come Roll With Us as we hire and build strong teams. You can <a href="https://www.adroll.com/about/careers/open-positions">learn more at our career site</a>.
</description>
</item>
<item>
<title>Announcing TrailDB - An Efficient Library for Storing and Processing Event Data</title>
<link>https://tech.nextroll.com/blog/data/2016/05/24/traildb-open-sourced.html</link>
<pubDate>Tue, 24 May 2016 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data/2016/05/24/traildb-open-sourced</guid>
<description><a href="http://traildb.io"><img src="/images/post_images/traildb_logo.png" style="display: block; width: 40%; margin-left: auto; margin-right: auto" /></a>
tl;dr Today, we are open-sourcing TrailDB, a core library powering AdRoll. TrailDB makes it fast and fun to handle event data. Find it at <a href="http://traildb.io">traildb.io</a>.
<h2 id="problem-event-data">Problem: Event Data</h2>
Imagine that you have a large amount of event data that looks like this:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2016-05-02T22:48:38 user023 view features
2016-05-02T22:49:01 user301 click graph
2016-05-02T23:03:02 user023 view pricing
2016-05-02T23:15:45 user187 submit signup
2016-05-02T23:35:23 user521 view about
2016-05-02T23:58:11 user004 click graph
2016-05-03T00:02:09 user023 submit signup
</code></pre></div></div>
Data like these could be generated by any web application: each event
has a timestamp, a user identifier, a token describing the action, and
its target.
One could store the events in a relational database, in a table <code class="language-plaintext highlighter-rouge">events</code>
with the above four columns. The table makes it easy to query aggregate
statistics, like signups by day:
<figure class="highlight"><pre><code class="language-sql" data-lang="sql">SELECT date_trunc('day', timestamp), count(*)
FROM events
WHERE target='signup'
GROUP BY 1;</code></pre></figure>
Aggregate queries like this are the bread and butter of data
analytics. There are many excellent, scalable databases such
as <a href="http://aws.amazon.com/redshift">Amazon Redshift</a> or
<a href="http://prestodb.io">Presto</a> which can handle such queries this with
ease.
<h4 id="user-level-analytics">User-level Analytics</h4>
There is another class of queries that are considerably harder to express
in SQL. Notice how in the example data above, <code class="language-plaintext highlighter-rouge">user023</code> first comes to the
site, views a page about product features, then views a page about pricing,
and finally signs up after about an hour since the first event.
Let’s say you wanted to count the number of users
who sign up in less than an hour since the first
event. One way of doing this in SQL is to use <a href="http://www.postgresql.org/docs/current/static/functions-windo
w.html">window
functions</a>. Window functions are a powerful feature of
SQL but unfortunately <a href="http://tapoueh.org/blog/2013/08/20-Window-Functions">they can be quite hard to
grasp</a>.
Over the past years at AdRoll, we have seen a growing number of use
cases for user-level analytics. The use cases vary from computing
metrics like bounce rate to fraud detection and feature extraction for
machine learning. Theoretically, all these use cases could be solved
with sufficiently advanced SQL queries but it is hardly practical.
Alternatively, the queries could be expressed as Hadoop or Spark jobs.
None of these solutions felt sufficiently flexible for all use cases.
Relational databases are not well-suited for computationally demanding
jobs: none of the existing relational databases can scale automatically
and smoothly based on the query load. Frameworks like Hadoop or Spark
impose their own constraints on how to express algorithms, and what
languages, libraries, and infrastructure can be used.
<h2 id="solution-traildb">Solution: TrailDB</h2>
TrailDB is a library implemented in C, which is optimized for storing
and querying series of events at the user level. Its <a href="http://traildb.io/docs/technical_overview/#data-model">data model</a> is specifically designed for use cases like those described above:
<img src="/images/post_images/traildb_datamodel.png" style="display: block; width: 80%; margin-left: auto; margin-right: auto" />
TrailDB’s secret sauce is <a href="http://traildb.io/docs/technical_overview/#internals">data
compression</a>. It
leverages predictability of time-based data to compress your data to a
fraction of their original sizes. In contrast to traditional compression,
you can query the encoded data directly, decompressing only the parts
you need.
TrailDB provides <a href="http://traildb.io/docs/api">a straightforward API</a>
for querying events by user efficiently. The API is
thoroughly performance-driven: Not only does it allow you to
query events quickly, but it also <a href="http://traildb.io/docs/technical_overview/#performance-best-practices">encourages programming
patterns</a>
that make the surrounding application fast.
<h4 id="designed-for-the-cloud">Designed for the Cloud</h4>
TrailDBs are immutable files which your application can create and
access using the TrailDB library. This is a deliberate design choice:
compressed, immutable files like these are a perfect match for modern,
distributed, elastic cloud environments. You can store TrailDBs as you
see fit; we find that Amazon S3 provides virtually unlimited throughput
for your data, and you can process them with an auto-scaling cluster of
(spot) instances which provides virtually unlimited computing power.
TrailDB by itself is a simple library, decoupled from infrastructure,
which is designed to do one thing well: compress event data and
provide high-performance access to them. It integrates seamlessly to
elastic data pipelines as described in a previous blog article,
<a href="http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-do
cker.html">Petabyte-Scale Data Pipelines with Docker, Luigi and Elastic Spot
Instances</a>.
All together, our elastic stack for processing event data in AWS looks
like this:
<div style="margin-top: 5px; font-size: 80%; border-radius: 5px; margin-left: 5em; margin-right: 5em; text-align: center; color: #333; border: 1px solid #ccc; padding: 5px; background: #eee">
Job scheduler, <a href="http://tech.adroll.com/blog/data/2015/10/15/luigi.html">Quentin</a>, executes jobs in an auto-scaling cluster
</div>
<div style="margin-top: 5px; font-size: 80%; border-radius: 5px; margin-left: 5em; margin-right: 5em; text-align: center; color: #333; border: 1px solid #ccc; padding: 5px; background: #eee">
<a href="http://github.com/spotify/luigi">Luigi</a> manages dependencies between jobs
</div>
<div style="margin-top: 5px; font-size: 80%; border-radius: 5px; margin-left: 5em; margin-right: 5em; text-align: center; color: #333; border: 1px solid #ccc; padding: 5px; background: #eee">
Language-agnostic jobs in <a href="http://docker.io">Docker containers</a> query TrailDBs
</div>
<div style="margin-top: 5px; margin-bottom: 2em; font-size: 80%; border-radius: 5px; margin-left: 5em; margin-right: 5em; text-align: center; color: #333; border: 1px solid #ccc; padding: 5px; background: #eee">
Event data are stored in <a href="http://traildb.io">TrailDBs</a> in S3
</div>
This stack allows us to perform a massive amount of computation
cost-efficiently without any performance bottlenecks. S3 has proven
to be able to provide amazing aggregate throughput to hundreds of
concurrent instances, so TrailDBs can be accessed quickly.
Thanks to TrailDB’s various language bindings, we can use the
best tool for each job. Some computationally intensive
jobs are written in <a href="http://github.com/traildb/traildb">C</a>
or <a href="http://github.com/traildb/traildb-d">D</a>. Some jobs
rely on external libraries which are convenient to access
in <a href="http://github.com/traildb/traildb-python">Python</a> or
<a href="http://github.com/traildb/traildb-r">R</a>. Naturally not all
use cases are batch jobs; we have services written
in <a href="http://github.com/traildb/traildb-go">Go</a> and even
<a href="http://github.com/traildb/traildb-haskell">Haskell</a> which access
TrailDBs.
As TrailDB is a library, not a framework, it allows you to structure
your program in the most idiomatic way, which is great for productivity.
Docker removes deployment headaches by encapsulating everything in
well-behaved containers.
Thanks to Quentin and auto-scaling groups, we can optimize instance
types for each workload, some of which are IO-bound while others require
more CPU. Finally, Luigi makes a job graph of tens of inter-dependent
jobs manageable.
<h2 id="traildb-at-adroll">TrailDB at AdRoll</h2>
<img src="/images/post_images/traildb_breadcrumbs.png" style="display: block; width: 80%; margin-left: auto; margin-right: auto" />
TrailDB has been used in production at AdRoll for about one and a half
years. During this time, we have stored tens of trillions of events in
TrailDBs. Today, the TrailDBs are queried by almost a thousand jobs
daily.
As TrailDB is a core component of our data infrastructure, we take
robustness and testing seriously. The unit test coverage of TrailDB is
nearly 100%. TrailDB is also hardened by a number of integration tests.
As a result of long-term production use, we take backwards compatibility
very seriously: not a single time during the history of TrailDB we have
broken backwards compatibility so that old TrailDBs could not be read
anymore. We intend to keep it this way.
We have a number of tools built on top of TrailDB which make computing
various user-level metrics easier. We are planning to open-source some
of these tools in the future. Adroll continues active development of
TrailDB and we hope that the wider community finds it useful as well.
<h2 id="learn-more">Learn More</h2>
You can find TrailDB with extensive documentation at <a href="http://traildb.io">traildb.io</a>.
The quickest way to get started is to read the <a href="http://traildb.io/docs/getting_started/">installation instructions</a> and walk through <a href="http://traildb.io/docs/tutorial">the TrailDB tutorial</a>.
If you want to learn more about the background and internals of TrailDB, you can <a href="https://www.youtube.com/watch?v=ondmDAMWEtg">watch
a presentation about TrailDB on YouTube</a> and
browse <a href="http://slides.com/villetuulos/intro-to-traildb#/">the slides of the presentation here</a>.
If you have any questions, you can <a href="https://gitter.im/traildb/traildb">find us in the TrailDB Gitter channel</a>.
</description>
</item>
<item>
<title>Collider: A New Frontier in Testing</title>
<link>https://tech.nextroll.com/blog/rtb/2016/04/29/collider.html</link>
<pubDate>Fri, 29 Apr 2016 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/rtb/2016/04/29/collider</guid>
<description><h4 id="introduction">Introduction</h4>
Advertising is a competitive market. This probably won’t be the first time you hear something like that, but it’s absolutely true. For every move you make, you have to consider your competitors are getting a step closer to you. Speed and precision are key to the success of the business, but how can we guarantee these with the stress of such a large system? Is that pricing strategy better than what we are running right now? Would we improve the performance of our bids, or just burn money? Handling billions of requests may make changing a simple digit an odyssey that can end extremely badly, costing a significant chunk of revenue.
Considering the need and the risk, there is a single solution: iterating fast, but in a controlled manner. Risking all of our revenue by deploying an idea would get, in the best case scenario, our business intelligence team to raise an eyebrow. So why don’t we risk just a fraction of our traffic? Let’s construct an experiment, specify our study group, and fire away. No complicated deployments, no significant risk of the whole system catching on fire, and immediate feedback on how that experiment is performing.
Regardless of how cumbersome and slow a regular deployment process is, that’s something to worry about at a later time. Now is the time to do some good, old-fashioned trial and error, but in a healthy, controlled manner.
<h4 id="collider">Collider</h4>
How can we iterate in that swift, controlled manner while taking less than 10 minutes to deploy a new set of tests? Easy: everything comes back to Erlang.
If you are unfamiliar with <a href="http://learnyousomeerlang.com">Erlang</a>, you are missing out on a magnificent language with myriad smart features. This functional language was developed with a simple idea in mind: being able to process thousands of parallel requests in a concurrent manner. It was a perfect fit for our real-time bidders. These bidders handle hundreds of thousands of requests per second and have to respond on rigid deadlines, so concurrent and lightweight processes are a must. We’ll focus on a couple smart features: isolation and “hot swap.”
Our experiment framework is called Collider. As we mentioned, we want to be able to run these experiments in as much isolation as possible. This framework splits the total traffic received by a single box into a set of slots. Do we think that an experiment would need a big sample for proving its efficacy? We assign a big percentage of our traffic to that experiment, but we rest assured that the untouched slots are completely safe and unaffected by any crazy ideas. Erlang guarantees isolation on each process, so even if we deploy a wild piece of code, the system will ensure only that fraction of traffic is affected. Also, as we are controlling strict deadlines internally (as we have timeout requirements for third parties), even if we try to connect to external components, the requests will be expired and cleaned before they can affect the rest of the system.
<center>
<img alt="Collider Traffic Flow" src="/images/post_images/collider.png" />
</center>
And what about that “hot swap” thing? Erlang allows us to hot reload code without any complicated ritual. We simply replace the previous version of our code on the filesystem and, poof!, the code will seamlessly be replaced for the whole runtime. It’s as easy as that. Instead of thinking about bringing down boxes and waiting for a new version to propagate, we simply pack our Collider experiment bundle and fire it off to S3. In no time at all, the boxes will be running the new set of tests and expiring those that are no longer needed. Zero impact, zero downtime, immediate success.
<h4 id="integrations">Integrations</h4>
“Okay, that sounds interesting, but what’s an ‘experiment?’ What can this Collider thing test?” This is one of the best aspects of Collider, as the infrastructure is prepared for handling all types of experiments.
You may have heard about real-time bidding that it’s not only about being fast, it’s also about being smart about where to spend money. “How probable is it that this person will click on this ad? Do we need to go all out or will our money be wasted on this potential impression?” Having to reply to these kinds of questions without the appropriate data is a gamble, and that’s why the bidders rely on our BidIQ machine learning system.
BidIQ trains on the data we have seen so far, extracting information from select features of our traffic. It then uses these features to return a performance prediction and bid price. But when we want to add new training features to that system, it’s not always easy to see which new criteria may be good indicators about the quality of a bid request. For these scenarios, Collider is a perfect fit, allowing us to do an A/B test in a controlled manner with live traffic. Once the data science peeps are happy with the results, that new model can be moved into production, becoming the default model that predicts prices for all of our traffic.
But our integrations don’t end here. For example, when several ads are available to display to the same user, what criteria should we use to decide which ad is best? Or, is this real traffic or fraud? All these variables take place in different parts of our real-time bid flow, but the Collider framework is sufficiently generalized to inject itself into all of these places. We can change the behavior of key parts of our system in a transparent way, and for a small percentage of traffic in a completely isolated way. In order to keep our fingers on the system’s pulse, all results can be monitored on different dashboards with our analytics tools.
Even better, when defining the slots for splitting our traffic, they can be reused, allowing us to combine different experimental behaviors across the system. Essentially, we can control whether we’re testing independent or correlated variables. For example, say we have a new, more aggressive BidIQ version and we want to combine it with a new internal auction strategy to select from all available ads. Reusing the same slots on both experiments would achieve the desired effect. In the image below, we can see how distributing slots appropriately will not affect our tests. The comparison of red, green, and white in the top layer is fine because each of those is subjected to proportionally the same amount of blue traffic. Similarly, on the second layer, the blue traffic is subjected to the same amount of red, green, and white traffic from the top layer as the white traffic in its layer.
<center>
<img alt="Collider Correlation Control" src="/images/post_images/collider-mix.png" />
</center>
<h4 id="can-it-get-any-better">Can it get any better?</h4>
Even with a swift deployment process, flexibility, and monitoring, we still have plenty of exciting features to implement on Collider and its environment.
Right now, testing locally can take a while, especially when tests get more complicated or require specific external components available on production. Most of the time would be spent setting everything up to replicate production in an accurate manner (and with high odds of something missing). One of our short-term goals is automatically creating staging environments to push a release of Collider and let it run with synthetic traffic. In this replica of a production environment, the new version would run for a time until a set of health indicators are achieved. The release would then be ready to be shot into production (with human intervention, of course), across the board.
This may look like a contradiction with a tight development cycle, but it allows for less tech-savvy parts of the company to create tests without requiring engineering time to debug them. For that reason, we would like to provide new, simple interfaces to set parameters on tests so anyone with the right permissions can fiddle with them.
If working in such a fast-paced environment sounds enticing and you are in love with new and exciting technologies for high traffic volumes, don’t think twice: <a href="https://www.adroll.com/about/careers">drop us a line!</a>
</description>
</item>
<item>
<title>The New AdRoll Data Analysis Platform</title>
<link>https://tech.nextroll.com/blog/data/2016/04/01/nada.html</link>
<pubDate>Fri, 01 Apr 2016 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data/2016/04/01/nada</guid>
<description>
<h4 id="introduction">Introduction</h4>
If there is one thing that has been made abundantly clear over the last decade in tech,
it’s that understanding your data is paramount to achieving your goals. Understandably, many
startups have sprung up promising to aid every company and their employees with data analytics
to fully optimize not only their work lives, but their personal lives too. Knowledge is power.
Of course, the more power you have, the more knowledge you can gain. After analyzing current market offerings, we on AdRoll Data Science
found only solutions that sluggishly accelerated our growth to the <a href="https://en.wikipedia.org/wiki/Technological_singularity">Singularity™</a>.
Some like to say they welcome our new robot overlords, but we have a policy of actively ushering them in. (Note: We
will soon be adding a lovable automaton, <a href="https://en.wikipedia.org/wiki/Gort_(The_Day_the_Earth_Stood_Still)">Gort</a>,
to our collection of <a href="https://www.adroll.com/about/careers">spirit animals</a>.) This is why we set to work and
developed the New AdRoll Data Analytics (NADA) platform.
<center>
<img alt="Gort in action." src="/images/post_images/gort.jpg" /> 
Gort in action.
</center>
<h4 id="technology">Technology</h4>
As some readers of our previous blog posts may know, AdRoll strongly encourages its engineers to explore new
technologies, and if you’ve been following the Data Science Engineering team in particular, you know that we
love to get to the metal to eke out as much performance as we can.
But this isn’t enough. AdRoll also has a long-standing philosophy of bringing the best to everyone, not just
those with the time and resources of enterprise businesses. We knew that NADA had to be fast, powerful, and most of all, keep
us close to our data. But this stuff is only great if the platform is usable, so we knew we wanted a nice,
clean API. Take a look for yourself at this k-means code:
I don’t know about you, but this is just about the clearest code I’ve ever seen. Yes,
we at AdRoll have become huge fans of <a href="http://compsoc.dur.ac.uk/whitespace/tutorial.html">Whitespace</a>.
This may seem like a controversial choice to some, but consider the benefits:
<ul>
<li>You can code for longer as eye strain is minimized, and with <a href="https://upload.wikimedia.org/wikipedia/commons/9/91/Sun_three_button_mouse.jpg">appropriate hardware</a>, the ergonomics are fantastic.</li>
<li>Rather than slogging through PRs, each diff is a colorful delight that warms the soul. See image below.</li>
<li>In a process of discovery, printing out your code gives no benefit to your legal opposition.</li>
<li>By inputting your data as essentially raw binary, you’re thinking much more like a machine. This makes Gort happy.</li>
<li>You want Gort to be happy.</li>
</ul>
These are just the benefits of Whitespace in and of itself. Let’s move on to NADA.
<center>
<img alt="Beautiful diff." src="/images/post_images/ws_diff.png" /> 
Beautiful diff.
</center>
<h4 id="nada-features">NADA Features</h4>
The core features of NADA are a boon to any data scientist, statistician, or Swiss army personnel.
You can regress your convex optimizations, trick your kernels into matrix factorizations, and
backpropagate your support vector machinations. All of this is so abstracted away, you can be sure
you won’t miss the random forest for the decision stumps.
For example, take a look at this snippet, which convolutes a neural network into a stochastic principal
component Lp space:
<figure class="highlight"><pre><code class="language-c" data-lang="c">start_with_your_neural_net_to_convolute...
stochasticizing...
now_for_components...
output_Lp_space!
repeat_for_recurrent_space.</code></pre></figure>
So much processing in such a small snippet really shows the expressive power of NADA.
This expressivity has allowed AdRoll to bootstrap our data science efforts faster
than we ever thought possible, because NADA is also incredibly efficient and scalable.
NADA’s underlying data framework is a parallelized, fault-tolerant, NoSQL, relational
data structure that guarantees 100% consistency. This plays a key part in our ability
to rip through tons more data than we have in the past. It’s a unique structure, for sure,
so there is some ramp up to take full advantage of its features. However, we went ahead
and implemented more standard APIs to get your feet wet.
For example, for our <a href="https://www.adroll.com/product/prospecting">Prospecting product</a> we
were asked to find out whether certain subsets of our data contained graphs of a particular
<a href="https://en.wikipedia.org/wiki/Clique_problem">size</a>. Tackling this somewhat naïvely, we have:
<figure class="highlight"><pre><code class="language-c" data-lang="c">good...
...but_slow</code></pre></figure>
It does all right even with the standard API calls, but that’s mainly a testament
to NADA’s computational ingenuity. However, as soon as we switch to NADA’s unique
brand of data structure:
<figure class="highlight"><pre><code class="language-c" data-lang="c">
Wow!</code></pre></figure>
“<a href="https://en.wikipedia.org/wiki/Wow!_signal">Wow,</a>” indeed! Even a cursory glance
reveals the speedup here. We’re expecting a <a href="http://www.claymath.org/millennium-problems/p-vs-np-problem">novelty check</a>
in the mail soon.
<h4 id="onwards-and-upwards">Onwards and Upwards!</h4>
We’ve only just scratched the surface of NADA here. While NADA is primarily
a data analytics framework, we have also found it useful in a variety of other
contexts, such as automatic intrusion detection, improving our WiFi signals,
supplanting the human race, and defragging our Hadoop clusters. Also, NADA is
absolutely delightful spread on a <a href="http://www.businessinsider.com/we-tried-the-fancy-4-toast-san-francisco-is-going-crazy-for-2015-6">thick piece of toast</a>
with a glass of orange juice.
So where do we go from here? Naturally, we’ll be open-sourcing NADA shortly.
Gort is looking forward to seeing what you can do with our framework
as soon as you get your hands on it. Gort does not like to be disappointed,
so please, contribute as much NADA as you can.
Happy coding!
<h4 id="ps">P.S.</h4>
</description>
</item>
<item>
<title>Bidding Credit Systems
</title>
<link>https://tech.nextroll.com/blog/dev/2016/02/28/bidding-credits.html</link>
<pubDate>Sun, 28 Feb 2016 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2016/02/28/bidding-credits</guid>
<description>The project I work on at AdRoll is mature. In a mature project the economics of
software flip around to where doing something right is more desirable that doing
something quickly. In 2015 we invested a lot of effort in resolving long-tail
issues with the bidder system, issues we’d recognized in our failure-mode
conversations but had left unresolved pending future work. Sometimes, a system
will evolve away from such things and sometimes you’ll have to put in the time
and energy to, not repair, but mitigate. Long-tail issues are often intrinsic,
those fun little sideshows Charles Perrow called Normal Accidents, failures
inherent to a system that can’t be removed but must be lived with. It’s good
times.
This blog post details one of my favorite resolutions in 2015, the bidding
credit system.
A brief run-down of the bidders. A bid comes in from an “exchange”–these are
companies like Google, AppNexus or OpenX–and we’re obligated to reply. Replies
can be either a “no bid”, essentially a bid of $0, or a “bid”, a bid of greater
than $0. If we win the auction the exchange sends us a specially coded
additional request, called a “win”. When this arrives, we reduce the amount of
money the system as a whole has to spend. If a “win” doesn’t arrive, we assume
that there’s still money to spend and the system does its best to do so.
Here’s a fun question: what happens if an actual win gets decoupled from “win”
events? By protocol contract with the exchanges they’re not supposed to be but,
you know, things go wrong. What happens?
The bidder used to simply assume, without a “win” that it had money to spend and
would continue to do so. This meant that it would continue to spend money, money
it may, were the “wins” arriving appropriately, have not spent. Worse, the
bidders might erroneously spend more money that in its budget, a worst-case
scenario we call “overspend”. Overspends are potentially catastrophic as there’s
no fixed upper bound.
Win decoupling took us by surprise in early 2015. Alarms fired alerting us that
the system’s rate of win had dropped below normal bounds but a side-channel in a
more delayed notification informed us that the system was still spending money.
Oops! Manual intervention killed all bidding and communication with the exchange
in question informed us that a bug in their latest release caused a failure of
win notifications. This was… not cheap.
In the lifetime of the system this decoupling had only been seen just the once
but it was bad enough to need a safety system to stop it from happening again.
My colleague
<a href="http://tech.adroll.com/blog/dev/2015/11/16/count-things-with-aws-lambda-python-and-dynamodb.html">Mike</a>
and I kicked around ideas and settled on a take off a classic feedback control
system.
The algorithm is this:
Grant a “bidding credit pool” of some initial capacity C, maximally bounded at
Cm, some bid rate Br (per second) and some win rate Wr (per second):
<ul>
<li>Each bid issued by the system to an exchange decrements C by B.</li>
<li>Each win received by the system from an exchange increases C by W.</li>
<li>If at bid time C =&lt; L a no-bid is automatically issued.</li>
<li>Every X seconds Bmp credits are added to the pool.</li>
</ul>
The first two points decide for the system how much a losing bid costs. Combine
that with the third point, it’s possible to set values such that a losing streak
will cause the pool to exhaust, shutting off bidding. The fourth point protects
against losing streaks and allows for automatic recovery in the event of pool
exhaustion.
It seemed to Mike and me that the algorithm would work. Say you have Cm = 100, C
= 100, Br = 100, Wr = 0.1, B = 1, W = 1, L = 0, X = 5 and Bmp = 100. Then you’ll
find that per second Br*Wr = 10 and so every second you’ll remove 90 credits
from C, or, C(t1) = C(t0) - Br(1-Wr).
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>C(0) = 100
C(1) = 100 - 100(1-0.1) = 90
C(2) = 90 - 100(1-0.1) = 0
C(3) = 0
C(4) = 0
C(5) = 0 + 100 - 100(1-0.1) = 90 = C(0).
</code></pre></div></div>
It’s a loop. In reality there’d be some variation as Br won’t always be a
constant but notice that for 3/5 of the time the system is in a state where it
can’t bid. That’s not good: the whole purpose of the system is to make bids.
Charles Perrow’s analysis in
<a href="http://www.huffingtonpost.com/brian-troutwine/peculiar-books-reviewed-c_b_5244093.html">Normal Accidents</a>
argued that by attempting to resolve failure in one subsystem with the addition
of another you introduce new failures elsewhere. Well, check it out, the bidding
credit system can be configured to cause a worse problem than the one we’re
attempting to fix.
Clearly, the trick here is finding the right parameters for the system. By team
consensus we wanted something that would be slow to shut off bidding, something
with a gradual wind-down, and quick to start bidding again, something with “get
up and go”. This bidding credit system is dangerous enough that it can’t be
released into the world behind a feature gate and fiddled with but we also
must have operational experience with it.
What to do?
We modeled it! What we had on our hands was a simple-to-understand system and a
search space for ideal configuration parameters. A small bit of python:
<figure class="highlight"><pre><code class="language-python" data-lang="python">import argparse
import math
import random
import sys
ONE_DAY_MS = 86400
def tri(mode):
return random.triangular(max(0.0, mode-0.3),
mode,
max(1.0, mode+0.3))
def log(n):
if not n:
return 0
else:
return math.log(n)
class World:
def __init__(self,
seed,
initial_credits=1000,
win_kill_point=43200,
win_recover_point=53200,
credit_cap=1000,
time_step=10,
time_end_lower_limit=86400,
bid_request_rate_per_second=100,
win_rate=0.10,
bid_rate=0.10,
decriment_per_bid=1,
incriment_per_win=10,
periodic_credit_bump=100,
periodic_credit_bump_amount=10
):
self.time_step = time_step
self.time_end_lower_limit = time_end_lower_limit
self.bid_request_rate_per_second = bid_request_rate_per_second
self.win_rate = win_rate
self.bid_rate = bid_rate
self.decriment_per_bid = decriment_per_bid
self.incriment_per_win = incriment_per_win
self.periodic_credit_bump = periodic_credit_bump
self.periodic_credit_bump_amount = periodic_credit_bump_amount
self.credit_cap = credit_cap
self.win_kill_point = win_kill_point
self.win_recover_point = win_recover_point
self.current_time = 0
self.outbound_bids = 0
self.credits = initial_credits
self.total_bids = 0
self.total_wins = 0
random.seed(seed)
return None
def step(self):
if self.current_time &gt;= self.time_end_lower_limit:
return None
instantaneous_max_bids = int(self.bid_request_rate_per_second * self.time_step)
time_of_day_adjustment = (math.sin(self.current_time * (1.0/(ONE_DAY_MS/8)))+1)/2.0
instantaneous_bids = int(instantaneous_max_bids * time_of_day_adjustment)
# Determine how many bids we can make given our credit pool and the
 # bids coming into the system. Can't bid more than we get traffic!
free_bids = math.ceil(self.credits / (self.decriment_per_bid*1.0))
bids_submitted = math.ceil(min(instantaneous_bids, free_bids) * tri(self.bid_rate))
# Based on the outbound bids, determine how many of those signal back as
 # wins. (We fake the delay to be one tick. Not realistic but I don't
 # think it matters.)
 #
 # If we're past the kill point, we have no wins coming into the system.
if self.win_kill_point &lt;= self.current_time &lt;= self.win_recover_point:
dead_air = True
instantaneous_wins = 0
else:
dead_air = False
instantaneous_wins = math.ceil(self.outbound_bids * tri(self.win_rate))
# Queue up our current bids to be outbound for the next tick.
 self.outbound_bids = bids_submitted
self.total_wins += instantaneous_wins
self.total_bids += bids_submitted
# Credit adjustments
 #
 # Remove credits per bid. I don't _think_ order of this matters in the
 # context of simulation, whether we incriment or decriment first.
 self.credits -= bids_submitted * self.decriment_per_bid
# Account for wins. We done good and deserve some credit.
 self.credits += instantaneous_wins * self.incriment_per_win
# Every fixed period we get an automatic credit bump.
if (self.current_time % self.periodic_credit_bump) == 0:
self.credits += self.periodic_credit_bump_amount
if self.credits &gt; self.credit_cap:
self.credits = self.credit_cap
assert self.credits &lt;= self.credit_cap
self.current_time += self.time_step
return [self.current_time, self.credits, free_bids, bids_submitted, self.outbound_bids, instantaneous_wins, instantaneous_bids, time_of_day_adjustment, self.total_bids, self.total_wins, dead_air]
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Simulate the "bidding credits" constrol system')
parser.add_argument('--time_step', type=int, default=10,
help='integer, unit in seconds: time simulation steps forward per tick')
parser.add_argument('--time_end_lower_limit', type=int, default=86400,
help='integer, unit in seconds: time simulation will not cease before; default 1 day')
parser.add_argument('--win_kill_point', type=int, default=(86400/2),
help='integer, unit in seconds: point in time when all win notifications will fail; default 1/2 day')
parser.add_argument('--win_recover_point', type=int, default=(86400/2)+10,
help='integer, unit in seconds: point in time when all win notifications will recover; default 1/2 + 10000 day')
parser.add_argument('--seed', dest='seed',
default="""I went to the woods because I wished to live deliberately, to front only the
essential facts of life, and see if I could
not learn what it had to teach, and not, when
I came to die, discover that I had not lived.
I did not wish to live what was not life,
living is so dear; nor did I wish to practice
resignation, unless it was quite
necessary.""",
help='hashable: seed for the simulation random number')
parser.add_argument('--initial_credits', type=int, default=1000,
help='integer: total number of credits available to the system on startup'),
parser.add_argument('--credit_cap', type=int, default=1000,
help='integer: maximum number of credits the system can accumulate'),
parser.add_argument('--bid_request_rate_per_second', type=int, default=100,
help='integer, units in bid/sec: total number of requests per second in the simulation')
parser.add_argument('--win_rate', type=float, default=0.10,
help='float, [0,1.0]: The percentage of time a bid wins.')
parser.add_argument('--bid_rate', type=float, default=0.10,
help='float, [0,1.0]: The percentage of time we make a bid.')
parser.add_argument('--decriment_per_bid', type=int, default=1,
help='integer: total credits a single bid consumes')
parser.add_argument('--incriment_per_win', type=int, default=10,
help='integer: total credits a single win provides')
parser.add_argument('--periodic_credit_bump', type=int, default=100,
help='integer, units in seconds: periodicity of automatic credit replenishment')
parser.add_argument('--periodic_credit_bump_amount', type=int, default=10,
help='integer: amount of credits per replenishment cycle')
args = parser.parse_args()
w = World(args.seed,
initial_credits=args.initial_credits,
win_kill_point=args.win_kill_point,
win_recover_point=args.win_recover_point,
credit_cap=args.credit_cap,
time_step=args.time_step,
time_end_lower_limit=args.time_end_lower_limit,
bid_request_rate_per_second=args.bid_request_rate_per_second,
win_rate=args.win_rate,
bid_rate=args.bid_rate,
decriment_per_bid=args.decriment_per_bid,
incriment_per_win=args.incriment_per_win,
periodic_credit_bump=args.periodic_credit_bump,
periodic_credit_bump_amount=args.periodic_credit_bump_amount
)
sep = '\t'
print sep.join(['# ', 'current_time', 'credits', 'free_bids', 'bids_submitted', 'outbound_bids', 'instantaneous_wins', 'instantaneous_bids', 'time_of_day_adjustment', 'total_bids', 'total_wins', 'dead_air'])
step = w.step()
while step:
print sep.join([str(x) for x in step])
step = w.step()</code></pre></figure>
and an even smaller bit of gnuplot:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/usr/bin/env gnuplot
#
# Plotting data about the credit model (see credit_model.dat)
reset
# png
set terminal png size 2560,1440 enhanced font 'Verdana,10'
set output 'credit_model.png'
set xlabel 'Time (seconds)'
set tics scale 0.75
plot 'credit_model.dat' using 1:2 smooth acsplines with lines title 'Credits Available', \
'credit_model.dat' using 1:4 smooth acsplines with lines title 'Bids Submitted', \
'credit_model.dat' using 1:5 smooth acsplines with lines title 'Outbound Bids', \
'credit_model.dat' using 1:6 smooth acsplines with lines title 'Instantaneous Wins'
</code></pre></div></div>
let the team understand how the system would behave in a no-risk manner. You’ll
notice that the python code is simulating more than the brief sketch above:
waning traffic through the day, random fluctuation around rates. Here’s
a similar periodic crash scenario graphed:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>blt&gt; python credit_model.py --time_end_lower_limit=1800
--bid_request_rate_per_second=1000 --initial_credits=1000 --time_step=1
--credit_cap=1000 --periodic_credit_bump=60
--periodic_credit_bump_amount=1000 --decriment_per_bid=1
--incriment_per_win=1 --bid_rate=1.0 --win_rate=0.1 &gt; credit_model.dat &amp;&amp;
./credit_model.gnu
</code></pre></div></div>
<img src="/images/post_images/bidding_credits/0000-boom_and_bust.png" alt="Boom and Bust Graph" />
Crummy, right?
Here’s another scenario with W = 10, effectively canceling out all the losing
bids to get to a win.
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>blt&gt; python credit_model.py --time_end_lower_limit=1800
--bid_request_rate_per_second=1000 --initial_credits=1000 --time_step=1
--credit_cap=1000 --periodic_credit_bump=60
--periodic_credit_bump_amount=1000 --decriment_per_bid=1
--incriment_per_win=10 --bid_rate=1.0 --win_rate=0.1 &gt; credit_model.dat &amp;&amp;
./credit_model.gnu
</code></pre></div></div>
<img src="/images/post_images/bidding_credits/0001-stable.png" alt="Stable Graph" />
Configured well, the system is stable in normal operation. Good! What about if
there’s a win decoupling event? The careful reader will have noticed that the
python script has the ability to simulate a decoupling. Here’s the stable system
with a decouple thrown in:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>blt&gt; python credit_model.py --time_end_lower_limit=1800
--bid_request_rate_per_second=1000 --initial_credits=1000 --time_step=1
--credit_cap=1000 --periodic_credit_bump=60 --periodic_credit_bump_amount=1000
--decriment_per_bid=1 --incriment_per_win=10 --bid_rate=1.0 --win_rate=0.1
--win_kill_point=900 --win_recover_point=1200 &gt; credit_model.dat &amp;&amp;
./credit_model.gnu
</code></pre></div></div>
<img src="/images/post_images/bidding_credits/0002-crash_recover.png" alt="Recover Graph" />
You’ll note it’s got the properties we desire, a clamping of bidding and a rapid
jump back to full capacity once wins start entering the system again. At no
point do bids ever permanently drop to zero–the periodic credit bump keeps us
issuing a moderate number bid submissions–but the fall-off is drastic enough to
put the loss in the realm of pennies.
And it turns out, it works well in practice, too:
<img src="/images/post_images/bidding_credits/0003-works_in_practice.jpg" alt="Real Recover Graph" />
</description>
</item>
<item>
<title>Spark on EMR</title>
<link>https://tech.nextroll.com/blog/spark/2016/01/25/spark-on-emr.html</link>
<pubDate>Mon, 25 Jan 2016 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/spark/2016/01/25/spark-on-emr</guid>
<description><h4 id="high-level-problem">High Level Problem</h4>
Our Dynamic Ads product is designed to show a user ads with specific, recommended products based on their browsing history. Each advertiser gives us a list of all of its products beforehand so that we can generate a “product catalog” for the advertiser. Sometimes, however, the product catalog that the advertiser has given us may not accurately reflect all of the products that exist on their website, and our recommendations become less accurate. Thus, we wanted to run a daily data processing job that would look at all products viewed on a given day and determine if we had a matching record in the advertiser’s product catalog. Then, we could aggregate that data to calculate a “match rate” for each advertiser, allowing us to easily target and troubleshoot troubled product catalogs.
In order to accomplish this task, we needed to perform the following actions:
<ul>
<li>Query <a href="http://tech.adroll.com/blog/adroll/2015/05/04/aws-summit-keynote.html">AdRoll’s raw logs</a> via <a href="https://prestodb.io/">Presto</a> for initial data set (8 million rows).</li>
<li>For each row in the data set, query DynamoDB.</li>
<li>Add each new row to a table in a PostgreSQL database (Amazon RDS).</li>
<li>Summarize (reduce) the data set to get an aggregated view.</li>
<li>Add each row to another aggregated table in the PostgreSQL database.</li>
</ul>
<h4 id="why-pyspark">Why PySpark?</h4>
Given that we were faced with a large-scale data processing task, it became clear that we would need to use a cluster computing solution. As a new engineer who is most familiar with Python, I was initially drawn to Spark (as opposed to Hadoop) because I would be able to use <a href="https://spark.apache.org/docs/0.9.0/python-programming-guide.html">PySpark</a>, a Python wrapper around Spark. While there are other Python libraries to perform mapreduce tasks (like <a href="https://pythonhosted.org/mrjob/">mrjob</a>), PySpark was the most attractive of the Python options due to its intuitive, concise syntax and its easy maintainability. Moreover, the performance gains from using Spark were also a major contributing factor.
While the PySpark code needed to build the necessary RDDs and write to the database ended up being rather trivial, I ran into numerous operational roadblocks around deploying and running PySpark on EMR. Many of these issues stemmed from the fact that despite the new EMR 4.1.0 AMI supporting Spark and PySpark out of the box, it was difficult to find documentation on configuring Spark to optimize resource allocation and deploying batch applications. Among the various issues I encountered and ultimately solved were: installing Python dependencies, setting Python 2.7 as the default, configuring executor and driver memory, and running the spark-submit step. In this post, I present a guide based on my experience that I hope serves to smooth the development process for future users.
<h4 id="spark">Spark</h4>
<ul>
<li>Transformations vs. Actions
<ul>
<li>Spark RDDs support two types of operations: <a href="http://spark.apache.org/docs/latest/programming-guide.html">transformations and actions</a>. As the documentation clearly explains, transformations (like <code class="language-plaintext highlighter-rouge">map</code>) create a new dataset from an existing one, but they are lazy, meaning that they do not compute their results right away. In other words, transformations simply remember which transformations were applied to the base data. Actions (like <code class="language-plaintext highlighter-rouge">foreachPartition</code>), on the other hand, return a value to the driver after running a computation. The computations associated with a transformation are not actually computed until an action is called.</li>
<li>An understanding of this difference is hugely important in actually leveraging Spark’s performance gains. For example, prior to understanding this crucial difference, I was actually making the 8 million DynamoDB calls twice, instead of just once. With this understanding, I was able to decrease my job runtime by more than 50%.</li>
<li>Below, I have shown the difference between the code before and after this realization. Depending on the size of your data set and the memory available, you can either call <code class="language-plaintext highlighter-rouge">.persist()</code> on the RDD that you will reuse (<code class="language-plaintext highlighter-rouge">rdd1</code> in our case) or write the data to disk. Due to the size of our data set and the fact that the intermediary RDD was precisely what I wanted to store in the database, I chose to write <code class="language-plaintext highlighter-rouge">rdd1</code> to the database, and then query the database to initiate a new RDD.</li>
</ul>
</li>
</ul>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> ##### INEFFICIENT CODE #####
 base_rdd = sc.parallelize(presto_data)
# We do not actually query DynamoDB until foreachPartition is called,
 # as map is a transformation and foreachPartition is an action.
 rdd1 = base_rdd.map(query_dynamo_db)
rdd1.foreachPartition(add_to_db)
# At this point, because query_dynamo_db was within a map function
 # (a transformation, which is remembered by rdd1), we actually query
 # DynamoDB all over again when foreachPartition is called.
rdd2 = rdd1.map(lambda x: (x[0], x[1])).reduceByKey(lambda x,y: x+y)
rdd2.foreachPartition(add_to_db)
</code></pre></figure>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> ##### IMPROVED CODE #####
 base_rdd = sc.parallelize(presto_data)
rdd1 = base_rdd.map(query_dynamo_db)
rdd1.foreachPartition(add_to_db)
# Here we build the second RDD based on the data which was added
 # to the database in the first foreachPartition action, instead of
 # based upon rdd1. Thus, we do not repeat the DynamoDB calls.
 db_rows = get_rows_from_db()
new_rdd = sc.parallelize(db_rows)
rdd2 = new_rdd.map(lambda x: (x[0], x[1])).reduceByKey(lambda x,y: x+y)
rdd2.foreachPartition(add_to_db)
</code></pre></figure>
<ul>
<li>Partitions
<ul>
<li>An important parameter when considering how to parallelize your data is the number of partitions to use to split up your dataset.</li>
<li>Spark runs one task (action) per partition, so it’s important to partition your data appropriately.</li>
<li>Spark’s documentation recommends 2-4 partitions for each CPU in the cluster.</li>
<li>RDDs created from <code class="language-plaintext highlighter-rouge">textFile</code> or <code class="language-plaintext highlighter-rouge">hadoopFile</code> will use the mapreduce API to determine the number of partitions and thus will have a reasonable predetermined number of partitions.</li>
<li>On the other hand, the data set returned from my Presto query returns a number of partitions determined by the <code class="language-plaintext highlighter-rouge">spark.default.parallelism</code> setting. By default, this setting sets the number of partitions to the number of executors (<code class="language-plaintext highlighter-rouge">spark.executor.instances</code>, 20 in our case) or 2, whichever is greater. Thus, in order to optimize, we manually tuned the number of partitions to be equal to 80 (20 executors * 4 partitions each).</li>
</ul>
</li>
</ul>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> base_rdd = sc.parallelize(presto_data, 80)
</code></pre></figure>
<ul>
<li>Importing from Self-created Modules
<ul>
<li>Unlike in base Python, where one is able to import from any other Python file within a given repository, in Pyspark, external self-created libraries need to be zipped up and then added to the global SparkContext object.</li>
<li>First the SparkContext object must be created, and then the zip file is added to the object via the <a href="https://spark.apache.org/docs/0.7.2/api/pyspark/pyspark.context.SparkContext-class.html#addPyFile">addPyFiles</a> method.</li>
</ul>
</li>
</ul>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> from pyspark import SparkContext
sc = SparkContext()
sc.addPyFile('/home/hadoop/spark_app.zip')
</code></pre></figure>
<ul>
<li>Using Sqlalchemy
<ul>
<li>Because of the distributed nature of Spark, a Sqlalchemy <code class="language-plaintext highlighter-rouge">Session</code> object can not be shared across multiple executors or paritions.</li>
<li>For this reason, a new <code class="language-plaintext highlighter-rouge">Session</code> object must be created per partition, and <code class="language-plaintext highlighter-rouge">sqlalchemy.create_engine</code> and <code class="language-plaintext highlighter-rouge">sqlalchemy.orm.sessionmaker</code> must be imported within each partion. All of this is done within a function called by <code class="language-plaintext highlighter-rouge">foreachPartition</code>.</li>
</ul>
</li>
</ul>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> def add_to_db(iterator):
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
engine = create_engine(DATABASE_URL)
session = sessionmaker(bind=engine)
Session = session()
for obj in iterator:
Session.add(obj)
Session.commit()
my_rdd.foreachPartition(add_to_db)
</code></pre></figure>
<h4 id="emr">EMR</h4>
<ul>
<li>Packaging &amp; Deploying Code
<ul>
<li>The PySpark code was organized as follows:
<ul>
<li>Use Presto to query raw logs on S3 each day for the count (<code class="language-plaintext highlighter-rouge">view_count</code>) of product views per product_id.</li>
<li>Convert the results of this query into an RDD.</li>
<li>Using Spark, map over this RDD to query DynamoDB to see if each row in the RDD has a matching record in DynamoDB. The result of this mapping function is a new RDD (named “Product RDD” in the diagram below), which now has a boolean field <code class="language-plaintext highlighter-rouge">matched</code> to indicate whether or not a matching record was found.</li>
<li>Write the contents of this new RDD to the Postgres database, an AWS RDS instance.</li>
<li>Reduce this data to the advertiser level, giving a summary of the number of matched views vs. total views by advertiser by day.</li>
</ul>
</li>
</ul>
<img src="/images/post_images/pyspark-script-overview-image.png" alt="PySpark Script Overview" title="PySpark Script Overview" />
<ul>
<li>As discussed above, in order to access self-created Python modules from within my main PySpark script, the modules had to be zipped up. My solution to this was to create a main module called <code class="language-plaintext highlighter-rouge">spark_app</code>, within which I had sub-modules <code class="language-plaintext highlighter-rouge">model</code> and <code class="language-plaintext highlighter-rouge">utils</code>. I then created a <code class="language-plaintext highlighter-rouge">deploy.py</code> script that serves to zip up the <code class="language-plaintext highlighter-rouge">spark_app</code> directory and push the zipped file to a bucket on S3.</li>
<li>Similarly, the <code class="language-plaintext highlighter-rouge">deploy.py</code> script also pushes <code class="language-plaintext highlighter-rouge">process_data.py</code> (the main PySpark script) and <code class="language-plaintext highlighter-rouge">setup.sh</code> (the bootstrap action script) to the same bucket on S3.</li>
<li>Bootstrap actions are run before your steps run. In this way, they are used to set up your cluster appropriately. For our purposes, that involved installing my necessary Python dependencies and copying <code class="language-plaintext highlighter-rouge">spark_app.zip</code> and <code class="language-plaintext highlighter-rouge">process_data.py</code> to the right directory.</li>
<li>Below is our <code class="language-plaintext highlighter-rouge">setup.sh</code> script:</li>
</ul>
</li>
</ul>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"> #!/bin/bash
# Install our dependencies - replace these libraries with your needs!
sudo yum install -y gcc python-setuptools python-devel postgresql-devel
sudo python2.7 -m pip install SQLAlchemy==1.0.8
sudo python2.7 -m pip install boto==2.38.0
sudo python2.7 -m pip install funcsigs==0.4
sudo python2.7 -m pip install pbr==1.8.1
sudo python2.7 -m pip install psycopg2==2.6.1
sudo python2.7 -m pip install six==1.10.0
sudo python2.7 -m pip install wsgiref==0.1.2
sudo python2.7 -m pip install requests[security]
# Download code from S3 and set up cluster
aws s3 cp s3://your-bucket/spark_app.zip /home/hadoop/spark_app.zip
aws s3 cp s3://your-bucket/process_data.py /home/hadoop/process_data.py
</code></pre></figure>
<ul>
<li>Running the Job
<ul>
<li>Ultimately, I chose to run this Spark application using a scheduled Lambda function on AWS (discussed below), but for the purposes of this blog post, I have included both the Lambda function which utilizes <a href="http://boto3.readthedocs.org/en/latest/reference/services/emr.html#EMR.Client.run_job_flow">boto3</a> and the equivalent cluster launch using the <code class="language-plaintext highlighter-rouge">aws cli</code>.</li>
<li>AWS Lambda Function:</li>
</ul>
</li>
</ul>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> import boto3
def lambda_handler(json_input, context):
client = boto3.client('emr', region_name='us-west-2')
client.run_job_flow(
Name='YourApp',
ReleaseLabel='emr-4.1.0',
Instances={
'MasterInstanceType': 'm3.xlarge',
'SlaveInstanceType': 'm3.xlarge',
'InstanceCount': 21,
'Ec2KeyName': 'ops',
'KeepJobFlowAliveWhenNoSteps': False,
'TerminationProtected': False,
'Ec2SubnetId': 'subnet-XXXX
},
Steps=[
{
'Name': 'YourStep',
'ActionOnFailure': 'TERMINATE_CLUSTER',
'HadoopJarStep': {
'Jar': 'command-runner.jar',
'Args': [
'spark-submit',
'--driver-memory','10G',
'--executor-memory','4G',
'--executor-cores','4',
'--num-executors','20',
'/home/hadoop/process_data.py'
]
}
},
],
BootstrapActions=[
{
'Name': 'cluster_setup',
'ScriptBootstrapAction': {
'Path': 's3://your-bucket/subfolder/setup.sh',
'Args': []
}
}
],
Applications=[
{
'Name': 'Spark'
},
],
Configurations=[
{
"Classification": "spark-env",
"Properties": {
},
"Configurations": [
{
"Classification": "export",
"Properties": {
"PYSPARK_PYTHON": "/usr/bin/python2.7",
"PYSPARK_DRIVER_PYTHON": "/usr/bin/python2.7"
},
"Configurations": [
]
}
]
},
{
"Classification": "spark-defaults",
"Properties": {
"spark.akka.frameSize": "2047"
}
}
],
VisibleToAllUsers=True,
JobFlowRole='EMR_EC2_DefaultRole',
ServiceRole='EMR_DefaultRole'
)
</code></pre></figure>
<ul>
<li>AWS CLI Equivalent:</li>
</ul>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"> aws emr create-cluster --name "YourApp" --release-label emr-4.1.0 \
--use-default-roles \
--ec2-attributes KeyName=YourKey,SubnetId=YourSubnet \
--applications Name=Spark \
--configurations https://bucket.s3.amazonaws.com/key/pyspark_py27.json \
--region us-west-2 --instance-count 21 --instance-type m3.xlarge \
--bootstrap-action Path="s3://your-bucket/path/to/script" \
--steps Type=Spark,Name="StepName",ActionOnFailure=TERMINATE_CLUSTER,\
Args=[--driver-memory,10G,\
--executor-memory,4G,\
--executor-cores,4,\
--num-executors,20,\
/home/hadoop/process_data.py]
</code></pre></figure>
<ul>
<li>Notes on Parameters:
<ul>
<li>Configurations
<ul>
<li>Because we were also accessing data from Presto within the main Pyspark script, it was necessary that all of our machines were using Python 2.7 as the default (Hive/Presto requirement). This involved configuring <code class="language-plaintext highlighter-rouge">PYSPARK_PYTHON</code> and <code class="language-plaintext highlighter-rouge">PYSPARK_DRIVER_PYTHON</code> environment variables.</li>
<li>Additionally, we needed to add a special configuration to increase the <code class="language-plaintext highlighter-rouge">spark.akka.frameSize</code> parameter. The maximum frame size is 2047, so that is what we have set it to. This may be unnecessary for some applications, but if you receive an error about frameSize, you will know that you need to add in a configuration for this parameter.</li>
<li>Note: the <code class="language-plaintext highlighter-rouge">--configurations</code> parameter in the AWS CLI example simply provides a url to a json file stored on S3. That file should contain the json blob from <code class="language-plaintext highlighter-rouge">Configurations</code> in the boto3 example above.</li>
</ul>
</li>
<li>Steps
<ul>
<li>Within the Spark step, you can pass in Spark parameters to configure the job to meet your needs. In our case, I needed to increase both the driver and executor memory parameters, along with specifying the number of cores to use on each executor. We also found that we needed to explicitly stipulate that Spark use all 20 executors we had provisioned.</li>
<li>The final argument in the list is the path to the main PySpark script (<code class="language-plaintext highlighter-rouge">process_data.py</code>) that runs the actual data processing job.</li>
</ul>
</li>
<li>Applications
<ul>
<li>You must explicitly specify that you intend to run a Spark application.</li>
</ul>
</li>
</ul>
</li>
<li>Scheduling the Job
<ul>
<li>With the <a href="https://aws.amazon.com/blogs/aws/aws-lambda-update-python-vpc-increased-function-duration-scheduling-and-more/">recent announcement</a> of Amazon’s newly available ‘Scheduled Event’ feature within Lambda, we were able to very easily set up a recurring cluster launch on a cron schedule.</li>
<li>As shown above, the code to launch the cluster was extremely simple, leveraging the power of boto3’s <code class="language-plaintext highlighter-rouge">run_job_flow</code> method.</li>
<li>The only wrinkle here was properly configuring the role for the lambda function, so that it would have the right permissions to launch an EMR cluster. It was necessary that the action “elasticmapreduce:RunJobFlow” had the “iam:PassRole” role associated with it.</li>
<li>Below is the policy for the role:</li>
</ul>
</li>
</ul>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"> {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"elasticmapreduce:RunJobFlow",
"iam:PassRole"
],
"Resource": "*"
}
]
}
</code></pre></figure>
<h4 id="closing-thoughts">Closing Thoughts</h4>
While there were several hurdles to overcome in order to get this PySpark application running smoothly on EMR, we are now extremely happy with the successful and smooth operation of the daily job. I hope this guide has been helpful for future PySpark and EMR users.
Finally, after taking a one-day Spark course, I have realized that for performance reasons, it likely makes sense to use Spark’s dataframes instead of RDDs for some parts of this job. Stay tuned for a future post on transitioning to dataframes and (hopefully) resulting performance gains!
</description>
</item>
<item>
<title>gulp-react-docs: From propTypes to Markdown in 3 seconds</title>
<link>https://tech.nextroll.com/blog/frontend/2015/12/21/gulp-react-docs.html</link>
<pubDate>Mon, 21 Dec 2015 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/frontend/2015/12/21/gulp-react-docs</guid>
<description>In a <a href="http://tech.adroll.com/blog/frontend/2015/11/12/rollup-react-and-npm-at-adroll.html">blog post a couple weeks ago</a> we talked about the developer tools we built for Rollup, our UI component library. One of those tools is responsible for <a href="http://tech.adroll.com/blog/frontend/2015/11/12/rollup-react-and-npm-at-adroll.html#automatic-documentation-generation">automatically generating documentation</a> for React components and we just released it as an open-source <a href="http://gulpjs.com/">gulp</a> plugin called <a href="https://www.npmjs.com/package/gulp-react-docs">gulp-react-docs</a>.
When we started working on Rollup, we knew that we needed to make it easy for application developers to figure out what <code class="language-plaintext highlighter-rouge">props</code> a component expects. To us that meant providing documentation so that application developers wouldn’t need to look for the entry <code class="language-plaintext highlighter-rouge">.jsx</code> file and then examine the <code class="language-plaintext highlighter-rouge">propTypes</code>. We first looked for packages that we could run our <code class="language-plaintext highlighter-rouge">.jsx</code> files through that would then output Markdown based on the component comments and props, but couldn’t find any that did just that. So we decided to build our own documentation generator based on <a href="https://www.npmjs.com/package/react-docgen">react-docgen</a>.
<code class="language-plaintext highlighter-rouge">react-docgen</code> parses React component files into an <a href="https://en.wikipedia.org/wiki/Abstract_syntax_tree">AST</a> in JSON format. We then take the output and pipe it through <a href="https://github.com/AdRoll/gulp-react-docs/blob/master/src/react-docgen-md.js">some clever Handlebars templates</a> to produce a Markdown document for each React component. This process is encapsulated in our <code class="language-plaintext highlighter-rouge">gulp-react-docs</code> plugin. We decided to implement our documentation generator as a <code class="language-plaintext highlighter-rouge">gulp</code> plugin to make it easy to generate documentation for multiple components at a time.
So, using <code class="language-plaintext highlighter-rouge">gulp-react-docs</code> we do something like this in our <code class="language-plaintext highlighter-rouge">gulpfile.js</code>:
<figure class="highlight"><pre><code class="language-js" data-lang="js">var gulpReactDocs = require('gulp-react-docs');
gulp.task('docs', function() {
var docsDest = 'docs';
return gulp.src('./components/**/*.jsx')
.pipe(gulpReactDocs({
// the `path` option is used for backlinking the
// output documentation to the source code / file
path: docsDest
}))
.pipe(gulp.dest(docsDest));
});</code></pre></figure>
To take components with inline documentation that looks like this:
<figure class="highlight"><pre><code class="language-js" data-lang="js">import React from 'react';
let DataTable = React.createClass({
propTypes: {
/**
* The columns you want the data table to have. Each column can
* have the following attributes:
* - `key` **(required)**: column identifier
* - `label` **(required)**: Display text for the column. Should
* already be translated when passed to the DataTable.
* - `accessor` **(required)**: Function that returns the
* relevant value from a given data item. Later passed to
* the column `render` function.
* - `render` **(required)**: Function that takes the output of
* the `accessor` and returns what should be rendered for a
* given data item in that column. Should return either a
* formatted value or can also be html. Columns without
* `render` functions will not be displayed but can be used
* for filtering (see the `filters` prop for more information).
* - `textAlignment`: Column is center-aligned by default. Use
* `DataTable.TEXT_ALIGN_LEFT` or `DataTable.TEXT_ALIGN_RIGHT`
* to override the center alignment.
* - `widthMultiplier`: Number to multiply the width of the column
* relative to other columns. By default, columns are of equal
* width.
* - `adminOnly`: Whether or not this is an admin-only or
* permission-gated column. `adminOnly` columns will only be
* shown if the table's `isAdmin` prop is `true`.
*/
columns: React.PropTypes.arrayOf(
React.PropTypes.shape({
key: React.PropTypes.string.isRequired,
label: React.PropTypes.string.isRequired,
accessor: React.PropTypes.func.isRequired,
render: React.PropTypes.func,
textAlignment: React.PropTypes.oneOf([
TEXT_ALIGN_LEFT,
TEXT_ALIGN_RIGHT
]),
widthMultiplier: React.PropTypes.number,
adminOnly: React.PropTypes.bool
})
),
// ... more documented props
}
});
export default DataTable;</code></pre></figure>
And output a markdown file that looks like this:
<center>
<img alt="Rollup Data Table prop documentation" src="/images/post_images/gulp_react_docs_example_output.png" />
</center>
For more information on usage see the <a href="https://www.npmjs.com/package/gulp-react-docs"><code class="language-plaintext highlighter-rouge">gulp-react-docs</code> plugin page on npm</a> or our <a href="https://github.com/AdRoll/gulp-react-docs/tree/master/example">example gulpfile on GitHub</a>. Thanks for reading and hope you find this plugin useful!
</description>
</item>
<item>
<title>Data Science Event Processing</title>
<link>https://tech.nextroll.com/blog/data-science/2015/12/08/data-science_event_processing.html</link>
<pubDate>Tue, 08 Dec 2015 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data-science/2015/12/08/data-science_event_processing</guid>
<description>AdRoll uses large-scale machine learning to bid intelligently in internet advertising auctions.
In this post, we explore some of the engineering behind the data pipelines that feed our learning algorithms,
in particular our real-time system that constructs features from event streams.
<hr />
#Overview
If you have been served a banner ad on the internet in the past few years, there is a good chance
that ad was served through a process known as real-time bidding (RTB). As its name suggests, in RTB-based
ad serving, there is an auction held for the ad space on a web page in real-time every
time a page is loaded. AdRoll bids in these auctions on behalf of our clients; ad exchanges
send us a bid request whenever a page is loading, and we respond to these requests with a bid price. There are
many engaging engineering and data science problems that arise in the context of participating in RTB
auctions. On AdRoll’s data science team, we mainly concern ourselves with the problem of determining the
value of impressions intelligently in real-time.
At a high level, the solution to this problem is essentially a large-scale machine learning product, BidIQ.
We learn the probability that particular impressions will click and convert based on historical features
available at bid time, use our models to forecast this probability for incoming
bid requests, and then bid as some function of these probabilities and other variables. If you’re interested in our
modeling and learning algorithms, check out Matt Wilson’s post on <a href="http://tech.adroll.com/blog/data-science/2015/08/25/factorization-machines.html" title="Factorization Machines">Factorization Machines</a>.
There are many features that we feed into our learning algorithm. One major source is those derived from the bid
request. These include features such as geo-location, ad network, inventory source, the customer that we would
serve this impression to, time, among many others. Another source of very valuable information is features related
to the cookie that this impression would be served to, such as past behavior of that user. Enter our event-processing system, internally dubbed chocolate chip (cookies!), which computes such features and makes them available at
bid time.
<hr />
#Chocolate Chip
One of the challenges of using data on individual cookies (profiles) in real-time is that, unless the profile
is stored in the cookie, it is not available in a bid request. This necessitates the use of a key-value store
mapping cookies to profiles of features. There are costs and benefits to both approaches. Storing data in the
cookie guarantees truly real-time data, whereas key-value stores allows greater flexiblity with offline testing
and data munging. We ultimately decided on the latter approach. For our backend key-value store,
we use AWS DynamoDB, which we have used with great success for similar use cases in the past.
At its core, chocolate chip is a system that takes events, processes them, and then writes profiles to
DynamoDB. Although originally designed and built by the data science team, it has found other uses throughout
AdRoll. Our use case on the data science team is to build predictive features from events in data streams and write
these profiles of features in as close to real-time as possible. These profiles are then fetched by our bidders
and fed into our machine learning model to produce more accurate click and conversion predictions.
In order to learn on these data, the profile at the time of the bid is logged. Chocolate chip was built from
the ground up in <a href="http://dlang.org" title="D Programming Language">D</a>, including our own high-performance DynamoDB client.
The high-level concept of building a profile is simple: read the current value of the profile in DynamoDB,
update this profile with information on the current event, and write this profile back. This, however, is
complicated due to the fact that in order to achieve the scale and performance needed at AdRoll, the system is
highly parallelised and supports several methods by which to distribute the computation. Additionally, we
run computations in mini-batches to reduce the number of HTTP requests and increase throughput.
<hr />
#Architecture
Chocolate chip ingests multiple streams of data, each stream comprised of a distinct event type. Each event
is in the form of an AdRoll log line, a unified log line format over all our event types. The cookie is then
handed off to one of several worker threads based on the cookie hash and event type. One of the main
considerations when dealing with concurrent read and writes is always going to be avoiding data races. In
particular, we never want more than one concurrent read-build-write cycle per cookie to DynamoDB, otherwise the
events processed by one of the cycles will be overwritten by the other. By having each worker thread only deal
with a partition of the cookie space, we eliminate this race condition. This race is exacerbated for us because
cookies tend to be active at certain times, and therefore have events close together in the event stream. If we
did not break up by cookies, it is likely multiple workers could deal with events from the same cookie
concurrently. Although sharding by cookie does scale over several nodes, it is not our primary form of sharding
as each of the data streams themselves are not broken up by cookie, and hence would have to be replicated.
When scaling out chocolate chip, the primary mechanism is to shard by event type. This exploits DynamoDB
hash-and-range key type, with the primary key being the cookie hash and the range key being the event type. Using
this property, we can break out separate nodes to deal with distinct event streams and build profiles for
different event types on separate subrange keys. Our input event streams do not guarantee
event ordering, so any features we build from events must be commutative. With the hash-and-range key method,
we’re also restricted to associative events. This basically means we can only build features which, even
if partially constructed from separate events stream, can be reconstructed to their true value. While this adds
some complexity at bid time, we have yet to find a feature we’d like to build that is commutative but not
associative, so sharding out the building of features by event has not impacted what features we construct.
Once a worker thread reads an event from its queue, it stores this event until it has events accumulated from
several distinct cookies. DynamoDB permits batch read and write requests, the latter having a batch size of 25, so
this is the size we use. Once 25 distinct cookies have been registered (perhaps with multiple events per cookie),
a batch read request is made to the master region, the profiles returned are updated or generated if nonexistent,
and then written back and replicated to slave regions when there is available capacity.
<hr />
#Example feature: Exponential Moving Average
One can imagine that for some event types, recent events are more valuable for learning than past ones.
A mechanism we use to capture such information is the exponential moving average. To understand
why this type of feature is used, let us first examine the simple moving average. Say we care about the amount
of ads we have shown to a particular cookie in the past hour (such a feature could be phrased as a simple moving
average). In order to calulate this feature, we would have to store the timestamps of all the impressions shown,
and count those in the past hour at bid time. The generation of the feature can lazily eliminate events over an
hour old, but the key here is that some information about all the events must be stored.
In the low-latency environment under which this system operates, the memory costs of this feature are prohibitive.
At bid time, we must fetch the the profile from DynamoDB over a TCP connection and the latency of fetching the
feature is proportional to the number of TCP packets needed to communicate the profile. If we store many
features like this, the number of packets needed and, therefore, latency would be unacceptable.
This is where the exponential moving average (EMA) comes in. EMAs can store a notion of number of events that
occurred recently in a fixed amount of memory.
The formal defintion of an EMA is:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{equation}
\hat{X}_{n,\alpha}\, = (1-\alpha) \sum_{i=1}^n \alpha^{n-i}x_i
\end{equation}
%]]>
</script>
With a little math, we can show that this is equivalent to following recursive definition:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{equation}
\hat{X}_{0,\alpha}\, = 0
\end{equation}
%]]>
</script>
<script type="math/tex; mode=display">
%<![CDATA[
\begin{equation}
\hat{X}_{n+1,\alpha}\, = (1-\alpha)x_i + \alpha \hat{X}_{n,\alpha}
\end{equation}
%]]>
</script>
And there we have it. All we need to calculate the current value of the EMA is that last value, the timestamp,
and the current timestamp. An observant reader will also note that we can add in events out of order to the EMA,
as we just need to apply an exponential decay factor to the event. We can build secondary features for free
by using the EMA feature; we have to store the timestamp of the last event, so we can extract this and build other
features out of it.
In our case, applying this to impression count, we can think of each passing second as an observation, which
takes the value 1 if an impressions occured in that second, or a 0 if not. Then we can say that:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{equation}
\hat{X}_{T_{n+1},\alpha}\, = (1-\alpha) + \alpha^{T_{n+1} - T_n} \hat{X}_{T_n,\alpha}
\end{equation}
%]]>
</script>
where
<script type="math/tex; mode=display">
%&lt;![CDATA[
\begin{equation}
T_i
\end{equation}
%]]&gt;
</script>
refers to the timestamp of the ith event.
The EMA described is essentially just a counter that decays over time. While we cannot guarantee the count of
events in the past hour, we get a feature which takes a high value when there are a large number of recent events,
and a low value when there are not.
<hr />
#Performance considerations in D
As mentioned earlier, this system was built from the ground up in the <a href="http://dlang.org" title="D Programming Language">D programming language</a>. Rather than plugging
a <a href="http://storm.apache.org" title="Apache Storm">storm bolt</a> into our existing data structure, this allows code sharing with our existing machine learning
infrastructure, and our already-developed in-house D expertise. An added benefit is our ability to
optimize all the areas of our stack, rather than just the feature generation layer.
With systems such as these, memory management is also going to be a major factor in performance. We found when
we profiled our code that the primary source of contention on a single machine was the global
lock that occurs around everything in D’s GC allocator. By restructuring our code to mostly use preallocated
buffers for computation, and by using a specialized JSON serializer and parser, we are able to achieve single
machine throughput of around 20,000 events per second on a 32 hyperthread instance. Another optimization we found
was that D’s GC seems to have poor heuristics for when to enter a GC cycle for this application. By disabling
automatic GC collections and running GC.collect() and GC.minimize() according to our own heuristics, we are able
to achieve 2-3x higher throughput. The runtime in D has been getting much love in recent releases, and we hope
that such measures will not be necessary in the future.
Ultimately, we didn’t really have to do anything fancy or micro-optimize to get solid performance out of the
system. A significant advantage of this solid performance is that we have not had to build chocolate chip as a
truly distributed system, and as such, we’ve had very good uptime and low maintenance costs. As a side note, for
performance reasons, we also use unique memory management techniques in our prediction server for real-time ad
pricing. That is a truly real-time system with millisecond SLAs and cannot afford any GC cycles.
<hr />
Chocolate chip has added tremendous predictive power and accuracy to our learning algorithms and has
drastically improved how intelligently we bid on exchanges. In the recent release of an EMA feature as outlined
above, we found cost-per-click to drop by 2% with no effect on reach.
</description>
</item>
<item>
<title>Rollup: What we have learned from sharing UI code at AdRoll</title>
<link>https://tech.nextroll.com/blog/frontend/2015/11/19/rollup-major-learnings.html</link>
<pubDate>Thu, 19 Nov 2015 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/frontend/2015/11/19/rollup-major-learnings</guid>
<description>This is the third post in a series of three blog posts about Rollup, AdRoll’s UI component library. This post covers what we learned from building a UI component library. For details on why we built the UI component library see the <a href="/blog/frontend/2015/11/05/rollup-shared-ui-components.html">first post</a> in the series and for how we built it see the <a href="/blog/frontend/2015/11/12/rollup-react-and-npm-at-adroll.html">second post</a>.
At end of Rollup component and developer tool development, we reflected on what we did and realized that we had learned four important lessons from the work:
<ul>
<li><a href="#making-reusable-components-for-app-developers-is-hard">Making reusable components for app developers is hard</a></li>
<li><a href="#making-reusable-components-for-contributors-is-hard">Making reusable components for contributors is hard</a></li>
<li><a href="#interfaces-are-hard">Interfaces are hard</a></li>
<li><a href="#making-everything-look-the-same-is-hard">Making everything look the same is hard</a></li>
</ul>
Let’s dig into each one separately.
<h2 id="making-reusable-components-for-app-developers-is-hard">Making reusable components for app developers is hard</h2>
When we first started working on Rollup, we wanted to make sure that individual components would be easy to use for any developer on any team. For app developers, we wanted them to be able to use Rollup components even if:
<ul>
<li>they’re new to JavaScript</li>
<li>they don’t use React in their application</li>
<li>they don’t have fancy build tools, e.g. <a href="gulpjs.com">Gulp</a> or <a href="http://browserify.org/">Browserify</a></li>
<li>they want to use the same version of a component forever</li>
<li>we never thought about their use case</li>
</ul>
To address the first three points, each published component has a CDN build. The CDN build, among other things, contains transpiled code. The transpiled JS means that other teams do not need to have a fancy build tool to use a component in their project and that app developers do not need to use <a href="https://facebook.github.io/react/docs/jsx-in-depth.html">JSX</a> in order to add the components to their application.
Not only does this mean that components can be used in projects not built in <a href="https://facebook.github.io/react/">React</a>, but also makes it easier to use for those who are new to JavaScript. The CDN build accomplishes this by allowing them to use whichever framework they feel most comfortable with (e.g. jQuery or plain JavaScript).
To allow app developers to use the same version of a component forever, components are published to <code class="language-plaintext highlighter-rouge">npm</code> according to <a href="http://semver.org/">SemVer</a> and the CDN assets are pushed to S3 under a versioned URL. Other applications using <code class="language-plaintext highlighter-rouge">npm</code> can leverage <code class="language-plaintext highlighter-rouge">npm</code>’s dependency management to use the same version forever. While other app developers loading components via the CDN can always load the same version through the versioned URL.
We also want app developers to be able to use the components even if we had not thought about their use case. We can accomplish this in two ways. The first way is to provide well defined and general interfaces for the components (see <a href="#interfaces-are-hard">interfaces are hard</a> for more details).
The second way we want to accomplish is through open communication channels. Even if interfaces are well-defined and general, we want to be able to customize and iterate on them quickly. In order to accomplish this we have created a <a href="https://slack.com/">Slack</a> channel, #frontend_helpdesk, dedicated to frontend and Rollup help. We track all bugs and feature requests received there using GitHub issues.
By keeping the lines of communications open and judiciously tracking feature requests and bug fixes, we aim to quickly address new use cases. The issue tracking allows us to track and implement the needed fixes ourselves, and serves as a place where we can foster further conversation on how to support a new use case.
<h2 id="making-reusable-components-for-contributors-is-hard">Making reusable components for contributors is hard</h2>
The developer tools we put in place surrounding the components aim to make it easier to contribute to the component library. The goal of the tooling was to enable Rollup contributors to:
<ul>
<li><a href="#focus-on-the-important-things-when-developing-a-component">focus on the important things when developing a component</a></li>
<li><a href="#focus-on-the-important-things-when-reviewing-a-component">focus on the important things when reviewing a component</a></li>
</ul>
<h4 id="focus-on-the-important-things-when-developing-a-component">Focus on the important things when developing a component</h4>
To develop and publish a new component, contributors should not have to worry about boilerplate like build configuration and file structure. To that end, we built a Rollup component generator using <a href="http://yeoman.io/generators/">Yeoman</a>. The generator takes the name of the component and then creates all files needed for a <a href="https://github.com/AdRoll/react-component-skeleton/tree/9a6f1e7df1965b342066ca029cc8cce0a3f41295">blank component</a>. After running the generator, contributors can just hit the ground running.
Each component has an <a href="https://github.com/AdRoll/react-component-skeleton/tree/9a6f1e7df1965b342066ca029cc8cce0a3f41295/examples">examples/</a> directory and a Gulp task for watching the changes made to the component and reloading the example in the browser. This makes it easy for contributors to test the functionality that they are working on in an isolated development and removes the hassle of having to set up a test application to interact with the component.
<h4 id="focus-on-the-important-things-when-reviewing-a-component">Focus on the important things when reviewing a component</h4>
We also want reviewers of pull requests in the Rollup repo to be able to focus on the important things when looking over someone else’s work. To that end, we have set up a global linter and <a href="https://facebook.github.io/jest/">Jest</a> tests. The linting and test suite are automatically run on every single PR by <a href="https://jenkins-ci.org/">Jenkins</a>. The reviewer no longer needs to remember to do these checks, since it is done for them.
Instead, the reviewer can focus on the code quality and changes to component interfaces. In addition and thanks to each component’s <code class="language-plaintext highlighter-rouge">examples/</code> directory, reviewers can also interact with the component and verify functionality, just like the component’s author(s) can.
<h2 id="interfaces-are-hard">Interfaces are hard</h2>
When starting work on Rollup and trying to use these components in other applications, we quickly realized that it mattered how easy it was to integrate a component. Enter the interface. Here are some of the best practices we learned while building them:
<ul>
<li>Limit the number of files that need to be included to get a component to work. One JS and one SCSS file is ideal.</li>
<li>Make sure component and <a href="https://facebook.github.io/react/docs/reusable-components.html">React prop</a> names make sense. For example, a click handler should have the word <code class="language-plaintext highlighter-rouge">click</code> in it somewhere.</li>
<li><code class="language-plaintext highlighter-rouge">prop</code> definitions that allow develops to achieve the most types of interactions with the least number of props.</li>
</ul>
The first point hopes to limit the pain for app developers integrating components into their applications.
The second and third points together form the essence of what we learned when designing component interfaces. The idea behind these two points is to encourage contributors to limit the number of <a href="https://facebook.github.io/react/docs/reusable-components.html">React props</a> each component needs to be minimally functional, but be careful to define each prop in such a way that also supports more complicated use cases at the same time.
To help explain what we mean by that, let’s take a look at the <code class="language-plaintext highlighter-rouge">columns</code> <code class="language-plaintext highlighter-rouge">prop</code> for our data table component:
<figure class="highlight"><pre><code class="language-js" data-lang="js">columns: [{
key: String
label: String
accessor: Function
render: Function
textAlignment: (TEXT_ALIGN_LEFT|TEXT_ALIGN_RIGHT)
widthMultiplier: Number
adminOnly: Boolean
}, ...]</code></pre></figure>
Let’s take a look at the <code class="language-plaintext highlighter-rouge">accessor</code> and <code class="language-plaintext highlighter-rouge">render</code> attributes that each column can have. In addition to the <code class="language-plaintext highlighter-rouge">columns</code> <code class="language-plaintext highlighter-rouge">prop</code>, the table also expects an array of <code class="language-plaintext highlighter-rouge">data</code>. The <code class="language-plaintext highlighter-rouge">accessor</code> in the <code class="language-plaintext highlighter-rouge">columns</code> <code class="language-plaintext highlighter-rouge">prop</code> is a function, that when given a data item will return a value that is then fed to the <code class="language-plaintext highlighter-rouge">render</code> function for that same column.
Together, the <code class="language-plaintext highlighter-rouge">accessor</code> and <code class="language-plaintext highlighter-rouge">render</code> support the simple use case of simply displaying information for a given data item without the data table component having to know the internal organization of each item.
However, because the <code class="language-plaintext highlighter-rouge">accessor</code> and <code class="language-plaintext highlighter-rouge">render</code> attributes are functions, they also support more complicated use cases. For example, these two attributes on a given column can be used to render line charts that summarize data for a particular data item. To do that, the <code class="language-plaintext highlighter-rouge">accessor</code> could return an array of data points, while the <code class="language-plaintext highlighter-rouge">render</code> can return anything as long as <a href="https://facebook.github.io/react/">React</a> can render it. Allowing the app developer using the component to return JSX or a React element for the line chart. For example:
<center>
<img alt="Rollup Data Table complicated use case" src="/images/post_images/rollup_data_table_use_case.png" />
</center>
In the above example, the <code class="language-plaintext highlighter-rouge">accessor</code> and <code class="language-plaintext highlighter-rouge">render</code> functions are general enough to support straightforward uses of the data table and more complicated ones as well.
<h2 id="making-everything-look-the-same-is-hard">Making everything look the same is hard</h2>
Making things look the same across products and components developed in isolation is not an easy feat. For example, the blue used in the date picker needs to be the same blue that’s used in our data table. As another example, the top navbar needs to be pixel perfect across products, so that the end user feels like they are navigating within the same application, even if that’s not how the product is implemented.
To solve this problem, we came up with a Rollup component that maintains styles shared by applications and components. This component is called <code class="language-plaintext highlighter-rouge">ar-style-base</code>, and all common styles are built into it:
<ul>
<li>Common colors are built in as <a href="http://sass-lang.com/">Sass</a> variables</li>
<li>Common icons and typography are also defined for components and different products</li>
<li>We also include bootstrap customizations in <code class="language-plaintext highlighter-rouge">ar-style-base</code> since many of our UI components use <a href="http://react-bootstrap.github.io/">React Bootstrap</a></li>
</ul>
Even though the base styles can be overridden, they provide a good starting point for new components and products that need to share colors, typography, and iconography.
<h2 id="so-whats-next">So what’s next?</h2>
We continue to encourage the adoption of our components at AdRoll and share the things we have learned. Not only will sharing internally the things we’ve learned help grow our contributor pool, but it will also help grow the frontend expertise of the company as a whole.
As we mentioned at the end of the <a href="/blog/frontend/2015/11/05/rollup-shared-ui-components.html">first blog post in this series</a>, we decided to build our own components since we were not happy with open-source solutions. So, the next big thing we have in mind for Rollup is to open source them one at a time.
Thanks for reading this series and we hope you have enjoyed what we have to share about Rollup!
</description>
</item>
<item>
<title>
Build a simple distributed system using AWS Lambda, Python, and DynamoDB
</title>
<link>https://tech.nextroll.com/blog/dev/2015/11/16/count-things-with-aws-lambda-python-and-dynamodb.html</link>
<pubDate>Mon, 16 Nov 2015 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2015/11/16/count-things-with-aws-lambda-python-and-dynamodb</guid>
<description><style type="text/css">/*<![CDATA[*/
.highlight code { background-color: #272822; }
.highlight code.sh .nb { color: #E6DB74; }
.highlight code.sh .nv { color: #4DFFFF; }
.highlight code.sh .s2 { color: #FDA533; }
.highlight code.sh .s1 { color: #FFA52F; }
.highlight code.sh .s { color: #FBE72E; }
.highlight code.sh .k { color: #61FF70; }
.highlight code.sh .c { color: #FED633; }
.highlight code.sh .o { color: #5CEA26; }
.highlight code.json .nt { color: #30F926; }
.highlight code.json .mi { color: #00FDFD; }
.highlight code.json .s2 { color: #FDA533; }
.highlight code.json .kc { color: #FFFC49; }
.highlight code.json .p { color: #AE81FF; }
.highlight code.python { background-color: #315050; }
.highlight code.python .c { color: #FED633; }
.highlight code.python .n { color: #CBBA77; }
.highlight code.python .o { color: #CBBA77; }
.highlight code.python .p { color: #CBBA77; }
.highlight code.python .s { color: #FD9226; }
.highlight code.python .nn { color: #CBBA77; }
.highlight code.python .kn { color: #4DFFFF; }
.highlight code.python .k { color: #4DFFFF; }
.highlight code.python .ow { color: #2DFFFE; }
.highlight code.python .nb { color: #29F89D; }
.highlight code.python .nf { color: #29F89D; }
.highlight code.python .bp { color: #FEA07E; }
.highlight code.python .mi { color: #CBBA77; }
table.cost-comparison thead { background-color: #BDE4F3; }
table.cost-comparison tbody { background-color: #D7F2FB; }
table.cost-comparison tbody tr:nth-child(even) { background-color: #D0F0F0; }
table.cost-comparison td, th { padding-right: 0.5em; padding-left: 0.5em; }
table.cost-comparison td ~ td, th ~ th { padding-left: 1em; }
table.cost-summary td, th { padding-right: 0.5em; padding-left: 0.5em; }
table.cost-summary td ~ td, th ~ th { padding-left: 1em; }
table.cost-summary td:nth-child(1) { background-color: #BDE4F3; font-weight: bold; }
table.cost-summary td:nth-child(1)::after { content: ":"; }
table.cost-summary td:nth-child(2) { background-color: #D7F2FB; }
table.math-legend td, th { padding-right: 0.5em; padding-left: 0.5em; }
table.math-legend td ~ td, th ~ th { padding-left: 1em; }
table.math-legend td:nth-child(1) { background-color: #BDE4F3; font-weight: bold; }
table.math-legend td:nth-child(1)::after { content: ":"; }
table.math-legend td:nth-child(2) { background-color: #D7F2FB; }
table.math-legend { margin-top: -1em; margin-left: auto; margin-right: auto; margin-bottom: 1em; }
/*]]>*/</style>
We have implemented a number of systems in support of our <a href="http://www.erlang.org/">Erlang</a>-based
real-time bidding platform. One of these is a <a href="http://www.celeryproject.org/">Celery</a> task system which runs
code implemented in Python on a set of worker instances running on <a href="https://aws.amazon.com/ec2/">Amazon
EC2</a>.
With the <a href="https://aws.amazon.com/blogs/aws/aws-lambda-update-python-vpc-increased-function-duration-scheduling-and-more/">recent announcement</a> of built-in support for Python
in <a href="http://aws.amazon.com/lambda/">AWS Lambda</a> functions (and upcoming access to VPC resources from
Lambda), we’ve started considering increased use of Lambda for a number of applications.
In this post, we’ll present a complete example of a data aggregation system using
Python-based Lambda functions, S3 events, and DynamoDB triggers; and configured using
the AWS command-line tools (<code class="language-plaintext highlighter-rouge">awscli</code>) wherever possible. (Note: Some of these steps
are handled automatically when using the AWS console.)
Here’s how our completed system will look:
<img src="/images/post_images/lambda-counter-aggregator-overview.png" alt="Lambda-based counter aggregation system overview, image made with draw.io" />
<h2 id="why">Why?</h2>
Each of our EC2 instances participating in RTB maintains a set of counter values: these
represent the health of an important aspect of our platform. Each instance reports
these values using Kinesis, and also periodically uploads its set of counters to a
per-instance key on S3 as part of a failsafe system. When we aggregate these counter
values, we have greater insight into whether our system as a whole is healthy (and we
can take action automatically when it isn’t). This post will focus on a system which
can process the data we upload to S3.
Here’s what we actually want to do with our per-instance counters:
<ol>
<li>
Have an up-to-date global view of total summed counter values across all instances.
</li>
<li>
Take action whenever the summed counter values change.
</li>
</ol>
<h3 id="current-solution">Current solution</h3>
One of our Celery tasks (A) implements counter aggregation, with the output consumed
by a periodic downstream task (B) implementing our business logic. Every few minutes,
task A does the following:
<ol>
<li>
Scans an S3 bucket prefix for keys having a certain naming convention. (Each of
these keys represents counter data uploaded by a single EC2 instance, and each key name
includes the current date and the ID of the instance which uploaded it. An instance
usually writes to the same key.)
</li>
<li>
Reads the contents of all these keys, which have a format like the following:
<pre>
COUNTER1 VALUE1
COUNTER2 VALUE2
COUNTER3 VALUE3</pre>
</li>
<li>
Sums all the values for each counter. (The same counter can appear in multiple
files.)
</li>
<li>
Writes the summed counter values to a single S3 key for use in task B, which
can then take action on the aggregate counter values.
</li>
</ol>
<h3 id="drawbacks-of-current-solution">Drawbacks of current solution</h3>
We’re essentially polling S3 to detect file changes, and we’re also reprocessing data
which has not necessarily changed. This is time-consuming and ties up resources which
could have been allocated elsewhere.
It also would be nice to react more quickly to updated counter values. With our current
implementation, there can be significant lag between a counter data upload and actually
observing its contribution to the global aggregate value.
<h2 id="enter-lambda">Enter Lambda</h2>
Lambda promises scalable, serverless, event-based code execution with granular billing.
This is a compelling value proposition for the following reasons:
<ul>
<li>
Standalone code bundles are the unit of deployment;
</li>
<li>
We can run our code while only paying for the time spent actually executing it instead
of spending money on idle CPU time;
</li>
<li>
We can reduce the peak load experienced by our task system (by migrating certain
frequent and spiky tasks to Lambda), allowing us to reduce the amount of resources
allocated to it while increasing utilization of the remaining resources;
</li>
<li>
Our counter system will perform its work in a more timely manner by responding to
events instead of scanning S3, and it will avoid reprocessing data which has not
changed.
</li>
</ul>
Using Lambda is now much easier thanks to the recently-announced built-in support for
Python. (Note: it has always been possible to run Python-based code, but now it’s
easier to get started.)
<h2 id="system-design">System design</h2>
We want a system which can react to S3 events, process and aggregate data associated
with those events, and eventually convey some processed data back to S3 after executing
some business logic.
<h3 id="cost-comparison">Cost comparison</h3>
We can do all of this using Lambda, but should we? Let’s do a rough cost estimate
assuming we have 100 instances each uploading to S3 a file containing 100 counters every
60 seconds.
<h4 id="original-system">Original system</h4>
Suppose (1) we created our task system only to support <a href="#current-solution">our two tasks A and
B</a>, (2) that we scan S3 every five minutes, (3) that task A
completes in 60 seconds, and (4) that we deploy four times.
We might then have the following costs:
<table class="cost-comparison">
<thead>
<tr>
<th style="text-align: right">Description</th>
<th style="text-align: left">Cost</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right">task scheduler (t2.micro)</td>
<td style="text-align: left">$0.013 / hour</td>
</tr>
<tr>
<td style="text-align: right">task worker (m3.medium)</td>
<td style="text-align: left">$0.067 / hour</td>
</tr>
<tr>
<td style="text-align: right">S3 usage</td>
<td style="text-align: left">negligible</td>
</tr>
<tr>
<td style="text-align: right">design/configure/test/deploy task infrastructure</td>
<td style="text-align: left">16 hours</td>
</tr>
<tr>
<td style="text-align: right">design/test this task</td>
<td style="text-align: left">4 hours</td>
</tr>
<tr>
<td style="text-align: right">deploy this task</td>
<td style="text-align: left">0.5 hours / deploy</td>
</tr>
<tr>
<td style="text-align: right">infrastructure maintenance</td>
<td style="text-align: left">1 hour / month</td>
</tr>
</tbody>
</table>
<table class="cost-summary">
<thead>
<tr>
<th style="text-align: right"> </th>
<th style="text-align: left"> </th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right">Initial developer time</td>
<td style="text-align: left">22 hours</td>
</tr>
<tr>
<td style="text-align: right">Ongoing developer time</td>
<td style="text-align: left">12 hours / year</td>
</tr>
<tr>
<td style="text-align: right">Recurring costs</td>
<td style="text-align: left">$701 / year</td>
</tr>
<tr>
<td style="text-align: right">Reaction time</td>
<td style="text-align: left">1 to 6 minutes</td>
</tr>
</tbody>
</table>
<h4 id="lambda-based-system">Lambda-based system</h4>
We’ll assume the following for our Lambda-based implementation:
<ul>
<li>
100 files are uploaded to S3 every 60 seconds, each containing 100 counters. We want
all data to be processed within 50 seconds.
</li>
<li>
We’ll write counter values to a DynamoDB table. To support 10,000 writes over 50
seconds, we’ll need 200 units of provisioned write capacity. We assume our library
(<a href="https://github.com/boto/boto3">boto3</a>) will handle retries and spread out our writes when we exceed our
provisioned write capacity and burst limits.
</li>
<li>
To support the implementation described later in this post, we’ll have a second table
which is updated as a result of each change made to the first. This second table
therefore will also need 200 units of provisioned write capacity. It will be updated
by a second Lambda function (which we assume has the same characteristics as the
first) triggered by events on a DynamoDB stream.
</li>
<li>
We’ll need to create some packaging / testing / deployment helper scripts to deploy
our code.
</li>
</ul>
<table class="cost-comparison">
<thead>
<tr>
<th style="text-align: right">Description</th>
<th style="text-align: left">Cost</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right">(table 1) provision 200 DynamoDB writes/sec</td>
<td style="text-align: left">$0.13 / hour</td>
</tr>
<tr>
<td style="text-align: right">(table 2) provision 200 DynamoDB writes/sec</td>
<td style="text-align: left">$0.13 / hour</td>
</tr>
<tr>
<td style="text-align: right">(table 1 stream) 200 GetRecords/sec (worst case)</td>
<td style="text-align: left">negligible</td>
</tr>
<tr>
<td style="text-align: right">Lambda invocation cost</td>
<td style="text-align: left">negligible</td>
</tr>
<tr>
<td style="text-align: right">Lambda duration cost @ 1536 MB, 2 * 100 * 50000 ms (worst case)</td>
<td style="text-align: left">$0.25 / hour</td>
</tr>
<tr>
<td style="text-align: right">design/test Lambda deployment scripts</td>
<td style="text-align: left">4 hours</td>
</tr>
<tr>
<td style="text-align: right">design/test needed Lambda functions</td>
<td style="text-align: left">4 hours</td>
</tr>
<tr>
<td style="text-align: right">deploy needed Lambda functions</td>
<td style="text-align: left">5 minutes / deploy</td>
</tr>
<tr>
<td style="text-align: right">infrastructure maintenance</td>
<td style="text-align: left">$0</td>
</tr>
</tbody>
</table>
<table class="cost-summary">
<thead>
<tr>
<th style="text-align: right"> </th>
<th style="text-align: left"> </th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right">Initial developer time</td>
<td style="text-align: left">8 hours</td>
</tr>
<tr>
<td style="text-align: right">Ongoing developer time</td>
<td style="text-align: left">20 minutes / year</td>
</tr>
<tr>
<td style="text-align: right">Recurring costs (DynamoDB)</td>
<td style="text-align: left">$2279 / year</td>
</tr>
<tr>
<td style="text-align: right">Recurring costs (Lambda)</td>
<td style="text-align: left">$2192 / year</td>
</tr>
<tr>
<td style="text-align: right">Reaction time</td>
<td style="text-align: left">0.1 to 51 seconds</td>
</tr>
</tbody>
</table>
<h4 id="worth-it-possibly">Worth it? Possibly.</h4>
<ul>
<li>
We’ve greatly improved the upper (7x) and lower (600x) bounds for hypothetical system
reaction time (length of time after a file upload before we can start observing its
effects). With the Lambda-based system, we start updating our stored state in
DynamoDB as soon as our function begins executing.
</li>
<li>
Less developer time is required to get a working system. The original design requires
almost 3x initial developer time, and almost 40x ongoing time to maintain. The Lambda
solution saves 3 developer-days of time in the first year.
</li>
<li>
More money is required on an ongoing basis—around 6.5x—but we don’t have
to manage any infrastructure, which leaves more time for other projects.
</li>
<li>
It’s easier to scale (up to a point):
<ul>
<li>
If we wanted to halve our maximum reaction time, we could double our provisioned
DynamoDB write capacity without changing anything else. (This would double our
DynamoDB costs and halve our worst-case Lambda costs, for a total recurring cost
increase of 26%.)
</li>
<li>
If our RTB system were to suddenly double in size, we could simply double our
provisioned write capacity using the console without exploring alternate
designs or modifying our infrastructure!
</li>
</ul>
</li>
</ul>
<h4 id="theres-more-than-one-way-to-do-it">There’s More Than One Way To Do It™</h4>
We could take a different approach which (1) doesn’t involve Lambda or DynamoDB, or (2)
uses these services in a different way. This cost estimate shows that using Lambda
for a system like the one described here is within the realm of possibility.
<h4 id="eventual-consistency">Eventual consistency</h4>
Working with distributed systems requires balancing trade-offs. It’s worth noting that
the Lambda-based system we describe exhibits eventual consistency: the state we can
observe at any particular moment in time may not yet include all updates we’ve made, but
all updates will eventually be observable if we stop performing them. This is due to
the use of DynamoDB and S3 as well as the system’s overall design.
<ul>
<li>
DynamoDB supports eventually-consistent reads (the default) as well as
strongly-consistent reads. The consistency time scale is relatively small (usually
“within one second”) compared to other parts of our system. Whenever we read an item,
we’ll observe a state which is the result of some sequence of atomic updates (but not
necessarily all of the most recent updates). We can live with this trade-off.
</li>
<li>
S3 supports eventual consistency for overwritten objects, and the typical consistency
time scale is again relatively small—on the order of seconds. We shouldn’t need
to worry about it as long as we don’t receive out-of-order event notifications for the
same key: we’ll always read data which was valid at one point in time, and we can
check timestamps and versions if we’re concerned about stale events or reads.
</li>
</ul>
Our system as a whole wouldn’t be strongly-consistent even if S3 and DynamoDB presented
only strongly-consistent interfaces, as we can observe our system’s output (contents of
second DynamoDB table) before all current input (S3 file uploads) has been processed.
We essentially have a view of an atomic update stream which periodically will be a few
seconds out of date; this is fine for our purposes.
<h2 id="implementation">Implementation</h2>
Enough background—let’s implement our system!
We need to react to S3 file uploads. We can configure our S3 bucket to emit events
which will cause a Lambda function to run. Our first new component will therefore look
like this:
<img src="/images/post_images/counter-upload-processor-1.png" alt="initial counter upload processor" />
<h4 id="sns-topic-creation">SNS topic creation</h4>
We’ll create an SNS topic to receive S3 events. While Lambda directly supports S3
events, using an SNS topic will allow us to more easily configure additional S3 event
handlers in the future for events having the same prefix. (S3 doesn’t currently
support event handlers dispatching on prefixes which overlap.)
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">REGION="us-west-2"
BUCKET="your-bucket-name"
TOPIC_NAME="your-file-upload-topic"
SOURCE_ARN="arn:aws:s3:::$BUCKET"
# creating a topic gives us its ARN. we'll need that later along with
# our account number:
TOPIC_ARN=$(aws sns create-topic \
--region $REGION \
--name $TOPIC_NAME \
--output text \
--query 'TopicArn')
ACCOUNT=$(echo $TOPIC_ARN | awk -F':' '{print $5}')</code></pre></figure>
We also need to configure the SNS topic’s policy to allow our S3 bucket to publish
events to it:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">POLICY=$(python -c "import json; print json.dumps(
{'Version': '2008-10-17',
'Id': 'upload-events-policy',
'Statement': [
{'Resource': '$TOPIC_ARN',
'Effect': 'Allow',
'Sid': 'allow_s3_publish',
'Action': 'SNS:Publish',
'Condition': {'ArnEquals': {'aws:sourceArn': '$SOURCE_ARN'}},
'Principal': '*'},
{'Resource': '$TOPIC_ARN',
'Effect': 'Allow',
'Sid': 'owner_sns_permissions',
'Action': ['SNS:Subscribe', 'SNS:ListSubscriptionsByTopic',
'SNS:DeleteTopic', 'SNS:GetTopicAttributes',
'SNS:Publish', 'SNS:RemovePermission',
'SNS:AddPermission', 'SNS:Receive',
'SNS:SetTopicAttributes'],
'Condition': {'StringEquals': {'AWS:SourceOwner': '$ACCOUNT'}},
'Principal': {'AWS': '*'}}
]})")
aws sns set-topic-attributes \
--region $REGION \
--topic-arn "$TOPIC_ARN" \
--attribute-name "Policy" \
--attribute-value "$POLICY"</code></pre></figure>
<h4 id="s3-bucket-configuration">S3 bucket configuration</h4>
We need to configure our S3 bucket to publish events to the topic. If any event
settings already exist for the bucket, we should instead do this in the S3 console:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"># we're interested in events occurring under this prefix:
PREFIX="path/to/data/"
# we don't want to accidentally overwrite an existing event
# configuration:
EXISTING=$(aws s3api get-bucket-notification-configuration \
--bucket $BUCKET)
if [ -z "$EXISTING" ]; then
# there was no existing configuration. install our new one:
EVENT_CONFIG=$(python -c "import json; print json.dumps(
{'TopicConfigurations': [
{ 'Id': '$TOPIC_NAME events',
'TopicArn': '$TOPIC_ARN',
'Events': ['s3:ObjectCreated:*'],
'Filter': {
'Key': { 'FilterRules': [
{ 'Name': 'prefix', 'Value': '$PREFIX' }
]}
}
}]})")
aws s3api put-bucket-notification-configuration \
--bucket $BUCKET \
--notification-configuration "$EVENT_CONFIG"
else
echo "bucket already has an event configuration."
echo "use the S3 console to configure events."
fi</code></pre></figure>
<h4 id="lambda-function-creation">Lambda function creation</h4>
We need to choose a name for our first Lambda function, create a role to use when
executing, and enable AWS Lambda to assume that role so it can actually execute our
function:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">FUNCTION1_NAME="counter-upload-processor"
ASSUMEROLE_POLICY=$(python -c "import json; print json.dumps(
{ 'Version': '2012-10-17',
'Statement': [{ 'Sid': '',
'Effect': 'Allow',
'Principal': {'Service': 'lambda.amazonaws.com'},
'Action': 'sts:AssumeRole'
}] })")
ROLE1_NAME="$FUNCTION1_NAME-execute"
# create the execution role and save its ARN for later:
EXEC_ROLE1_ARN=$(aws iam create-role \
--region $REGION \
--role-name "$ROLE1_NAME" \
--assume-role-policy-document "$ASSUMEROLE_POLICY" \
--output text \
--query 'Role.Arn')</code></pre></figure>
Lambda functions normally emit logs using CloudWatch to a log group whose name is
derived from the function’s name. Let’s create a log group for our function:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">LOG_GROUP1="/aws/lambda/$FUNCTION1_NAME"
aws logs create-log-group \
--region $REGION \
--log-group-name "$LOG_GROUP1"</code></pre></figure>
To actually emit logs, we need to grant some permissions to the role used by our Lambda
function:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">CLOUDWATCH_POLICY1=$(python -c "import json; print json.dumps(
{ 'Version': '2012-10-17',
'Statement': [
{ 'Effect': 'Allow',
'Action': ['logs:PutLogEvents',
'logs:CreateLogStream'],
'Resource': 'arn:aws:logs:$REGION:$ACCOUNT:log-group:$LOG_GROUP1:*' }
]})")
aws iam put-role-policy \
--region $REGION \
--role-name "$ROLE1_NAME" \
--policy-name "emit-cloudwatch-logs" \
--policy-document "$CLOUDWATCH_POLICY1"</code></pre></figure>
We can now create a Python Lambda function. This naming convention and zip file layout
will allow it to be edited using the AWS Lambda console:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">cat &gt;lambda_function.py &lt;&lt;EOF
import logging
def lambda_handler(event, context):
logging.getLogger().setLevel(logging.INFO)
logging.info('got event: {}'.format(event))
logging.info('got context: {}'.format(context))
return False
EOF
zip -j lambda-example-1.zip lambda_function.py
# we'll need the function ARN later:
FUNCTION1_ARN=$(aws lambda create-function \
--region $REGION \
--runtime python2.7 \
--role "$EXEC_ROLE1_ARN" \
--description "counter file reverse mapper" \
--timeout 10 \
--memory-size 128 \
--handler lambda_function.lambda_handler \
--zip-file fileb://lambda-example-1.zip \
--function-name $FUNCTION1_NAME \
--output text \
--query 'FunctionArn')</code></pre></figure>
Let’s manually test the function:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">aws lambda invoke \
--region $REGION \
--function-name $FUNCTION1_ARN \
--payload '{}' \
--log-type 'Tail' \
--output text \
--query 'LogResult' \
- \
| base64 --decode</code></pre></figure>
The result should look like this:
<pre>
START RequestId: e7739dcd-7380-11e5-aa63-c740254a89b5 Version: $LATEST
[INFO] 2015-11-16T21:08:38.830Z e7739dcd-7380-11e5-aa63-c740254a89b5 got event {}
[INFO] 2015-11-16T21:08:38.830Z e7739dcd-7380-11e5-aa63-c740254a89b5 got context &lt;__main__.LambdaContext object at 0x7f61bfd246d0&gt;
END RequestId: e7739dcd-7380-11e5-aa63-c740254a89b5
REPORT RequestId: e7739dcd-7380-11e5-aa63-c740254a89b5 Duration: 0.41 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 13 MB
</pre>
<h4 id="putting-it-all-together">Putting it all together</h4>
We now have a working Lambda function implemented in Python. We can subscribe it to the
SNS topic we created earlier:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">aws sns subscribe \
--region $REGION \
--protocol lambda \
--topic-arn $TOPIC_ARN \
--notification-endpoint $FUNCTION1_ARN</code></pre></figure>
But before things will actually work, we need to grant the SNS topic permission to
invoke our Lambda function:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">aws lambda add-permission \
--region $REGION \
--function-name $FUNCTION1_ARN \
--statement-id "$FUNCTION1_NAME-invoke" \
--principal "sns.amazonaws.com" \
--action "lambda:InvokeFunction" \
--source-arn $TOPIC_ARN</code></pre></figure>
As objects are uploaded to our S3 bucket, we should start seeing records of invocations
and logs appearing in the CloudWatch log group we created earlier. Let’s upload a test
file:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">echo "COUNTER1 12345" \
| aws s3 cp - s3://$BUCKET/${PREFIX}$(date +'%Y-%m-%d')/values_i-deadbeef</code></pre></figure>
We can view our CloudWatch logs from the command line. We should see some output
corresponding to the key we just uploaded:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">for STREAM_NAME in $(aws logs describe-log-streams \
--region $REGION \
--log-group-name $LOG_GROUP1 \
--descending \
--order-by 'LastEventTime' \
--output text \
--query 'logStreams[].logStreamName'); do
aws logs get-log-events \
--region $REGION \
--log-group-name $LOG_GROUP1 \
--log-stream-name $STREAM_NAME \
--start-from-head \
--output text \
--query 'events[].message'
done</code></pre></figure>
If you see something like this, it worked:
<pre>
[INFO] 2015-11-16T23:44:15.876Z 1ad60ef6-3417-12c5-92b0-0f211eba5fc1 got event: {u'Records': [{u'EventVersion': u'1.0', u'EventSubscriptionArn': ... u'EventSource': u'aws:sns', u'Sns': { ... u'Message': u'{"Records":[{"eventVersion":"2.0","eventSource":"aws:s3", ...
</pre>
<h3 id="data-persistence">Data persistence</h3>
Our Lambda function is now hooked up to events emitted from S3! However, it doesn’t yet
do anything useful. We want our system to react to events and to only reprocess data
which has changed. We’ll achieve this by (1) storing some intermediate state in
DynamoDB and (2) using a second Lambda function to process updates to this state.
Our updated system will look like this:
<img src="/images/post_images/counter-upload-processor-2.png" alt="updated counter upload processor" />
<h4 id="create-a-dynamodb-table">Create a DynamoDB table</h4>
We want to store items which look like this in our intermediate state table:
<pre>
{ 'Counter': { 'S': 'XXXXXXXXXXXXXXXXXXXXXX' },
'Date': { 'S': 'YYYY-MM-DD' },
'InstanceValues': { 'M': { 'i-deadbeef': { 'N': '1234' },
'i-beefdead': { 'N': '5678' },
...
}
}
}
</pre>
Using a <code class="language-plaintext highlighter-rouge">map</code> type (“M”) and <a href="http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.AccessingItemAttributes.html#DocumentPaths">document paths</a> to store
per-instance counter values will allow concurrently-executing Lambda function invocations to
update the same DynamoDB item without interfering with each other.
Our table schema will look like this:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">TABLE="$FUNCTION1_NAME-state"
aws dynamodb create-table \
--region $REGION \
--table-name $TABLE \
--provisioned-throughput ReadCapacityUnits=10,WriteCapacityUnits=10 \
--key-schema AttributeName=Counter,KeyType=HASH \
AttributeName=Date,KeyType=RANGE \
--attribute-definitions AttributeName=Counter,AttributeType=S \
AttributeName=Date,AttributeType=S</code></pre></figure>
<h4 id="s3-sns-event-handling">S3-SNS event handling</h4>
We want to update our intermediate state whenever an event comes in. Our SNS events
will look like this:
<figure class="highlight"><pre><code class="language-json" data-lang="json">{"Records": [
{"EventVersion": "1.0",
"EventSubscriptionArn": "SUBSCRIPTION_ARN",
"EventSource": "aws:sns",
"Sns": {
"SignatureVersion": "1",
"Timestamp": "2015-11-16T22:08:36.726Z",
"Signature": "SIG_DATA",
"SigningCertUrl": "SIG_CERT_URL",
"MessageId": "MESSAGE_ID",
"Message": "JSON_MESSAGE_DATA",
"MessageAttributes": {},
"Type": "Notification",
"UnsubscribeUrl": "UNSUBSCRIBE_URL",
"TopicArn": "$TOPIC_ARN",
"Subject": "Amazon S3 Notification"
}
}
]}</code></pre></figure>
The <code class="language-plaintext highlighter-rouge">JSON_MESSAGE_DATA</code> value in these events will be a json-encoded S3 event. It will
look like this when decoded:
<figure class="highlight"><pre><code class="language-json" data-lang="json">{"Records": [
{"eventVersion": "2.0",
"eventTime": "2015-11-16T22:08:36.682Z",
"requestParameters": {"sourceIPAddress": "1.2.3.4"},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "$TOPIC_NAME events",
"object": {"versionId": "NEW_OBJECT_VERSION_ID",
"eTag": "NEW_OBJECT_ETAG",
"sequencer": "SEQUENCER_VALUE",
"key": "${PREFIX}2015-11-16/values_i-deadbeef",
"size": 16},
"bucket": { "arn": "$SOURCE_ARN",
"name": "$BUCKET",
"ownerIdentity": {"principalId": "OWNING_PRINCIPAL_ID"}
}},
"responseElements": { "x-amz-id-2": "AMZ_ID_1",
"x-amz-request-id": "AMZ_ID_2" },
"awsRegion": "$REGION",
"eventName": "ObjectCreated:Put",
"userIdentity": {"principalId": "CREATING_PRINCIPAL_ID"},
"eventSource": "aws:s3"
}
]}</code></pre></figure>
(Note: the <code class="language-plaintext highlighter-rouge">key</code> value in the event is url-encoded.)
Let’s update the code in <code class="language-plaintext highlighter-rouge">lambda_function.py</code> to process these events:
<figure class="highlight"><pre><code class="language-python" data-lang="python">import logging, boto3, botocore.exceptions, json, urllib
TABLE = "counter-upload-processor-state"
FILENAME_PREFIX = 'values_'
def lambda_handler(event, context):
logging.getLogger().setLevel(logging.INFO)
for record in event['Records']:
if 'aws:sns' == record['EventSource'] and record['Sns']['Message']:
handle_sns_event(json.loads(record['Sns']['Message']), context)
return True
def handle_sns_event(event, context):
for record in event['Records']:
logging.info('looking at {}'.format(record))
if 'aws:s3' == record['eventSource'] \
and record['eventName'].startswith('ObjectCreated:'):
region = record['awsRegion']
bucket_name = record['s3']['bucket']['name']
key_name = urllib.unquote(record['s3']['object']['key'])
key_vsn = record['s3']['object'].get('versionId')
logging.info('new object: s3://{}/{} (v:{})'.format(bucket_name,
key_name,
key_vsn))
key = boto3.resource('s3', region_name=region) \
.Bucket(bucket_name) \
.Object(key_name)
data = key.get(**{'VersionId': key_vsn} if key_vsn else {})
process_key(region, key, data, context)
def process_key(region, key, data, context):
filename = key.key.split('/')[-1]
if filename.startswith(FILENAME_PREFIX):
date = key.key.split('/')[-2]
instance_id = filename.split('_')[1]
logging.info('processing ({}, {})'.format(date, instance_id))
for line in data['Body'].read().splitlines():
counter, value = [x.strip() for x in line.split()]
update_instance_value(region, date, instance_id, counter, value)
def update_instance_value(region, date, instance_id, counter, value):
logging.info('updating instance counter value: {} {} {}'.format(
instance_id, counter, value))
tbl = boto3.resource('dynamodb', region_name=region) \
.Table(TABLE)
key = {'Counter': counter,
'Date': date}
# updating a document path in an item currently fails if the ancestor
 # attributes don't exist, and multiple SET expressions can't
 # (currently) be used to update overlapping document paths (even with
# `if_not_exists`), so we must first create the `InstanceValues` map
 # if needed. we use a condition expression to avoid needlessly
 # triggering an update event on the stream we'll create for this
 # table. in a real application, we might first query the table to
 # check if these updates are actually needed (reads are cheaper than
 # writes).
 lax_update(tbl,
Key=key,
UpdateExpression='SET #valuemap = :empty',
ExpressionAttributeNames={'#valuemap': 'InstanceValues'},
ExpressionAttributeValues={':empty': {}},
ConditionExpression='attribute_not_exists(#valuemap)')
# we can now actually update the target path. we only update if the
 # new value is different (in a real application, we might first query
 # and refrain from attempting the conditional write if the value is
 # unchanged):
 lax_update(tbl,
Key=key,
UpdateExpression='SET #valuemap.#key = :value',
ExpressionAttributeNames={ '#key': instance_id,
'#valuemap': 'InstanceValues'},
ExpressionAttributeValues={':value': int(value)},
ConditionExpression='NOT #valuemap.#key = :value')
def lax_update(table, **kwargs):
try:
return table.update_item(**kwargs)
except botocore.exceptions.ClientError as exc:
code = exc.response['Error']['Code']
if 'ConditionalCheckFailedException' != code:
raise</code></pre></figure>
We can now update our Lambda function’s code:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">zip -j lambda-example-2.zip lambda_function.py
aws lambda update-function-code \
--region $REGION \
--function-name $FUNCTION1_ARN \
--zip-file fileb://lambda-example-2.zip</code></pre></figure>
Before this updated code will work, we need to grant permission to the execution role we
created earlier for our Lambda function—otherwise, it won’t be able to access our S3
bucket or update our DynamoDB table:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">REV_MAPPER_POLICY=$(python -c "import json; print json.dumps(
{ 'Version': '2012-10-17',
'Statement': [
{ 'Effect': 'Allow',
'Action': ['dynamodb:UpdateItem'],
'Resource': 'arn:aws:dynamodb:$REGION:$ACCOUNT:table/$TABLE' },
{ 'Effect': 'Allow',
'Action': ['s3:GetObject',
's3:GetObjectVersion'],
'Resource': '$SOURCE_ARN/${PREFIX}*' }
]})")
aws iam put-role-policy \
--region $REGION \
--role-name "$ROLE1_NAME" \
--policy-name "fetch-data-and-update-dynamo" \
--policy-document "$REV_MAPPER_POLICY"</code></pre></figure>
As new files appear in our bucket under the target prefix, we should see our table being
updated accordingly. Let’s give it a try:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">echo "COUNTER1 12345" \
| aws s3 cp - s3://$BUCKET/${PREFIX}$(date +'%Y-%m-%d')/values_i-deadbeef
echo "COUNTER1 12345" \
| aws s3 cp - s3://$BUCKET/${PREFIX}$(date +'%Y-%m-%d')/values_i-beefdead
echo "COUNTER1 54321
COUNTER2 54321" | aws s3 cp - s3://$BUCKET/${PREFIX}$(date +'%Y-%m-%d')/values_i-foobazzled</code></pre></figure>
If we scan our intermediate state table, we should now see that it has been updated by
our Lambda function:
<pre>
aws dynamodb scan \
--region $REGION \
--table-name $TABLE
</pre>
<figure class="highlight"><pre><code class="language-json" data-lang="json">{
"Count": 2,
"Items": [
{
"Date": {
"S": "2015-11-16"
},
"InstanceValues": {
"M": {
"i-foobazzled": {
"N": "54321"
}
}
},
"Counter": {
"S": "COUNTER2"
}
},
{
"Date": {
"S": "2015-11-16"
},
"InstanceValues": {
"M": {
"i-beefdead": {
"N": "12345"
},
"i-deadbeef": {
"N": "12345"
},
"i-foobazzled": {
"N": "54321"
}
}
},
"Counter": {
"S": "COUNTER1"
}
}
],
"ScannedCount": 2,
"ConsumedCapacity": null
}</code></pre></figure>
If you see something like this, it worked. Huzzah! We’ve completed the first part of
our system:
<img src="/images/post_images/counter-upload-processor-2.png" alt="updated counter upload processor" />
<h3 id="enter-dynamodb-streams">Enter DynamoDB Streams</h3>
In our intermediate state table we’re now maintaining an inverted map of the original
data uploaded to S3, converting per-instance lists of per-counter values on S3 to
per-counter lists of per-instance values in DynamoDB. Our goal is to sum up all
per-instance values for each counter and to take action when these values exceed a
per-counter limit, only processing data which has changed.
DynamoDB Streams is a <a href="https://aws.amazon.com/blogs/aws/dynamodb-update-triggers-streams-lambda-cross-region-replication-app/">recently-released feature</a>
which grants a view of change events on a DynamoDB table (akin to a Kinesis stream). We
can use “DynamoDB Triggers” (the combination of DynamoDB Streams and Lambda functions)
to achieve our goal.
The current version of the <code class="language-plaintext highlighter-rouge">awscli</code> tools doesn’t support enabling or modifying DynamoDB
stream settings, so for now we must use the console to configure a table stream:
<img src="/images/post_images/enable-dynamo-streams.png" alt="Enable DynamoDB streams" />
Since we’ll be performing an operation on the entire new value of updated items, select
“new image” as the stream view type:
<img src="/images/post_images/dynamo-stream-view-new-image.png" alt="Configure new DynamoDB stream" />
After configuring the table stream in the DynamoDB console, we can fetch its ARN:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">STREAM_ARN=$(aws dynamodbstreams list-streams \
--region $REGION \
--table-name $TABLE \
--query 'Streams[0].StreamArn' \
--output text)</code></pre></figure>
<h4 id="putting-it-all-together-again">Putting it all together (again)</h4>
We’ll need another Lambda function to process change events on our state table. We’ll
use the same recipe (and initial code) as our first Lambda function.
Here’s how the next stage of our system will look:
<img src="/images/post_images/counter-upload-aggregator-1.png" alt="counter upload aggregator" />
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">FUNCTION2_NAME="counter-upload-aggregator"
ROLE2_NAME="$FUNCTION2_NAME-execute"
EXEC_ROLE2_ARN=$(aws iam create-role \
--region $REGION \
--role-name "$ROLE2_NAME" \
--assume-role-policy-document "$ASSUMEROLE_POLICY" \
--output text \
--query 'Role.Arn')
LOG_GROUP2="/aws/lambda/$FUNCTION2_NAME"
aws logs create-log-group \
--region $REGION \
--log-group-name "$LOG_GROUP2"
CLOUDWATCH_POLICY2=$(python -c "import json; print json.dumps(
{ 'Version': '2012-10-17',
'Statement': [
{ 'Effect': 'Allow',
'Action': ['logs:PutLogEvents',
'logs:CreateLogStream'],
'Resource': 'arn:aws:logs:$REGION:$ACCOUNT:log-group:$LOG_GROUP2:*' }
]})")
aws iam put-role-policy \
--region $REGION \
--role-name "$ROLE2_NAME" \
--policy-name "emit-cloudwatch-logs" \
--policy-document "$CLOUDWATCH_POLICY2"
FUNCTION2_ARN=$(aws lambda create-function \
--region $REGION \
--runtime python2.7 \
--role "$EXEC_ROLE2_ARN" \
--description "counter data aggregator" \
--timeout 10 \
--memory-size 128 \
--handler lambda_function.lambda_handler \
--zip-file fileb://lambda-example-1.zip \
--function-name $FUNCTION2_NAME \
--output text \
--query 'FunctionArn')</code></pre></figure>
We need to grant some additional permissions to allow access to our table stream:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">STREAM_POLICY=$(python -c "import json; print json.dumps(
{ 'Version': '2012-10-17',
'Statement': [
{ 'Effect': 'Allow',
'Action': ['dynamodb:GetRecords',
'dynamodb:GetShardIterator',
'dynamodb:DescribeStream'],
'Resource': '$STREAM_ARN' },
{ 'Effect': 'Allow',
'Action': ['dynamodb:DescribeStreams'],
'Resource': '*' }
]})")
aws iam put-role-policy \
--region $REGION \
--role-name "$ROLE2_NAME" \
--policy-name "dynamodb-stream-access" \
--policy-document "$STREAM_POLICY"</code></pre></figure>
We can now create an event source mapping between our table stream and our new Lambda
function. This will cause Lambda to poll the DynamoDB stream and execute our new
function with events read from it:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">aws lambda create-event-source-mapping \
--region $REGION \
--event-source-arn $STREAM_ARN \
--function-name $FUNCTION2_ARN \
--starting-position LATEST \
--enabled</code></pre></figure>
Let’s upload another test file. (Note: there may be a delay after configuring the
event source mapping before this upload will result in the second function being
triggered.)
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">echo "COUNTER1 123456" \
| aws s3 cp - s3://$BUCKET/${PREFIX}$(date +'%Y-%m-%d')/values_i-deadbeef</code></pre></figure>
This upload should cause our first Lambda function to update the intermediate state
table. Our second Lambda function should receive an event like this:
<figure class="highlight"><pre><code class="language-json" data-lang="json">{"Records": [
{ "awsRegion": "us-east-1",
"eventName": "MODIFY",
"eventSourceARN": "$STREAM_ARN",
"eventSource": "aws:dynamodb",
"eventID": "SOME_EVENT_ID",
"eventVersion": "1.0",
"dynamodb": { "StreamViewType": "NEW_IMAGE",
"SequenceNumber": "SOME_SEQUENCE_NUMBER",
"SizeBytes": 125,
"Keys": {
"Date": {"S": "2015-11-16"},
"Counter": {"S": "COUNTER1"}
},
"NewImage": {
"Counter": {"S": "COUNTER1"},
"Date": {"S": "2015-11-16"},
"InstanceValues": {"M": {"i-foobazzled": {"N": "54321"},
"i-beefdead": {"N": "12345"},
"i-deadbeef": {"N": "123456"}}}
}
}
}
]}</code></pre></figure>
If we repeat the same upload, our second Lambda function should not be executed a second
time—our first Lambda function only updates an item if it would result in a new or
changed value. This is good, but it doesn’t quite fulfill our goal of only processing
changed data: if a new instance comes online and emits a value of zero for a counter,
our second Lambda function will still see an update despite the global value for that
counter remaining unchanged.
<h3 id="rinse-and-repeat">Rinse and Repeat</h3>
To fulfill our goal of only acting on changed data, we’ll create a second DynamoDB table
to hold actual per-counter aggregate values. (A third Lambda function monitoring this
second table’s event stream could contain the business logic acting on changed counter
values; all of our goals would then have been met.)
Here’s how this part of our updated system will look:
<img src="/images/post_images/counter-upload-aggregator-2.png" alt="updated counter upload aggregator" />
In our second table, we want to store items which look like this:
<pre>
{ 'Counter': { 'S': 'XXXXXXXXXXXXXXXXXXXXXX' },
'Date': { 'S': 'YYYY-MM-DD' },
'Value': { 'N': '1234567' }
}
</pre>
Our second DynamoDB table has the following schema. Including the <code class="language-plaintext highlighter-rouge">ByDate</code> global
secondary index will allow us to query the table for all items having a particular date,
which might come in handy later (although it will double our table cost, because all
attributes participate in the index):
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">TABLE2="aggregate-counters"
aws dynamodb create-table \
--region $REGION \
--table-name $TABLE2 \
--provisioned-throughput ReadCapacityUnits=10,WriteCapacityUnits=10 \
--key-schema AttributeName=Counter,KeyType=HASH \
AttributeName=Date,KeyType=RANGE \
--attribute-definitions AttributeName=Counter,AttributeType=S \
AttributeName=Date,AttributeType=S \
--global-secondary-indexes '[{
"IndexName": "ByDate",
"KeySchema": [{ "AttributeName": "Date",
"KeyType": "HASH" }],
"ProvisionedThroughput": { "ReadCapacityUnits": 1,
"WriteCapacityUnits": 10 },
"Projection": { "ProjectionType": "ALL" }
}]'</code></pre></figure>
Let’s update our second Lambda function’s <code class="language-plaintext highlighter-rouge">lambda_function.py</code> file:
<figure class="highlight"><pre><code class="language-python" data-lang="python">import logging, boto3, botocore.exceptions
TABLE = "aggregate-counters"
def lambda_handler(event, context):
logging.getLogger().setLevel(logging.INFO)
for record in event['Records']:
if 'aws:dynamodb' == record['eventSource'] \
and 'MODIFY' == record['eventName'] \
and 'NEW_IMAGE' == record['dynamodb']['StreamViewType']:
region = record['awsRegion']
keys = record['dynamodb']['Keys']
date = keys['Date']['S']
counter = keys['Counter']['S']
new_item = record['dynamodb']['NewImage']
instance_values = new_item['InstanceValues']['M']
total_value = sum(int(v['N'])
for v in instance_values.values())
logging.info('updated counter: {} {} {}'.format(
date, counter, total_value))
# go! thou art counted:
lax_update(boto3.resource('dynamodb', region_name=region) \
.Table(TABLE),
Key={'Counter': counter,
'Date': date},
UpdateExpression='SET #value = :value',
ExpressionAttributeNames={'#value': 'Value'},
ExpressionAttributeValues={':value': total_value},
ConditionExpression='NOT #value = :value')
return True
def lax_update(table, **kwargs):
try:
return table.update_item(**kwargs)
except botocore.exceptions.ClientError as exc:
code = exc.response['Error']['Code']
if 'ConditionalCheckFailedException' != code:
raise</code></pre></figure>
And update our second Lambda function’s code:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">zip -j lambda-example-3.zip lambda_function.py
aws lambda update-function-code \
--region $REGION \
--function-name $FUNCTION2_ARN \
--zip-file fileb://lambda-example-3.zip</code></pre></figure>
As before, we need to grant permission to the execution role used by our second Lambda
function so it can update our second DynamoDB table:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">AGGREGATOR_POLICY=$(python -c "import json; print json.dumps(
{ 'Version': '2012-10-17',
'Statement': [
{ 'Effect': 'Allow',
'Action': ['dynamodb:UpdateItem'],
'Resource': 'arn:aws:dynamodb:$REGION:$ACCOUNT:table/$TABLE2' }
]})")
aws iam put-role-policy \
--region $REGION \
--role-name "$ROLE2_NAME" \
--policy-name "update-dynamo-aggregate-counters" \
--policy-document "$AGGREGATOR_POLICY"</code></pre></figure>
When we upload new counter data, our second table should be updated with the total
summed value:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">echo "COUNTER1 1000" \
| aws s3 cp - s3://$BUCKET/${PREFIX}$(date +'%Y-%m-%d')/values_i-flub</code></pre></figure>
This upload should have triggered the following sequence of events:
<ol>
<li>
Our S3 bucket emits an <code class="language-plaintext highlighter-rouge">ObjectCreated</code> event to our SNS topic.
</li>
<li>
Our topic wraps the S3 event in an SNS notification and delivers it to our first
Lambda function (invoking it).
</li>
<li>
Our first Lambda function executes and updates our first table.
</li>
<li>
An event appears on our first table’s event stream, which (Lambda on behalf of) our
second Lambda function has been polling.
</li>
<li>
Our second Lambda function executes and updates our second table.
</li>
</ol>
We should see a value of <code class="language-plaintext highlighter-rouge">12345+123456+1000+54321 = 191122</code> in our second table, which
is the sum of all uploaded values for <code class="language-plaintext highlighter-rouge">COUNTER1</code>. We haven’t updated any other counter
values since setting up our second table and function, so they shouldn’t yet exist in
our second table:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">aws dynamodb query \
--region $REGION \
--table-name $TABLE2 \
--index-name "ByDate" \
--key-condition-expression '#date = :date' \
--expression-attribute-names '{"#date": "Date",
"#value": "Value",
"#counter": "Counter"}' \
--expression-attribute-values '{":date":
{"S": "'$(date +'%Y-%m-%d')'"}}' \
--projection-expression "#date, #counter, #value"</code></pre></figure>
<figure class="highlight"><pre><code class="language-json" data-lang="json">{
"Count": 1,
"Items": [
{
"Date": {
"S": "2015-11-16"
},
"Value": {
"N": "191122"
},
"Counter": {
"S": "COUNTER1"
}
}
],
"ScannedCount": 1,
"ConsumedCapacity": null
}</code></pre></figure>
If you see this, it worked! Our final system:
<img src="/images/post_images/lambda-counter-aggregator-overview.png" alt="Lambda-based counter aggregation system overview" />
<h2 id="onward-and-upward">Onward and Upward</h2>
We have successfully implemented a basic system which can count things we care about by
reacting appropriately to S3 uploads. We also eliminated the need to worry about server
configuration tools and management!
By implementing two simple Python modules and making some configuration calls, we have:
<ul>
<li>
Decreased coupling between components: Instead of a set of cooperating periodic
tasks, we have some small Lambda functions each doing one simple thing—reacting
to events. Each is also easily testable (1) locally with a test suite, (2) in the
Lambda console, and (3) from the command line using <code class="language-plaintext highlighter-rouge">awscli</code>.
</li>
<li>
Improved responsiveness and scalability: Our new system can begin reacting to
updates with less latency than the original, and we can realize significant
performance gains simply by deciding to spend more money—or less, if our
requirements decrease—without any need for a redesign.
</li>
</ul>
<h3 id="overcoming-failure">Overcoming Failure</h3>
The example system described in this post glosses over certain issues which would be
problematic in a real application, and no discussion involving distributed systems
would be complete without addressing some of the ways in which they can fail.
We expect our system to finish processing each event within 50 seconds. What if it
doesn’t?
With the system we’ve implemented here, the most likely failures are:
<ol>
<li>
Incomplete file processing due to reaching the Lambda invocation time limit. There
might be too many counters in each file for us to process even after making improvements
to our implementation and paying for more DynamoDB capacity.
</li>
<li>
Incomplete file processing due to a provisioned throughput exception. Our library
will perform retries, but only up to a point before giving up and raising an exception.
</li>
</ol>
Both timing out and raising an exception are treated as execution errors by Lambda and
will result in our function being invoked again with the same event some time later (up
to a certain number of retry attempts).
While the system we’ve described is idempotent—each instance uploads its current
set of counter values, not counter deltas—this retry behavior can still be
problematic for our system as-written: a failing invocation C might be followed by a
successful invocation D, which is then followed by a successful retry of C (“lost
update” problem). Unless we take steps to mitigate this, older data from C could wipe
out newer data from D if our bucket uses versioning and they correspond to the same
uploading instance. We can handle this by adding a per-instance event timestamp to
our first table’s items and including it in our condition expression.
We could also encounter a situation where we can never catch up because the incoming
data rate exceeds the maximum sustained processing rate. If this happens after we’ve
already made our implementation more efficient and increased our provisioned capacity,
we should consider adjusting our schema (i.e., so that we may use <code class="language-plaintext highlighter-rouge">BatchWriteItem</code>
operations on our first table) and/or rethink our system’s design. (We also control
the systems producing this data, so nothing is set in stone!)
<h4 id="a-word-on-latency">A word on latency</h4>
We listed reaction times earlier in our cost estimate, based on the following system
latency formula:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{equation}
L = u + d_1 + max(p_1, d_2 + p_2)
\end{equation}
%]]>
</script>
<table class="math-legend">
<thead>
<tr>
<th style="text-align: right"> </th>
<th style="text-align: left"> </th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right">$L$</td>
<td style="text-align: left">latency</td>
</tr>
<tr>
<td style="text-align: right">$u$</td>
<td style="text-align: left">time until upload</td>
</tr>
<tr>
<td style="text-align: right">$d_1$</td>
<td style="text-align: left">delivery time (first lambda function)</td>
</tr>
<tr>
<td style="text-align: right">$p_1$</td>
<td style="text-align: left">processing time (first lambda function)</td>
</tr>
<tr>
<td style="text-align: right">$d_2$</td>
<td style="text-align: left">delivery time (second lambda function)</td>
</tr>
<tr>
<td style="text-align: right">$p_2$</td>
<td style="text-align: left">processing time (second lambda function)</td>
</tr>
</tbody>
</table>
We estimated our system’s reaction time by ignoring time until upload, assuming minimal
delivery time values (0-0.1s), the same processing time values (50s), and a small fudge
factor, giving a range of 0.1-51s. This ignores two important details: (1) concurrency
limits for Lambda (100 concurrent executions in each account region by default) and (2)
the sharding of our DynamoDB stream (maximum stream reader invocation concurrency is the
number of shards in the stream).
Item (2) is most problematic in our system—compared to our first Lambda function,
our second function will be invoked more frequently (smaller batches of counter updates)
and with less concurrency (limited by number of stream shards); therefore, the
assumption we made in our cost estimate about “similar function characteristics” was
flawed.
Experimentation reveals that even with tuning and implementation improvements, our
system’s best total latency is 64s (with 100 file uploads of 100 counters every 60s).
If this is due to our second Lambda function taking more than 60s to finish processing
all events, this is the “never catch up” situation described earlier. Updating our
second Lambda function to use <code class="language-plaintext highlighter-rouge">BatchWriteItem</code> instead of <code class="language-plaintext highlighter-rouge">UpdateItem</code> requires no
schema change and greatly improves performance, bringing total system latency back in
line with our expectations.
<h2 id="final-words">Final words</h2>
While no tool is right for every job, I had fun writing this post and I look forward to
using Lambda more often in situations where it makes sense to do so. I hope you found
this discussion and example interesting!
Exercises for the reader:
<ol>
<li>
Improve the efficiency and implementation of both Lambda functions. Try using
<code class="language-plaintext highlighter-rouge">multiprocessing</code> in the first (but note that <code class="language-plaintext highlighter-rouge">Pool</code> and <code class="language-plaintext highlighter-rouge">Queue</code> won’t work) and
implement exponential backoff/retry for provisioned throughput exception handling. Use
<code class="language-plaintext highlighter-rouge">BatchWriteItem</code> in the second Lambda function.
</li>
<li>
Implement a third Lambda function using the recipes in this post which watches our
second DynamoDB table for per-counter updates, and which emits the summed values for all
of a particular day’s counters back to S3 when any counter’s summed value changes.
</li>
</ol>
Want to learn more about AdRoll? <a href="https://www.adroll.com/about/careers/">Roll with Us</a>!
</description>
</item>
<item>
<title>Rollup: How we use React.js and npm to share UI code at AdRoll</title>
<link>https://tech.nextroll.com/blog/frontend/2015/11/12/rollup-react-and-npm-at-adroll.html</link>
<pubDate>Thu, 12 Nov 2015 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/frontend/2015/11/12/rollup-react-and-npm-at-adroll</guid>
<description>This is the second in a series of three blog posts about Rollup, AdRoll’s UI component library. This post covers how we build individual components and the developer tools supporting them. For background on why we built a UI component library, see <a href="/blog/frontend/2015/11/05/rollup-shared-ui-components.html">last week’s post</a>. For a discussion on what we learned from building Rollup see <a href="/blog/frontend/2015/11/19/rollup-major-learnings.html">next week’s post</a>.
All the shared UI components we use at AdRoll live in a private GitHub repository called Rollup. In the repo, each component has its own directory whose name matches the component’s <code class="language-plaintext highlighter-rouge">npm</code> package name. All components, and by extension Rollup <code class="language-plaintext highlighter-rouge">npm</code> package names, are prefixed with <code class="language-plaintext highlighter-rouge">ar-</code> for <code class="language-plaintext highlighter-rouge">AdRoll</code>.
The repo also contains developer tools to help maintain components. Some developer tools simply need a configuration file, while others may be more complicated and will have their own directory in the repo. Directories for developer tools are not prefixed with <code class="language-plaintext highlighter-rouge">ar-</code> and so can easily be told apart from the component directories.
Now that we’ve covered the overall structure of the repo, we’re going to talk in more detail about the following:
<ul>
<li><a href="#rollup-components">Rollup components</a>
<ul>
<li><a href="#src">src/</a>
<ul>
<li><a href="#jsx-to-scss-file-ratio">JSX to SCSS file ratio</a></li>
<li><a href="#importing-the-component-in-another-application">Importing the component in another application</a></li>
<li><a href="#es6-transpilation-in-other-applications">ES6 transpilation in other applications</a></li>
</ul>
</li>
<li><a href="#dist">dist/</a></li>
</ul>
</li>
<li><a href="#rollup-developer-tools">Rollup developer tools</a>
<ul>
<li><a href="#developer-tools-for-rollup-contributors">Developer tools for Rollup contributors</a>
<ul>
<li><a href="#automated-testing">Automated testing</a></li>
<li><a href="#live-component-examples">Live component examples</a></li>
<li><a href="#new-component-generator">New component generator</a></li>
</ul>
</li>
<li><a href="#developer-tools-for-rollup-users">Developer tools for Rollup users</a>
<ul>
<li><a href="#automatic-documentation-generation">Automatic documentation generation</a></li>
<li><a href="#component-asset-bundles">Component asset bundles</a></li>
<li><a href="#example-applications">Example applications</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h2 id="rollup-components">Rollup components</h2>
Components are published as <code class="language-plaintext highlighter-rouge">npm</code> packages to a private <code class="language-plaintext highlighter-rouge">npm</code> registry set up using <a href="http://www.jfrog.com/open-source/">Artifactory</a>, and the compiled code is made available via AdRoll’s CDN on <a href="https://aws.amazon.com/s3/">S3</a>. The version of each component is updated according to <a href="http://semver.org/">Semver</a> and version-specific CDN URLs for components look like the following:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">&lt;cdn_url&gt;/rollup/ar-component/&lt;version_number&gt;/</code></pre></figure>
Components are written as <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import">ES6 modules</a>. UI components in Rollup (not utility modules like <code class="language-plaintext highlighter-rouge">i18n</code>) are written in <a href="http://facebook.github.io/react/">React</a> and rely heavily on <a href="http://react-bootstrap.github.io/">React Bootstrap</a>. Rollup components that require styling have styles written in <a href="http://sass-lang.com/">Sass</a>. React is not a requirement for building UI components in the Rollup repo but it is highly encouraged. We chose this framework over others for the following reasons:
<ul>
<li>React components do not require separate template files, thanks to <a href="https://facebook.github.io/react/docs/jsx-in-depth.html">JSX</a></li>
<li>Each component defines its interface using <a href="https://facebook.github.io/react/docs/reusable-components.html">React propTypes</a></li>
<li>Component rendering is deterministic based on component props / state instead of direct DOM manipulation</li>
</ul>
Each component has three directories:
<ul>
<li><code class="language-plaintext highlighter-rouge">src/</code></li>
<li><code class="language-plaintext highlighter-rouge">dist/</code></li>
<li><code class="language-plaintext highlighter-rouge">examples/</code> (covered in <a href="#developer-tools-for-rollup-contributors">Developer tools for Rollup contributors</a>)</li>
</ul>
In this section we’re going to talk about the <code class="language-plaintext highlighter-rouge">src/</code> and <code class="language-plaintext highlighter-rouge">dist/</code> directories, using our data table component as an example.
<center>
<img alt="Rollup Data Table" src="/images/post_images/rollup_data_table.png" />
</center>
We’re also going to link to equivalent files in a <code class="language-plaintext highlighter-rouge">react-component-skeleton</code> component we have published <a href="https://github.com/AdRoll/react-component-skeleton/tree/9a6f1e7df1965b342066ca029cc8cce0a3f41295">here</a>.
<h3 id="src"><code class="language-plaintext highlighter-rouge">src/</code></h3>
The source code for all components (regardless of type, UI component vs. utility module) lives in each component’s <a href="https://github.com/AdRoll/react-component-skeleton/tree/9a6f1e7df1965b342066ca029cc8cce0a3f41295/src">src/ directory</a>. This directory is included in the <code class="language-plaintext highlighter-rouge">npm</code> package as well as made available via AdRoll’s CDN.
We’re going to go over the following three aspects of the <code class="language-plaintext highlighter-rouge">src/</code> directory:
<ul>
<li><a href="#jsx-to-scss-file-ratio">JSX to SCSS file ratio</a></li>
<li><a href="#importing-the-component-in-another-application">Importing the component in another application</a></li>
<li><a href="#es6-transpilation-in-other-applications">ES6 transpilation in other applications</a></li>
</ul>
<h4 id="jsx-to-scss-file-ratio">JSX to SCSS file ratio</h4>
Component JavaScript is written as an ES6 module and lives in <a href="https://github.com/AdRoll/react-component-skeleton/tree/9a6f1e7df1965b342066ca029cc8cce0a3f41295/src/components">src/components</a>, while component styles are written in Sass and live in <a href="https://github.com/AdRoll/react-component-skeleton/tree/9a6f1e7df1965b342066ca029cc8cce0a3f41295/src/styles">src/styles</a>.
Components may have as many style sheets or React components as the authors like, but only the <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/src/components/ReactComponentSkeleton.jsx">.jsx</a> and <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/src/styles/ar-react-component-skeleton.scss">.scss</a> files that share the same name as the component are intended for external use. Other files are intended for internal use only. For example, consider our <code class="language-plaintext highlighter-rouge">ar-data-table</code> component that has the following <code class="language-plaintext highlighter-rouge">.jsx</code> and <code class="language-plaintext highlighter-rouge">.scss</code> files in its <code class="language-plaintext highlighter-rouge">src</code> directory:
<center>
<img alt="Rollup Data Table src/" src="/images/post_images/rollup_data_table_src.png" />
</center>
As you’ll notice above, we have a 1:1 correlation of <code class="language-plaintext highlighter-rouge">.jsx</code> to <code class="language-plaintext highlighter-rouge">.scss</code> files (with the exception of our Sass variables file, <code class="language-plaintext highlighter-rouge">_ar-data-table-vars.scss</code>). For Rollup developers this helps keep files organized, and it’s also obvious where the styling for a given React component lives.
All CSS classes in our stylesheets are also prefixed with the name of the component to avoid name-spacing conflicts with applications using the components. E.g. the class given to rows in the data table is <code class="language-plaintext highlighter-rouge">ar-data-table-row</code>. This way of organizing stylesheets was inspired by <a href="https://gist.github.com/bobbygrace/9e961e8982f42eb91b80">Trello’s CSS Guide</a>.
<h4 id="importing-the-component-in-another-application">Importing the component in another application</h4>
Having multiple files for one Rollup component like we do above aims to simplify the lives of Rollup developers. But what about developers integrating Rollup components into their projects and applications? What are they supposed to do with all of those files?
We want to make it easy for developers to integrate these components into their application. So we have it set up so that consuming developers only need to include one <code class="language-plaintext highlighter-rouge">.jsx</code> and one <code class="language-plaintext highlighter-rouge">.scss</code> file to fully integrate the component code and styles into their application.
The <code class="language-plaintext highlighter-rouge">npm</code> package for Rollup components’ <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/package.json#L6">main file points to the React component that has the same name</a>. For example, the <code class="language-plaintext highlighter-rouge">package.json</code> of <code class="language-plaintext highlighter-rouge">ar-data-table</code> has a <code class="language-plaintext highlighter-rouge">main</code> field that points to <code class="language-plaintext highlighter-rouge">ar-data-table/src/DataTable.jsx</code>. So importing the component is as easy as:
<figure class="highlight"><pre><code class="language-js" data-lang="js">import DataTable from 'ar-data-table';</code></pre></figure>
In terms of documenting the component, all props required by the components and its subcomponents will be documented in the top-level component (<code class="language-plaintext highlighter-rouge">DataTable.jsx</code> in our example). We do this even if the top-level React component doesn’t use that <code class="language-plaintext highlighter-rouge">prop</code> directly. This reduces the number of files developers would need to look at if they were trying to figure out what type of <code class="language-plaintext highlighter-rouge">props</code> a component expects.
We also want to make it easy to import component styles and handle static assets. To that end, <code class="language-plaintext highlighter-rouge">ar-data-table.scss</code> imports all stylesheets associated with subcomponents, SASS variables, and mixins. But what about static assets? How do we make that easy for Rollup consumers?
Say you want to use an image for a Rollup component that you are developing. In a stylesheet, it might look like:
<figure class="highlight"><pre><code class="language-scss" data-lang="scss">.class-with-background {
background: url('../../images/background-image.png');
}</code></pre></figure>
However, the relative path above becomes a problem when you try to use the component in another project. The relative path works locally when developing a Rollup component, but is not guaranteed to work in other project setups depending on their directory structures. For that reason, we do not publish static assets (e.g. images and fonts) in the component’s <code class="language-plaintext highlighter-rouge">npm</code> package. Rather, we push all statics assets to our CDN and replace all URLs for static assets in our stylesheets with the following:
<figure class="highlight"><pre><code class="language-scss" data-lang="scss">.class-with-background {
background: url('/* @echo STATIC_ASSETS_URL*/background-image.png');
}</code></pre></figure>
We then use <a href="https://www.npmjs.com/package/gulp-preprocess">gulp-preprocess</a>. Gulp-preprocess gives us the ability to replace occurrences of <code class="language-plaintext highlighter-rouge">/* @echo STATIC_ASSETS_URL */</code> with relative paths for local developemt and with absolute paths (ponting to the corresponding static assets on our CDN) for the <code class="language-plaintext highlighter-rouge">npm</code> or publication build. The absolute URLs that are published in the <code class="language-plaintext highlighter-rouge">npm</code> package, means that component users do not need to worry about the relative paths that would otherwise be in the <code class="language-plaintext highlighter-rouge">src/styles/</code> files.
One side-effect of the preprocessing, is that component users need to load the stylesheets from <code class="language-plaintext highlighter-rouge">dist/</code> instead of <code class="language-plaintext highlighter-rouge">src/</code>. They still only have to load one <code class="language-plaintext highlighter-rouge">.scss</code> file in order to include all the styles needed for the component, but instead of loading them from <code class="language-plaintext highlighter-rouge">src/</code> the import looks like the following:
<figure class="highlight"><pre><code class="language-scss" data-lang="scss">@import 'ar-data-table/dist/scss/ar-data-table';</code></pre></figure>
<h4 id="es6-transpilation-in-other-applications">ES6 transpilation in other applications</h4>
Now let’s go over how we enable Rollup consumers to deal with components written in ES6. We use <a href="https://github.com/babel/babelify">babelify</a> to transpile the <code class="language-plaintext highlighter-rouge">src/</code> ES6 code into <code class="language-plaintext highlighter-rouge">dist/</code> code. Currently, the assumption is that other teams are also using <a href="http://browserify.org/">browserify</a> to compile their production code. So in the <code class="language-plaintext highlighter-rouge">npm</code> package for each Rollup component we have <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/package.json#L18-L22">added the following to its package.json</a>:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">"browserify": {
"transform": [
"babelify"
]
}</code></pre></figure>
…and we have also listed <code class="language-plaintext highlighter-rouge">babelify</code> as a <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/package.json#L24">peer dependency</a> so the browserify transform plugin will be <code class="language-plaintext highlighter-rouge">npm install</code>-ed at the root of the parent application. This makes it so that other projects and applications using Rollup components do not need to setup the ES6 → ES5 transpilation step themselves, provided they’re also using <code class="language-plaintext highlighter-rouge">browserify</code>.
<h3 id="dist"><code class="language-plaintext highlighter-rouge">dist/</code></h3>
The <code class="language-plaintext highlighter-rouge">dist/</code> directory is the component’s CDN build. <a href="https://github.com/AdRoll/react-component-skeleton/tree/9a6f1e7df1965b342066ca029cc8cce0a3f41295/dist">This folder</a> contains:
<ul>
<li>the component <code class="language-plaintext highlighter-rouge">.scss</code> files with absolute URLs we can load static assets from the CDN</li>
<li>an ES5 <code class="language-plaintext highlighter-rouge">.js</code> file for each component</li>
<li>CSS for the component</li>
</ul>
The build step that produces the <code class="language-plaintext highlighter-rouge">.js</code> and <code class="language-plaintext highlighter-rouge">.css</code> files in <code class="language-plaintext highlighter-rouge">dist/</code> allows Rollup developers to implement the components in a build-system and framework-agnostic way. This allows components to be compatible with a variety of project setups (differing build systems, or lack of a build system completely) and makes them accessible to all engineering teams, regardless of what type of frontend resources they have or build system they are (or are not) using.
So that Rollup components can be used easily using the <code class="language-plaintext highlighter-rouge">dist/</code> code, components are attached to a global Rollup namespace. For example, a data table component would be accessible to a developer loading the compiled <code class="language-plaintext highlighter-rouge">.js</code> file via <code class="language-plaintext highlighter-rouge">Rollup.DataTable</code>.
<figure class="highlight"><pre><code class="language-js" data-lang="js">var properties = { /* ... */ };
React.render(
React.createElement(Rollup.DataTable, properties),
document.getElementById('data-table-container')
);</code></pre></figure>
We use <a href="http://browserify.org/">browserify</a> and <a href="https://babeljs.io">Babel</a> to compile our distribution JavaScript from a <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/src/dist.js">dist.js entry file</a>. The <code class="language-plaintext highlighter-rouge">dist.js</code> file for our data table component looks like this:
<figure class="highlight"><pre><code class="language-js" data-lang="js">import DataTable from '../.';
window.Rollup = window.Rollup || {};
window.Rollup.DataTable = DataTable;</code></pre></figure>
The component CDN build does not include external dependencies, and all dependencies are <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/src/package.json#L5-L9">assumed to exist in the global scope</a>.
The CSS in the <code class="language-plaintext highlighter-rouge">dist/</code> folder is compiled from the <code class="language-plaintext highlighter-rouge">gulp-preprocess</code>ed Sass using <a href="https://www.npmjs.com/package/node-sass">node-sass</a>.
<h2 id="rollup-developer-tools">Rollup developer tools</h2>
Each component has quite a lot going on inside of its directories. Now imagine trying to maintain many components, and also make it easy for other developers to find their way around using several of them. To ease the pain of these tasks, we built developer tools as needed. These tools live in the Rollup repo alongside the components.
As we added more and more components, we wanted to accomplish the following:
<ul>
<li>reduce the number of steps required for initial component setup</li>
<li>reduce the number of things we needed to remember and worry about while reviewing changes to existing components</li>
<li>maintain component stability across version updates</li>
<li>improve the documentation and tooling for those teams interested in using Rollup components</li>
</ul>
To address these issues the Rollup repo has the following tools:
<ul>
<li><a href="#automated-testing">automated testing</a></li>
<li><a href="#live-component-examples">live component examples</a></li>
<li><a href="#new-component-generator">new component generator</a></li>
<li><a href="#automatic-documentation-generation">automatic documentation generation</a></li>
<li><a href="#component-asset-bundles">component asset bundles</a></li>
<li><a href="#example-applications">example applications</a></li>
</ul>
With that in mind, we’re going to talk about two categories of developer tools:
<ul>
<li><a href="#developer-tools-for-rollup-contributors">developer tools for Rollup contributors</a></li>
<li><a href="#developer-tools-for-rollup-users">developer tools for Rollup users</a></li>
</ul>
<h3 id="developer-tools-for-rollup-contributors">Developer tools for Rollup contributors</h3>
One set of developer tools was built to help Rollup contributors focus on the important things when adding new components, and also when reviewing changes to existing components. These developer tools were also put in place to help maintain the stability of components across version updates.
<h4 id="automated-testing">Automated testing</h4>
One of the things we believe to be important is stability and consistency across the repo. For stability, we set up repo-wide infrastructure for <a href="https://facebook.github.io/jest/">jest</a> tests. For consistency we added a linter to the Rollup repo using <a href="http://eslint.org/">eslint</a>.
The linting and testing can be run individually for each component, but they are also run on every pull request by a <a href="https://jenkins-ci.org/">Jenkins</a> “PR Builder” job that updates the GitHub status of the most recently pushed commit. The jenkins job also checks to make sure the automatically generated component documentation up to date.
The tests and linter were put in place to help reviewers and code authors. The linter and test suite help the reviewer by allowing them focus on the important things. In other words, the reviewer can focus on the architectural and functional changes, and not have to worry about code style. With <code class="language-plaintext highlighter-rouge">jest</code> tests that test the behavior of a component when given certain <code class="language-plaintext highlighter-rouge">props</code>, the reviewer and author can also be sure that internal code changes do not affect the component interface. It’s an added bonus that these tests are run automatically, so neither the reviewer or author need to locally run all the checks that Jenkins runs, and better yet they do not need to worry about remembering to.
<h4 id="live-component-examples">Live component examples</h4>
In addition to the <code class="language-plaintext highlighter-rouge">src/</code> and <code class="language-plaintext highlighter-rouge">dist/</code> directories, each component also has an <code class="language-plaintext highlighter-rouge">examples/</code> directory. The examples directory has three files:
<ul>
<li><a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/examples/example.jsx">example.jsx</a>: imports the component code</li>
<li><a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/examples/example.scss">example.scss</a>: imports the component styles</li>
<li><a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/examples/index.html">index.html</a>: loads the example code after it’s been compiled.</li>
</ul>
Every component has a <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/gulpfile.js#L25">gulp dev</a> task that compiles the example JavaScript and CSS. The <code class="language-plaintext highlighter-rouge">gulp dev</code> task also sets up a local server that loads <code class="language-plaintext highlighter-rouge">index.html</code> and live reloads the page when there are any changes to the example or the component <code class="language-plaintext highlighter-rouge">src/</code> code.
<center>
<img alt="Rollup Data Table live example" src="/images/post_images/rollup_data_table_example.png" />
</center>
The live component examples help the reviewer and author by giving them an easy way to interact with the component. Both the reviewer and author can interact with the component, without having to worry about how to work with an example implementation of the code. The author or contributor also is given an isolated environment in which they can test their changes during development.
<h4 id="new-component-generator">New component generator</h4>
This tool was put in place to simplify the process of creating a new component. Based on what you’ve read about the <code class="language-plaintext highlighter-rouge">dist/</code> and the <code class="language-plaintext highlighter-rouge">src/</code> directories, it’s clear that there are several configuration files needed in order for all the assets to be built and for the component to eventually be published. Also, these files and configuration options are almost identical across components.
As we added more components to the repo, it seemed tedious to create very similar files and directory structures over and over again. Also, with the number of files and configs that are needed, the initial setup phase took about an hour. Not to mention, that it’s easy to forget some of the files and configuration options as you go.
So we created a new component generator using <a href="http://yeoman.io/">Yeoman</a>. This generator simply asks for the name of the component you would like to create, validates that it starts with <code class="language-plaintext highlighter-rouge">ar-</code> (per our convention for Rollup components), and then creates all the necessary files and directories for you. The generator also <code class="language-plaintext highlighter-rouge">npm install</code>’s some dependencies that are commonly needed for Rollup component development. For example, React, React Bootstrap, babelify, and others. The generator even adds a failing <code class="language-plaintext highlighter-rouge">jest</code> test, to encourage developers to add tests for stability (see <a href="#automated-testing">Automated Testing</a> for more information) from the get go.
At the end of the component generation, the developer has <a href="https://github.com/AdRoll/react-component-skeleton/tree/9a6f1e7df1965b342066ca029cc8cce0a3f41295">a skeleton</a> with all the right files they need to develop and even publish the component later. <a href="http://gulpjs.com/">Gulp</a> tasks are already defined for the developer, and the component “entry” <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/src/styles/ar-react-component-skeleton.scss">.scss</a> and <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/src/components/ReactComponentSkeleton.jsx">.jsx</a> files are created as well. The <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/package.json">package.json</a> for the component even already has a <a href="https://github.com/AdRoll/react-component-skeleton/blob/9a6f1e7df1965b342066ca029cc8cce0a3f41295/package.json#L6">main field</a> pointing to the right <code class="language-plaintext highlighter-rouge">.jsx</code> file in <code class="language-plaintext highlighter-rouge">src/</code>. The idea here is that the developer can hit the ground running. Ideally, this will also reduce the barrier to entry for new Rollup contributors.
This generator not only cuts down on our development time by removing that one hour needed just to get a new component off the ground, but it also cut down on our review time. Reviewers no longer needed to worry that all files for new components were added manually by the author.
<h2 id="developer-tools-for-rollup-users">Developer tools for Rollup users</h2>
The other half of our developer tools were put in place to make it easy for teams to use Rollup components, regardless of what build system they are (or are not) using, and how comfortable they are with frontend work.
<h4 id="automatic-documentation-generation">Automatic documentation generation</h4>
In order to make it easier for users of Rollup to tell what <code class="language-plaintext highlighter-rouge">props</code> components expect and what each <code class="language-plaintext highlighter-rouge">prop</code> is responsible for (without them having to dig into the source code) we have set up a repo-level gulp task that automatically generates component <code class="language-plaintext highlighter-rouge">prop</code> documentation based on the component <a href="https://facebook.github.io/react/docs/reusable-components.html#prop-validation">prop validation</a> and the comments above them.
Using <a href="https://www.npmjs.com/package/react-docgen">react-docgen</a> we can generate a json file that represents the prop types each component expects. With the json file output by react-docgen and a little <a href="http://handlebarsjs.com/">handlebars</a> magic we can go from this:
<figure class="highlight"><pre><code class="language-js" data-lang="js">/**
* The columns you want the data table to have. Each column can have the
* following attributes:
* - `key` **(required)**: column identifier
* - `label` **(required)**: Display text for the column. Should already
* be translated when passed to the DataTable.
* - `accessor` **(required)**: Function that returns the relevant value
* from a given data item. Later passed to the column `render`
* function.
* - `render` **(required)**: Function that takes the output of the
* `accessor` and returns what should be rendered for a given data item
* in that column. Should return either a formatted value or can also
* be html. Columns without `render` functions will not be displayed
* but can be used for filtering (see the `filters` prop for more
* information).
* - `textAlignment`: Column is center-aligned by default. Use
* `DataTable.TEXT_ALIGN_LEFT` or `DataTable.TEXT_ALIGN_RIGHT` to
* - `widthMultiplier`: Number to multiply the width of the column
* relative to other columns. By default, columns are of equal width.
* - `adminOnly`: Whether or not this is an admin-only or
* permission-gated column. `adminOnly` columns will only be shown if
* the table's `isAdmin` prop is `true`.
*/
columns: React.PropTypes.arrayOf(
React.PropTypes.shape({
key: React.PropTypes.string.isRequired,
label: React.PropTypes.string.isRequired,
accessor: React.PropTypes.func.isRequired,
render: React.PropTypes.func,
textAlignment: React.PropTypes.oneOf([
TEXT_ALIGN_LEFT,
TEXT_ALIGN_RIGHT
]),
widthMultiplier: React.PropTypes.number,
adminOnly: React.PropTypes.bool
})
),</code></pre></figure>
To this:
<center>
<img alt="Rollup Data Table prop documentation" src="/images/post_images/rollup_data_table_prop_docs.png" />
</center>
To generate the documentation, we have defined a handlebars partial for each possible <code class="language-plaintext highlighter-rouge">React.PropType</code>. For example, a prop of type <code class="language-plaintext highlighter-rouge">React.PropTypes.string</code> has its own handlebars partial. We can then recursively generate documentation for complicated prop types, e.g. shapes with attributes that can be arrays of strings…
The automatically generated documentation should be updated for each component version. So, in addition to running <code class="language-plaintext highlighter-rouge">jest</code> tests and our linter (see <a href="#automated-testing">Automated testing</a>), our Jenkins PR Builder also checks to see if the documentation is up to date with the code in the PR. The author of a PR can update the documentation based on their changes, by running a top-level gulp task we have set up to do just that.
<h4 id="component-asset-bundles">Component asset bundles</h4>
To help with Rollup adoption and further usage, we created component asset bundles. Each component can have a variety of dependencies that are assumed to be in the global scope when using the code from the <code class="language-plaintext highlighter-rouge">dist/</code> directory. Teams may also want to use multiple components at a time.
So, we have published <code class="language-plaintext highlighter-rouge">components[.min].{js,css}</code> and <code class="language-plaintext highlighter-rouge">vendor[.min].{js,css}</code> asset bundles.
The component bundles make it easy for others to use several components at a time. The vendor bundles also makes it easy to use one component, or multiple components at a time. It saves Rollup users the time of having to figure out which dependencies they need to load for the components to work, and also saves them the time of having to find publicly accessible versions of those dependencies.
<h4 id="example-applications">Example applications</h4>
As developers, sometimes code speaks for itself better than documentation can. In Rollup we have an <code class="language-plaintext highlighter-rouge">examples/</code> directory (not to be confused with the component-specific <code class="language-plaintext highlighter-rouge">examples/</code> directories) with two example admin applications in them.
<center>
<img alt="Rollup Data Table Admin example" src="/images/post_images/rollup_admin_app.png" />
</center>
The two examples look identical with a form and a data table, but one is built using a simple gulp build, and the other is built by loading the component and vendor bundles from the CDN. Both examples are interactable.
We wanted to implement the same application with different build systems (or lack thereof) so that Rollup users can choose what type of build system they would like to use (or not use), and have an example to work off of for both cases. The two applications not only serve as examples, but the code can be easily copied and serve as a good starting point for a new project.
In addition to serving as a good template for new projects or pages, both applications are meant to be educational for users of Rollup just starting to work on frontend-leaning projects. Both examples are heavily commented, and the comments are aimed at developers just beginning to learn about frontend work, as opposed to those who already know how <code class="language-plaintext highlighter-rouge">gulp</code> works, for example.
The example applications serve as a nice compliment to the examples for individual components. The examples for individual components show usage of each one individually, while the full applications demonstrate how several could be included on one page.
<h2 id="major-learnings">Major Learnings</h2>
As you can tell there is a lot going on here and a lot of thought that went into the technical details of the Rollup repo. <a href="/blog/frontend/2015/11/19/rollup-major-learnings.html">Next week</a> we’ll talk about:
<ul>
<li>what we learned</li>
<li>how some of the technical pieces we talked about here fit into those learnings</li>
<li>how those learnings influenced some of the technical decisions</li>
</ul>
</description>
</item>
<item>
<title>Rollup: Shared UI components at AdRoll</title>
<link>https://tech.nextroll.com/blog/frontend/2015/11/05/rollup-shared-ui-components.html</link>
<pubDate>Thu, 05 Nov 2015 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/frontend/2015/11/05/rollup-shared-ui-components</guid>
<description>This is the first in a series of three blog posts about Rollup, AdRoll’s UI component library. This post covers how and why we built a component library. See the <a href="/blog/frontend/2015/11/12/rollup-react-and-npm-at-adroll.html">second post</a> for details on the technical implementation, and the <a href="/blog/frontend/2015/11/19/rollup-major-learnings.html">third post</a> for a discussion of what we learned.
The <a href="http://blog.adroll.com/product/adroll-prospecting">AdRoll Prospecting launch</a> forced us to rethink the way we build UIs here at AdRoll. For the June 17th launch, we deployed two new and similar dashboards: the Prospecting dashboard and a new multi-product landing page.
<h2 id="the-old-way-of-doing-things">The old way of doing things</h2>
Until recently, the frontend code for our client-facing dashboard lived in one Git repository and was built using <a href="http://backbonejs.org/">Backbone.js</a>. This made it easy for different projects and features to share UI components. But the downside was that as new features were added to the UI, the code became tightly coupled and the implementation of a single UI component would often end up fragmented across several files. DOM manipulations and application state updates were happening in any one of those files. It was not only hard to figure out how user interactions were implemented, but became very difficult to maintain.
<h2 id="the-new-way-of-doing-things">The new way of doing things</h2>
Having learned our lesson from the monolithic and complex codebase, the two new dashboards were being built as separate web applications in their own repositories. But because the UI wireframes for both dashboards were so similar, we also needed a scalable way to maintain the consistent look and feel across products.
<center>
<img alt="Product Comparison" src="/images/post_images/rollup_dashboard_comparison.png" />
</center>
 
So we knew we had to find a way for the UI components to be shared by both projects as easily as we could share code in a single repository. Sharing the code across repositories would eliminate the need for duplicate code and work for the initial implementation and when the UX specification for a component changes.
What we ended up with is the Rollup repository (not to be confused with the <a href="http://rollupjs.org">other Rollup</a>) - a reusable component library that now contains implementations for all major components a data-oriented UI should have. Those components include:
<ul>
<li>Line chart</li>
<li>Multi Select dropdown</li>
<li>Date picker</li>
<li>Data Table</li>
<li>User alerts</li>
<li>Navbar</li>
<li>Data summary bar with spark lines</li>
<li>a shareable i18n module (localizes strings, dates, and currencies)</li>
</ul>
Each component in Rollup is implemented in <a href="https://facebook.github.io/react/">React.js</a> and were developed so that they could be used in a variety of frameworks and by anyone at AdRoll with a basic understanding of JavaScript. Each Rollup component also uses <a href="http://semver.org/">Semantic Versioning</a> and is published as an <a href="https://www.npmjs.com/">npm</a> package to a private <code class="language-plaintext highlighter-rouge">npm</code> registry we have set up in <a href="http://www.jfrog.com/open-source/">Artifactory</a>.
While the framework components are built in is not set in stone, React is our current convention. Components built in React can be self contained and have a well-defined interface. We chose React for a few reasons:
<ul>
<li>React components do not require separate template files, thanks to <code class="language-plaintext highlighter-rouge">jsx</code></li>
<li>Each component defines its interface using <a href="https://facebook.github.io/react/docs/reusable-components.html">React propTypes</a></li>
<li>Component rendering is deterministic based on component props / state instead of direct DOM manipulation</li>
</ul>
<h2 id="why-we-like-the-rollup-repository-so-much">Why we like the Rollup repository so much</h2>
Establishing the Rollup repository for the AdRoll Prospecting launch worked just as expected, if not better. Before the launch we had a resource problem: three experienced frontend engineers had a month and a half to develop and deploy two separate dashboards. By creating the Rollup repo, we were able to fully parallelize the work required to build the two dashboards between all three engineers.
In addition to providing an easy way to share code between projects regardless of whether they are in the same repository or not, the component library has a few other benefits:
<ul>
<li>Each component is developed independently from other projects, build systems, and even other components</li>
<li>Each component’s source and compiled code are published so that components are compatible with a variety of project setups</li>
<li>The Rollup repo serves as a place at AdRoll where we can safely adopt new frontend practices, tools, and frameworks</li>
<li>Teams responsible for building UIs can move faster thanks to shared improvements and bug fixes.</li>
<li>Having a unified component library makes maintaining a consistent look and feel across all products easy</li>
</ul>
<h2 id="whats-next-for-rollup">What’s next for Rollup</h2>
So, what’s next?
Now that the implementations of many of the components have stabilized, we have a few things in mind that we’d like to do:
<ul>
<li>Drive component adoption</li>
<li>Continue to improve the documentation for the repository and individual components</li>
<li>Encourage contributions to the Rollup repository, in the form of bug fixes or entirely new components</li>
<li>Continue to improve the repository tooling and share frontend knowledge with other teams (supports the bullet point above)</li>
<li>Open-source Rollup components. When we started work on Rollup, we took a look at open-source implementations for some of the more complicated UI components, e.g. the data table and date picker. However, we were not happy with open-source implementations we found since they did not support our use cases and were not easily extensible.</li>
</ul>
Next week, we’ll be publishing the <a href="/blog/frontend/2015/11/12/rollup-react-and-npm-at-adroll.html">second post</a> in this series, covering the technical details of the Rollup repository.
</description>
</item>
<item>
<title>My Journey to Engineering</title>
<link>https://tech.nextroll.com/blog/personal/2015/10/29/my-journey-to-engineering.html</link>
<pubDate>Thu, 29 Oct 2015 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/personal/2015/10/29/my-journey-to-engineering</guid>
<description><h4 id="adroll--part-i">AdRoll — Part I</h4>
As a high school math teacher, I never imagined that I would become a software engineer, but here I am.
After teaching and running a math tutoring business for five years, I began to yearn for a new challenge and started to tackle <a href="https://www.codecademy.com/learn/python">Codecademy’s Python course</a>. By the end of the section on for loops, I was hooked; the structured way of thinking that programming required jived with my math brain, and I knew that I needed to find a job that would allow me to explore this new interest. I began to apply to analyst roles, while simultaneously cramming SQL into my brain. After a couple months of working on these basics on my own, I managed to pass the AdRoll interviews and convince the CFO to let a math teacher help him with company analytics.
In my Business Intelligence (BI) role, my appreciation for programming soared to even greater heights. I devised scripts, in both R and Python, that ultimately allowed me to automate elements of my job, eliminating approximately 60 hours of work per month. This role afforded me the freedom to learn new technical skills that allowed me to excel in my work, while also helping me discover my passion for coding.
Given this revelation, I decided that I wanted to transition to a full-time engineering role, but I was acutely aware that there were fundamental gaps in my knowledge base that I would need to fill before qualifying for such a transition. In order to acquire this knowledge, I knew that I would either need to engage in several months of self-study outside of work or fork over thousands of dollars to attend an accelerated engineering bootcamp. Ultimately, I chose the riskier approach, leaving my secure job and selling my Mini Cooper to fund <a href="https://hackbrightacademy.com/">Hackbright</a>, a 10-week software engineering fellowship.
<h4 id="hackbright">Hackbright</h4>
At Hackbright, I had the luxury of focusing all of my time on coding for 10 weeks, which allowed me to build a more cohesive foundation of knowledge. The first five weeks were classroom and lab-based education focused on the fundamentals of modern day web development, and in the second five weeks, I built a full-stack web application from scratch.
With an enriched skillset, bolstered confidence, and a web app built, I began the process of studying for technical interviews. This was an incredibly daunting task - how would I learn all of the things?! Did I need to know pre, post, and in order traversals of binary trees? Did I need to know how to write merge and quick sort?
I quickly came to realize that I would never learn all of the things. It’s not possible, and frankly, it’s what I love about this field of work. There is an endless supply of things to learn, and for this reason, I’ll never get bored. So for the purposes of studying, I just started chipping away at a list of concepts. Repetition proved to be deceivingly valuable, stumbling through a problem the first time, doing it faster the second time, and maybe even discovering a more optimal solution the third time.
Once I had accepted that I would never know all of the things, I understood that I would also never feel “ready” to start interviewing. So I just had to start. This was the most terrifying part of the whole journey, as I knew that I would fail some interviews, and I was very concerned that I wouldn’t get back up after a rejection. What if I didn’t know something in an interview, knew something but completely blanked, thought of the more optimal solution as soon as I left the building? All of these things happened, but shockingly, it didn’t hurt that badly when I fell, and I did get back up because I was hungry and determined.
<h4 id="adroll--part-ii">AdRoll — Part II</h4>
After interviewing and receiving a couple of offers, I decided to return to AdRoll’s engineering team, most significantly because of the supportive culture here. Throughout my time on BI, my hiatus at Hackbright, and my interview process, there was always a team of my peers and superiors who were cheering for me. Knowing that I would have that encouragement as I embarked on this career transition was truly pivotal in my decision.
In addition to the aid of the supportive environment in which I have the privilege of working everyday, I have also personally taken measures to ensure my smooth transition to engineering. Firstly, I quickly constructed a support team that I could go to with a wide array of questions - everything from setting up my dev environment to codebase specific concerns. I took a “spread the love” approach with these folks and would rotate through them. In this way, I was able not only to be proactive about my learning, but also to establish relationships and allies within engineering immediately.
Additionally, without the warm, fuzzy blanket of Hackbright wrapped around me, I quickly learned to be my own cheerleader and celebrate my victories, even the tiny ones. I chose this path because it is hard, and the challenge is part of the thrill, so it is hugely important that I allow myself some (smug) satisfaction when I finally get that unit test to pass. Those are the glorious moments that make this type of work so rewarding.
Lastly, I had to fight off the self-constructed notion that I was an impostor or a fraud and convince myself that I could contribute right away. Whether it’s teaching someone a new built-in Python method or my fresh perspective on a piece of code that uncovers a gap in test coverage, I have been able to add value, and I will just continue to add more.
In closing, I can honestly say it has all been worth it. The long hours, the leaps of faith, and the various unknowns have all led me to what I have been searching for: unending intellectual stimulation (with a paycheck). I am eternally grateful to those who first gave me a chance at AdRoll and to those who have continued to support me throughout my tenure here.
</description>
</item>
<item>
<title>Managing Containerized Data Pipeline Dependencies With Luigi</title>
<link>https://tech.nextroll.com/blog/data/2015/10/15/luigi.html</link>
<pubDate>Thu, 15 Oct 2015 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data/2015/10/15/luigi</guid>
<description>This is the <a href="http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html" title="Docker @AdRoll">second article</a> in a series that describes how we built AdRoll Prospecting. This time, we’ll talk about managing batch job dependencies.
Batch processing and data pipeline coordination is one of the oldest application of computer systems. In late 1950s US Navy and NASA first started using computers for task scheduling—tasks being assembly jobs performed by people. Controlling people that way is what <a href="https://en.wikipedia.org/wiki/Topological_sorting" title="Topological sorting">topological sorting algorithms</a> were invented for. Later, in mid-1960s, one of the killer features of the IBM’s novel operating system OS/360 was its extensive mainframe task management capabilities.
<img src="/images/post_images/luigi_RTOS360.png" alt="RTOS/360" title="RTOS/360" />.
Back then, system level limitations, like memory and disk space, were major concerns. Therefore, scheduling required a lot of manual hints and tweaks.Nowadays—well, maybe over the last 40 years—this became less of an issue. One can easily buy hundreds of gigabytes and terabytes of RAM and disk space, enough to store, say, bios and photos of every person on Earth for the price of a family car.
Building effective and robust data processing pipelines became much easier compared to 1960s; there are quite a few tools available to do this today. To name some, since it is hard to keep a comprehensive list:
<ul>
<li><a href="https://github.com/mesos/chronos" title="Chronos">Chronos</a> from AirBnB</li>
<li><a href="https://github.com/airbnb/airflow" title="Airflow">Airflow</a> from AirBnB</li>
<li><a href="https://github.com/spotify/luigi" title="Luigi">Luigi</a> from Spotify</li>
<li><a href="https://aws.amazon.com/datapipeline/" title="AWS Data Pipeline">AWS Data Pipeline</a></li>
</ul>
Many of them can fit the bill for a lot of cases, but still, there are some trivial things that in our experience are easy to overlook and get oneself in trouble. Specifically:
<h4 id="problem-with-time-based-scheduling">Problem with time based scheduling</h4>
Time-based scheduling is something everyone starts with. For example, analysts want a daily revenue report to be generated at 9AM daily. We can just use <code class="language-plaintext highlighter-rouge">cron(8)</code> or try something fancier like Chronos for this. And that works fine for simple cases.
<img src="/images/post_images/luigi_timebased_1.png" alt="timebased_1" title="Scheduling 1" />
<h4 id="next-stop-dependencies">Next stop: dependencies</h4>
Not too long after setting up <code class="language-plaintext highlighter-rouge">cron</code>, it turns out that we need our task to run after a daily database ETL job runs. A simple solution comes to mind: just run ETL job at 6AM, then trigger daily revenue report generation job as ETL job ends.
<img src="/images/post_images/luigi_timebased_2.png" alt="timebased_2" title="Scheduling 2" />
And a bit later, but sooner than one may think, our pipeline will need to support multiple dependencies. Now these are not that easy to express in a time-based system: one cannot unconditionally trigger jobs downstream anymore, as multiple prerequisites must be met before they run.
<img src="/images/post_images/luigi_timebased_3.png" alt="timebased_3" title="Scheduling 3" />
<h4 id="failures-and-reruns">Failures and reruns</h4>
It wouldn’t be that hard to work around these multiple dependencies if not for the fact that jobs fail. Or even worse, they don’t fail, but instead produce incorrect results due to a software bug somewhere upstream.
<img src="/images/post_images/luigi_timebased_4.png" alt="timebased_4" title="Scheduling 4" />
And with our fancy time-based scheduling, things get complicated: what if we need rerun yesterday’s report? Or rather, if yesterday’s report failed, if we rerun it will we get today’s or yesterday’s one? Should we backfill failed jobs or just abandon them and go on? What if that happens across midnight? Will it trigger “today’s” jobs downstream or “yesterday’s”? What if the pipeline starts to take too long? Can we restart or disable part of it?
Correct answer is to sit back and maybe reflect a little. Job pipelines are all about inputs, outputs, and dependencies, but not about time—that can easily be made another virtual dependency expressed as “wallclock time &gt; X.”
<img src="/images/post_images/luigi_timebased_5.png" alt="timebased_5" title="Scheduling 5" />
<h4 id="make1">make(1)</h4>
Some 40 years ago, someone already solved this problem in UNIX. If you define your pipelines as a directed acyclic graph of dependencies, they become trivial to reason about. Simply follow a few rules:
<ul>
<li>
Jobs need to be re-runnable. I would like to say <a href="https://en.wikipedia.org/wiki/Side_effect_(computer_science)" title="Idempotence">idempotent</a> but that’s not always possible in a strict sense. Ideally, they should be pure functions of their inputs. And these inputs should be files.
</li>
<li>
If they do interact with the outside world (call external APIs, and such), try to turn that into a materialized snapshot stored somewhere persistently as a file. That way, job failures are easier to reproduce.
</li>
<li>
Jobs should be transactional, in the sense that they either completely fail or completely succeed.
</li>
<li>
Yesterday’s report generation job and today’s should be separate jobs, meaning they should not output to exactly the same place. Add timestamp or some kind of job id there, treating <a href="https://aws.amazon.com/s3/" title="Amazon Simple Storage Service">S3</a> as write only storage. One could draw some parallels with <a href="https://en.wikipedia.org/wiki/Static_single_assignment_form" title="Static single assignment form">SSA</a> here—this makes it much easier to reason about what’s going on in the code.
</li>
</ul>
Following these principles makes it much simpler than your average makefile, assuming you have enough disk space (which, as we know, is cheap) to write all outputs.
<h3 id="luigi">Luigi</h3>
That’s why we now use Spotify’s <a href="https://github.com/spotify/luigi" title="Luigi">Luigi</a>. It is simple enough, extensible in Python, and doesn’t have weird implicit rules like <code class="language-plaintext highlighter-rouge">make(1)</code>—make is a build system, after all, and has its own quirks. All inputs and outputs are in <a href="https://aws.amazon.com/s3/" title="Amazon Simple Storage Service">S3</a>, which provides great persistence guarantees, unlimited storage space, and read bandwidth that scales more or less linearly with the number of readers.
<h4 id="docker">Docker</h4>
We also wanted to give users the ability to use tools they want for data processing: R comes to mind first; C, which we use extensively; and Lua and Haskell make appearances as well. This is where <a href="https://www.docker.com/" title="Docker">Docker</a> comes handy: we use <a href="https://github.com/spotify/luigi" title="Luigi">Luigi</a> to manage dependencies and schedule containerized jobs, but normally, no data processing happens in the Luigi worker process itself. See our previous <a href="http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html" title="Docker @AdRoll">blog post</a> for more details.
Here’s a typical Luigi job:
<figure class="highlight"><pre><code class="language-python" data-lang="python">class PivotRunner(luigi.Task):
blob_path = luigi.Parameter()
out_path = luigi.Parameter()
segments = luigi.Parameter()
def requires(self):
return BlobTask(blob_path=self.blob_path)
def output(self):
return luigi.s3.S3Target(self.out_path)
def run(self):
q = {
"cmdline" : ["pivot %s %s" % (self.out_path, self.segments)],
"image": 'docker:5000/pivot:latest',
"caps" : "type=r3.4xlarge"
}
quentin.run_queries('pivot', [json.dumps(q)], max_retries=1)</code></pre></figure>
As one can see, it is conceptually very similar to a <code class="language-plaintext highlighter-rouge">Makefile</code>, but with Python syntax.
The only missing piece is a system that would run containers on a cluster of <a href="https://aws.amazon.com/ec2/" title="Amazon EC2">EC2</a> instances, so that one doesn’t have to care about provisioning and scaling the cluster in the Luigi job. We built an in-house solution called Quentin; it is a simple job queue with a REST API and UI on top.
<img src="/images/post_images/luigi_quentin.png" alt="Quentin" title="Quentin" />.
Quentin maintains the task queue and feeds metrics to Cloudwatch so AWS Autoscaling can take care of scaling clusters up and down.
<h4 id="nice-to-have-dynamic-dependencies">Nice to have: dynamic dependencies</h4>
One feature of Luigi that we found very useful is the ability to create nodes in acyclic job graphs dynamically. This makes it possible to shard jobs dynamically based on volume, use Hadoop-like map reduce patterns, and even recursive reduce schemes, with a few lines of code. We use this heavily so our typical pipeline consists of thousands of containerized jobs orchestrated by Luigi, created dynamically by a higher level, “virtual” job.
<img src="/images/post_images/luigi_luigi.png" alt="Luigi" title="Luigi" />.
<h4 id="not-necessary-high-availability">Not necessary: high availability</h4>
Following the practices above, making jobs (mostly) idempotent, and storing all inputs and outputs in a resilient store like <a href="https://aws.amazon.com/s3/" title="Amazon Simple Storage Service">S3</a>, we find that it is not necessary for components like the Luigi server or queue service (Quentin) to be highly available. Distributed systems are <a href="https://en.wikipedia.org/wiki/Byzantine_fault_tolerance" title="Byzantine Failures">hard</a>, and, in this case it costs us much more in development and operational efforts to ensure zero downtime survival during server and data center failures.
Currently, if anything fails, a new Luigi server comes up and continues from where it stopped, based on results that are already in S3. This works perfectly as long as the data pipeline is not very sensitive to small delays.
<h3 id="conclusion">Conclusion</h3>
There are certainly things to improve in this ecosystem (like our friends at <a href="http://pachyderm.io" title="Pachyderm">Pachyderm</a> are trying to do), but even with the existing tools, it really pays off when you keep things simple: S3 as a data store, <code class="language-plaintext highlighter-rouge">make</code>-like dependency scheduler on top, simple non-HA job queue, and Docker as a working horse works. This combination works very well for us in practice.
If you’d like more details on our experience, here are <a href="http://www.slideshare.net/AmazonWebServices/cmp310-data-processing-pipelines-using-containers-spot-instances" title="slides">slides</a> and <a href="https://www.youtube.com/watch?v=bNkfdU_LDHg" title="video">video</a> of the talk on our data pipeline at <a href="https://reinvent.awsevents.com/" title="AWS re:Invent 2015">re:Invent 2015</a>.
<iframe src="//www.slideshare.net/slideshow/embed_code/key/JTiSYzYQ76qa93" width="425" height="355" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen=""> </iframe>
Video:
<iframe width="560" height="315" src="https://www.youtube.com/embed/bNkfdU_LDHg" frameborder="0" allowfullscreen=""></iframe>
</description>
</item>
<item>
<title>Petabyte-Scale Data Pipelines with Docker, Luigi and Elastic Spot Instances</title>
<link>https://tech.nextroll.com/blog/data/2015/09/22/data-pipelines-docker.html</link>
<pubDate>Tue, 22 Sep 2015 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data/2015/09/22/data-pipelines-docker</guid>
<description>This is the first article in a series that describes how we
built a new data-intensive product, <a href="http://blog.adroll.com/product/adroll-prospecting" title="AdRoll Prospecting">AdRoll Prospecting</a>, using
an architecture based on Docker containers.
We <a href="http://www.meetup.com/Bay-Area-Big-Data-with-Docker/" title="Big Data with Docker meetup">organized a meetup</a> about this topic with <a href="http://pachyderm.io" title="Pachyderm">Pachyderm.io</a>.
You can watch a recording of our meetup talk here:
<iframe width="560" height="315" src="https://www.youtube.com/embed/4Nv_DXMNoN0" frameborder="0" allowfullscreen=""></iframe>
and see the slides here:
<iframe src="//slides.com/villetuulos/pb-scale-workflows/embed" width="576" height="420" scrolling="no" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
We will elaborate different aspects of the architecture in upcoming blog
posts. The one below will focus on Docker.
<h2 id="a-modern-data-driven-product">A Modern Data Driven Product</h2>
On June 17th, we launched a new product, <a href="http://blog.adroll.com/product/adroll-prospecting" title="AdRoll Prospecting">AdRoll Prospecting</a>,
to public beta. A remarkable thing about the launch is that the product
was built from scratch by a small engineering team of six people, in
about six months, and it was released on time.
The product does something that is practically the Holy Grail of
marketing: The core of AdRoll Prospecting is <a href="http://blog.adroll.com/trends/predictive-behavioral-targeting" title="Lookalike Model">a massive-scale machine
learning model</a> that is able to predict who is most likely
interested in your product, amongst billions of cookies AdRoll knows
something about, and thus can find new customers for your business.
A modern, data-driven product like Prospecting is not only about
machine learning. We provide also an easy-to-use dashboard (built
using <a href="https://facebook.github.io/react/" title="React.js">React.js</a>) that allows you to view performance of your
prospecting campaigns in detail. Behind the scenes, we are connected to
AdRoll’s <a href="http://blog.adroll.com/news/valentino-volonghi-aws-summit-2015" title="RTB">Real-Time Bidding engine</a>, and we have dozens of checks
and dashboards monitoring the health of the product internally, so we
can be proactive about any issues affecting customer accounts.
Thanks to our experience with AdRoll’s existing <a href="https://www.adroll.com/product/web-retargeting" title="AdRoll Retargeting">retargeting
product</a>, we had an idea of what building a complex system like
this would entail. When we started building AdRoll Prospecting, we
were able to look back at the lessons learned, and think how we could
build a flexible and sustainable backend architecture for a massively
data-driven product like this <a href="http://firstround.com/review/speed-as-a-habit/" title="Speed As A Habit">as quickly as possible</a> without
sacrificing robustness or cost of operation.
<h2 id="managing-complexity">Managing Complexity</h2>
We have been very happy with the result, which is the motivation for
this series of blog articles. Not only has it allowed us to build and
release the product on time, but we are planning to migrate many existing
workloads to the new system as well.
Probably the most important feature of the new architecture is its lack
of features, i.e. simplicity. Knowing that the problem we are
solving is so complex, we did not want to complicate it further with
a framework that would force us to model the problem <a href="http://martinfowler.com/bliki/InversionOfControl.html" title="Frameworks">in terms of a
framework</a>.
Our architecture is based on a stack of three complementary layers, which
heavily rely on well-known, battle-hardened components:
<ol>
<li>
At the lowest level, we use <a href="https://aws.amazon.com/ec2/spot/" title="Spot Instances">AWS Spot Instances</a> and
<a href="https://aws.amazon.com/autoscaling/" title="AutoScaling">Auto-Scaling Groups</a> to provide computing resources on a demand
basis. Data is stored in <a href="https://aws.amazon.com/s3/" title="Amazon Simple Storage Service">AWS Simple Storage Service (S3)</a>. We have
built a simple in-house job queue, Quentin, so we can leverage <a href="http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/policy_creating.html" title="CloudWatch">custom
CloudWatch metrics</a> to trigger scaling based on the actual
length of the job queue.
</li>
<li>
We orchestrate a complex graph of interdependent batch jobs using
<a href="https://github.com/spotify/luigi" title="Luigi">Luigi</a>, a Python-based, open-source tool for workflow management.
</li>
<li>
At the highest level, each individual task (batch job) is packaged as
a <a href="https://www.docker.com/" title="Docker">Docker container</a>.
</li>
</ol>
This stack allows anyone to build new tasks very quickly using Docker,
define their dependencies in terms of inputs and outputs using Luigi,
and get them executed on any number of EC2 instances without having to
worry about provisioning thanks to our scheduler and auto-scaling groups,
as illustrated below.
<img src="/images/post_images/docker-luigi-arch.png" alt="Docker, Luigi and Quentin" title="Architecture Diagram" />.
This seemingly simple architecture makes a vast amount of complexity
manageable. The Docker containers encapsulate jobs written in seven
different programming languages. Luigi is used to orchestrate a tightly
connected graph of about 50 of these jobs, and Quentin and Auto-Scaling
Groups allow us to execute the jobs on an elastic fleet of hundreds of
the largest EC2 spot instances in a very cost-effective manner.
The main benefit of embracing this heterogeneous, <a href="https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar" title="The Cathedral and The Bazaar">bazaar-like
approach</a> is that we can safely use the most suitable language,
instance type, and distribution pattern for each task.
<h2 id="old-new-paradigm">Old New Paradigm</h2>
Containerized batch jobs have been used for decades. Mainframes have
pioneered batch jobs and virtualization since the 1960s. Outside
mainframes, for example even by the the early 2000s, Google was
isolating batch workloads using operating-system level virtualization
using their in-house system, <a href="http://blog.kubernetes.io/2015/04/borg-predecessor-to-kubernetes.html" title="Google Kubernetes">Borg</a>. A few years after, this
approach became more widely available using open-source tools such as
<a href="http://openvz.org" title="OpenVZ">OpenVZ</a> and <a href="https://en.wikipedia.org/wiki/LXC" title="Linux Containers">LXC</a>, and later with managed services such as
<a href="https://www.joyent.com/blog/hello-manta-bringing-unix-to-big-data" title="Joyent Manta">Joyent Manta</a> that is based on <a href="http://en.wikipedia.org/wiki/Solaris_Containers" title="Solaris Containers">Solaris Zones</a>.
Containers solve three tricky issues in batch processing, namely:
<ol>
<li>
Job packaging - a job may depend on a multitude of third party
libraries that have dependencies of their own. In particular, if the job
is written in a scripting language, such as Python or R, encapsulating
the whole environment in a self-contained package is non-trivial.
</li>
<li>
Job deployment - packaged jobs need to be deployable on a host
machine, and they need to be able to execute on the host without
altering system-wide resources.
</li>
<li>
Resource isolation - if multiple jobs execute on the same host
simultaneously, they must share resources nicely and they must not
interfere with each other.
</li>
</ol>
All these issues have been technically solvable using existing
virtualization techniques for decades prior to Docker. What happened
with Docker is that creating containers became so easy, and socially
acceptable, that today it is realistic to expect that every analyst,
data scientist, and junior software engineer is able to package their
code in a container on their laptop.
A result of this is that we can allow, and even encourage, each user
of the system to use their favorite, most appropriate tools for the
job, instead of learning a new language or computing paradigms such
as MapReduce, whatever makes them most productive. Each person is
naturally responsible for packaging their jobs using Docker, and fixing
them if anything fails in the container.
The result is not only faster time to market, thanks to a more efficient
use of different skillsets and battle-hardened tools such as R, but also
a feeling of empowerment across the organization; everyone can access
data, test new models, and push code to production using the tools they
know best.
<h2 id="good-behavior-expected">Good Behavior Expected</h2>
Containerized batch jobs are not only about peace, love, and continuous
deployment. We expect jobs to adhere to certain ground rules.
The basic pattern that most of our jobs follow is that they only
ingest immutable data from S3 as their input, and produce immutable
data in S3 as their output. If the job sticks with this simple
pattern, it becomes <a href="https://en.wikipedia.org/wiki/Side_effect_(computer_science)" title="Idempotence">idempotent and side-effect free</a>.
In effect, each container becomes a function in the sense of
functional programming. We have found that it is very natural to write
containerized batch jobs with this mindset, from the simplest shell
scripts to the most complex data manipulation jobs.
Another related requirement is that jobs should be atomic. We expect
that jobs produce a <code class="language-plaintext highlighter-rouge">_SUCCESS</code> file, similar to Hadoop, upon successful
completion. Operations on single files are atomic in S3, so this
requirement is easy to fulfill. Our task dependencies are set up with
Luigi so that the output data is considered valid only if the success
file exists, so partial results are not a concern.
We have found this straightforward reliance on files in S3 to be easy
to explain, troubleshoot, and reason about. S3 is a nearly perfect data
fabric: it is extremely scalable, has an amazing record of uptime,
and it is cheap to use. The empowering effect of Docker would be much
diminished if data were less easily accessible.
<h2 id="next-up-luigi">Next Up: Luigi</h2>
Containerized batch jobs benefit from a clear <a href="https://en.wikipedia.org/wiki/Separation_of_concerns" title="Separation of Concerns">separation of
concerns</a>. Not only is it easier to write jobs this way, but
it also helps to ensure that each job takes only minutes, or at most a
few hours, to execute, which is crucial when dealing with ephemeral spot
instances.
An inevitable result of this is that the system becomes a complex
hairball of interdependent jobs. <a href="https://github.com/spotify/luigi" title="Luigi">Luigi</a> has proven to be a
direct way to manage this dependency graph, which is the topic of
a future blog post.
</description>
</item>
<item>
<title>Factorization Machines</title>
<link>https://tech.nextroll.com/blog/data-science/2015/08/25/factorization-machines.html</link>
<pubDate>Tue, 25 Aug 2015 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data-science/2015/08/25/factorization-machines</guid>
<description>The data science team at AdRoll is constantly working to improve our programmatic bidding algorithm, BidIQ. One
recent improvement to BidIQ has been the introduction of a novel modeling technique called Factorization Machines
(FM). The FM model allows us to consider interactions between all predictive variables, rather than only certain
manually selected interactions. Thanks to the FM model, we are able to use the exact same feature information to
more accurately value every potential impression that comes our way. As a result, our advertisers are collecting almost
7% more clicks for only 2% more cost.
To motivate the FM model, let’s consider a simplified ad bidding algorithm that bids using three variables:
web domain, advertiser, and user location. In the first iteration of our modeling, we would learn a weight ${w_i}$
for each web domain, advertiser, and user location. Our bid, $B_{linear}$, would then be based on a function
$f$ of a linear combination of the weight vector $\vec{w}$. That is:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{equation}
B_{linear} = f\left(w_0 + \sum_{i=1}^n w_i x_i\right)
\end{equation}
%]]>
</script>
where $w_0$ is an intercept term and $x_i$ is the value of the $i$th feature. This modeling technique
proved extremely powerful, but the drawback was that the model only learned the effect of our three variables individually
rather than in combination. But what if an advertiser’s ads perform particularly well on some specific domains? What
if users in San Francisco are much more interested in some advertisers than others? What if a particular domain tends
to receive exceptionally valuable traffic from New York? One popular way of solving this problem at scale is to supplement
the standard linear model equation with an additional term to model pairwise feature interactions.
The simplest such strategy is to learn a weight $w_{ij}$ for each feature combination. We would then make our bid,
$B_{quadratic}$, according to:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{equation}
B_{quadratic} = f\left(w_0 + \sum_{i=1}^n w_i x_i + \sum_{i=1}^n \sum_{j=i+1}^n w_{ij} x_i x_j\right)
\end{equation}
%]]>
</script>
where $\mathbf{W} \in \mathbb{R}^{n \times n}$ are additional parameters to be learned. Unfortunately, this naïve approach
will not work for two main reasons. First, the size of the model is now $\mathcal{O}(n^2)$, which has terrible
implications for both the amount of memory needed to store the model, and the time it takes to train the model. Second,
our dataset is too sparse for us to learn the all of the weights $\mathbf{W}$ reliably. That is, for almost all pairs
$(i,j)$ we would not have enough training examples to learn the weight $w_{ij}$ well. To improve the pairwise feature
interaction modeling of $B_{quadratic}$, we could include only the weights $w_{ij}$ with corresponding
features sufficiently dense and informative. In the second iteration of our modeling, we did just this. The problem is
selecting the valuable weights $w_{ij}$ is difficult to do algorithmically, and does not scale to a large number of possible
feature combinations.
FM solves the problem of considering pairwise feature interactions. Indeed, it allows us to bid based on
reliable information from every pairwise combination of variables in the model. Just as important, FM allows us to do
this in a remarkably efficient way both in terms of both time and space complexity. So how exactly does FM work? FM models
pairwise feature interactions as the inner product of low dimensional vectors. More precisely, our bid with the FM model,
$B_{FM}$, becomes:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{equation}
B_{FM} = f\left(w_0 + \sum_{i=1}^n w_i x_i + \sum_{i=1}^n \sum_{j=i+1}^n \left<\vec{v}_i,\vec{v}_j \right> x_i x_j\right)
\end{equation}
%]]>
</script>
where $\mathbf{V} \in \mathbb{R}^{n \times k}$ are additional parameters to be learned, and $\vec{v}_i$ is the $i$th
row of $\mathbf{V}$. Notice that the FM model replaces the weights $w_{ij}$ by $\left&lt;\vec{v}_i,\vec{v}_j\right&gt;$. From
a modeling perspective, this is powerful because each feature ends up embedded in an inner product space, with similar features
embedded near one another. As a result, the FM model is even able to learn the effect of interactions between features which
appear together very infrequently in the training data. Furthermore, the size of the FM model is now a more reasonable
$\mathcal{O}(kn)$, where the latent dimension $k$ is a hyperparameter of the FM model.
However, the FM model is not an obvious improvement from a computation time perspective. In fact, computation of the pairwise
feature interaction term now appears to require $\mathcal{O}(kn^2)$ operations rather than the $\mathcal{O}(n^2)$ of the naïve
interaction modeling solution. Yet this is not the case; after some manipulation we may rewrite the nonlinear FM term
as follows <a href="#footnotes">(1)</a>:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{equation}
\sum_{i=1}^n \sum_{j=i+1}^n \left<\vec{v}_i,\vec{v}_j \right> x_i x_j =
\frac{1}{2} \sum_{j=1}^k \left( \left(\sum_{i=1}^n v_{ij}x_i \right)^2 - \sum_{i=1}^n v_{ij}^2 x_i^2 \right)
\end{equation}
%]]>
</script>
The right hand side of this equation can clearly be computed in $\mathcal{O}(kn)$ time. This is the magic of FM: we are able to
compute the term that models all pairwise interactions in linear time. As a result, we are able to train the FM model in a time
proportional to the time needed to train the linear model.
Apart from the optimization described above, we used several other strategies to reduce training time and improve convergence.
First, the FM model is trained using stochastic gradient descent (SGD), an algorithm known for its speed. <a href="#footnotes">(2)</a>
Second, we parallelized SGD using the lock-free so-called “HOGWILD!” scheme <a href="#references">[2]</a>. This lock-free, parallel SGD is notable
because it avoids blocking any threads by allowing race conditions during model updates. Additionally, we found AdaGrad to be a very effective
learning-rate schedule for training an FM model with SGD <a href="#references">[3]</a>. Finally, we used single instruction multiple data (SIMD)
computation to vectorize calculations involving the matrix $\mathbf{V}$, quartering FM model training time. See below for a D programming
language (pseudocode) example of the vectorized calculation of the FM model equation. With the optimizations described we are able to train
an FM model in approximately the same amount of time it took to train our previous model. This is outstanding as the FM model is
significantly more predictive.
<figure class="highlight"><pre><code class="language-d" data-lang="d">class FMModel
{
uint k; // FM latent dimension
float w0;
float[] w_vector;
float4[][] v_matrix;
// ... other FM model variables and functions ...
float model_equation(Observation obs)
{
float linear_term = w0;
foreach(Feature feat; obs)
linear_term += w_vector[feat.id];
float4 non_linear_term4 = 0;
for(int j = 0; j &lt; (k/4); j++)
{
float4 first_term4 = 0.toFloat4();
float4 second_term4 = 0.toFloat4();
foreach(Feature feat; obs)
{
float4 update = v_matrix[j][feat.id] * feat.weight;
first_term4 += update;
second_term4 += (update * update);
}
first_term4 *= first_term4;
non_linear_term4 += (first_term4 - second_term4);
}
float non_linear_term = non_linear_term4.sum_float4();
return 0.5 * (linear_term + non_linear_term);
}
}</code></pre></figure>
Moving forward, we continue to search for modeling and model training improvements. If Factorization Machines or machine learning
are the types of things that interest you, please consider applying to <a href="https://www.adroll.com/about/careers">work at AdRoll</a>!
<h4 id="-references"><a name="references"></a> References</h4>
<ol>
<li>Rendle, Steffen. Factorization Machines.</li>
<li>Recht, Benjamin and Re, Christopher and Wright, Stephen and Niu, Feng. Hogwild: A Lock-Free Approach to Parallelizing
Stochastic Gradient Descent.</li>
<li>Duchi, John and Hazan, Elad and Singer, Yoram. Adaptive Subgradient Methods for Online Learning and Stochastic
Optimization.</li>
</ol>
<h4 id="-footnotes"><a name="footnotes"></a> Footnotes</h4>
<ol>
<li>For details, please see <a href="#references">[1]</a>.</li>
<li>The main downside to using FM is the resulting optimization problem is no longer convex. As a result, many effective
optimization techniques are no longer at our disposal when learning the parameters $\vec{w}$ and $\mathbf{V}$ of the FM model.
Fortunately, SGD still works quite well in this non-convex setting.</li>
</ol>
</description>
</item>
<item>
<title>Easy (and slightly crazy) way of writing bash scripts</title>
<link>https://tech.nextroll.com/blog/terminal/2015/08/24/bash-command-runner.html</link>
<pubDate>Mon, 24 Aug 2015 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/terminal/2015/08/24/bash-command-runner</guid>
<description>I’ve always been interested in creating and improving our developer tools.
Things like <a href="http://tech.adroll.com/blog/web/2014/03/05/adding-jscs-to-your-commit-hook.html">git
hooks</a>,
<a href="https://chrome.google.com/webstore/detail/commit-filter-for-github/lbapcjnnpkmjkfdenpigkpmaelpgldmk?hl=en">browser
extensions</a> or <a href="http://tech.adroll.com/blog/terminal/2014/09/26/introducing-aircontrol-control-airplay-through-terminal.html">command line scripts</a>.
When I work on such tools, one of my goals is to make them easy but also
pleasant to use. “Pleasantness” typically comes in the form of being the least
verbose necessary, displaying nice and meaningful colors or making informed
assumptions.
In this post, I talk about a <code class="language-plaintext highlighter-rouge">bash</code> “command runner”. It allows me to easily
write informative and complicated scripts in a way that satisfies my
“pleasantness” requirement.
To keep you reading, here it is in action:
<img src="/images/post_images/command-runner-demo.gif" alt="command_runner in action" />
<h1 id="the-problem">The problem</h1>
I recently worked on streamlining some steps of our internationalization
(a.k.a
“<a href="http://en.wikipedia.org/wiki/Internationalization_and_localization#Naming">i18n</a>”)
process.
For the most part, our process is as described by LingoHub in <a href="http://blog.lingohub.com/2013/03/integrating-internationalization-in-a-git-workflow/">this
post</a>.
I.e. we have a branch named <code class="language-plaintext highlighter-rouge">i18n</code> that starts as a copy of <code class="language-plaintext highlighter-rouge">master</code>. All
feature branches are merged into <code class="language-plaintext highlighter-rouge">i18n</code> when they are ready to be merged into
<code class="language-plaintext highlighter-rouge">master</code>. That lets us get translations started for all the feature branches
being reviewed.
This process has a few recurring tasks for developers, like merging our current
branch into the <code class="language-plaintext highlighter-rouge">i18n</code> branch or pushing translation files to an external API.
The two common themes with these tasks are that:
<ul>
<li>They have a significant number of steps.</li>
<li>They involve git operations: checking a branch out, pulling from the remote,
merging, etc.</li>
</ul>
I’ll use the <code class="language-plaintext highlighter-rouge">push</code> example: our new strings are ready to be sent to
translators, so we need to compile our messages into a POT file and push it to
<a href="http://www.smartling.com/">Smartling</a>, the online software we use to manage
work with our translators.
For that one task we need to run the following commands:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"># Starting from any branch
# In case something has been changed:
git stash
git checkout i18n
git pull origin i18n
# Extract messages from codebase using PyBabel
python setup.py extract_messages
# Pushes to our translation SaaS (using separate command line tool)
smartling push
# Commit the whole thing
git add adroll/dotcom/i18n/adroll.pot
git commit -m 'Update POT file'
git push origin i18n
# Get back to previous state
git checkout -
git stash pop # maybe?</code></pre></figure>
Obviously, this is too many steps to type by hand every time. We’re programmers,
let’s automate this!
<h2 id="the-naive-solution">The naive solution</h2>
The first option would be to write a naive script: take all the above commands
in a file, append <code class="language-plaintext highlighter-rouge">#!/usr/env bash -e</code>, <code class="language-plaintext highlighter-rouge">chmod +x</code> and you’re good to go!
Well… until you realize that you first need to check that you did stash
something, otherwise you’d be popping an unrelated stash. You also need to be
confident that all the <code class="language-plaintext highlighter-rouge">git</code> commands will succeed. In reality, they can each
fail in many different ways. For example, maybe your local <code class="language-plaintext highlighter-rouge">i18n</code> branch is
ahead of your <code class="language-plaintext highlighter-rouge">origin</code>.
With the <code class="language-plaintext highlighter-rouge">-e</code> flag, <code class="language-plaintext highlighter-rouge">bash</code> will stop at the first problem, but then you need
to pick up where you left off and that naive script won’t let you do that.
<h2 id="the-trying-to-be-smarter-solution">The (trying-to-be-)smarter solution</h2>
A second option then is to write a smarter script: any time there’s ambiguity,
you check the exit code and keep track of the state of the commands. Something
like this:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">#!/usr/env bash -e
git stash # this shoud be safe
git checkout i18n # this one too
git pull origin i18n # hmm, that could fail for a few reasons...
if [ $? = 0 ]; then
python setup.py extract_messages
if [ $? = 0 ]; then
smartling push # Pushes to our translation SaaS
# ...now what?
if [ $? = 0 ]; then
git add adroll/dotcom/i18n/adroll.pot
git commit -m 'Update POT file'
git push origin i18n
git checkout -
git stash pop # maybe?</code></pre></figure>
This gets very hairy very fast. It’s not even clear how that script would be
smarter, and what to do with the error codes and how to handle errors
gracefully. And it still won’t let you go pick up where you left off. Plus, that
would be that one task, and I would need to do the same work for all of them.
Another thing I dislike with the solutions above is that they’re very verbose.
These tasks should be the least disruptive possible for the developers’
workflow. They’re tasks that need to be done, yes. But they’re secondary and if
all goes well, I’d rather not see a screenful of output of the various commands.
<h1 id="enter-command_runner">Enter command_runner</h1>
Instead, I wrote <a href="https://gist.github.com/Timothee/71e653a6a8e18ae17fb7">a
script</a> that gives me the
following:
<ul>
<li>an easy way to create new tasks, without having to manage the various exit
codes of each step</li>
<li>if everything goes well, as little output as necessary</li>
<li>if anything goes wrong, a way to fix the issue and pick up where I left off</li>
</ul>
It defines a <code class="language-plaintext highlighter-rouge">run_commands</code> function which takes a function name as an argument.
The function name is the name of a function that prints out all the commands,
one per line. E.g.
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">foo () {
cat &lt;&lt;-EOL
ls
ls foo
cat foo
EOL
}</code></pre></figure>
Then you’d call that as <code class="language-plaintext highlighter-rouge">run_commands foo</code>.
You can also pass it a step number as a second argument to start at that step
instead, allowing you to pick up where you left off after an error.
E.g. in the above example, the commands would be numbered this way:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">0. ls
1. ls foo
2. cat foo</code></pre></figure>
So, calling <code class="language-plaintext highlighter-rouge">run_commands foo 2</code> would only call <code class="language-plaintext highlighter-rouge">cat foo</code>.
<hr />
The meat of <code class="language-plaintext highlighter-rouge">run_commands</code> can be summed up in two parts:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"> while read -r line; do
# do something with $line
done &lt; &lt;(echo "$($1)")</code></pre></figure>
<code class="language-plaintext highlighter-rouge">&lt;(echo "$($1)")</code> makes a call to the function name given to <code class="language-plaintext highlighter-rouge">run_commands</code>,
captures its output and feeds it into the <code class="language-plaintext highlighter-rouge">while</code> loop. In that loop, each line
(i.e. command to run) can be handled.
The second part is the handling of <code class="language-plaintext highlighter-rouge">$line</code> in the loop. The barebone version is:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"> echo -e "$line"
OUT=$($line 2&gt;&amp;1)</code></pre></figure>
In other words, it’s printing the command as a string first, then using that
string in a command substitution to run it.
This is where readers become split between the ones that think it’s pretty cool
and the ones that think I’m a <code class="language-plaintext highlighter-rouge">bash</code> heretic for using a variable as both a
string and a command.
It can work but it’s limited and a bit unpredictable because of the way <a href="http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html">shell
expansion</a> works.
Take this example:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">&gt; echo "foo"
foo
&gt; a="echo \"foo\""
&gt; echo $a
echo "foo"
&gt; $a
"foo"</code></pre></figure>
<code class="language-plaintext highlighter-rouge">echo $a</code> outputs exactly the first command, i.e. <code class="language-plaintext highlighter-rouge">echo "foo"</code>. But running <code class="language-plaintext highlighter-rouge">$a</code>
and <code class="language-plaintext highlighter-rouge">echo "foo"</code> doesn’t end with the same result. This is due to how the
command is parsed and expanded. In this case, it’s very benign but it
illustrates the difficulties you can run into.
For the same reason, some commands won’t work or will need some tweaking or
further wrapping-in-a-function. E.g. <code class="language-plaintext highlighter-rouge">git commit -m "This will fail"</code> should be
wrapped into something like that:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">commit () {
git commit -m 'This will work'
}</code></pre></figure>
Then you’d use <code class="language-plaintext highlighter-rouge">commit</code> in your list of commands rather than <code class="language-plaintext highlighter-rouge">git commit -m
"foo"</code>.
<hr />
Beyond that, there’s more code to handle the exit code, stop if necessary, start
at a specific step, add nice colors, etc. You can check the full script out in
<a href="https://gist.github.com/Timothee/71e653a6a8e18ae17fb7">this gist</a>.
<h1 id="here-is-how-to-use-command_runner">Here is how to use <code class="language-plaintext highlighter-rouge">command_runner</code>:</h1>
Let’s work through it with a simple example. I don’t use <code class="language-plaintext highlighter-rouge">git rebase</code> much and
instead I “merge forward”:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">git checkout master
git pull
git checkout -
git merge -</code></pre></figure>
(of course, you can also just do <code class="language-plaintext highlighter-rouge">git merge origin/master</code> from your feature
branch but let’s use the long version for this example)
1. Take the list of commands and wrap it in <code class="language-plaintext highlighter-rouge">cat</code>, inside a function:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">mergeforward () {
cat &lt;&lt;-EOL
git checkout master
git pull
git checkout -
git merge -
EOL
}</code></pre></figure>
2. You can then wrap the call to <code class="language-plaintext highlighter-rouge">run_commands</code> into your own script (let’s
call it <code class="language-plaintext highlighter-rouge">git-mfw</code>) like so:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">#!/usr/bin/env bash
source command_runner
mergeforward () {
cat &lt;&lt;-EOL
git checkout master
git pull
git checkout -
git merge -
EOL
}
run_commands mergeforward $@</code></pre></figure>
3. Call <code class="language-plaintext highlighter-rouge">git-mfw list</code> to list the steps that are going to happen:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">$ ./git-mfw list
0.git checkout master
1.git pull
2.git checkout -
3.git merge -</code></pre></figure>
4. Imagine you have a local change to some file and run the script:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">$ ./git-mfw
✓ 0.git checkout master
✗ 1.git pull
2.git checkout -
3.git merge -
Command git pull failed with exit code 1
Output:
From github.com:adroll/test-branch
error: Your local changes to the following files would be overwritten by merge:
foo/bar/modified.py
Please, commit your changes or stash them before you can merge.
Aborting
Updating 1df3f18..efa27b</code></pre></figure>
5. You would then correct the issue and start where you left off by giving the
step number to start at:
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">$ git reset --hard
$ ./git-mfw 1
~ 0.git checkout master
✓ 1.git pull
✓ 2.git checkout -
✓ 3.git merge -</code></pre></figure>
6. It’s trivial to extend your script by adding one function per task you need
to accomplish. In the <code class="language-plaintext highlighter-rouge">i18n</code> example, we ended up with <code class="language-plaintext highlighter-rouge">push</code>, <code class="language-plaintext highlighter-rouge">pull</code> and
<code class="language-plaintext highlighter-rouge">merge</code>.
<h1 id="run-commands-run">Run, commands, run!</h1>
While working on this, I came across a lot of webpages that warned against
wanting to both print a command and execute it. And for good reasons! It’s
obscure, prone to errors and can potentially harm your system if you’re not
careful. (to be fair, a lot of things happening in your terminal can)
Like anything, it has limitations but overall it’s been incredibly useful! You
can write a nice-looking script that benefits your whole team very very fast
and you’ll be a hero.
</description>
</item>
<item>
<title>How We Transparently Track NPS</title>
<link>https://tech.nextroll.com/blog/adroll/2015/08/05/how-we-nps.html</link>
<pubDate>Wed, 05 Aug 2015 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/adroll/2015/08/05/how-we-nps</guid>
<description>It has been a long time outstanding, “to do item”, to start collecting feedback and net promoter scores from the tens of thousands of customers that AdRoll has. Once at scale, it becomes surprisingly challenging to ensure that feedback from users is sent to Product, Engineering and SMB teams.
Last month we turned <a href="https://satismeter.com/">Satismeter</a> on via our <a href="http://segment.com">Segment</a> dashboard. Satismeter injects a prompt at the bottom of our Dashboard asking how likely you are to refer AdRoll to a friend - in order to gather what’s more formally known as a Net Promoter Score or NPS. It looks like this:
<img src="/images/post_images/nps-prmpt.png" alt="Image" />
Instantly we were deluged with feedback from our customers. The issue was that this feedback was now contained within Satismeter, but I wanted it in front of everyones faces. We did two cool things:
<ul>
<li>
Firstly, we sent the NPS comments back to Segment. This ensured that the data was then sent to every product we were using via Segment. For example, Customer Success and SMB teams can now see the comments and scores in <a href="http://intercom.io">Intercom</a>, Zendesk. We can even now ensure that those users receive different emails via <a href="http://customer.io">customer.io</a> and different modals via <a href="http://appcues.com">Appcues</a>. Maybe very satisfied customers are the perfect testing cohort for some of our Growth experiments.
</li>
<li>
We also sent this data into our #product <a href="http://slack.com">Slack</a> channel. This channel is one of our highest population channels, with approx. 120 people in it. The virtues and extraordinary execution of Slack is a post for another day, but it is worth highlighting that this channel has everyone in product in it, it has all of the executive team, it has our VP of SMB, it has our entire customer support team in it and dozens of other highly relevant teams.
</li>
</ul>
<img src="/images/post_images/nps-comment.png" alt="Image" />
Short and sweet.
It was trivial to set up. We created a Zapier account, configured a web-hook zap, and placed that URL in the Satismeter Web-hooks panel:
<img src="/images/post_images/nps-webhook.png" alt="Image" />
We only wanted scores with comments to be sent to Slack, a trivial custom filter:
<img src="/images/post_images/nps-filter.png" alt="Image" />
And this was the output for Zapier:
<img src="/images/post_images/nps-output.png" alt="Image" />
A few things to note that I think are cool. If you click the email address, it loads that user in Intercom. This allows you to always see if someone else has reached out to the user in a central place - much much better than just opening in your email client. We also made the avatar an image of Grandpa Simpson, because… The Simpsons.
To recap:
<ul>
<li>We collect NPS and feedback</li>
<li>Most importantly, that feedback is outputted in a highly visible Slack channel</li>
<li>We’ve empowered users to reach out and fix things via Intercom</li>
<li>Segment has allowed us to propagate this data everywhere relevant</li>
<li>The Simpsons rules.</li>
</ul>
</description>
</item>
<item>
<title>How We Built the AdRoll Growth Team</title>
<link>https://tech.nextroll.com/blog/adroll/2015/07/27/how-we-built-the-adroll-growth-team.html</link>
<pubDate>Mon, 27 Jul 2015 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/adroll/2015/07/27/how-we-built-the-adroll-growth-team</guid>
<description>Eight years ago, AdRoll was founded on this principle: to make display advertising work for everyone. As we’ve grown to six offices and over 25,000 advertisers worldwide, our challenge has been to make sure that we continue to identify and respond to the needs of businesses large and small.
The Growth team was created to ensure that as AdRoll continues to build out more great products for marketers in more regions, we stay true to our founding philosophies. As we ourselves grow, we want the SMB advertisers that make up the core of our business to grow along with us.
Peter - I grew up in England, studied Computer Science, and then went through Y Combinator. I’m the product manager. 
Etan - Formerly a Data Scientist at Boost Media, have a love for both theoretical and applied mathematics. Business Analyst Support for Growth 
Anil - I’m a software engineer that was one of the founding employees at Udemy. I’m Turkish. 
Matt - AdRoll acquired my mobile analytics company, before that I worked as a software engineer in the finance industry. I’m Australian. 
Patrick - VP of Engineering that has scaled a number of fast growing companies since I moved to the US from Ireland. 
Here’s a recap of our first 30 days:
Q: How did you build the team? 
Patrick: Our first task was to identify the right people. How do you pull together a team that will be effective, efficient, and enthusiastic in meeting this challenge? One that both cares about this problem, and is empowered to solve it?
Everyone in our org cares about our customers but we needed to find individuals that also had a desire and drive to produce new solutions for persistent problems, and who get things done quickly. We needed a team that could experiment, test, implement, reflect, and improve at extreme velocity. Because we wanted this team to move fast while working with many organizations in the company, we wanted individuals who could operate under the philosophy: “It’s easier to ask forgiveness, than it is to get permission”.
Q: How would you define the problem this team is solving for? 
Peter: Our overarching themes are to reduce friction during initial campaign setup, provide ongoing support, and increase lifetime value. For small to medium size businesses (SMB), you don’t have a lot of leeway to allow for an account not being set up correctly. In other words, the negative impact of having a campaign that is setup sub-optimally can be very significant for an SMB advertiser. We want to make sure that small businesses get the most out of the platform by utilizing the best of what AdRoll has to offer, in the best ways it can offer.
Q: What are the KPIs, and what are the things we want to focus on? What does a successful result look like? 
Etan: Our first step to start answering these questions was to go through the exercise of identifying the signs and signals that make a successful customer. We sat down to identify the customers within the SMB segment that were ‘successful’ vs. ones that were not, then had to describe what exactly that success looks like.
For example, we found that customers that were using more than one product, had multiple ads uploaded, and had mobile ad sizes saw stronger campaign performance and lower churn. Having asked the question, “What does a successful customer look like?”, we could better measure whether our efforts were being effective down the line, according to these standards.
Q: How did you prioritize the different needs to address? How did you decide where to start? 
Etan: Once we had characterized our picture of the successful SMB customer, our next step was to identify the pain points within the product offering that could be impeding advertisers from getting started or achieving outstanding results. These were anything that might be causing friction or problems for clients. For example, the early conversation of measuring return on investment for the user is dependent on specific tracking having been attributed by the customer. Without the ability to understand the value of the AdRoll service, this naturally led to churn and unhappiness.
Having identified these pain points, we then tried to map them against retention. Essentially, we plotted a 1:1 relationship between negative events and churn, and used this to direct our product roadmap as an immediate way to prioritize different pain points.
Q: How did you go about this? 
Anil: We started by imagining what it would be like to land on the AdRoll dashboard as a first-time advertiser. In doing this, there were a number of things that came to our attention that could be potentially significant challenges for someone who has never set up an advertising campaign. There were certain things that could be made more intuitive, and a lot of opportunity to offer additional guidance and help small business owners set up for success.
We worked closely with our SMB team to learn about the most common problems they encounter. Our strategy was data plus conversation - collecting information both quantitatively, and through our own experience in trying to navigate the platform.
Q: In your opinion, what is growth trying to do? 
Peter: We want to increase the threshold for which we give dedicated service. Our hope is to reach an ideal state of frictionless campaign setup flow - which is to allow people to utilize and setup the product themselves, to full potential.
Matt: People equate early-stage startups with their founding philosophies, and what the company is trying to build. Generally, as a company grows in size, it becomes harder to ingrain these philosophies into every new employee, and there’s a lot of day-to-day work that is done without these founding principles in mind.
For us, we’re trying to ensure that everything we do is, in fact, aligned with this goal of what we want to build. Velocity is key - we want to address the challenges that advertisers are coming up against, now. The customer success signals we identified help keep us on track and recognize when we’ve gone awry; we use them to adjust and react.
Q: How does this fit back into meeting the needs of small businesses? 
Peter: There are a lot of small improvements we can make that can drive significant results for SMB advertisers. For these businesses, a small improvement in campaign setup can mean huge business impact as far as % gains. For small businesses that are just getting started and trying to reach and retain new customers, companies like AdRoll can actually have a significant impact on that business’s success, if utilized correctly.
Q: What makes this team unique? What makes it work? 
Peter: We’re operating from a different perspective than other teams in the organization. Rather than focusing on a single product, we’re focused on a single problem. So, we’re able to see the larger picture and push out more updates, more quickly, to solve for this problem. There’s also more latitude to adjust our strategy and what we’re working on, to make sure we’re in line with our vision.
What does success for this team look like 1 year from now? 
One of the nice things about Growth is that it is so data driven. In fact, if you can’t track something then a Growth team should not really do it in the first place. So when we look at Growth 1 year from now, we’ll be comparing how users are signing up and activating and sticking around compared with today. One of the big metrics that we track each month is a simple one: how many people that signup go on to spend? In my opinion this metric alone speaks for the health of a Growth team. It conveniently thinks about all touch points of onboarding, which is really a key part of Growth.
Given how international the team is, I think 3 green cards would be appreciated too. Another easy metric to track.
</description>
</item>
<item>
<title>Streaming Petabytes of Data in Realtime with Kinesis</title>
<link>https://tech.nextroll.com/blog/data/2015/06/26/kinesis.html</link>
<pubDate>Fri, 26 Jun 2015 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data/2015/06/26/kinesis</guid>
<description>AdRoll’s real-time data pipeline drives many systems crucial to our business. These systems inform the decisions made by our real-time bidding infrastructure in response to the tens of billions of daily requests we receive from ad exchanges. They also refresh our predictive models, guard against overspend, and deliver up to the minute campaign metrics through our dashboard. To function properly these systems need to maintain 100% uptime while inhaling about 1.3 petabytes of data per month across 5 regions worldwide.
We’ve found that improving real-time data recency directly correlates with improved performance and is an enabler for future features and products. As such, we recently found ourselves motivated to overhaul our real-time pipeline in an effort to reduce end to end latency as much as possible. By leveraging Amazon’s scalable data streaming service <a href="http://aws.amazon.com/kinesis/">Kinesis</a> we were able to drop our real-time pipeline’s processing latency from 10 minutes to just under 3 seconds (while simultaneously cutting costs and improving system stability!).
Our pipeline involves feeding event logs generated by our real-time bidders and ad-servers into a set of <a href="https://storm.apache.org/">Storm</a> topologies and a few other proprietary applications which do our real-time processing. Originally, we ingested data by reading log files rotated to S3 every few minutes from each bidder and ad-server. But this mode of ingestion caused a tension between S3 performance and data processing latency - flushing logs more often improves recency but yields a larger number of smaller log files, which degrades list and read performance. We found that in practice we were bounded by a 10 minute average processing delay with this setup.
<img src="/images/post_images/kinesis_architecture_1.png" alt="Old Pipeline" />
To break the 10 minute recency barrier we needed a continuous data streaming service. But not just any off the shelf solution would suffice. Whatever we chose would have to be able to reliably handle our <a href="http://tech.adroll.com/blog/adtech/2014/11/24/adroll-puts-the-p-in-big-data.html">volume</a> while providing redundancy/fault tolerance. Additionally, we would need this service deployed near each of our geographically distributed data centers. The motivation here is to minimize data loss in the event of a machine failure – we wouldn’t be able to offload logs as frequently or as reliably with transcontinental network calls involved. Presumably, once in our streaming service, data will be more durable, so our real-time processing applications (which live in a single region) would be better suited to stream logs across oceans.
<img src="/images/post_images/kinesis_architecture_2.png" alt="New Kinesis" />
Initially we explored Kafka but we found it difficult to configure properly and it fell over every time we tried to scale. We kept running into issues with tasks like setting up cross datacenter mirroring and allocating the appropriate amount of disk space per topic. Eventually we started looking for an easier to manage solution.
Enter Kinesis. Very similar to Kafka, Kinesis is a partitioned streaming service. In Kinesis data producers push small data blobs called records into streams which are broken into partitions called shards. Kinesis has built in redundancy and is arbitrarily scalable by design so it can handle our volume. And it’s an AWS service so we get streaming locally to the regions where our log producers reside for free because all of our infrastructure already lives on AWS. Also, it’s a managed service so don’t have to worry about configuring and tuning and we won’t be losing any sleep from ops work!
We were able to get up and running on Kinesis fairly quickly with some plug and play libraries. On our log producers, which are written in Erlang, we used <a href="https://github.com/AdRoll/kinetic">Kinetic</a>, an open source Erlang Kinesis client developed here at AdRoll. Kinetic runs on a logging process which lives in each of our real-time bidders and ad-servers and handles about 5,000 logs per second (per machine). We designed this process to flush data in small batches which are formed by concatenating logs until either Kinesis’s 1MB record size limit is reached or 1 second has passed since the last flush. This allows us to minimize put record requests which are expensive due to network calls …and the fact that we’re literally <a href="http://aws.amazon.com/kinesis/pricing/">billed by Amazon for them</a>.
On the Kinesis consumer side, we connected our Storm topologies to Kinesis by forking Amazon’s <a href="https://github.com/awslabs/kinesis-storm-spout">Kinesis storm spout</a>. For those unfamiliar with Storm, spouts are components which ingest data from outside sources and send it to other Storm components to be processed. Out of the box we found the spout streamed data too slowly because polling from Kinesis was tightly coupled with Storm’s requests for the spout to emit data. This is shown in the (simplified) snippet of code from Amazon’s library below:
<figure class="highlight"><pre><code class="language-java" data-lang="java">// poll records for a Kinesis stream shard
public class KinesisShardGetter {
...
AmazonKinesisClient kinesisClient;
...
// read and emit Kinesis records
public List&lt;Record&gt; getNext(int numRecords) {
GetRecordsRequest request = new GetRecordsRequest();
request.setLimit(numRecords);
GetRecordsResult result = kinesisClient.getRecords(request);
return result.getRecords();
}
...
}</code></pre></figure>
In the vanilla library, Storm requests data from <code class="language-plaintext highlighter-rouge">KinesisShardGetter</code> objects via <code class="language-plaintext highlighter-rouge">getNext(...)</code> and each of these requests kicks off a slow, blocking network call to Kinesis made by the <code class="language-plaintext highlighter-rouge">kinesisClient</code>. This doesn’t scale well and a single spout will quickly reach its network bounded throughput cap even when consuming from a small number of shards. To make the spout more performant we wrote our own <code class="language-plaintext highlighter-rouge">KinesisShardGetter</code> shown (simplified) below.
<figure class="highlight"><pre><code class="language-java" data-lang="java">// poll and buffer records for a Kinesis stream shard
public class KinesisAsyncShardGetter extends KinesisShardGetter {
...
AmazonKinesisAsyncClient kinesisClient;
final ConcurrentLinkedQueue&lt;Record&gt; recordsQueue;
...
// read Kinesis records
public void readRecords() {
final GetRecordsRequest request = new GetRecordsRequest();
kinesisClient.getRecordsAsync(request, new AsyncHandler&lt;GetRecordsRequest, GetRecordsResult&gt;() {
@Override
public void onSuccess(GetRecordsRequest request, GetRecordsResult result) {
recordsQueue.addAll(result.getRecords());
readRecords();
}
...
}
}
...
// emit Kinesis records
@Override
public List&lt;Record&gt; getNext(int numRecords) {
List&lt;Record&gt; records = new ArrayList&lt;&gt;();
for (int i = 0; i &lt; numRecords; i++) {
if (!recordsQueue.isEmpty()) {
records.add(recordsQueue.pop());
} else { break; }
}
return records;
}
...
}</code></pre></figure>
To avoid a blocking Kinesis network call with each <code class="language-plaintext highlighter-rouge">getNext(...)</code> we asynchronously poll and buffer data from Kinesis in a background thread via <code class="language-plaintext highlighter-rouge">readRecords()</code>. This way we need only <code class="language-plaintext highlighter-rouge">recordsQueue.pop()</code> to emit data into Storm from memory. This allowed us to increase the spout’s throughput to the point where we were bounded by data preprocessing instead of network latency (which was around a 10x speed boost for us).
The final deployment hurdle we had to clear was setting up some Kinesis stream management tooling/automation – namely we wanted to be able to quickly and easily create or scale all of our streams across all our AWS regions. We tried using an existing <a href="https://github.com/awslabs/amazon-kinesis-scaling-utils">AWS labs tool</a> but it seemed that it’s requests would rarely succeed due to poor error handling. We ended up just writing our own.
Programatically creating streams was a fairly straight forward process but very slow (it takes around 30 minutes to set up all our streams in serial). We made sure to thread create requests such that we were hitting the 5 simultaneous Kinesis stream operations per account limit.
Scaling streams was a bit of a headache because the current API makes it a very manual process. For some context, each Kinesis stream has a hash key space which is broken into sections, one for each shard. Each put request is given a random value in the stream’s hash key space to determine which shard the request will be sent to. Typically all shards own an equal slice of the hash key space so requests will be distributed evenly across all shards.
Ideally streams would be scaled by a single API call specifying a stream and a target number of shards and Amazon would handle splitting the hash key space into new, evenly sized sections. However, in the current API you are required to split and merge sections of the hash key space manually, each split and merge operation being a separate request. Moreover you need to specify the actual point in the hash key space where you want to draw new split boundaries. Properly adding a single shard to a stream with two shards takes three distinct requests! For example, given a stream with two shards owning sections <code class="language-plaintext highlighter-rouge">[1-15], [16-30]</code> of hash key space <code class="language-plaintext highlighter-rouge">[1-30]</code>, to add a new shard – yielding sections <code class="language-plaintext highlighter-rouge">[1,10], [11, 20], [21,30]</code> – you would need to:
<blockquote>
<ol>
<li>Split shard <code class="language-plaintext highlighter-rouge">[1,15]</code> at 10, into shards <code class="language-plaintext highlighter-rouge">[1,10]</code> and <code class="language-plaintext highlighter-rouge">[11, 15]</code></li>
<li>Split shard <code class="language-plaintext highlighter-rouge">[16,30]</code> at 20, into shards <code class="language-plaintext highlighter-rouge">[16,20]</code> and <code class="language-plaintext highlighter-rouge">[21, 30]</code></li>
<li>Merge shards <code class="language-plaintext highlighter-rouge">[11,15]</code> and <code class="language-plaintext highlighter-rouge">[16,20]</code> into new shard <code class="language-plaintext highlighter-rouge">[11,20]</code></li>
</ol>
</blockquote>
…I suggest avoiding scaling between relatively prime numbers of shards (stick to powers of two).
Annoyingly, split or merge requests involving unrelated shards can’t be run at the same time on the same stream. So each operation must be performed in serial and scaling a stream is very slow. Again, we made sure to thread our scaling logic to operate on multiple streams simultaneously in the event that we needed to quickly respond to a global traffic increase.
Eventually we got all of the migration details sorted out, and we hit the switch to turn on Kinesis. Immediately saw our event processing latency drop from 10 minutes to just under 3 seconds. Also, continuously streaming data smoothed out a few previously bursty workloads (can you spot the Kinesis pipeline release from the graph below of our DynamoDB writes?). This improved system stability and allowed us to cut our costs nearly in half for managed services that bill based on provisioned throughput such as DynamoDB. These savings actually more than paid for the cost of using Kinesis!
<img src="/images/post_images/kinesis_dynamo.png" alt="New Kinesis" />
We ran into one fairly painful Kinesis service constraint though. The scale of each shard in Kinesis is based on throughput. Each shard provides a maximum write capacity of 1MB/s and a maximum read capacity of 2MB/s. This means that unless you overprovision or use multiple layers of streams you are limited to having two distinct real-time applications consuming from a Kinesis stream at any given time. Because we could potentially have around 10 real-time applications trying to consume from the same Kinesis stream we decided not to try to expose our real-time data directly from Kinesis.
In the past S3 had worked well for serving data to many concurrent consumers but it had become a performance bottleneck because we were writing out a separate log file for each of our 500+ log producers. Once we had Kinesis, however, we were able to write an application which aggregates logs from all of our data producers in a single location – allowing us to flush all logs to S3 in a single file. By doing this we can flush much more frequently and therefore expose much fresher data to all of our real-time applications without sacrificing S3 performance. By using this Kinesis to S3 architecture we were able to achieve an end to end latency of about 30 seconds for all real-time applications. Also the aggregator application only takes up one Kinesis consumer application slot so we were able to have one other application which was particularly sensitive to recency connect directly to Kinesis and enjoy the aforementioned 3 second latency.
<img src="/images/post_images/kinesis_architecture_3.png" alt="Old Pipeline" />
Overhauling our data pipeline with Kinesis ended up being very fruitful and relatively painless. We were able to expose massive recency gains to our entire real-time data pipeline while at the same time improving system stability and cutting costs.
At AdRoll we’re looking for ways to improve our backend systems. If you have some ideas or have been itching to work with huge amounts of data, <a href="https://www.adroll.com/about/careers/open-positions">let us know</a>!
</description>
</item>
<item>
<title>Tech Talks: Lee Byron on Immutable.js</title>
<link>https://tech.nextroll.com/blog/adroll/2015/05/13/tech-talk-immutable-lee-byron.html</link>
<pubDate>Wed, 13 May 2015 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/adroll/2015/05/13/tech-talk-immutable-lee-byron</guid>
<description>Since June 2014, the AdRoll Engineering team has been hosting Tech Talks every
other Tuesday – give or take – where members of our team share about a
technology, project, or tool they’re excited about. We’ve been lucky enough to
have a number of developers from within the larger SF / Bay Area community stop
by our offices for these Tuesday “Lunch and Learn” hours to share new and
interesting projects they’re pursuing, and now we’d like to share them with you!
Most recently, Lee Byron, author of Immutable.js and other great things at
Facebook, joined us for a talk on Immutable.js, a library of persistent
immutable data structures that make Flux and React.js development even more
awesome. Lee Byron has been making things at Facebook since 2008 including
React, Immutable.js, and GraphQL.
Check out the full video below:
<iframe width="500" height="281" src="https://www.youtube.com/embed/kbnUIhsX2ds?feature=oembed" frameborder="0" allowfullscreen=""></iframe>
Interested in learning more about life in AdRoll Engineering? Check out our <a href="https://www.adroll.com/about/careers">Careers page</a>!
</description>
</item>
<item>
<title>AWS Summit Keynote in San Francisco</title>
<link>https://tech.nextroll.com/blog/adroll/2015/05/04/aws-summit-keynote.html</link>
<pubDate>Mon, 04 May 2015 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/adroll/2015/05/04/aws-summit-keynote</guid>
<description><div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/JRODD1_jBww?start=2894&amp;end=3386" frameborder="0" allowfullscreen=""></iframe></div>
AdRoll is building products that allow customers of any size, big or
small, with a lot of marketing experience or none, to run high
performing marketing campaigns. And with over 20,000 customers in over
150 countries, we’ve been fairly successful in this.
Our main product has been Retargeting, the basic concept is very simple:
when one of your customers leaves your site without buying, we enable
you to reach them with the most appropriate messaging while they browse
other sites or use their phone. This is great because it’s easily
measurable and it performs as well as search.
The part that is not easy in Retargeting is the complexity of the
infrastructure that lies behind it, which is called Real Time Bidding or
RTB. The basic requirement in RTB, is that we are latency bound. When we
talk about real time we mean it in the literal sense of a classic firm
real time system. Our machines receive about 60B requests each day and
they need to respond to each one of them in less than 100ms. 1% of
errors with 60B requests means 600M errors and that’s too much
opportunity cost.
100ms max latency is a very challenging problem globally though. We have
customers from over 150 countries and light in fiber goes at 120k miles
per second. That’s a real problem because this ray of light actually
takes about 60ms to roundtrip between New York and Paris. That’s the
majority of the time we have available to bid on a single impression.
If you want to operate in RTB, you need global presence from the
beginning. It would have been extremely capital intensive for us to go
across the many regions of the world that we needed to start data center
or bandwidth contracts, hire engineers and create entities in all of
those different nations. With our infrastructure in Amazon AWS we never
have to worry about that. We just opened our business in Japan and it
only takes us a few minutes to bootstrap our infrastructure and start
serving that region of the world with the required low latency.
But we can’t just have our machines across the world, our data needs to
be there too and it needs to be able to be queried with the same low
latency. Operating a massively scalable data storage to handle over 1M
requests per second is certainly an interesting task but not that
differentiating for our customers. We’d rather focus on product and our
algorithms. So we built the parts of the infrastructure that we needed
and started using <a href="http://aws.amazon.com/dynamodb/">DynamoDB</a> to store information. Today we have almost
500B items stored in DynamoDB overall. No engineers working on its
operation, except all of those inside Amazon, while at the same time we
see consistently low latencies of less than 5ms, throughout this growth
period.
Just having data sitting statically however is not enough, it needs to
be analyzed and you need to be able to store a lot of it. Moreover we
can only achieve our full potential if all of AdRoll teams are able to
access the data when they need it and quickly. All of our teams
coordinate using Amazon S3. We’re now storing over 300TB of new
compressed data in S3 every month, over 17M new files, and in March 2015
we stored as much data as we did the first 4 months of 2014 combined.
This is our core asset and the more accessible it is the more value we
can extract from it, this is why we generate over 10B requests per month
on our buckets across all of our teams.
We also like to play around with new services when they come out and
<a href="http://aws.amazon.com/lambda/">Lambda</a> was no exception. Polling is not a scalable strategy to figure
out when new files are added to S3, especially when you add 17M of them
per month. When Lambda was released, a couple engineers quickly
prototyped a change in flow and moved Lambda in front of S3. Very
quickly we were able to go in production with this change and now every
team receives notifications for the files they are interested in, as
soon as they are added, solving a major bottleneck.
<hr />
</description>
</item>
<item>
<title>Announcing Hologram, taking EC2 Instance Roles everywhere</title>
<link>https://tech.nextroll.com/blog/ops/2014/12/22/announcing-hologram.html</link>
<pubDate>Mon, 22 Dec 2014 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/ops/2014/12/22/announcing-hologram</guid>
<description>We love Amazon Web Services here at AdRoll. All of our production, test, and staging infrastructure runs on AWS. Really, the only systems we operate outside of EC2 are developers’ laptops. Most of the time this isn’t an issue, as we use platform-agnostic tools like Python, Java and Erlang. When it came to interacting with AWS APIs, though, we had a challenge: how do we manage sensitive AWS Keys across dozens of developer’s laptops and many teams?
Key management in EC2 is best handled using “IAM Roles” where a special endpoint in the Instance Metadata service (http://169.254.169.254/…) exposes temporary AWS API access credentials that have permissions defined by the instance’s Role, configured at launch time. This way applications can be designed that do not require checking secret keys into their repositories, reducing the chance of malicious key usage. This service only exists in EC2, though, so many developers who use AWS keep long-lived, highly privileged credentials in a file on their local development machine. This isn’t particularly secure, and it creates a difference between development and production that can lead to broken deployments.
Several months ago, a couple of us embarked on a project to bridge this gap. Today, we’re proud to announce the release of Hologram, a system for bringing EC2 Role-like key management to non-EC2 hosts. Hologram exposes an imitation of the EC2 Instance Metadata service on developer workstations that exposes temporary credentials to your software the same way that EC2 does. It behaves just like EC2, so your code can use the same process in both development and production. The keys that Hologram provisions are temporary. EC2 access is then centrally controlled without direct administrative access to developer workstations.
In the past, I’ve worked at companies that give up on local development and require developers to run all of their code, even when developing, on an EC2 instance. We could have gone down that path here at AdRoll, but then we would have lost the richness of local development: powerful IDE tooling, instant access to files, interactive graphical debugging, and no waiting for instances to come up. Instead, we’ve decided to make local development the best experience we can, and Hologram enables us to do that.
Hologram is <a href="https://github.com/AdRoll/hologram">available on GitHub</a> under the Apache License, Version 2.0. The current implementation depends on having an LDAP server in your organization and isn’t super easy to set up, but we plan to add support for simpler authentication backends and streamline installation over the coming months. Pull Requests are, of course, welcome!
</description>
</item>
<item>
<title>AdRoll Puts the “P” in Big Data: Processing Petabytes</title>
<link>https://tech.nextroll.com/blog/adtech/2014/11/24/adroll-puts-the-p-in-big-data.html</link>
<pubDate>Mon, 24 Nov 2014 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/adtech/2014/11/24/adroll-puts-the-p-in-big-data</guid>
<description>Originally published on the AdRoll Blog on <a href="http://blog.adroll.com/adroll-puts-the-p-in-big-data-processing-petabytes">November 12, 2014</a>.
The advertising industry has undeniably become a data play, as consumers are generating valuable data with every digital interaction. We hear buzzwords like “big data,” “machine learning” and “real-time algorithms,” but little about how these puzzle pieces fit together to help marketers achieve their business objectives. Over the last few years, the ad tech industry has lead the way in turning big data concepts into solutions that solve real business challenges.
<img src="/images/post_images/big-data-feature-img.png" alt="Big Data Feature Image" />
In its most basic form, data science is the extraction of knowledge from data, and machine learning powers this process making programmatic buying possible. In performance advertising, the predictive power of your model grows as you increase the amount of data that is flowing through the system. And <a href="http://venturebeat.com/2014/11/12/adroll-hits-gigantic-130-terabytes-of-ad-data-processed-daily-says-size-matters/">AdRoll has a lot of data to work with</a>.
Customer intent data is the biggest competitive advantage at a company’s disposal if it can collect, analyze, and use that data in real time. A few months ago our systems generated and ingested 40-50 terabytes of data each day, but recently we’ve reached new heights in data volume at AdRoll, processing 130TB of data, about 30TB compressed, every single day. Basically, we operate at a data volume that is 150 times bigger than all of the US stock exchanges combined, by two orders of magnitude, and over 10 times the volume of events. In three days we generate as much data as the US stock exchanges generate in one year.
In fact, this year we hit the petabyte mark, processing over 10 petabytes (1 PB = 1000000000000000 bytes) = 1000 terabytes), a 1,200% increase year over year. To put that in perspective, in order to accommodate the storage and processing capacity of 10PB, we would need a space at least the size of AT&amp;T Park. Given San Francisco real estate prices, it’s a good thing we’ve been able to utilize globally distributed, cloud-based data warehouses thanks to our friends at <a href="http://aws.amazon.com/">Amazon AWS</a>.
<div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/FekGOiOGB8Y" frameborder="0" allowfullscreen=""></iframe></div>
Our partnership with Amazon allows us to grow our inventory sources and securely store, process, and scale our data volume. We can keep our data science and engineering talent focused on product innovation without the time and management resources required by legacy infrastructure. This has allowed us to serve our customers better while providing a steady revenue stream for publishers, which subsidizes the free web consumers have come to expect.
Consumers are spending more time online and on mobile devices, and we are incorporating more and more signals and digital interactions (time on site, quality of ad space, emails opened) into our RTB algorithm for better targeting and optimization. Our machines now process over 60 billion events on a daily basis.
So what does this massive increase in data volume indicate for the industry as a whole? It’s simple- predictive analytics are the future of RTB and data will drive the next generation of advertising.
Retargeting has become a powerful tool, providing the masses with real-time predictive capabilities that were once only reserved for the likes of Google. More inventory sources are popping up, more digital interactions are being captured, and more intelligent algorithms are being developed as a result.
Retargeting was one of the first ways marketers could leverage the customer data available for collection on their websites and it continues to outpace the industry in terms of innovation in RTB. Big data has moved from a buzzword to a staple in advertising, and it’s importance and profitability will only grow.
Who knows how many baseball fields of data we’ll be processing in 2015.
</description>
</item>
<item>
<title>D is for Data Science</title>
<link>https://tech.nextroll.com/blog/data/2014/11/17/d-is-for-data-science.html</link>
<pubDate>Mon, 17 Nov 2014 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data/2014/11/17/d-is-for-data-science</guid>
<description><a href="http://dlang.org">The D programming language</a> has quickly become our language of choice on the Data Science
team for any task that requires efficiency, and is now the keystone language for our critical infrastructure.
Why? Because D has a lot to offer.
<hr />
<h2 id="a-brief-introduction">A Brief Introduction</h2>
One of the clearest advantages of using D compared to other typical data science workflows is that it compiles down
into machine code. Without an interpreter or virtual machine layer, we can rip through data significantly
faster than other tools like a Java hadoop framework, R, or python would allow. But D’s compiler is fast
enough that in many cases it can be run as if it were a scripting language. Let’s try comparing to python by
generating a million uniform random variates, sorting, and finding the deciles:
<figure class="highlight"><pre><code class="language-python" data-lang="python">from random import uniform
variates = [uniform(0.0, 1.0) for i in range(1000000)]
variates.sort()
for i in range(0, 1000000, 100000):
print variates[i]</code></pre></figure>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; time python deciles.py &gt; some_deciles
real 0m0.857s
user 0m0.825s
sys 0m0.032s</code></pre></figure>
And similarly for D:
<figure class="highlight"><pre><code class="language-d" data-lang="d">import std.random;
import std.stdio;
void main(string[] args) {
double[] variates = new double[1_000_000];
foreach(ulong i, double _; variates) {
variates[i] = uniform(0.0, 1.0);
}
variates.sort;
foreach(int i; 0..10) {
writeln(variates[i * 100_000]);
}
}</code></pre></figure>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; time rdmd deciles.d &gt; some_deciles
real 0m0.929s
user 0m0.725s
sys 0m0.177s</code></pre></figure>
Wait… what? It took longer, but that’s because I just ran it the once and it includes
the compilation time. If I don’t change anything and run it again:
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; time rdmd deciles.d &gt; some_deciles
real 0m0.293s
user 0m0.291s
sys 0m0.004s</code></pre></figure>
That’s better. <code class="language-plaintext highlighter-rouge">rdmd</code> won’t bother to recompile if there’s no code change. These savings
can add up quite significantly once your code becomes more computationally complex and you
need to perform many computations over a substantial amount of data.
But of course there’s no real revelation here. If we want hyper-efficient code, we know
that it’s best to drop into a compiled language. The key thing here that separates D from
other efficient languages like the oft-suggested C or C++ is that D frees you to program
in the style you feel most comfortable with at the given time.
The code above shows how we can write a quick “script” in D without incurring any additional
headache over the simpler python. But D is also just as clean if you want to start writing
some object-oriented code:
<figure class="highlight"><pre><code class="language-d" data-lang="d">import std.random;
import std.range;
import std.stdio;
class Rectangle {
double width;
double height;
this(double width, double height) {
this.width = width;
this.height = height;
}
double area() {
return width * height;
}
}
void main(string[] args) {
Rectangle[] rs = new Rectangle[1_000_000];
rs = rs.map!(x =&gt; new Rectangle(uniform(0.0, 1.0),
uniform(0.0, 1.0))).array;
rs.sort!((x, y) =&gt; x.width &lt; y.width);
foreach(int i; 0..10) {
writefln("%0.4f\t%0.4f\t%0.4f",
rs[i * 100_000].width,
rs[i * 100_000].height,
rs[i * 100_000].area());
}
}</code></pre></figure>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; rdmd rectangle_deciles.d
0.0000 0.9382 0.0000
0.1000 0.2020 0.0202
0.1996 0.3612 0.0721
0.2995 0.3947 0.1182
0.3994 0.7440 0.2972
0.4993 0.8733 0.4360
0.5997 0.0221 0.0132
0.6997 0.6624 0.4634
0.7997 0.6204 0.4961
0.9003 0.4640 0.4177</code></pre></figure>
And D is as ready as you are for your high-performance needs. For example, if we want to calculate a fast
<a href="http://en.wikipedia.org/wiki/Fast_inverse_square_root">inverse square root</a>, we can use some pointer
voodoo (very lightly modified from the linked Wikipedia article):
<figure class="highlight"><pre><code class="language-d" data-lang="d">import std.conv;
import std.stdio;
void main(string[] args) {
//computes 1/sqrt(x)
auto x = to!float(args[1]);
int i;
float x2, y;
const float threehalves = 1.5f;
x2 = x * 0.5f;
y = x;
i = *cast(int*)&amp;y;
i = 0x5f3759df - ( i &gt;&gt; 1 );
y = *cast(float*)&amp;i;
y = y * (threehalves - (x2 * y * y));
writeln(y);
}</code></pre></figure>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; rdmd invsqrt.d 1
0.998307
%&gt; rdmd invsqrt.d 2
0.70693
%&gt; rdmd invsqrt.d 4
0.499154
%&gt; rdmd invsqrt.d 16
0.249577</code></pre></figure>
D will even let you write <a href="http://dlang.org/iasm.html">inline assembly</a> if you
really want to squeeze the most performance out of it. But this is all just fun
and games. How can D help in a real-world scenario?
<hr />
<h2 id="ripping-through-data">Ripping Through Data</h2>
During the course of our work at AdRoll, we had some infrastructure in D that was
running fine for a while, but at a certain point, when our data problems exceeded
the scope this code was designed for, we had to optimize. And, believe it or not,
the problem was as banal as pulling some fields out of delimited files. This is
the gist of what we did.
This particular log file contains some ad data delimited by the ASCII record
separator. Let’s say we want to pull out some timestamps and the country whence
these data come. As you can probably imagine by now, the naïve D solution is
quite readable, but is perhaps not as snappy as we’d like:
<figure class="highlight"><pre><code class="language-d" data-lang="d">import std.stdio;
import std.string;
static immutable uint TIMESTAMP_INDEX = 19;
static immutable uint COUNTRY_INDEX = 42;
static immutable char DELIMITER = cast(char)30;
void main(string[] args) {
auto file = File(args[1], "r");
foreach(char[] line; file.byLine()) {
char[][] fields = split(line, DELIMITER);
writefln("%s\t%s",
fields[TIMESTAMP_INDEX],
fields[COUNTRY_INDEX]);
}
file.close();
}</code></pre></figure>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; time rdmd parser.d log.txt &gt; country_info
real 0m13.421s
user 0m13.270s
sys 0m0.153s</code></pre></figure>
Thirteen seconds? That’s an eternity! This one log file is but a mere sample
of what we’re producing at AdRoll—uncompressed, we generate about <a href="http://venturebeat.com/2014/11/12/adroll-hits-gigantic-130-terabytes-of-ad-data-processed-daily-says-size-matters/">130TB of
log files each day</a>.
Ok, we could potentially distribute the problem, but spinning up clusters takes
time, and one of our mottos is to “do more with less.” How can we improve the
performance here to keep our scalability down?
One of the best things we can do is minimize the amount of memory we’re
allocating; we allocate a new <code class="language-plaintext highlighter-rouge">char[]</code> every time we read a line. But even
beyond that, we read through the line to put into this <code class="language-plaintext highlighter-rouge">char[]</code>, and read
through it again to split it by our record separator. But this splitting
creates an array of all of the fields in the line, and allocates memory for
each and every one of them. From an efficiency standpoint, this clean,
straightforward code is actually quite a mess.
To address our first concern with memory allocation, we can have a buffer
that’s already allocated to read into. The trick here is that the next line
will immediately follow the previous line in the same buffer, and the end of
it may not fit into the buffer. Worse, we may only get a partial bit of a
field that we care about.
The solution is to have two buffers which we swap between. The fields we care
about may straddle both buffers, so we’ll also construct a double-length buffer
for simple reconstruction. When we reach the end of the current buffer, we’ll
make it the “last buffer,” and then load more data into the other buffer,
promoting it to the current buffer. The caveat is that we need to make sure
that none of our lines has a length longer than both of these buffers combined.
Our second concern had to do with the inefficiency of splitting. Instead of
breaking apart the line in totality, once it’s in memory, let’s just
sequentially read through it. We’ll keep track of our progress through both
the buffer (index of the array) and our current line (number of delimiters
we’ve seen). Once we hit the right number of delimiters, we just need to find
the next delimiter, and we know our field is the contents in between. If we
hit the end of a line by reading a newline, we’ll reset our line progress.
Finally, once we’ve collected all our fields, we just need to rip through
the buffer until we find the next newline.
Enough chit-chat; let’s look at some code:
<figure class="highlight"><pre><code class="language-d" data-lang="d">import std.stdio;
immutable static ulong READ_BUFFER_SIZE = 32768;
immutable static char DELIMITER = cast(char)30;
immutable static char NEWLINE = cast(char)10;
immutable static uint TIMESTAMP_INDEX = 19;
immutable static uint COUNTRY_INDEX = 42;
immutable static char[] UNKNOWN_FIELD = "unknown";
class FastReader {
File file;
char[] bufferA;
char[] bufferB;
char[] double_buffer;
char[] current_buffer;
char[] last_buffer;
ulong num_buf;
ulong index;
uint num_del;
bool line_end;
this(string filename) {
file = File(filename, "r");
bufferA = new char[READ_BUFFER_SIZE];
bufferB = new char[READ_BUFFER_SIZE];
double_buffer = new char[2 * READ_BUFFER_SIZE];
current_buffer = bufferA;
last_buffer = bufferB;
num_buf = 0;
index = 0;
num_del = 0;
line_end = false;
}
void reset_progress() {
num_del = 0;
line_end = false;
}
void swap_and_load() {
last_buffer = current_buffer;
num_buf++;
if((num_buf &amp; 1) == 0)
current_buffer = bufferA;
else
current_buffer = bufferB;
current_buffer = file.rawRead(current_buffer);
index = 0;
}
void get_field(uint field_id, ref char[] field) {
if(line_end) {
field = UNKNOWN_FIELD.dup;
return;
}
ulong start = index;
while(num_del &lt; field_id) {
if(start == current_buffer.length) {
swap_and_load();
start = 0;
}
if(current_buffer[start] == NEWLINE) {
line_end = true;
break;
}
if(current_buffer[start] == DELIMITER)
num_del++;
start++;
}
if(start == current_buffer.length) {
swap_and_load();
start = 0;
}
if(line_end) {
field = UNKNOWN_FIELD.dup;
index = start + 1;
return;
}
ulong end = start;
bool swapped = false;
while(current_buffer[end] != DELIMITER) {
end++;
if(end == current_buffer.length) {
swap_and_load();
swapped = true;
end = 0;
}
if(current_buffer[end] == NEWLINE) {
line_end = true;
num_del--; //Don't count as delimiter
break;
}
}
num_del++;
if(!swapped)
field = current_buffer[start .. end];
else {
ulong size_end = last_buffer.length - start;
double_buffer[0 .. size_end] = last_buffer[start .. $];
double_buffer[size_end .. (size_end + end)] =
current_buffer[0 .. end];
field = double_buffer[0 .. (size_end + end)];
}
index = end + 1;
}
void advance_to_next() {
ulong start = index;
if(start == current_buffer.length) {
swap_and_load();
start = 0;
}
while(current_buffer[start] != NEWLINE) {
start++;
if(start == current_buffer.length) {
swap_and_load();
start = 0;
}
}
index = start + 1;
reset_progress();
}
bool eof() {
return file.eof &amp; (index == current_buffer.length);
}
void close() {
file.close();
}
}
void process_file(string file) {
char[] timestamp;
char[] country;
FastReader frd = new FastReader(file);
while(!frd.eof) {
frd.get_field(TIMESTAMP_INDEX, timestamp);
frd.get_field(COUNTRY_INDEX, country);
frd.advance_to_next();
writefln("%s\t%s", timestamp, country);
}
frd.close();
}
void main(string[] args) {
process_file(args[1]);
}</code></pre></figure>
This really beefs up our simple program from before to accomplish the same task.
Let’s break it down a bit. <code class="language-plaintext highlighter-rouge">bufferA</code> and <code class="language-plaintext highlighter-rouge">bufferB</code> are the two backing buffers,
and they will be pointed to by either <code class="language-plaintext highlighter-rouge">current_buffer</code> or <code class="language-plaintext highlighter-rouge">last_buffer</code>,
depending on our state, which we track with <code class="language-plaintext highlighter-rouge">num_buf</code>. To keep track of our progress
through <code class="language-plaintext highlighter-rouge">current_buffer</code> we use <code class="language-plaintext highlighter-rouge">index</code>, and to keep track of our progress through
a line we use <code class="language-plaintext highlighter-rouge">num_del</code> to represent the number of delimiters we’ve seen thus far.
Finally, we want to know if we’ve hit the end of a line with <code class="language-plaintext highlighter-rouge">line_end</code>.
The instantiation of <code class="language-plaintext highlighter-rouge">FastReader</code> is straightforward: we start with <code class="language-plaintext highlighter-rouge">bufferA</code>.
<code class="language-plaintext highlighter-rouge">reset_progress()</code> gets called when we hit the start of a new line and it just
updates the state to reflect that. <code class="language-plaintext highlighter-rouge">swap_and_load()</code> is our toggling method. Note
that the <code class="language-plaintext highlighter-rouge">num_buf</code> operation to determine our <code class="language-plaintext highlighter-rouge">current_buffer</code> is equivalent to
<code class="language-plaintext highlighter-rouge">num_buf % 2</code>.
There’s also a quick catch here with the <code class="language-plaintext highlighter-rouge">file.rawRead()</code> method: though it
takes a <code class="language-plaintext highlighter-rouge">char[]</code> in and populates it from the <code class="language-plaintext highlighter-rouge">File</code>, if we hit an end of file,
the <code class="language-plaintext highlighter-rouge">char[]</code> will have its previous contents past the EOF. For example:
<figure class="highlight"><pre><code class="language-d" data-lang="d">import std.stdio;
void main(string[] args) {
auto file = File("test.txt", "r");
char[] buffer = "0123456789".dup;
writeln("buffer length:\t", buffer.length);
writeln("buffer content:\t", buffer);
file.rawRead(buffer);
writeln("buffer length:\t", buffer.length);
writeln("buffer content:\t", buffer);
}</code></pre></figure>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; echo -n "test" &gt; test.txt
%&gt; rdmd test.d
buffer length: 10
buffer content: 0123456789
buffer length: 10
buffer content: test456789</code></pre></figure>
For our purposes, this is an undesirable behavior because we want to know
when we’ve actually hit the end of the <code class="language-plaintext highlighter-rouge">File</code> and the end of the last line.
It turns out that <code class="language-plaintext highlighter-rouge">file.rawRead()</code> also returns a <code class="language-plaintext highlighter-rouge">char[]</code> which is the slice
of the passed in array that has new content. Just changing the line to
<code class="language-plaintext highlighter-rouge">buffer = file.rawRead(buffer)</code> fixes the issue:
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; rdmd test.d
buffer length: 10
buffer content: 0123456789
buffer length: 4
buffer content: test</code></pre></figure>
We use the same construction in <code class="language-plaintext highlighter-rouge">swap_and_load()</code>, finally resetting our
progress through <code class="language-plaintext highlighter-rouge">current_buffer</code> to 0.
<code class="language-plaintext highlighter-rouge">get_field()</code> is the trickiest method, but still intelligible. If
we’ve already hit the end of the line, there’s no field to find, so we
write out that it’s unknown. Starting from where we are in <code class="language-plaintext highlighter-rouge">current_buffer</code>,
we just start counting delimiters. If we hit the end of <code class="language-plaintext highlighter-rouge">current_buffer</code>, it’s
time to <code class="language-plaintext highlighter-rouge">swap_and_load()</code>. Once we’ve hit the correct number of delimiters,
we need to find the next one. This is essentially the same code, but we need
to know if we swap buffers during this process. If we hit a newline, we count
that as the end of the field.
Constructing the field is simple: if we didn’t swap, it’s just the slice of
<code class="language-plaintext highlighter-rouge">current_buffer</code> from our start to end indices. Otherwise, we piece it
together from both <code class="language-plaintext highlighter-rouge">last_buffer</code> and <code class="language-plaintext highlighter-rouge">current_buffer</code>.
The <code class="language-plaintext highlighter-rouge">advance_to_next()</code> method also works in a similar way to searching out
delimiters, but instead we look for newline characters, move into the next
line, and then <code class="language-plaintext highlighter-rouge">reset_progres()</code>.
There are a couple of catches at this point. First, there’s our <code class="language-plaintext highlighter-rouge">eof()</code> method.
It’s possible that we hit EOF but there are more lines to read. We
check for this by ensuring that we’re only truly done when our <code class="language-plaintext highlighter-rouge">index</code> is
the same as our <code class="language-plaintext highlighter-rouge">current_buffer</code> length. Finally, in our <code class="language-plaintext highlighter-rouge">process_file()</code> call,
we make sure that we move on to the next line after getting all the fields we
require. Critically, we search for the fields in their numerical order.
We need to do this because we run through our lines in sequence and never
look back.
Ok, so our code gained some weight. But fortunately, it gained weight in
terms of muscle mass instead of bloat. Check it out:
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; time rdmd parser_fast.d log.txt &gt; country_info_fast
real 0m2.159s
user 0m2.093s
sys 0m0.068s</code></pre></figure>
Hey, that’s better than a 6x improvement! We can now run over way more
data in the same amount of time. Just to make sure nothing fishy is going
on:
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; diff country_info country_info_fast
%&gt;</code></pre></figure>
Awesome. But hey, y’know what? It would be really cool if we could exploit
the fact that I’m running on a multi-core machine and read in multiple files
at once. How would we do that?
<figure class="highlight"><pre><code class="language-d" data-lang="d">import std.parallelism;
import std.stdio;
import std.string;
/* Then it's all the same until main()... */
void main(string[] args) {
auto files = split(args[1], ",");
foreach(string file; parallel(files)) {
process_file(file);
}
}</code></pre></figure>
Well, that was pretty easy. What type of performance do we get?
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; for i in 2 3 4; do cp log.txt log$i.txt; done
%&gt; time rdmd parser_parallel.d log.txt &gt; more_country_info
real 0m2.219s
user 0m2.153s
sys 0m0.070s
%&gt; time rdmd parser_parallel.d log.txt,log2.txt &gt; more_country_info
real 0m2.308s
user 0m4.340s
sys 0m0.228s
%&gt; time rdmd parser_parallel.d log.txt,log2.txt,log3.txt &gt; more_country_info
real 0m2.458s
user 0m6.821s
sys 0m0.354s
%&gt; time rdmd parser_parallel.d log.txt,log2.txt,log3.txt,log4.txt &gt; more_country_info
real 0m2.634s
user 0m9.213s
sys 0m0.781s</code></pre></figure>
Wow! That’s over five time faster than our original solution while running over
four times the data, for a 20x performance boost! And we’re not done yet…
<code class="language-plaintext highlighter-rouge">rdmd</code> doesn’t perform as many optimizations as it could. Once our code is in
the state that we want it, it makes sense to compile with <code class="language-plaintext highlighter-rouge">dmd</code> rather than
running it under a scripting idiom. To be fair, we’ll also compile down our
original naïve solution and our non-parallelized one:
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">%&gt; dmd -O -release -inline parser.d
%&gt; time ./parser log.txt &gt; country_info
real 0m8.352s
user 0m8.224s
sys 0m0.128s
%&gt; dmd -O -release -inline parser_fast.d
%&gt; time ./parser_fast log.txt &gt; country_info_fast
real 0m0.697s
user 0m0.637s
sys 0m0.060s
%&gt; dmd -O -release -inline parser_parallel.d
%&gt; time ./parser_parallel log.txt &gt; more_country_info
real 0m0.950s
user 0m0.759s
sys 0m0.181s
%&gt; time ./parser_parallel log.txt,log2.txt &gt; more_country_info
real 0m0.857s
user 0m1.467s
sys 0m0.196s
%&gt; time ./parser_parallel log.txt,log2.txt,log3.txt &gt; more_country_info
real 0m1.127s
user 0m2.469s
sys 0m0.617s
%&gt; time ./parser_parallel log.txt,log2.txt,log3.txt,log4.txt &gt; more_country_info
real 0m1.527s
user 0m3.792s
sys 0m1.264s</code></pre></figure>
After compiling everything down, our fast solution has nearly a 12x performance
boost over the naïve one, and our parallelized solution, when run over four times
the data, has nearly a 22x boost.
At AdRoll Data Science, we’ve become big fans of D, and it’s easy to see why.
We can rapidly prototype new infrastructure and analysis tasks, and when
efficiency becomes a core concern, we have the ability to refactor that same
code base to squeeze as much performance out as possible. If you’re interested
in tackling big data problems with an eye for lean, mean code, you should
<a href="http://www.adroll.com/about/careers">let us know</a>.
</description>
</item>
<item>
<title>Shiny at AdRoll</title>
<link>https://tech.nextroll.com/blog/external/2014/10/29/shiny.html</link>
<pubDate>Wed, 29 Oct 2014 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/external/2014/10/29/shiny</guid>
<description>Read the post here: <a href="http://glossy.adroll.com/shiny_blogpost/">http://glossy.adroll.com/shiny_blogpost/</a>
</description>
</item>
<item>
<title>Introducing AirControl: AirPlay Mirroring from your terminal</title>
<link>https://tech.nextroll.com/blog/terminal/2014/09/26/introducing-aircontrol-control-airplay-through-terminal.html</link>
<pubDate>Fri, 26 Sep 2014 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/terminal/2014/09/26/introducing-aircontrol-control-airplay-through-terminal</guid>
<description>AirControl is a tool to control AirPlay Mirroring from the command line.
TL;DR: it’s open source and available in our <a href="https://github.com/AdRoll/AirControl">GitHub
repo</a>.
If you’re interested in the details of how we got it working, read on.
<hr />
Once a month, the Engineering team has a Hack Day. Anybody can work on anything
they like, be it discovering a new programming language, speeding up our test
suite, or, in the case at hand, building a terminal tool to control AirPlay
Mirroring.
The only constraint we put on these Hack Days is to work with other people on
your project.
<hr />
I use vim, Vimium, have custom mappings on my keyboard, wrote a Chrome extension
to log you out of websites with a keyboard shortcut: in short, I like typing
more than moving my mouse and clicking around. That’s why a couple of months
ago I realized that having to use the mouse to activate AirPlay Mirroring was
inconvenient… nay! Barbaric! Something needed to be done.
I recruited <a href="http://hackerengineer.net/">Eric</a> and
<a href="http://www.math.missouri.edu/~evanslc/">Chris</a> to work on this last Friday.
We worked through the problem in stages:
<h1 id="activate-airplay-mirroring-programmatically">Activate AirPlay Mirroring programmatically</h1>
First, we got a short AppleScript to activate the AirPlay menu in the menu bar:
<figure class="highlight"><pre><code class="language-applescript" data-lang="applescript">tell application "System Events"
tell process "SystemUIServer"
click (menu bar item 1 of menu bar 1 whose description contains "Displays")
click menu item 4 of menu 1 of result
end tell
end tell</code></pre></figure>
You can open AppleScript Editor, paste this and run it, if you have an AirPlay
device on your network, it should connect you.
Changing <code class="language-plaintext highlighter-rouge">item 4</code> to <code class="language-plaintext highlighter-rouge">item "Living-room"</code> works too.
Now we needed to have this AppleScript run from the terminal and take an
argument.
<h1 id="to-the-terminal">To the terminal!</h1>
The easiest way for that is to use <code class="language-plaintext highlighter-rouge">osascript</code>. <code class="language-plaintext highlighter-rouge">osascript</code> lets you run any
<a href="https://developer.apple.com/library/mac/documentation/applescript/conceptual/applescriptx/concepts/osa.html">Open Scripting
Architecture</a>
script and in particular AppleScript.
So, instead of using AppleScript directly, we let <code class="language-plaintext highlighter-rouge">bash</code> invoke <code class="language-plaintext highlighter-rouge">osascript</code> with
the above script filled in with the first argument passed in for AirPlay device
name:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">#!/usr/bin/env bash
tvname=\"$1\"
read -d '' APPLESCRIPT &lt;&lt;EOF
tell application "System Events"
tell process "SystemUIServer"
click (menu bar item 1 of menu bar 1 whose description contains "Displays")
click menu item $tvname of menu 1 of result
end tell
end tell
EOF
osascript -e "$APPLESCRIPT" &gt; /dev/null</code></pre></figure>
You can run the above as <code class="language-plaintext highlighter-rouge">aircontrol 2nd\ Floor\ Lounge</code> and you’ll connect to
the AppleTV with that name.
NB: you’ll need to enable your terminal for Accessibility settings as shown below:
<img src="/images/post_images/aircontrol-accessibility.png" alt="accessibility settings" />
<h1 id="picking-up-the-tab">Picking up the tab</h1>
At this point, we have something that does what we wanted: starting AirPlay
Mirroring from the terminal. But we want more! We <s>want</s> need to
tab-complete this thing!
<h2 id="dns-discovery">DNS Discovery</h2>
The first thing is to find the available device names. I had previously gathered
<a href="https://github.com/pstadler/non-terminating-bash-processes">some</a>
<a href="http://nto.github.io/AirPlay.html">references</a> that led us towards <code class="language-plaintext highlighter-rouge">dns-sd</code> to
find out what AirPlay devices are available.
Namely <code class="language-plaintext highlighter-rouge">dns-sd -B _aiplay._tcp</code> will print out all the AirPlay-enabled devices
on the network:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">$ dns-sd -B _airplay._tcp
Browsing for _airplay._tcp
DATE: ---Fri 26 Sep 2014---
11:09:47.758 ...STARTING...
Timestamp A/R Flags if Domain Service Type Instance Name
11:09:47.759 Add 3 4 local. _airplay._tcp. Honor Room (right)
11:09:47.759 Add 3 4 local. _airplay._tcp. Good Times Room
11:09:47.759 Add 3 4 local. _airplay._tcp. Dev
11:09:47.759 Add 3 4 local. _airplay._tcp. Unagi Room
...</code></pre></figure>
Using <code class="language-plaintext highlighter-rouge">cut</code> would trim that down to the useful info.
(and make sure you check out <a href="https://github.com/pstadler/non-terminating-bash-processes">this
post</a> on
non-terminating bash processes)
<h2 id="you-complete-me">You complete me</h2>
With that list in hand, we needed to tell <code class="language-plaintext highlighter-rouge">bash</code> how to tab-complete the
command.
The short story is that you run <code class="language-plaintext highlighter-rouge">complete -o nospace -F _aircontrol aircontrol</code>
where <code class="language-plaintext highlighter-rouge">_aircontrol</code> is a function that builds an array called <code class="language-plaintext highlighter-rouge">COMPREPLY</code> with
the completion candidates.
Here’s our whole script to activate the tab-completion:
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">_aircontrol()
{
local cur=${COMP_WORDS[COMP_CWORD]}
COMPREPLY=()
i=0
while read -r line; do
i=`expr $i + 1`
if [ $i -lt 5 ]; then continue; fi # skip the header lines
room=$( echo $line | cut -d ' ' -f 7-100 )
if `echo "$room" | grep ^${cur}.*$ &gt; /dev/null`; then
COMPREPLY+=("$(printf "%q" "$room")")
fi
# break if no more items will follow (e.g. Flags != 3)
if [ $(echo $line | cut -d ' ' -f 3) -ne '3' ]; then
break
fi
done &lt; &lt;(dns-sd -B _airplay._tcp)
}
complete -o nospace -F _aircontrol aircontrol</code></pre></figure>
To wrap it up, we added an option to stop the mirroring: <code class="language-plaintext highlighter-rouge">-k</code>.
Here it is in action:
<img src="/images/post_images/aircontrol.gif" alt="aircontrol in action" />
The final result is available here:
<a href="https://github.com/AdRoll/AirControl">https://github.com/AdRoll/AirControl</a>
As you can see, not that much code at all.
Starting from this, one could build a workflow for
<a href="http://www.alfredapp.com/">Alfred</a> or a <a href="https://developer.apple.com/library/mac/documentation/Carbon/Conceptual/MDImporters/MDImporters.html#//apple_ref/doc/uid/TP40001279-BBCFBCAG">plugin for
Spotlight</a>.
A Spotlight extension would be an interesting hack since it’s designed for
files, but that will have to wait for another Hack Day…
</description>
</item>
<item>
<title>Bulk Loading Multiple Tables</title>
<link>https://tech.nextroll.com/blog/data/2014/07/15/multi-table-bulk-import.html</link>
<pubDate>Tue, 15 Jul 2014 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data/2014/07/15/multi-table-bulk-import</guid>
<description>AdRoll’s customer dashboard is powered by our <a href="http://hbase.apache.org/">HBase</a> cluster, populated using both <a href="http://storm.incubator.apache.org/">Storm</a> and <a href="http://hadoop.apache.org/">Hadoop MapReduce</a>. Using Storm allows us to provide real-time statistics to our users, while Hadoop gives us the accuracy guarantees needed for billing. Throughout the day, Storm emits a steady stream of writes to HBase. Our MapReduce jobs however, run once a day over the previous day’s data. This generates a huge spike in write traffic that can drastically slow down, if not render the entire cluster unresponsive. To counteract this, we switched to having our MapReduce jobs <a href="http://hbase.apache.org/book/arch.bulk.load.html">bulk load</a>, skipping the write path entirely and saving both CPU and network IO. Unfortunately, we could not use the built-in bulk loading tools because of our non-standard use case.
<hr />
Each type of event we see is demultiplexed into a write for one or more tables. For example, a click event might lead to incrementing a counter based on which ad and campaign generated the click. For bidding purposes, we may also want to keep track of which bidding strategy was used and which site the ad was placed.
<img src="/images/post_images/hbase_demultiplex.png" alt="Multiplex" />
Unfortunately, with the way <a href="http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hbase/hbase/0.94.17/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java">HFileOutputFormat</a> is written, there was no way to do what we wanted without running over the same input multiple times. This is because HFileOutputFormat can only generate output for one table at a time. The amount of new data AdRoll collects and generates (on the order of 25TB/day) makes this a very unattractive, if not untenable option. Luckily, we have access to the source code so it was fairly painless to subclass the output format and add the ability to generate HFiles for multiple tables in one pass.
Before we dive into the technical details, we need to understand at a high level how HFileOutputFormat.<a href="http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hbase/hbase/0.94.17/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java#HFileOutputFormat.configureIncrementalLoad%28org.apache.hadoop.hbase.mapreduce.Job%2Corg.apache.hadoop.hbase.client.HTable%29">configureIncrementalLoad()</a> works. Cloudera has a <a href="http://blog.cloudera.com/blog/2013/09/how-to-use-hbase-bulk-loading-and-why/">great overview of bulk loading</a> – we mostly care about section 2 copied below.
<blockquote>
The job [Mapper] will need to emit the row key as the Key, and either a KeyValue, a Put, or a Delete as the Value. The Reducer is handled by HBase; you configure it using HFileOutputFormat.configureIncrementalLoad() and it does the following:
<ol>
<li>Inspects the table to configure a total order partitioner</li>
<li>Uploads the partitions file to the cluster and adds it to the DistributedCache</li>
<li>Sets the number of reduce tasks to match the current number of regions</li>
<li>Sets the output key/value class to match HFileOutputFormat’s requirements</li>
<li>Sets the reducer up to perform the appropriate sorting (either KeyValueSortReducer or PutSortReducer)</li>
</ol>
At this stage, one HFile will be created per region in the output folder.
</blockquote>
Our multi-table HFileOutputFormat (<a href="https://gist.github.com/wesleychowadroll/d3b0ee1e85c445379a32">source</a>) is going to do exactly this, but demultiplex each key into the appropriate HFile depending on destination table. The convention we have adopted is for each row key to be prepended with the table name. Colons are not legal in HBase table names, so they are safe to be used as separators. Step 1 from above then changes a little bit, but the general idea is to get each region start key and prepend the table name to it.
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby">partition_keys = []
for table_name in tables:
start_keys = get_region_start_key(table_name)
for start_key in start_keys:
partition_key.append(table_name + ':' + start_key)
end
end
configure_total_order_partitioner(partition_keys)</code></pre></figure>
Step 3 will set the number of reduce tasks to the total number of regions across all tables. The actual HFile demultiplexing happens in the output format RecordWriter (lines <a href="https://gist.github.com/wesleychowadroll/d3b0ee1e85c445379a32#file-multitablehfileoutputformat-java-L292-L304">292-304</a> and <a href="https://gist.github.com/wesleychowadroll/d3b0ee1e85c445379a32#file-multitablehfileoutputformat-java-L322">322</a>). Instead of just writing to the output path, we separate out the table name and the row key, then write each table’s HFiles into it’s own folder. To use our new output format, all we have to do is change the mapper to emit <code class="language-plaintext highlighter-rouge">&lt;table&gt;:&lt;row key&gt;</code> instead of just <code class="language-plaintext highlighter-rouge">&lt;row key&gt;</code>.
<hr />
The standard usage of HFileOutputFormat doesn’t allow any work to be done in the reduce step, as configureIncrementalLoad will set the reducer to be either KeyValueSortReducer or PutSortReducer. If we need to do work in the reduce step, we have two options. The first option is to have a two-step MapReduce job. The first reducer writes to a sequence file, which the second map reads and emits into a sorting reducer configured by HFileOutputFormat. The second option is a bit more efficient as it does all of the necessary work in one reducer. If we take a look at what the sorting reducers are doing, we see that all it does is take the KeyValues and emit them in sorted order. It’s easy then, to modify our reducer to do some useful work combining KeyValues, then sort and emit all in one step.
<figure class="highlight"><pre><code class="language-java" data-lang="java">public void reduce(ImmutableBytesWritable kw, Iterable&lt;Writable&gt; values, Context context) {
List&lt;KeyValue&gt; kvs = someUsefulWork(kw, values);
TreeSet&lt;KeyValue&gt; map = new TreeSet&lt;KeyValue&gt;(KeyValue.COMPARATOR);
for (KeyValue kv: kvs) {
map.add(kv.clone());
}
// Write out the values in order
for (KeyValue kv : map) {
context.write(kw, kv);
}
}</code></pre></figure>
That’s all there is to it! With these simple modifications, AdRoll is able to efficiently process and load large amounts of data into our HBase cluster. We are free to denormalize our data as needed, without worrying about many of the associated costs.
<hr />
</description>
</item>
<item>
<title>Hadoop at AdRoll</title>
<link>https://tech.nextroll.com/blog/events/2014/06/11/june-sf-hadoop-meetup.html</link>
<pubDate>Wed, 11 Jun 2014 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/events/2014/06/11/june-sf-hadoop-meetup</guid>
<description>Earlier this week, AdRoll hosted a group of Hadoop users and developers for their <a href="http://www.meetup.com/hadoopsf/">monthly meetup</a>. A member of our data team presented AdRoll’s infrastructure and described how we combine Hadoop with Storm and HBase to achieve both high accuracy and low latency. Here are the slides from that talk.
<script async="" class="speakerdeck-embed" data-id="30863080d49101316bc66ac2ad2297d3" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script>
</description>
</item>
<item>
<title>Valentino Volonghi Presents AdRoll's RTB Infrastructure at AWS Summit 2014</title>
<link>https://tech.nextroll.com/blog/web/2014/04/29/valentino-presents-adrolls-rtb-infrastructure.html</link>
<pubDate>Tue, 29 Apr 2014 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/web/2014/04/29/valentino-presents-adrolls-rtb-infrastructure</guid>
<description>Continuing with our video posts, here is Valentino once again, this time presenting at the 2014 AWS Summit in San Francisco. Valentino discusses how to handle 50 billion requests per day without breaking the bank by optimizing our AWS operations.
In addition to low cost, AWS allows AdRoll’s RTB infrastructure to achieve nearly 100% uptime with sub-100ms latency and less than 0.15% timeouts. Learn how we built out a globally scaled, highly available architecture that powers our advertising operations and pushes the limits of cloud computing.
<div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/0TsnCR6JMIM?start=1159&amp;end=2750" frameborder="0" allowfullscreen=""></iframe></div>
<hr />
</description>
</item>
<item>
<title>Werner Vogels' Fireside Chat with Valentino</title>
<link>https://tech.nextroll.com/blog/web/2014/04/22/werner-vogels-fireside-chat-with-valentino.html</link>
<pubDate>Tue, 22 Apr 2014 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/web/2014/04/22/werner-vogels-fireside-chat-with-valentino</guid>
<description>Better late than never, but we wanted to share a video from last fall’s AWS re:Invent conference where our own chief architect Valentino Volonghi shared the stage with Amazon CTO <a href="http://en.wikipedia.org/wiki/Werner_Vogels">Werner Vogels</a> for a fireside chat. Valentino discusses AdRoll’s use of <a href="http://aws.amazon.com/dynamodb/">DynamoDB</a>, one of the core technologies we leverage to build our real-time bidding system that handles more than 50 billion bid requests per day.
The resulting system is a highly-reliable, low-latency, seamlessly scalable framework that allows AdRoll to crunch data, determine a bid-price, and select an ad to serve within a 40ms auction window. So pull up a seat with Valentino and Werner and learn how AdRoll does all this while spending less on DynamoDB than we do on snacks.
<div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/ltMoyUADr7g?start=2244" frameborder="0" allowfullscreen=""></iframe></div>
<hr />
</description>
</item>
<item>
<title>How to add JSCS to your pre-commit hook</title>
<link>https://tech.nextroll.com/blog/web/2014/03/05/adding-jscs-to-your-commit-hook.html</link>
<pubDate>Wed, 05 Mar 2014 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/web/2014/03/05/adding-jscs-to-your-commit-hook</guid>
<description>I recently came across <a href="https://github.com/mdevils/node-jscs">JSCS</a> and thought
it would be a great addition to our git flow to enforce consistent code style
for all of our JavaScript code.
JSCS is a code style checker for JavaScript. It lets you decide if you prefer
your <code class="language-plaintext highlighter-rouge">if</code>’s with or without a space after them; if you’re OK with declaring
multiple variables with one <code class="language-plaintext highlighter-rouge">var</code> keyword; if your lines shouldn’t go over 80
characters; etc. A lot of it is a matter of taste of course, but the main goal
is consistency.
So far we’ve been pretty good at having a fairly homogeneous style in our
JavaScript files, but as our team grows, new people need to adapt to the implied
preferences of our code base. Adding JSCS means that our style guide is codified
in JSCS’ config file and less prone to interpretation.
Having it as part of our pre-commit hook results in less time spent in code
reviews saying things like “if you don’t mind, you missed a space here”. Some
might think it’s nitpicky, but we’re looking at it with the <a href="http://en.wikipedia.org/wiki/Broken_windows_theory">broken windows
theory</a> in mind.
<hr />
So, here is the addition we made to our precommit hook to run JSCS on any
JavaScript files that are staged:
<noscript><pre># Extract staged files to a temp directory
TMPDIR=
TMPFILE=
if [ &quot;$IS_LINUX&quot; == &quot;true&quot; ]; then
TMPFILE=`mktemp jscs_tmp_XXXXXX`
TMPDIR=`mktemp -d jscs_tmp_XXXXXX`
else
TMPFILE=`mktemp -t tmp/jscs_tmp_XXXXXX`
TMPDIR=`mktemp -t tmp/jscs_tmp -d`
fi
git diff --cached --name-only --diff-filter=ACMR | xargs git checkout-index --prefix=$TMPDIR/ --
# Check JavaScript code style
RUN_JSCS=1
TOTAL_ERRORS=0
JSFILES=$(git diff-index --name-status --cached HEAD | grep -v ^D | egrep &#39;.js$&#39; | cut -c3-)
if [ -z &quot;$JSFILES&quot; ]; then
# No JavaScript file changed for this commit
RUN_JSCS=0
elif [ -z &quot;$JSCS_PATH&quot; ]; then
echo &quot;Warning: You can&#39;t check the JS coding style.&quot;
echo &quot;You need to download and install jscs and set JSCS_PATH to its path.&quot;
RUN_JSCS=0
fi
# Ensuring proper coding style
if [ $RUN_JSCS -ne 0 ]; then
echo -n &quot;Checking JS style errors...&quot;
OUT=`$JSCS_PATH -r text $TMPDIR`
CODE=$?
# Erase last output line
echo -ne &#39;\r\033[K&#39;
if [ $CODE -ne 0 ]; then
# Replace temp file name with real filename with color
# in sed we use commas as separators for clarity, and execute echo
# in the replacement part to get colors
# Probably cleaner ways exist but I don&#39;t know them at the moment
OUT=`echo -e &quot;$OUT&quot; | sed &quot;s,$TMPDIR/$[^ ]*$,\`echo -e \&quot;\033[1;32m\1\033[0m\&quot;\`,&quot;`
# grab the number of errors for that file
# (keeps only numbers and takes the last line)
TOTAL_ERRORS=`echo -e &quot;$OUT&quot; | sed &#39;s/[^0-9]//g&#39; | tail -1`
# echo output minus last line
echo -e &quot;\033[1;37mJavaScript style errors found:\033[0m&quot;
echo -e &quot;$OUT&quot; | sed &#39;$ d&#39;
echo &quot;$TOTAL_ERRORS code style errors found.&quot;
echo &quot;Please fix and stage the files before commiting again.&quot;
rm -Rf $TMPDIR $TMPFILE
exit $CODE
else
echo &quot;No JS code style errors found.&quot;
fi
fi
# Clean up
rm -Rf $TMPDIR $TMPFILE
</pre></noscript>
<script src="https://gist.github.com/Timothee/8326f5544e7645f605b5.js?file=pre-commit.sh"> </script>
It’s fairly simple: we use <code class="language-plaintext highlighter-rouge">git checkout-index</code> to output the staged files to a
temporary folder, run JSCS against it and print out any errors, and stop the
commit if there are any errors.
The main drawback is that you can only really run JSCS against a full file and
not just the diff. So even if you change only one line in a file, you’ll have to
fix all the existing errors in that file, and the related pull request will be
more cluttered than necessary. But that should only be a transitional period:
once all your files have been cleaned up, you will get errors only on the
changes you’ve made.
Here are some tips to keep the initial cleanup manageable:
<ul>
<li>Cleanup your whole codebase in one big pull request containing only style
fixes. We did that mostly but split the cleanup by modules: models first, then
collections, etc.</li>
<li>On GitHub, on a pull request or commit page, you can add <code class="language-plaintext highlighter-rouge">?w=1</code> to the URL to
hide all whitespace-related diffs (from <a href="https://github.com/blog/967-github-secrets">GitHub’s
blog</a>). This won’t cover all the
changes caused by JSCS, but probably most of them.</li>
<li>Improve the hook to only keep errors for the staged lines. This is a bit
tricky to write and probably not worth the hassle since eventually we’d like
to have the whole project properly styled.</li>
</ul>
<h2 id="how-to-add-this-to-your-pre-commit-hook">How to add this to your pre-commit hook</h2>
<ol>
<li>Install JSCS (see
<a href="https://github.com/mdevils/node-jscs#installation">instructions</a>).</li>
<li>Create a file named <code class="language-plaintext highlighter-rouge">.jscs.json</code> with your <a href="https://github.com/mdevils/node-jscs#configuration">preferred
options</a>. As a starting
point, you can check <a href="https://gist.github.com/Timothee/96e944f99f2b97fc162f">our current
configuration</a>.</li>
<li>Paste the script above to your pre-commit hook in <code class="language-plaintext highlighter-rouge">.git/pre-commit</code>. (see the
<a href="http://git-scm.com/book/en/Customizing-Git-Git-Hooks">git documentation</a> for
more details) You’ll need to add <code class="language-plaintext highlighter-rouge">JSCS_PATH</code> to your environment, pointing to
where <code class="language-plaintext highlighter-rouge">jscs</code> lives.</li>
<li>Proceed to commit perfectly styled code!</li>
</ol>
</description>
</item>
<item>
<title>Speaking of Erlang and AdRoll...</title>
<link>https://tech.nextroll.com/blog/erlang/2014/01/27/speaking-of-erlang-and-adroll.html</link>
<pubDate>Mon, 27 Jan 2014 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/erlang/2014/01/27/speaking-of-erlang-and-adroll</guid>
<description>Speaking of <a href="http://tech.adroll.com/blog/erlang/2014/01/22/monitoring-with-exometer-at-adroll.html">Erlang and AdRoll</a>… now is probably a good time to let you know AdRoll will be acting as a sponsor at the Erlang Factory SF Bay Area Conference this coming March. Our very own Brian Troutwine (<a href="https://twitter.com/bltroutwine">@bltroutwine</a>) will be giving a talk on the complex problems involved in live monitoring our real-time bidding system. And just to put that in perspective a bit, we’re talking about live monitoring everything that could possibly go wrong on a system that receives 500K bid requests per second; requests which have the potential to spend a non-trivial amount of actual real-life company dollars in the blink of an eye.
Be sure to get your ticket to the conference now, so you can catch Brian’s talk and find out how we try to get as close to bulletproof monitoring as possible, not to mention all the other great talks on the schedule.
<hr />
<img src="/images/post_images/erlfactory-logo.png" alt="ErlFactory Logo" />
<hr />
The Erlang Factory SF Bay area is the largest event in the US dedicated to the Erlang programming language, and will feature over 50 speakers on 8 tracks. Talk topics will cover areas such as Reactive Programming, Analytics, Big Data, DevOps, Interoperationabilty, Infrastructure, Multi-core and Architectures.
Speakers include:
<ul>
<li>Bruce Tate - author of ‘Seven Languages in Seven weeks’</li>
<li>Dave Thomas - author of The Pragmatic Programmer</li>
<li>Bob Ippolito - founder of Mochi Media</li>
<li>Rick Reed - software engineer at WhatsApp</li>
<li>Brett Cameron - senior software architect with HP’s corporate Cloud System</li>
<li>Stuart Bailey - CTO of Infoblox</li>
<li>Erik Stenman - chief scientist at Klarna</li>
<li>Duncan McGregor - senior manager at Rackspace</li>
<li>and many, many more</li>
</ul>
Elixir’s inventor, José Valim – also a Ruby on Rails Core Team member, will give a joint keynote with Dave Thomas - author of The Pragmatic Programmer. Mike Williams, co-inventor of the Erlang programming language, will deliver the second keynote.
Besides the already traditional Erlang Express and Erlang OTP courses, this year participants are also offered courses on Elixir, Cowboy, Riak and Kazoo.
Use the code <code class="language-plaintext highlighter-rouge">ADROLL10</code> to get a 10% discount from the conference price. The discount applies to Early Bird rates until 11 February and to regular rates until the 6th of March.
You can read more and get your tickets at the <a href="http://www.erlang-factory.com/conference/show/conference-6/home/#home">Erlang Factory Conference Site</a>.
</description>
</item>
<item>
<title>Monitoring with Exometer: An Ongoing Love Story</title>
<link>https://tech.nextroll.com/blog/erlang/2014/01/22/monitoring-with-exometer-at-adroll.html</link>
<pubDate>Wed, 22 Jan 2014 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/erlang/2014/01/22/monitoring-with-exometer-at-adroll</guid>
<description>Welcome, friends, to the first AdRoll tech blog of 2014! Last night AdRoll
hosted <a href="http://www.meetup.com/ErlangSF/events/159719882/">Erlounge</a> here in our
Mission street offices; seventeen Erlangers showed up and three talks were
given. Juan Puig of Linden Labs gave a lightening tutorial on the use of
quickcheck in Erlang, the slides of which you can find
<a href="https://docs.google.com/presentation/d/1QQUwyJwBvv6yPt_FLOgo4ZoOXKkHil6JK1_O_-IPX90/edit?usp=sharing">here</a>.
Marc Sugiyama of Erlang Solutions–boy, they have nice business cards–spoke
about debugging techniques in long-lived Erlang systems. To my knowledge the
slides aren’t yet online, but if you can corner Marc at a conference do so.
As I completely spaced on getting these talks recorded–sorry, next time–I
present my slides with comments interspersed.
<hr />
<img src="/images/post_images/mon_exo/mon_exo.001.jpg" alt="Slide the 001" />
For my part, I spoke about the Real-Time Bidding (RTB) monitoring effort I’ve
been leading here at AdRoll and, in particular, the role of Ulf Wiger and Magnus
Feuer’s <a href="https://github.com/Feuerlabs/exometer">exometer</a> library in this work.
<hr />
<img src="/images/post_images/mon_exo/mon_exo.002.jpg" alt="Slide the 002" />
The dirty thing about monitoring is that it’s tangential to value-added work. It
provides no direct value to customers and is, well, expensive. Now, that’s not
to say monitoring isn’t important. It is. What we’re looking to do here is
safeguard our investments: investments in engineering time, customer
acquisition, in computing hardware and human life. (Happily, AdRoll doesn’t have
the responsibility of life and death, but we can engineer our systems to waste
people’s time, which they’ll never get back.) Monitoring complex systems is
playing the game of keeping ahead of work-stopping disaster. (Incidentally, I
highly recommend Charles Perrow’s
<a href="http://www.amazon.com/Normal-Accidents-Living-High-Risk-Technologies/dp/0691004129">Normal Accidents: Living with High-Risk Technologies</a>
and Eric Schlosser’s more recent
<a href="http://www.amazon.com/Command-Control-Damascus-Accident-Illusion/dp/1594202273">Command and Control: Nuclear Weapons, the Damascus Accident, and the Illusion of Safety</a>
if you’d like to read more in-depth on this subject.)
<hr />
<img src="/images/post_images/mon_exo/mon_exo.003.jpg" alt="Slide the 003" />
In our Erlang systems we’re on the look-out for two things primarily: red-line VM
statistics and metrics that say our applications have gone sideways. Any system
that’s going to help us monitor these things has to report to multiple
backends–meaning, I need to pump data through our sophisticated Big Data
pipeline which I barely understand, out to on-disk logs and to our metrics
aggregation partner, <a href="http://www.datadoghq.com/">DataDog</a>, via their statsd
client–and cannot introduce excessive overhead into the systems.
The RTB system, for instance, handles approximately 500,000 bid events per
second and must respond to each one, without fail, in 100 milliseconds. Let that
sink in for a bit. A few days ago we peaked 30 billion bid events in a single
day. Let that sink in. Any monitoring library we use is going to have to keep
up with that load and, as few people have that load to play with during
development, needs to be hackable enough for us to make changes. This implies
the authors have to be friendly and open to patches and blind “uh, I dunno, it
just doesn’t work” issue tickets.
<hr />
<img src="/images/post_images/mon_exo/mon_exo.004.jpg" alt="Slide the 004" />
We know what we need from a system–the ability to track two kinds of
statistics, configurable reporting and low-overhead–but what is it we’re
spending all this time building to avoid? What are the disasters inherent in the
system?
<hr />
<img src="/images/post_images/mon_exo/mon_exo.005.jpg" alt="Slide 005" />
There’s the usual suspects of VM killers: atom tables that hit the max, hitting
max processes or ETS tables. Monitoring these is a matter of going through the
<a href="http://www.erlang.org/doc/efficiency_guide/advanced.html#id69282">System Limits</a>
documentation a few times so you know what you’re up against. More difficult, if
only because it requires the intrinsic knowledge of the system engineers, are
identifying performance regressions–what’s the benchmark, anyway–and
‘abnormal’ behavior. What’s abnormal? Usually they’re identified after the fact
but smack you right in the face once they show up. For instance, it’s possible
to bid millions of dollars for individual ad slots. Exchanges don’t provide much
validation of bids except to assert that they’re well-formed. Bidding without a
safety net means opening yourself up to the possibility of blowing through your
company’s entire budget in a few minutes.
When something goes wrong at 500,000 bids a second it goes wrong real fast.
That’s where the general class of “Surprises” comes in. Insight into such a
monstrous system is hard-won and safety-nets are specific to the class of
problems you were able to anticipate or have survived. Effective monitoring
gives engineers a chance to examine the live system and discover behavior they
didn’t expect to create. Some of the unexpected behaviors are good and you
codify them, others are disasters waiting to happen and you dutifully fix them
before it comes to tears.
<hr />
<img src="/images/post_images/mon_exo/mon_exo.006.jpg" alt="Slide 006" />
<img src="/images/post_images/mon_exo/mon_exo.007.jpg" alt="Slide 007" />
Like I said, we’re using exometer and it’s very easy to get started yourself.
First, it’s important to understand exometer’s terminology about itself as it’s
a bit unlike any of the other monitoring libraries available for Erlang.
An “entry”, as the slides say, is a “receiver and aggregator of metrics”.
Consider the histogram entry. You stream number metrics into it–maybe Erlang
messages per second–and the histogram entry will build a histogram of this
data, discarding the original numbers as needed. A “reporter” is a exometer
entity capable of taking values from entries and shipping them elsewhere, across
a network in the case of the statsd reporter or, say, to your TTY. The
“subscription” defines how often a reporter–or reporters–will take values from
which entries and ship them off.
<hr />
<img src="/images/post_images/mon_exo/mon_exo.008.jpg" alt="Slide 008" />
As of this writing, entries are created dynamically. Here we’re creating the
<code class="language-plaintext highlighter-rouge">[rtb, bodhi, metrics_srv, packets_in]</code> metrics–which is a histogram whose
values expire after 60s–and a function entry. The function entry exometer
supplies is brilliant. Let’s break it down:
<ul>
<li>The exometer application will create an entry which calls <code class="language-plaintext highlighter-rouge">erlang:system_info/1</code> and</li>
<li>this entry will have four ‘values’: <code class="language-plaintext highlighter-rouge">port_count</code>, <code class="language-plaintext highlighter-rouge">process_count</code>, <code class="language-plaintext highlighter-rouge">thread_pool_size</code>.</li>
<li>These values will be be fed, by name, into the entry function so the metric
of the entry will the the result of <code class="language-plaintext highlighter-rouge">erlang:system_info(Value)</code> at
inspection time.</li>
</ul>
For instance, on one of our systems so instrumented, I find that:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt; exometer:get_value([erlang, system_info], process_count).
1065
</code></pre></div></div>
Function entries mean that exometer natively supports application specific and
VM specific metric gathering; it’s all just a matter of finding the right
functions to call. Even more brilliant, exometer supports a mini-language to
slice and dice function returns as needed because, remember, exometer entries
are number based. You’ll have to inspect the
<a href="https://github.com/Feuerlabs/exometer/blob/master/src/exometer_function.erl#L48">inline source documentation</a>
to learn more.
<hr />
<img src="/images/post_images/mon_exo/mon_exo.009.jpg" alt="Slide 009" />
Subscriptions are statically configured in exometer’s application configuration.
Here’ we’re creating four subscriptions: three for the
<code class="language-plaintext highlighter-rouge">[rtb, bodhi, metrics_srv, packets_in]</code> histogram and one for the
<code class="language-plaintext highlighter-rouge">[erlang, system_info]</code> function entry. The values of the tuple, by index, are:
<ol>
<li>The reporter we’re subscribing to an entry, here <code class="language-plaintext highlighter-rouge">exometer_report_statsd</code>,</li>
<li>the “entry name” which matches that supplied in the entry’s creation,</li>
<li>the “entry value” of which <code class="language-plaintext highlighter-rouge">max</code>, <code class="language-plaintext highlighter-rouge">median</code> and <code class="language-plaintext highlighter-rouge">mean</code> are stock for
histograms while we created the <code class="language-plaintext highlighter-rouge">port_count</code> value for the function entry,</li>
<li>the rate on which the reporter will poll the entry for values, in
milliseconds and</li>
<li>a “retry” flag which is sort of a workaround for races between entry and
subscription creation. (Consult the docs but set this to true.)</li>
</ol>
What we’ve got here now are entries collecting metrics and subscriptions to get
them shipped off to other–possibly multiple, depending on how many reporters
you configure–locations. All that’s left is configuring the reporter.
<hr />
<img src="/images/post_images/mon_exo/mon_exo.010.jpg" alt="Slide 010" />
Reporter configuration is also, itself, statically set in the exometer
application configuration. This is terrifically nice in the face of exometer
application restarts, made somewhat less effective by the purely dynamic creation
of entries. (Static all the way through is under discussion.) Above we’re just
defining the network location of our friendly statsd daemon and none of that
should be terribly surprising. What is worth pointing out is the <code class="language-plaintext highlighter-rouge">type_map</code>.
Exometer entries have no knowledge of the systems they’ll eventually be reported
to and, as such, carry no information about their ultimate coercion. The
reporter is the point of integration for this information.
Statsd has the following native aggregation types (which exometer would call an
“entry”):
<ul>
<li>gauges</li>
<li>counters</li>
<li>timers</li>
<li>histograms</li>
<li>meters</li>
</ul>
You can read more <a href="https://github.com/b/statsd_spec">here</a>. You can see in the
above that I’m shipping everything over as a ‘gauge’, including entries which
exometer is storing itself as a histogram. I’ll come back to this shortly.
<hr />
<img src="/images/post_images/mon_exo/mon_exo.011.jpg" alt="Slide 011" />
A side benefit, which you may have noticed, about exometer is that it’s
extremely configurable even to the point of totally avoiding its stock reporters
and entries. When I developed <code class="language-plaintext highlighter-rouge">exometer_report_statsd</code> I did so by ensuring that
the module was loaded by the code-server ahead of exometer being started and
configured exometer to report into that module. Non-default entries can be
slipped in similarly via configuration to the <code class="language-plaintext highlighter-rouge">exometer_admin</code> module. I haven’t
done this yet–it’s on the schedule for two sprints from now–but it looks
straightforward enough.
Exometer is configurable enough to the point of supporting your goofy, one-off
reporters without having to fork the library yourself. That is remarkably handy
when you need it.
I would also be remiss if I didn’t point out that Ulf and Magnus have been
extremely helpful and responsive to my needs: long email questions have
received detailed answers, non-trivial issues have been worked promptly and
they’ve been receptive to the (admittedly meager) code I’ve contributed upstream
so far. I can’t thank them enough.
<hr />
<img src="/images/post_images/mon_exo/mon_exo.012.jpg" alt="Slide 012" />
There are other monitoring libraries, better known and more venerable. Why did
we not choose to use them?
<ul>
<li>A lack of built-in reporters, to use the exometer term, is killer. Folsom
deployments tend to carry around bespoke code to pull metrics out and route
it around, with various degrees of reliability.</li>
<li>A lack of multi-reporting, in addition, unless you build that.</li>
<li>A relative lack of flexibility in adding new entry types.</li>
<li>A lack of ‘function’ entries, which are super handy for dealing with VM
statistics.</li>
<li>A lack of configurable polling periods for those projects that do ship with
limited reporting.</li>
</ul>
None of the other projects are totally lacking in all of the above, but each is
just enough to be problematic. Those that are venerable tend to be slower to
change, naturally.
<hr />
<img src="/images/post_images/mon_exo/mon_exo.013.jpg" alt="Slide 013" />
If exometer is so great–and it is, go use it–then what are the downsides?
Remember earlier when we created a histogram entry but then configured our
reporter to pump out select values as statsd gauges? Why is this? Well, statsd
assumes that it will receive the stream of numbers along with a bit of
metadata that assigns an aggregation type to it. Exometer assumes that it will
have this responsibility. As there’s no way to just push the state of a
histogram into statsd wholesale we fake it by pushing key points over. The
statsd ‘gauge’ is the only statsd type that assumes its “calculated at the
client rather than the server”. Honestly, I’m not sure how to resolve this. I
did <a href="https://github.com/Feuerlabs/exometer/issues/8">make an issue</a> for it.
As of this writing, there’s a serious space-leak in the histogram entry. This is
being <a href="https://github.com/Feuerlabs/exometer/issues/7">actively worked on</a>.
(Update: looks like this is resolved. Fun!)
I pointed out above that both subscriptions and reporters can be statically
configured. Entries, however, cannot be. There’s an
<a href="https://github.com/Feuerlabs/exometer/issues/5">issue for that</a>.
There are some reports that exometer won’t compile under R16B03 due to an
erl_syntax bug, but I believe this
<a href="https://github.com/Feuerlabs/exometer/commit/ec18c9ae2d7733c18971cf5e3bd57ed62b1432e4">has been resolved</a>.
<hr />
<img src="/images/post_images/mon_exo/mon_exo.014.jpg" alt="Slide 014" />
<hr />
<img src="/images/post_images/mon_exo/mon_exo.015.jpg" alt="Slide 015" />
<hr />
<img src="/images/post_images/mon_exo/mon_exo.016.jpg" alt="Slide 016" />
<hr />
<img src="/images/post_images/mon_exo/mon_exo.017.jpg" alt="Slide 017" />
<a href="http://www.erlang-factory.com/conference/show/conference-6/home/#brian-troutwine">
<img src="http://www.erlang-factory.com/static/upload/media/1390389464268463300x250speaker.png" alt="speaker badge" />
</a>
</description>
</item>
<item>
<title>Lazy loading Backbone collections with Promises</title>
<link>https://tech.nextroll.com/blog/web/2013/11/12/lazyloading-backbone-collection-with-promises.html</link>
<pubDate>Tue, 12 Nov 2013 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/web/2013/11/12/lazyloading-backbone-collection-with-promises</guid>
<description>Until recently, when our customers created a campaign, they could choose to target the whole world, the U.S. or create a list of specific countries and U.S. metros (e.g. the San Francisco Bay Area, the Atlanta metro, etc.).
Last week, we added the ability to target or exclude countries, metros, regions (e.g. states in the U.S., provinces in Canada,…), cities and postal codes. That meant doing a significant refactor of our geolocation code, which lead me to (finally) use <code class="language-plaintext highlighter-rouge">Deferred</code> objects.
We have under 200 countries and 250 metros in our database, so it was not a problem to bootstrap the whole list in our Backbone app on page load. However, regions are in the thousands and cities in the hundreds of thousands. We couldn’t just simply load everything anymore: we turned to lazy loading.
<hr />
With perfect timing, <a href="https://github.com/jashkenas">Jeremy Ashkenas</a>’ keynote presentation for <a href="http://backboneconf.com/">BackboneConf</a> was put online. He talks about a few Backbone patterns that he came across, including one to lazily load models in a collection. Here is the segment: (duration: ~1’30”)
<div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/P0YIdsJqKV4?start=1742&amp;end=1844" frameborder="0" allowfullscreen=""></iframe></div>
The idea is that the collection may or may not have loaded the model you’re interested in at this moment. If it’s loaded, the collection can give it to you right away. But if it’s not, it needs to get it asynchronously from the server. One way would be to have your view manage this:
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript">var user = users.get(id);
if (user) {
this.renderUser(user);
} else {
user = new User({id: id});
user.fetch({
success: function(user, response, options) {
this.renderUser(user);
}
});
}</code></pre></figure>
This works, but it’s hard to read, it’s messy and it doesn’t separate concerns well: the view shouldn’t need to check if a model is loaded or not. On top of that, you would need the same thing in any views that want to get a model.
<hr />
The solution of course is to move this logic into the collection itself, and Jeremy discusses using <code class="language-plaintext highlighter-rouge">Deferred</code> objects instead of passing a callback to the <code class="language-plaintext highlighter-rouge">lookup</code> method. Here is the code from his slides:
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript">// View code
users.lookup(id).then(function() {
...
});
// Collection code
lookup: function(id) {
var model;
if (model = this.get(id)) {
return $.Deferred().resolveWith(this, model);
} else {
model = new Model({id: id});
return model.fetch();
}
}</code></pre></figure>
The interesting part is in this line: <code class="language-plaintext highlighter-rouge">return $.Deferred().resolveWith(this, model);</code>. I.e. even if we have the model loaded locally, the collection returns a <code class="language-plaintext highlighter-rouge">Deferred</code> object, but it resolves right away with the local model. This way, the interface is the same in both cases and the view doesn’t need to worry about this at all. Very neat solution.
<hr />
However, the code above doesn’t work as-is, so I wanted to write down how I got it to work. (admittedly, I might have missed something or Jeremy just put the slide as an illustration, not to be taken literally)
The issue is that in one case the method returns a <code class="language-plaintext highlighter-rouge">Deferred</code> object resolved with a Backbone model, while in the second case we return what <code class="language-plaintext highlighter-rouge">model.fetch()</code> returns, which is a <a href="http://api.jquery.com/jQuery.ajax/#jqXHR"><code class="language-plaintext highlighter-rouge">jqXHR</code></a>. For our purposes, we just need to know that a <code class="language-plaintext highlighter-rouge">jqXHR</code> is an <code class="language-plaintext highlighter-rouge">XMLHttpRequest</code> that implements the <code class="language-plaintext highlighter-rouge">Promise</code> interface. When that <code class="language-plaintext highlighter-rouge">jqXHR</code> finally resolves (when the AJAX request finishes), the function passed to <code class="language-plaintext highlighter-rouge">then()</code> gets <code class="language-plaintext highlighter-rouge">data, textStatus, jqXHR</code> as arguments.
What we want to return however is a Backbone model to be consistent with the first case. After some slight refactoring, here is what I have:
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript">lookup: function(id) {
var model;
var deferred = $.Deferred();
if (model = this.get('id')) {
deferred.resolveWith(this, model);
} else {
model = new Model({id: id});
model.fetch({
success: function(model, response, options) {
deferred.resolveWith(this, model);
}
});
}
// Returning a Promise so that only this function can modify
// the Deferred object
return deferred.promise();
}</code></pre></figure>
One of the things to note is that I don’t return a <code class="language-plaintext highlighter-rouge">Deferred</code> object anymore but a <code class="language-plaintext highlighter-rouge">Promise</code>. The difference is that the <code class="language-plaintext highlighter-rouge">Promise</code> lets you add callbacks to the <code class="language-plaintext highlighter-rouge">Deferred</code> but doesn’t let you resolve it. It’s best practice in most cases to let whatever code created the object resolve it as well.
I like this approach a lot. It’s elegant and easy to read and adapts well to different cases.
<hr />
For example, in my geolocation case, I don’t want only one model at a time, but instead I want to do auto-completion and get a list of geolocations that match a specific string. In some cases, the geolocation type will be completely pre-loaded (for countries and metros); in others, it will be fetched on-demand.
The difference with the code above is that the <code class="language-plaintext highlighter-rouge">success</code> callback option for <a href="http://backbonejs.org/#Collection-fetch"><code class="language-plaintext highlighter-rouge">Backbone.Collection.fetch</code></a> doesn’t return the list of models that were fetched. Instead the callback arguments are <code class="language-plaintext highlighter-rouge">collection, response, options</code>. So I need a bit more code.
Here is a modified version for when you want to call <code class="language-plaintext highlighter-rouge">collection.fetch</code>:
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript">// `type`: type of geolocation
// `query`: query string you want to match your models against
// `this.prefetched`: array of what types are prefetched (set in initialize())
search: function(type, query) {
var matchingModels;
var deferred = $.Deferred();
// Here we check if the requested type is fully prefetched
if (this.prefetched[type]) {
// match() would be implemented however you need it to
matchingModels = this.match(type, query);
deferred.resolveWith(this, matchingModels);
} else {
var queryData = {
type: type,
query: query
};
this.fetch({
data: queryData,
success: function(collection, response, options) {
// At this point, the new models will have been added to the
// collection
var matchingModels = collection.match(type, query);
deferred.resolveWith(this, matchingModels);
}
});
}
// Returning a Promise so that only this function can modify
// the Deferred object
return deferred.promise();
}</code></pre></figure>
I can now create my collection with some fully bootstrapped geolocation types (countries and metros) and some that need to be fetched on-demand and the views don’t need to know which is which.
One drawback of having a <code class="language-plaintext highlighter-rouge">match</code> method in the collection is that I’m duplicating some server logic on the front end. The two could potentially clash, e.g. if my server takes the query string and sends back all cities whose name starts with the query string, while the collection matches a model if the query string is in the name, the displayed results won’t be consistent and may vary based on the order in which I make various queries.
I haven’t come up with a good solution to that. I could cache the list of IDs sent back by the server for each query string but a different problem appears for the prefetched types: I would need to also create list of IDs for any substring for the prefetched models.
For now, since I’ll be keeping a simple matching logic, duplicating the logic will do.
<hr />
Finally, I need to add a cache to my collection to keep track of the models I have already fetched from the server, because with the code above you could be making the same request multiple times.
When <code class="language-plaintext highlighter-rouge">search</code> is called, I just need to check if the requested type is fully prefetched. If it isn’t, I need to check if I have already queried that string for that type:
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript">// `type`: type of geolocation
// `query`: query string you want to match your models against
// `this.prefetched`: array of what types are prefetched (set in initialize())
// `this.requestCache`: object that keeps a cache of the strings that have already
// been queried from the server for each geo type
search: function(type, query, limit) {
var matchingModels;
var cache = this.requestCache[type];
var deferred = $.Deferred();
if (this.prefetched[type] ||
cache &amp;&amp; cache[query] &amp;&amp; cache[query] &gt;= limit) {
// match() would be implemented however you need it to
matchingModels = this.match(type, query, limit);
deferred.resolveWith(this, matchingModels);
} else {
var queryData = {
type: type,
query: query,
limit: limit
};
this.requestCache[type] = this.requestCache[type] || {};
this.requestCache[type][query] = limit;
this.fetch({
data: queryData,
success: function(collection, response, options) {
// At this point, the new models will have been added to the
// collection
var matchingModels = collection.match(type, query, limit);
deferred.resolveWith(this, matchingModels);
}
});
}
// Returning a Promise so that only this function can modify
// the Deferred object
return deferred.promise();
}</code></pre></figure>
I also added a <code class="language-plaintext highlighter-rouge">limit</code> attribute to be able to request a certain number of objects. I save that value in a dictionary keyed on the query string, so that I know how many of the already-cached objects will match this query for that geolocation type. For example, I could have some parts of the application that need only 20 items and some where I want to show 100 items. If I already searched for 100 items for a specific query string, I don’t need to make a new query for the part of the app where I only need 20 items. This saves some additional roundtrips to the server and make the application feel snappier.
<hr />
I could still improve this further by delaying the call to <code class="language-plaintext highlighter-rouge">fetch</code> so that I can gather up queries on multiple types at once. So if I were to call:
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript">collection.search('city', 'san').then(printGeos);
collection.search('region', 'san').then(printGeos);</code></pre></figure>
only one server call could be made to get results for both types. But I’ll leave that as an exercise for now.
Another area of improvement would be to add ways to manage the cache and the size of the collection. It’s probably a good idea to be able to clear out the cache over time and to make sure the collection doesn’t grow too big. But, this too will have to wait.
<hr />
To conclude, I now have a lazy-loaded collection that I can partially prefetch as needed, and I limit the number of server queries by caching the query terms. <code class="language-plaintext highlighter-rouge">Deferred</code> and <code class="language-plaintext highlighter-rouge">Promise</code> objects let me present a single, elegant interface to the views.
</description>
</item>
<item>
<title>Why we joined AdRoll Engineering</title>
<link>https://tech.nextroll.com/blog/adroll/2013/10/18/why-we-joined-adroll.html</link>
<pubDate>Fri, 18 Oct 2013 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/adroll/2013/10/18/why-we-joined-adroll</guid>
<description>AdRoll is going gangbuster when it comes to hiring engineers. We’re adding headcount in a very competitive market on an ongoing basis, and we continue to differentiate ourselves with the depth and complexity of our engineering problems. We take a very collaborative approach to engineering but give individuals a high degree of project ownership. While we bounce ideas off one another and learn together, each engineer is tackling large problems directly tied to the company’s bottom line.
Below you’ll meet a few of our college hires and learn why they chose to join AdRoll Engineering. Wesley Chow joins our data engineering team as a recent graduate of UC Berkeley EECS. Marianne “Mars” Jullian joins us from Princeton and is currently working on the front-end team.
Intro written by <a href="mailto:jesse@adroll.com">Jesse Lauro</a>, an engineer with AdRoll’s Real-time Bidding team.
<hr />
<h1 id="wesley-chow"><a href="mailto:wesley.chow@adroll.com">Wesley Chow</a></h1>
The CS job market for new graduates is very hot right now. While I don’t have any numbers to back this up, I would guess that many companies have trouble finding enough candidates to fill their open positions. Knowing this, many college seniors are picky (some would say spoiled) in choosing where they will work after graduation. Keeping this in mind, I wanted to share why I chose specifically to work at AdRoll over my other offers.
I had two major criteria during my job hunt. The first was that I really wanted a chance to grow into a large role within the company. My hope was that this would involve building core projects right from the start. Ideally I would have the opportunity to work on projects that would have a measurable effect on the success of the company. Clearly this is much easier to accomplish at a startup than at a more established company. With only twelve engineers at the time I interviewed (AdRoll engineering has since grown to about twenty), I felt that AdRoll fit the bill perfectly. Also, it definitely helped that AdRoll had a successful and well defined business model, contributing towards an overall sense of job security.
My second criterion was to continue to pursue my specific interests within the field. During my last year at school, I had developed an interest in working with distributed systems. At the risk of being cliche, I wanted to work with “Big Data”. With around 500TB of data stored in Amazon S3 and approximately 10TB of data being handled per day, AdRoll certainly qualified.
Two months in, I’ve already had the chance to touch the typical tools for storing and handling large amounts of data (Hadoop, HBase, S3, and DynamoDB), in addition to getting to play with some more cutting edge realtime processing tools such as <a href="http://storm-project.net/">Storm</a> and <a href="http://kafka.apache.org/">Kafka</a>. My next project promises to be a system I will get to build from the ground up that involves AdRoll’s push to go global. How’s that for an exciting challenge to a new engineer? I’m ecstatic. AdRoll has already proven to be the right choice, with a lot of long term potential.
<hr />
<h1 id="mars-jullian"><a href="mailto:mars.jullian@adroll.com">Mars Jullian</a></h1>
When I first started looking for jobs in October of last year I was a little nervous about how the process would go, but by the end of it I had received four comparable offers all located in the Bay Area. Everyone’s work and mission interested me; so why did I choose AdRoll over the other three companies? Between the four offers, it eventually came down to the culture of each company and opportunities for learning.
From a previous summer internship experience with a startup I already knew I wanted to work in a small company. In smaller startups, a fewer number of people aspire to tackle the same magnitude of problems that larger companies face. I found that there were more opportunities to learn and a greater chance of being able to make an impact. On the other hand, I was also interested in joining a larger company with some social support as I was facing a move to not only a new city, but also a completely different coast.
AdRoll was able to offer me both: a small team where I would have an opportunity to learn and grow as a developer, being only one of a handful of engineers working on big problems, and also the support of a larger company. AdRoll has about 250 employees spread across three offices in SF, NY, and Dublin with an engineering team of only about twenty people and the team I work on, the front-end engineering team, is even smaller than that: only four people (including me). It was the startup feel I was looking for in the context of a larger and social company.
Now that I am here at AdRoll, and six weeks in, I am so glad I made the decision that I did. Not only am I glad to be working on a small team that is part of a larger company, but the people are great and no one ever could have explained how interesting the problems the engineering team is working on are until I arrived. It’s going to be quite an adventure working here…
</description>
</item>
<item>
<title>Multi-Region DynamoDB</title>
<link>https://tech.nextroll.com/blog/ops/2013/10/02/dynamodb-replication.html</link>
<pubDate>Wed, 02 Oct 2013 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/ops/2013/10/02/dynamodb-replication</guid>
<description>In rolling out our real time bidding infrastructure, we were faced with
the task of syncing data for every user we could possibly target across
four regions. We have on the order of hundreds of millions of users,
and tens of thousands of writes per second. Not only do we have to deal
with the daunting task of writing this data out in real time, the
bidding system has a hard cap of 100ms for every bid request, so we need
strong guarantees on read performance.
<img src="/images/post_images/dynamodb-replication-cloudwatch.png" alt="Cloudwatch" />
With DynamoDB we are able to have consistent, fast performance without
having to worry about scaling out our own database infrastructure.
However, DynamoDB does not support any concept of replication so we
needed to build a system to write data to multiple data centers. Most
replication solutions generate a log of every write, and then sends that
log to all slaves. In our case, our application servers generate a list
of writes that act as our source of truth, and then we can treat each
region’s DynamoDB table as a slave reading from those logs.
The data we are storing is a mapping of user ids to segments, along with
a timestamp. We can then look up which ad to serve from the list of
valid segments.
<img src="/images/post_images/dynamodb-replication-userlist.png" alt="User Lists" />
The first replication solution was implemented using map reduce.
Periodically, a job would gather all the (<code class="language-plaintext highlighter-rouge">userID</code>, <code class="language-plaintext highlighter-rouge">segmentID</code>, <code class="language-plaintext highlighter-rouge">timestamp</code>)
tuples from logfiles, with an output format that wrote to every region
that needed this data. As we scaled up to more and more users, this job
would take many hours to run. Also, due to the nature of batch jobs,
there would be long idle periods, followed by large spikes in writes to
DynamoDB. Since billing was based on throughput, we were interested in
a much more consistent write load. We began searching for solutions to
speed this system up.
We have been moving away from batch jobs towards more realtime
processing systems using <a href="http://storm-project.net">Storm</a>. The process for converting the batch
job to Storm was very straight-forward. We already had a system for
ingesting our data into Storm; we just had to translate the data into
partial updates to update the time for a particular segment.
This scheme ran into some performance issues. In the map reduce job, we
were able to combine all the updates per user over a couple hour time
period into a single update. However, with Storm we were issuing them
one at a time, and suffering from the large overhead of each http
request. Since most of the time we were idle waiting for requests, the
first performance enhancement was to switch to the asynchronous Java
API. We were then able to issue more requests in parallel, but it still
was not able to get the throughput we expected.
We really wanted to use batching, but the API does not support batch
updates, only batch puts. This required a slight change to our schema.
<img src="/images/post_images/dynamodb-replication-userprofile.png" alt="User Profiles" />
By using range keys each row is independent of the others. Updates can
be done using the batch API, which lets us send 25 updates at a time,
effectively multiplying the throughput by that amount. Reads then just
have to query by user id with no range key, which will fetch all the
same data as the old schema.
One important thing to keep in mind is that the batch API can be
partially successful, and it returns the subset of write requests that
failed. And since each region has its own request, a given write may
succeed in one region and not the other. Our writes can safely be
repeated, so we err on the side of replaying a given write, which fits
well with Storm’s promise of at-least-once processing. We maintain a
list of all writes that are in flight, along with the number of regions
it has successfully been written to. When a request returns, that count
is incremented for all successful values. Once that count equals the
number of expected regions written to, it is acknowledged in Storm as
successful. Any failed values get retried a couple of times within the
same process. After the retries are finished, if there are any failed
writes left, we mark it as failed in Storm, which will cause it to be
put in the back of the write queue to be retried later. As most
failures are temporal, generally due to throughput limit throttling,
this process can repeat until all writes complete successfully.
While DynamoDB does not inherently support multiple regions, given a
stream of writes it’s relatively easy to send those data across the
world. Combining batch, asynchronous writes with the speed of DynamoDB
yields us an average of over ten thousand writes per second on a single
Storm worker, with plenty of room to handle spikes. Add to that linear
scaling of throughput with the number of workers, and we have a system that will
handle any load we can throw at it.
</description>
</item>
<item>
<title>DNS-less setup in AWS</title>
<link>https://tech.nextroll.com/blog/ops/2013/09/30/dns-less-setup-in-aws.html</link>
<pubDate>Mon, 30 Sep 2013 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/ops/2013/09/30/dns-less-setup-in-aws</guid>
<description>Anybody that has worked with <a href="http://aws.amazon.com/">AWS</a> must deal
with the arbitrary hostnames that AWS generates for each instance. These
names are difficult to remember and not particularly useful to
understand what tasks any particular node is assigned to.
The classic solution to this problem is to use a DNS server, but
unfortunately setting up and maintaining a DNS system can be a pain and,
when you are a quickly moving startup, it’s a lot easier to spend your
time maintaining only the bare necessities. Of course eventually you’ll
need a proper setup, but until then, there are plenty of work arounds that
do their job until you’re big enough to have more time to refine your
infrastructure.
Setting up a Private DNS is <a href="http://security.stackexchange.com/questions/39504/pros-cons-private-dns-vs-public-dns">not a simple task</a>.
Using Amazon Route53 could be an option, but it <a href="http://aws.amazon.com/route53/faqs/#Manage_private_IP_address">doesn’t manage private
ip addresses</a> any
differently so, unless you want to leak your private addresses to the
outside world, you’ll have to set up your own DNS. In addition to all of
this, although there are many options for cheap private DNS service,
nothing is cheaper than free.
Usually you live through this by setting up entries in the <code class="language-plaintext highlighter-rouge">/etc/hosts</code>
file but a great advantage of the cloud is that instances come up and
go down and scale up and down. Manually changing all of the <code class="language-plaintext highlighter-rouge">/etc/hosts</code>
or pre-configuring them is hard and time consuming, and worst of all
it’s hard to change in case of failures.
We’ve tried to work around the immediate necessity for a DNS server
while still obtaining all of its benefits by using tags in AWS. The
system works in two parts: on one side you set tags like <code class="language-plaintext highlighter-rouge">dns: HOSTNAME</code>
on every machine that you want to reach with a simple hostname, on the
other side you need a simple script that can build a <code class="language-plaintext highlighter-rouge">/etc/hosts</code> file
by using the information obtained from the EC2 Tags service.
<figure class="highlight"><pre><code class="language-python" data-lang="python">#!/usr/bin/env python

import os
from optparse import OptionParser
from boto import ec2
from boto.s3.connection import S3Connection
from boto.s3.key import Key
def put_file(bucket, key_name, content):
k = Key(bucket)
k.key = key_name
k.set_contents_from_string(content,
{'Content-Type': 'text/plain'},
replace=True)
def main(options):
aws_api_key = options.aws_access_key
aws_secret_key = options.aws_secret_key
os.environ['AWS_ACCESS_KEY_ID'] = options.aws_access_key
os.environ['AWS_SECRET_ACCESS_KEY'] = options.aws_secret_key
regions = ['us-west-1', 'us-west-2', 'us-east-1', 'eu-west-1', 'ap-southeast-1']
s3conn = S3Connection(aws_access_key_id=aws_api_key,
aws_secret_access_key=aws_secret_key)
bucket = s3conn.get_bucket('my-configuration-bucket')
region_instances = {}
for region in regions:
conn = ec2.connect_to_region(region)
all_reservations = conn.get_all_instances(filters={ 'tag-key': 'dns' })
instances = {}
for reservation in all_reservations:
for instance in reservation.instances:
if instance.state == "running" and 'dns' in instance.tags:
for dns_name in instance.tags['dns'].strip().split(","):
instances[dns_name] = instance
region_instances[region] = instances
#Generate an etc host for each region, local instances using 10. and others using public
 for dest_region in regions:
hosts_lines = [
'127.0.0.1 localhost',
'::1 ip6-localhost ip6-loopback',
'fe00::0 ip6-localnet',
'ff00::0 ip6-mcastprefix',
'ff02::1 ip6-allnodes',
'ff02::2 ip6-allrouters',
'ff02::3 ip6-allhosts'
]
for src_region in regions:
for dns_name, instance in region_instances[src_region].iteritems():
if src_region == dest_region:
hosts_lines.append("%s %s.internal %s" % (instance.private_ip_address, dns_name, dns_name))
else:
hosts_lines.append("%s %s" % (instance.ip_address, dns_name))
# make sure we replace hosts file as atomically as possible.
etc_hosts = '#! /bin/bash\n\necho """\n' + '\n'.join(hosts_lines) + '\n""" | tee /tmp/etc_hosts &amp;&amp; cp /tmp/etc_hosts /etc/hosts'
put_file(bucket, 'etc_hosts/etc_hosts.%s.sh' % dest_region, content=etc_hosts)
#Generate an etc hosts with all public ip addresses, for developers
 hosts_lines = []
for src_region in regions:
for dns_name,instance in region_instances[src_region].iteritems():
hosts_lines.append("%s %s %s.internal" % (instance.ip_address, dns_name, dns_name))
etc_hosts = '\n'.join(hosts_lines)
put_file(bucket, 'etc_hosts/etc_hosts.public', content=etc_hosts)
if __name__ == '__main__':
"""
e.g, update_hosts.py --secret=&lt;SECRET_KEY&gt; --api=&lt;API_KEY&gt;
"""
usage = "usage: %prog [options]"
parser = OptionParser(usage=usage)
parser.add_option("--secret", action="store", dest="aws_secret_key",
default=os.environ.get('AWS_SECRET_ACCESS_KEY'), help="AWS Secret Access Key")
parser.add_option("--api", action="store", dest="aws_access_key",
default=os.environ.get('AWS_ACCESS_KEY_ID'), help="AWS Access Key")
(options, args) = parser.parse_args()
main(options)</code></pre></figure>
This simple script will create and upload to S3 multiple versions of
your <code class="language-plaintext highlighter-rouge">/etc/hosts</code> file to accomodate for all the different regions that
your infrastructure could be running on. In each regional hosts file
there will be private addresses for local instances and public addresses
for remote ones, but in all cases always accessible through the same
simple name.
At this point all of your instances can just download the script from S3
and execute it to obtain a proper and up-to-date version of the
<code class="language-plaintext highlighter-rouge">/etc/hosts</code> file.
In addition, to simplify development, there is also a fully public
version of the hosts file. By using this version of the hosts and
loading it on your computer you will now be able to access any machine
of your cluster without needing to lookup their AWS hostname.
Of course there are a number of alternatives to solve this issue. By
using this this trick at AdRoll, we manage our current installation with
more than one hundred instances without many headaches.
Soon we’ll need to look into a proper DNS server but until then…
</description>
</item>
<item>
<title>Unexpected behavior with jQuery</title>
<link>https://tech.nextroll.com/blog/web/2013/07/22/playing-with-jquery-append.html</link>
<pubDate>Mon, 22 Jul 2013 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/web/2013/07/22/playing-with-jquery-append</guid>
<description>As perfect as jQuery may seem, sometimes its default behaviors end up
being very surprising to the uninitiated.
During a routine code review, these few lines were pointed out as being
weird because, even though there are 2 <code class="language-plaintext highlighter-rouge">append()</code> calls, nothing was
evidently wrong from the UI.
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript">if (this.summaryBar) {
this.$('.slot-one').append(this.summaryBar.el, this.campaignGraph.el);
}
this.$('.slot-one').append(this.campaignGraph.el);</code></pre></figure>
(NB: we’re using <a href="http://backbonejs.org/">Backbone.js</a> and these are Backbone views, so <a href="http://backbonejs.org/#View-el"><code class="language-plaintext highlighter-rouge">.el</code></a> represents the DOM element for these views)
Dan wrote:
<blockquote>
will this append the campaignGraph twice in the case that there is a summary bar?
</blockquote>
Reading the code, it seemed to be that way but running the code, <code class="language-plaintext highlighter-rouge">campaignGraph</code> was added only once.
This called for some digging.
<hr />
The reason is actually <a href="http://api.jquery.com/append/">documented</a>:
<blockquote>
You can also select an element on the page and insert it into another:
<code class="language-plaintext highlighter-rouge">$('.container').append($('h2'));</code>
If an element selected this way is inserted into a single location elsewhere in the DOM, it will be moved into the target (not cloned).
If there is more than one target element, however, cloned copies of the inserted element will be created for each target after the first.
</blockquote>
This actually makes some sense: an element is passed, so why not use it. But when called on multiple elements, we need to clone the argument to append it to multiple DOM locations.
However, because of that, <code class="language-plaintext highlighter-rouge">append()</code> doesn’t behave like you would expect when called multiple times.
Take this HTML:
<figure class="highlight"><pre><code class="language-html" data-lang="html">&lt;ul&gt;&lt;/ul&gt;</code></pre></figure>
What do you expect this JavaScript to do:
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript">var li = $('&lt;li&gt;Hello&lt;/li&gt;');
$('ul').append(li);
$('ul').append(li);
$('ul').append(li);
$('ul').append(li);</code></pre></figure>
Until this morning, I would have expected to get four <code class="language-plaintext highlighter-rouge">&lt;li&gt;</code> elements inside the <code class="language-plaintext highlighter-rouge">&lt;ul&gt;</code> but it will have only the one I created since it’s never cloned:
<figure class="highlight"><pre><code class="language-html" data-lang="html">&lt;ul&gt;
&lt;li&gt;Hello&lt;/li&gt;
&lt;/ul&gt;</code></pre></figure>
Even though it follows the documentation (<a href="#fn1">mostly</a>), the behavior is more unexpected when your selector matches multiple elements.
Let’s use the following HTML instead:
<figure class="highlight"><pre><code class="language-html" data-lang="html">&lt;ul&gt;&lt;/ul&gt;
&lt;ul&gt;&lt;/ul&gt;
&lt;ul&gt;&lt;/ul&gt;</code></pre></figure>
With the same JavaScript, the result is:
<figure class="highlight"><pre><code class="language-html" data-lang="html">&lt;ul&gt;
&lt;li&gt;Hello&lt;/li&gt;
&lt;li&gt;Hello&lt;/li&gt;
&lt;li&gt;Hello&lt;/li&gt;
&lt;li&gt;Hello&lt;/li&gt;
&lt;/ul&gt;
&lt;ul&gt;
&lt;li&gt;Hello&lt;/li&gt;
&lt;li&gt;Hello&lt;/li&gt;
&lt;li&gt;Hello&lt;/li&gt;
&lt;li&gt;Hello&lt;/li&gt;
&lt;/ul&gt;
&lt;ul&gt;
&lt;li&gt;Hello&lt;/li&gt;
&lt;/ul&gt;</code></pre></figure>
I followed the tracks of this behavior in <a href="http://bugs.jquery.com/ticket/8070">jQuery’s bug tracker</a> and <a href="https://github.com/jquery/jquery/commit/0a0cff9d29c4bd559da689d96c532d06c03fce09/#L0R319">commit log</a> and it dates from about two years ago. The reason is to avoid memory leaks. Sadly, the current side-effect is unintuitive.
Here is a pen you can play with:
<blockquote>
See the Pen <a href="http://codepen.io/adrolldev/pen/zmonw">jQuery.append() unexpected behavior</a> by AdRoll Dev (<a href="http://codepen.io/adrolldev">@adrolldev</a>) on <a href="http://codepen.io">CodePen</a>
</blockquote>
1. The documentation is incorrect since it's actually the last element, not the first that ends up with the original element. But the main idea remains. (as a good open-source citizen, I submitted an edit to the documentation)
Photo credit: <a href="http://www.flickr.com/photos/eschipul/">Ed Schipul</a>
</description>
</item>
<item>
<title>HyperLogLog and MinHash</title>
<link>https://tech.nextroll.com/blog/data/2013/07/10/hll-minhash.html</link>
<pubDate>Wed, 10 Jul 2013 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/data/2013/07/10/hll-minhash</guid>
<description>At AdRoll, we have a lot of data to deal with. As we keep accumulating all of this data, our scaling issues become
more complicated, and even something as simple as counting becomes a bit of a chore. After using Bloom filters to
count uniques, we eventually wanted to find something more space efficient.
We started researching, and
implemented a form of HyperLogLog, which gives us the ability count uniques with good accuracy, do it in a
distributed way, and keep our memory and storage requirements down. One of the drawbacks though, was that we
couldn’t take intersections with these structures, and so we tacked on the additional structure of MinHash.
This blog post briefly covers our research. For the full, gory mathematical details, I also wrote a
<a href="/media/hllminhash.pdf">much more thorough, yet casual, paper</a>, with way more graphs and analysis. If that’s what you’re
looking for, go straight to the paper: there’s nothing in the blog post that isn’t covered in the paper.
<h3 id="hyperloglog">HyperLogLog++</h3>
The first stop we made was the Google paper <a href="http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/40671.pdf">“HyperLogLog in Practice: Algorithmic Engineering of a State of The Art
Cardinality Estimation Algorithm”</a>
by Heule, Nunkesser, and Hall. It describes some improvements to the HyperLogLog algorithm, which the authors
now call HyperLogLog++. Double double.
Essentially, it works like this: When you go to add an element to the structure, you hash it into 64 bits. From there,
you divide up the hash as:
<script type="math/tex; mode=display">
%<![CDATA[
\overbrace{101\ldots 011}^{p\ \rm{bits}}\overbrace{001\ldots 110}^{64 - p\ \rm{bits}}
%]]>
</script>
Where $p$ is your defined precision. The first chunk defines the index $i$ into an array $M$ which has $2^p$ bins,
all initialized to zero. Now, for the second chunk, count the number of leading zeros plus one (in this case, three),
which we’ll call $z$. Then you set $M[i] = \max\left\{M[i], z\right\}$. By the end, you have a bunch of bins with
a bunch of numbers in them. Now what?
Well, it turns out that you can get an estimate of how many unique elements were added to this structure. And the
way you do this is:
<script type="math/tex; mode=display">
%<![CDATA[
E = 2^{2p} \left(2^p \int_{0}^{\infty} \left( \log_2 \left( \frac{2 + u}{1 + u} \right) \right)^{2^p}
{\rm{d}}u\right)^{-1} \left( \sum_{i=0}^{2^p - 1} 2^{-M[i]} \right)^{-1}
%]]>
</script>
Pretty nasty, but that second term converges. The Google paper recommends:
<script type="math/tex; mode=display">
%<![CDATA[
\begin{matrix}
p & & \rm{second\ term} \\
4 & & 0.673 \\
5 & & 0.697 \\
6 & & 0.709 \\
> 7 & & 0.7213/(1 + 1.079/2^p)
\end{matrix}
%]]>
</script>
And we’re still not done. If there are any zeros left in $M$, we count them (call it $V$) and output
$E = 2^p \log\left(\frac{2^p}{V}\right)$. And if $E &lt; 5 \cdot 2^p$, then we go through a bias correction step; the
Google paper has an addendum of empirically derived values for this. Otherwise, we can just output $E$.
The Google paper has some other enhancements as well, such as a sparse representation, but that’s beyond the scope
of this blog post. Anyway, this turns out to work pretty well. Here’s a graph of relative errors up to a
cardinality of 200 million, run 128 times for each cardinality tested, for $p = 18$ with $3\sigma$ ranges overlaid:
<img src="/images/post_images/hll-perf.png" alt="hll-perf" />
One of the really nice things about these HyperLogLog structures is that you can get unions for free; with two
such structures, $M_1$ and $M_2$, you just need $M[i] = \max\left\{M_1[i], M_2[i]\right\}$. This means that we can
tally up uniques in a distributed computing environment. It’s also possible to reduce the precision of a HyperLogLog
structure if you need to union two of them that have different array sizes, but you can check out the full paper for
those details.
Intersections are a different story. Some other blogs suggest using the <a href="http://en.wikipedia.org/wiki/Inclusion-exclusion_principle">inclusion-exclusion principle</a>,
but this can get somewhat unwieldy, and potentially give some <a href="http://blog.aggregateknowledge.com/2012/12/17/hll-intersections-2/">wild results</a>.
At AdRoll, we decided that we were willing to sacrifice some space efficiency by effectively tacking on a completely
separate structure.
<h3 id="minhash">MinHash</h3>
MinHash is an algorithm that computes an estimate of something called the Jaccard Index, which is defined as:
<script type="math/tex; mode=display">
%<![CDATA[
J(A,B) = \frac{|A \cap B|}{|A \cup B|}
%]]>
</script>
It functions as a measure of how similar two sets are. But let’s abandon this notion of similarity, and extend the
metric, so that we can talk about interesecting lots of sets:
<script type="math/tex; mode=display">
%<![CDATA[
J(A_1, A_2,\ldots,A_n) = \frac{\left|\bigcap_{i=1}^{n}A_i\right|}{\left|\bigcup_{i=1}^{n}A_i\right|}
%]]>
</script>
The variant of MinHash that we’ll be using employs only one hash function, $h(\cdot)$, and for each input set, we
keep the $k$ lowest values after going through this hash function (I’ll be using the notation $\min_k$ to represent
this, which overloads its usual mathematical meaning). If our hash function is uniformly distributed and is not
prone to collisions, we can say that:
<script type="math/tex; mode=display">
%<![CDATA[
\min_k\left\{\bigcup\min_k\left\{h\left(A_i\right)\right\}\right\} = \min_k\left\{h\left(\bigcup A_i\right)\right\}
%]]>
</script>
is a random sample of our total space. That means that we can check each element in this sample, and see if it is in
every $\min_k\left\{h\left(A_i\right)\right\}$. By tallying up all the elements that satisfy this criterion
(call it $y$), we can say:
<script type="math/tex; mode=display">
%<![CDATA[
\frac{y}{k} \approx J(A_1,\ldots,A_n)
%]]>
</script>
Now that we have this estimate, we know that:
<script type="math/tex; mode=display">
%<![CDATA[
\left|\bigcap A_i \right| = J(A_1,\ldots,A_n)\cdot\left|\bigcup A_i\right| \approx
\rm{MinHash\ Result} \cdot \rm{HLL\ Result}
%]]>
</script>
Hey, we’re hashing all this stuff with HyperLogLog++ anyway, right?
We tested this assertion with a HyperLogLog++ structure of $p = 18$ and a MinHash $k = 98304$, which in our implementation
adds up to a 1MB structure. We took two randomly generated sets, each with 100 million elements, and then intersected them
for a variety of Jaccard Indices, repeating the process 128 times. This is the result:
<img src="/images/post_images/mh-perf.png" alt="mh-perf.png" />
The blue region represents our theoretical 99% confidence bounds (you can read about the theory in the paper, along with
how to pick a good $k$ for your own purposes), and we see that our experimental results corroborate our predictions well.
You can see that the more the sets overlap, the better we are at predicting the size of that overlap. However, even for
small intersections, we do pretty well: we can be 99% sure that if the overlap is 0.01%, we’ll report that between 0.00%
and 0.02%, which is quite reasonable. A 1% overlap is 99% likely to be reported between 0.9195% 1.0824%.
<h3 id="conclusion">Conclusion</h3>
If you’re already using HyperLogLog structures in your work, but have been longing for a
means of intersecting them, it’s worth checking out out extending your code with a MinHash
implementation. While it does require extra processing power to deal with collecting all
the minima, it’s possible to get satisfactory performance out of the structure for a
relatively low storage or memory footprint.
Last, as a final plug, I get to tackle interesting problems like this all the time throughout
the course of my work. If this type of stuff fires you up, AdRoll is hiring, and we would love
to <a href="https://www.adroll.com/about/careers">hear from you</a>.
</description>
</item>
<item>
<title>Announcing Our Acquisition of Data Analytics Startup, Bitdeli</title>
<link>https://tech.nextroll.com/blog/adroll/2013/06/12/bitdeli-acquisition.html</link>
<pubDate>Wed, 12 Jun 2013 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/adroll/2013/06/12/bitdeli-acquisition</guid>
<description>Over the last year, AdRoll’s mission of making retargeting easy and effective for everyone has really caught fire. Last month alone, we welcomed over 1,250 new advertisers to our platform. The secret to our success has always been layering the industry’s most intuitive tools over a powerful data backend to help companies collect, analyze and (most importantly) act on their customer data.
Today, I’m excited to announce we’ve taken another big step forward in our mission with the acquisition of data analytics platform Bitdeli, and the addition of the company’s founders, the Tuulos brothers, to our engineering team.
<blockquote>
<img src="/images/post_images/bitdeli_group_photo.jpg" alt="BitDeli Group Photo" />
From left to right: Valentino Volonghi, Aaron Bell, Ville Tuulos, Jyri Tuulos
</blockquote>
Ville Tuulos (brother #1) first popped onto our radar five years ago when our Chief Architect, Valentino, learned of Disco, the Erlang-based MapReduce framework that Ville created:
<blockquote>
<img src="/images/post_images/disco.png" alt="Disco - an Erlang-based MapReduce framework" />
</blockquote>
Clearly, this epic “awesomeness night” set Valentino aflutter. Soon after, he struck up correspondence with Ville and became one of the first contributors to the Disco open source project. Years later, Ville and Jyri (brother #2) took their concept further and launched Bitdeli with a vision to democratize data and make it easy for anyone to build their own custom analytics.
When we met Jyri, we quickly realized he was the perfect complement to Ville, and had the unique front-end skills to make complex data mining technology simple through innovative visualizations and interfaces.
Once we all began talking, we realized we had matching world views and aspirations. It was a natural decision to join forces.
Over the coming months, we will be rolling the Bitdeli technology into the AdRoll product. With this, our users will see:
<blockquote>
<ul>
<li>New, richer ways to slice data for analysis and targeting</li>
<li>New, beautiful visualizations of data</li>
<li>Greater transparency across all sorts of data sets</li>
<li>Proactive insights and suggestions</li>
<li>Smarter customer scoring and optimizations</li>
</ul>
</blockquote>
As always, we’ll do our best to do the heavy lifting and make it quick-and-easy for you to run successful campaigns. While Bitdeli’s core product will be rolled into AdRoll, Bitdeli’s GitHub Badge, a free product that provides analytics specifically for GitHub users, will remain unchanged and continue to be maintained independently of AdRoll’s platform.
Welcome aboard, Ville and Jyri!
</description>
</item>
<item>
<title>Hankering for Some Hadoop?</title>
<link>https://tech.nextroll.com/blog/events/2013/06/11/june-sf-hadoop-meetup.html</link>
<pubDate>Tue, 11 Jun 2013 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/events/2013/06/11/june-sf-hadoop-meetup</guid>
<description>Tomorrow (Wednesday, June 12th) from 6:00-8:00pm, we’ll be hosting a group of Hadoop users and developers here at our SOMA headquarters, ShangRolla, for <a href="http://www.meetup.com/hadoopsf/events/120561212/">their monthly meetup</a>. Hadoop is an “open-source software for reliable, scalable, distributed computing” used by top tech companies like Yahoo!, Facebook, Amazon, Apple, Google and Microsoft. According to a recent <a href="http://tdwi.org/articles/2013/04/23/hadoop-usage-exploding.aspx">TDWI survey</a>, three-quarters of respondents have either deployed or expect to deploy HDFS (Hadoop Distributed File System) in production.
Our very own Derek Nelson will present on the “Real-World Use Cases of Hadoop/HBase Synergy” and will be followed by Christophe Taton of <a href="http://www.wibidata.com/">WibiData</a>, for his presentation “Under the Hood: How KijiSchmea extends HBase.”
This event follows a very successful, content-rich <a href="http://www.meetup.com/hbaseusergroup/events/103587852/">April meetup</a> in which Derek Nelson joined <a href="https://twitter.com/tsunanet">Benoit Sigoure</a> of Arista Networks, Francis Liu of Yahoo!, and <a href="https://twitter.com/sershe84">Sergey Shelukhin</a> of HortonWorks, to share the Big Data challenges of the online advertising space and and discuss how HBase fits into our engineering toolkit.
For Derek, advertising “combines big data, real-time processing, machine learning and money. It helps the internet be free and fluid, and it happens to be one of the most interesting places to sink your teeth into and build cool things.” Here at AdRoll, “we have all of our raw data being processed, aggregated, summarized and sliced-and-diced, in many, many different ways with Hadoop and all of that data is dumped into HBase.”
Want more information on April’s meetup? Check out Derek’s slideshow <a href="https://speakerdeck.com/dereknelson/rolling-with-hbase">here</a> or watch the recordings from all four of the April HBase Meetup speakers below:
<div class="youtube-wrapper"><iframe class="youtube-embed" width="574" height="323" src="http://www.youtube.com/embed/AYERq4pBRqQ" frameborder="0" allowfullscreen=""></iframe></div>
</description>
</item>
<item>
<title>The Secrets to Startup Staffing Success</title>
<link>https://tech.nextroll.com/blog/careers/2013/03/11/secrets-to-startup-staffing-success.html</link>
<pubDate>Mon, 11 Mar 2013 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/careers/2013/03/11/secrets-to-startup-staffing-success</guid>
<description>Throughout my career, I have had the honor to work with innovative, fast-paced and industry-disrupting companies that have attracted some of the best talent around the world. When I was given the opportunity to build AdRoll’s internal recruitment machine, I jumped at this chance last December. <a href="http://blog.adroll.com/inc-500-fastest-growing-advertising-company">Named the fastest growing advertising company in 2012 by Inc. Magazine</a> and already at over 100 employees when I started, we were on the right track to creating a best-in-class staffing initiative.
It’s now halfway into the first quarter of 2013 and things are moving even faster. We are expected to double in size and, having just opened a New York office, expand in other locations. Though we’re growing at a breakneck pace, we’ve been uncompromising when it comes to bringing on top talent. Over the past few months, I’ve had many colleagues in the industry ask me: how did AdRoll grow so fast? How are you going to scale this new growth?
In my experience, building a strong internal team, company culture and brand, and an employee referral program are key areas that companies have to focus on in order to build an effective staffing initiative. Also, having leaders that are on board with these areas is crucial to success.
<blockquote>
<ul>
<li>
Internal Team: a strong internal team is going to be the pillar to your company staffing initiative success. They are going to partner with you to drive a variety of initiatives, from developing a great candidate experience to sourcing to events, etc.
</li>
<li>
Culture: Culture fit is a very important piece and can be a challenge in hiring – it’s often the differentiating factor in a candidate’s decision-making process. This also contributes to companies making very costly hiring mistakes. In my experience recruiting in the U.S. and around the world, culture is something that you just can’t compromise. If you have a great culture, the other pieces will most likely fall into place. 
</li>
<li>
Brand: This goes hand-in-hand with a great company culture. Your employee brand relates directly to how you develop your internal culture. How you present your company on the web and in person will impact the employment brand image. Word spreads fast and a few negative remarks can really do damage.
</li>
<li>
Employee Referrals: Great people know great people and your company should have a “everyone recruits” culture. Great employee referrals are the backbone and often critical to a company’s overall success.
</li>
</ul>
</blockquote>
Interested in Rollin’ with us? <a href="http://www.adroll.com/about/careers">Check out our careers page today</a>!
<center>
<img src="/images/post_images/secrets_to_startup_staffing.jpg" />
</center>
</description>
</item>
<item>
<title>Out With the Old, In With the New Dashboard</title>
<link>https://tech.nextroll.com/blog/dev/2013/01/29/new-dashboard.html</link>
<pubDate>Tue, 29 Jan 2013 00:00:00 -0800</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/dev/2013/01/29/new-dashboard</guid>
<description>We’ve been really busy over here at <a href="http://blog.adroll.com/adrolls-hq-shangrolla">ShangRolla</a>. One of the largest projects that we’ve tackled is a complete redesign of the interface used to set up, measure, and manage your AdRoll campaigns. We’re excited to offer you access, and we hope that you’ll take it for a spin.
<center>
<img alt="New Dashboard Screenshot" src="/images/post_images/new_dashboard_1.png" />
</center>
This version is just the beginning. It’s in beta, so it’s a work in progress, and there could be an occasional bug. But even in its early state, we believe that it will improve your AdRoll experience and makes it even easier to run incredibly effective campaigns. Over the next few weeks, we’ll be adding a ton of new features, many of which you’ve requested. We’ve also made it very easy to switch back and forth between the old and new dashboard.
<center>
<img alt="New Dashboard Screenshot" src="/images/post_images/new_dashboard_2.png" />
</center>
At AdRoll, we truly value feedback. We could not have done this without the customers who offered up great ideas and constructive ways to improve our product. Ultimately we want to hear what you think. The more you speak up, the better we can make this. So use the feedback forum in the footer of the new dashboard. Let us know if you find bugs. Make feature requests. And if you’ve got some great ideas, we’d love to speak to you directly. Just email beta@adroll.com and we’ll schedule a time to chat.
<h3 id="a-few-highlights">A Few Highlights:</h3>
<blockquote>
1. Self service Facebook Retargeting: Easily retarget your customers on the new Facebook Exchange.
<img src="/images/post_images/self_service_facebook_retargeting.png" alt="Self Service Facebook Retargeting" />
2. Multi-ad upload: drag and drop multiple files or browse to select multiple ads at a time.
<img src="/images/post_images/multi-ad-upload.png" alt="Multi-ad Upload" />
3. Simplified campaign setup and editing
<img src="/images/post_images/simplified_campaign_setup_and_editing.png" alt="Simplified Campaign Setup and Editing" />
4. Improved creative management tools
<img src="/images/post_images/improved_creative_management.png" alt="Improved Creative Management Tools" />
</blockquote>
</description>
</item>
<item>
<title>New Roller in the House</title>
<link>https://tech.nextroll.com/blog/careers/2012/06/22/new-roller-in-house.html</link>
<pubDate>Fri, 22 Jun 2012 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/careers/2012/06/22/new-roller-in-house</guid>
<description>Meet Andrew Pascoe, data scientist!
<blockquote>
<img src="/images/post_images/pascoe_joins.jpg" alt="Andrew Pascoe" />
</blockquote>
At AdRoll, product is everything. In order to continue offering our customers the most innovative and effective platform in the retargeting space, we’ve made some <a href="http://blog.adroll.com/hires-new-branding-more">rockstar additions</a> to our engineering team as of late, growing our eng. team 50% this year.
Andrew will contribute to the algorithms powering our efficient real-time bidder (RTBoodah) and the intelligent crunching of the vast amounts of data we collect. Before becoming a Roller, Andrew worked as mathematician for Dean Kamen’s DEKA Research &amp; Development. Unfamiliar with DEKA? It’s the leading pioneer in innovative health and medical devices as well as the creator of the fan favorite Segway. Andrew applied fluid mechanics, Hamiltonian dynamics, and many more highly evolved niche sciences to new inventions.
Before his time at DEKA, Andrew was also selected to work at the National Security Agency where he parsed huge data sets and applied statistical techniques and machine learning algorithms. Later he published his research internally and presented his findings to the Director of the NSA himself. Our team is still intently working on prying our nation’s security secrets out of him.
</description>
</item>
<item>
<title>BuiltWith Lists AdRoll as Fastest Growing Ad Tech Company</title>
<link>https://tech.nextroll.com/blog/adroll/2011/10/13/builtwith-lists-adroll-as-fastest-growing-ad-tech-company.html</link>
<pubDate>Thu, 13 Oct 2011 00:00:00 -0700</pubDate>
<author></author>
<guid isPermaLink="false">https://tech.nextroll.com/blog/adroll/2011/10/13/builtwith-lists-adroll-as-fastest-growing-ad-tech-company</guid>
<description>There are many ways you can tell your startup is growing. Of course, we have our own internalmetrics around the hundreds of new retargeting advertisers who start using AdRoll every month. We’ve also found that two additional growth indicators are an increase in number of ramen packages consumed and an increase in boxes of gear delivered.
As much as we love ramen and gear, there’s one thing we love more: neutral, 3rd-party data validation.
This week we’re especially excited about topping <a href="http://trends.builtwith.com/nostats/growth#!oneYear">BuiltWith.com</a>’s list of fastest growing online advertising technologies over thepast year. BuiltWith is a leading provider of website technology trends analysis and competitor intelligence products. They crawl the web, inspecting each site, and figure out which tech said sites are “built with”, so to speak. It’s a great feeling to be acknowledged by esteemed peers in the tech world. Without further ado…
<center>
<img alt="Fastest Growing Advertising Technology" src="/images/post_images/builtwith.jpg" width="400" />
</center>
</description>
</item>
</channel>
</rss>

Home · About · News · Docs · Terms