FEED Validator

for Atom and RSS and KML

Sorry

This feed does not validate.

line 18, column 14: Invalid email address: (8 occurrences) [help]

      <author></author>
              ^

line 1500, column 26: XML parsing error: <unknown>:1500:26: not well-formed (invalid token) [help]

      <title>UITableView & UICollectionView: Killing stringly typed cells wi ...
                          ^

In addition, interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

line 59, column 0: description should not contain data-lang attribute (31 occurrences) [help]
```
&lt;div class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;langu ...
```
line 596, column 0: description should not contain relative URL references: %5Bhttps://www.postgresql.org/docs/current/functions-json.html%5D(https://www.postgresql.org/docs/current/functions-json.html) (2 occurrences) [help]
```
</description>
```

Source: http://techblog.thescore.com/feed.xml

<?xml version="1.0"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>theScore Tech Blog</title>
<link>https://techblog.thescore.com</link>
<atom:link href="https://techblog.thescore.com/feed.xml" rel="self" type="application/rss+xml" />
<description>Things We Learn While Building the Ultimate, Personalized Mobile Sports Experience</description>
<language>en-us</language>
<pubDate>Tue, 09 May 2023 17:31:30 +0000</pubDate>
<lastBuildDate>Tue, 09 May 2023 17:31:30 +0000</lastBuildDate>
<item>
<title>Repo.transact/2 (The Case Against Ecto.Multi)</title>
<link>https://techblog.thescore.com/2023/05/01/repo-transact/</link>
<pubDate>Mon, 01 May 2023 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2023/05/01/repo-transact</guid>
<description>After reading <a href="https://medium.com/very-big-things/towards-maintainable-elixir-the-core-and-the-interface-c267f0da43">Towards Maintainable
Elixir</a>
by Saša Jurić and hearing about his famous <code>Repo.transact</code> in some of his
talks, I decided it was time to explore this for myself.
This post takes into account that you (the reader) are aware and know why and
when to use <a href="https://hexdocs.pm/ecto/Ecto.Multi.html">Ecto.Multi</a>. But for
those unfamiliar, the TL;DR is you would use an <code>Ecto.Multi</code> when you want to
perform multiple transaction that you want to be committed to the database in
one shot. Meaning, if one of the transactions fails, you would want to revert
all other transactions in the run.
<h2>The Problem with Ecto.Multi</h2>
Lets get something straight, there is nothing wrong with using Ecto.Multi
in your codebase. If it works for you, then it works. However after working
with it in multiple codebases I have started to see a common theme: it is very
noisy and can be sometimes hard to follow and DRY up. You can get around it by
using a lot of private functions to support the Ecto.Multi, but then your
module just has a tons of wrapper functions.
<h2>What is Repo.transact/2?</h2>
<blockquote>
The function Repo.transact is our small wrapper around
<a href="https://hexdocs.pm/ecto/Ecto.Repo.html#c:transaction/2">Repo.transaction/2</a>.
This function commits the transaction if the lambda returns <code>{:ok, result}</code>,
rolling it back if the lambda returns <code>{:error, reason}</code>. In both cases, the
function returns the result of the lambda. We chose this approach over
Ecto.Multi, because we’ve experimentally established that multi adds a lot of
noise with no real benefits for our needs.
-- <cite>Saša Jurić</cite>
</blockquote>
<h2>Function definition</h2>
Saša never gives out the implementation of the function but I came up with
this, and it works great; Exactly as you would expect.
<div class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule MyApp.Repo do
 use Ecto.Repo,
 otp_app: :my_app,
 adapter: Ecto.Adapters.Postgres
 @doc &quot;&quot;&quot;
A small wrapper around `Repo.transaction/2&#39;.
Commits the transaction if the lambda returns `{:ok, result}`, rolling it
back if the lambda returns `{:error, reason}`. In both cases, the function
 returns the result of the lambda.
 &quot;&quot;&quot;
@spec transact((() -&gt; any()), keyword()) :: {:ok, any()} | {:error, any()}
def transact(fun, opts \\ []) do
 transaction(
 fn -&gt;
 case fun.() do
{:ok, value} -&gt; value
 :ok -&gt; :transaction_commited
{:error, reason} -&gt; rollback(reason)
 :error -&gt; rollback(:transaction_rollback_error)
 end
 end,
 opts
 )
 end
end
</code></pre></div>
<h2>Ecto.Multi vs Repo.transact</h2>
Lets say we want to take the typical user registration flow as an example. If
we want to insert a user, log the action to an audit table and also enqueue a job
to send a confirmation email.
We would have something like this using an Ecto.Multi:
<h4>Ecto.Multi implementation</h4>
<div class="highlight"><pre><code class="language-elixir" data-lang="elixir">def register_user(params) do
 Multi.new()
|&gt; Multi.insert(:user, Accounts.new_user_changeset(params))
|&gt; Multi.insert(:log, fn %{user: user} -&gt;
Logs.log_action(:user_registered, %{user: user})
 end)
|&gt; Multi.insert(:email_job, fn %{user: user} -&gt;
 Mailer.enqueue_email_confirmation(user)
 end)
 |&gt; Repo.transaction()
 |&gt; case do
{:ok, %{user: user}} -&gt;
 {:ok, user}
{:error, _failed_operation, failed_value, _changes_so_far} -&gt;
 {:error, failed_value}
 end
end
</code></pre></div>
As we can see, it is not the worst, but once we see the <code>Repo.transact/2</code> way,
it will be clear which is better.
<h4>Repo.transact implementation</h4>
<div class="highlight"><pre><code class="language-elixir" data-lang="elixir">def register_user(params) do
 Repo.transact(fn -&gt;
with {:ok, user} &lt;- Accounts.create_user(params),
{:ok, _log} &lt;- Logs.log_action(:user_registered, user),
{:ok, _job} &lt;- Mailer.enqueue_email_confirmation(user) do
 {:ok, user}
 end
 end)
end
</code></pre></div>
As you can see, it is much shorter and easier to read. Another big benefit is
that we do not need to go down to the changeset level for inserting, we could
use our functions that perform <code>Repo.insert</code>s in them
(<code>Accounts.new_user_changeset/1</code> vs <code>Accounts.create_user/1</code>). This lets us
compose many functions together from outside the context modules without having
the need to expose your changeset functions.
The end result is the same, but it is a lot easier to read what is going on IMHO.
</description>
</item>
<item>
<title>Testing a Java-based Kubernetes Admission Webhook Locally</title>
<link>https://techblog.thescore.com/2022/12/23/testing-kubernetes-webhook-locally/</link>
<pubDate>Fri, 23 Dec 2022 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2022/12/23/testing-kubernetes-webhook-locally</guid>
<description>A Kubernetes Admission webhook can be an effective way to enforce and apply custom system policies. Admission webhooks
are callbacks which allow admission requests (creating or updating a kubernetes resource) to first be forwarded to
a custom external service. In the external service, administrators can enforce constraints (Validating Admission Webhook) or apply changes (Mutating
Admission Webhook) to their Kubernetes resources.
The Platform Core team at theScore exposes some of our internal tooling as Kubernetes custom resources. We use Validating
Admission Webhooks to enforce that any resource and schema changes in a service&#39;s custom resource files are valid. This
allows users to catch configuration issues quickly and avoid their instances transitioning into a bad state.
However, one issue with Kubernetes Webhooks is that is can be tedious to test them in a local dev environment as they
are called directly from the Kubernetes API Server and to properly test, we would like to mimic this behaviour. This
outlines how the Platform Team&#39;s setup to test a Validating Webhook.
<h2>Prerequisites</h2>
Before being able to test a webhook, the service that is invoked from the Kubernetes API server needs to be implemented. For more information on
implementing webhooks, see the kubernetes documentation <a href="https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/">here</a>
For the sake of this blog, it is assumed you have a Admission Webhook Service implementation that can be called via the command
line and accepts keystore and trust store paths as arguments. This is helpful as kubernetes requires webhooks to have
a valid TLS certificate to be successfully called.
On the Platform Core Team at theScore, our webhook can be invoked using these commands:
<div class="highlight"><pre><code class="language-shell" data-lang="shell"># Build the webhook with maven
mvn package -DskipTests
datadex webhook --port=&lt;port num&gt; --keystore-path=&lt;path to keystore&gt; --keystore-pass=&lt;keystore password&gt; --truststore-path=&lt;truststore path&gt; --truststore-pass=&lt;truststore password&gt;
</code></pre></div>
<h2>Testing Process</h2>
<h3>Webhook Resource Definition</h3>
As part of the development of your Admission Webhook, you should have created a CRD file for the custom service to be
called by the API Server. For local testing, there are some particular configurations which are needed that will be pointed
out here. 
Below is an example webhook definition that we use on the Platform Team at theScore. As seen by the <code>webhooks.rules</code>
section, this webhook gets invoked whenever a Datadex Instance (a Platform Team custom resource) get created or updated.
For local testing, you should pay attention to the <code>webhooks.clientConfig</code> section:
* <code>url</code> is set to <code>https://host.docker.internal:443/validate</code>
* This defines the location of the locally running webhook. We will be using docker-desktop to create a local kubernetes cluster and the <code>https://host.docker.internal</code> allows docker to access services running on the local machine outside the docker container. In this example, calls to this url will hit the server running on localhost at port 443.
* <code>caBundle</code> defines the CAs (Certificate Authorities) used to verify the TLS connection of the webhook. See the &quot;Generating Local TLS Certificate Files&quot; section below for more details
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml">apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
 name: &quot;datadex-operator.thescore.com&quot;
webhooks:
 - name: datadex-operator.thescore.com
 rules:
- apiGroups: [&quot;datadex.thescore.com&quot;]
 apiVersions: [&quot;v1&quot;]
operations: [&quot;CREATE&quot;, &quot;UPDATE&quot;]
 resources: [&quot;datadexinstances&quot;]
 scope: &quot;Namespaced&quot;
 clientConfig:
 url: &quot;https://host.docker.internal:443/validate&quot;
 caBundle: &lt;This field will be populated later in this blog&gt;
 admissionReviewVersions: [&quot;v1&quot;]
 sideEffects: None
 timeoutSeconds: 30
</code></pre></div>
<h3>Generating Local TLS Certificate Files</h3>
As mentioned above, webhook services must have a valid TLS certificate. The cert can be obtained in several ways but
the easiest for local testing is to generate a self-signed certificate. They can be requested at any time by any developer,
and never expire making testing easy. However, this is only for local testing. For other live instances and especially
production, be sure to obtain a certificate from an approved Certificate Authority.
To generate a self-signed cert, openssl can be used. The following bash script can be used to generate all the necessary
data. It is originally from Slack&#39;s open source <a href="https://github.com/slackhq/simple-kubernetes-webhook/blob/main/dev/gen-certs.sh">code</a> and modified to this specific use case.
<div class="highlight"><pre><code class="language-shell" data-lang="shell">#!/bin/bash
openssl genrsa -out ca.key 2048
openssl req -new -x509 -sha256 -days 3650 -key ca.key \
 -subj &quot;/C=CA/CN=datadex-webhook&quot;\
 -out ca.crt
openssl req -newkey rsa:2048 -nodes -sha256 -keyout server.key \
 -subj &quot;/C=CA/CN=datadex-webhook&quot; \
 -out server.csr
openssl x509 -req \
 -extfile &lt;(printf &quot;subjectAltName=DNS:host.docker.internal&quot;) \
 -sha256 \
 -days 3650 \
 -in server.csr \
 -CA ca.crt -CAkey ca.key -CAcreateserial \
 -out server.crt
cat server.crt ca.crt &gt; server.bundle.crt
echo
echo &quot;&gt;&gt; Generating kube secrets...&quot;
kubectl create secret tls validating-webhook-tls \
 --cert=server.bundle.crt \
 --key=server.key \
 --dry-run=client -o yaml \
 &gt; webhook-secret.yaml
formatted_crt=$(cat ca.crt | base64 | fold)
# yq is a command line yaml processor. See https://github.com/mikefarah/yq for info and install instructions
yq -y -i -e &quot;.webhooks[0].clientConfig.caBundle = \&quot;$formatted_crt\&quot;&quot; webhook-configuration.yaml
echo
echo &quot;&gt;&gt; Generating Keystore and truststore files&quot;
openssl pkcs12 -export -inkey server.key -in server.crt -out key.pkcs12 -password pass:123456
keytool -importkeystore -noprompt -srckeystore key.pkcs12 -srcstoretype pkcs12 -destkeystore key.jks -storepass 123456 -srcstorepass 123456
csplit -f crt- -n 2 server.bundle.crt &#39;/-----BEGIN CERTIFICATE-----/&#39;
for file in crt-*; do
keytool -delete -noprompt -keystore trust.jks -trustcacerts -storepass 123456 -alias &quot;service-$file&quot;
keytool -import -noprompt -keystore trust.jks -file &quot;$file&quot; -storepass 123456 -alias &quot;service-$file&quot;
 rm &quot;$file&quot;
done
rm ca.crt ca.key ca.srl server.crt server.csr server.key server.bundle.crt key.pkcs12
</code></pre></div>
This script does a couple of things
1. Outputs two files; <code>key.jks</code> and <code>trust.jks</code>. These are stores for the self-signed server key and TLS certificate which will later be provided to the webhook service as arguments
2. Creates a YAML file (<code>webhook-secret.yaml</code>) which defines a Kubernetes TLS secret resource
3. Populates the <code>caBundle</code> field of the local Validating Webhook Kubernetes CRD (<code>webhook-configuration.yaml</code>). This defines the local CA which allows the self-signed cert to be verified.
<h3>Local Kubernetes Environment Setup</h3>
Now we must set up and configure our local kubernetes cluster to enable it to call our Custom Webhook.
<h4>Docker-Desktop Setup</h4>
For this example we will use docker-desktop. It provides out of the box kubernetes so to get a local kubernetes cluster
running, all we need to do is download docker-desktop and enable Kubernetes. See docker&#39;s <a href="https://docs.docker.com/desktop/kubernetes/">documentation</a> for more
information on this.
<h4>Kubernetes Setup</h4>
Once docker is all setup, next we need to install <code>kubectl</code> if not done already:
* MacOS: <code>brew install kubectl</code>
* See <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/">documentation</a> for other installation method
<h4>Webhook Configuration</h4>
The final step is to configure the kubernetes cluster to actually call our webhook whenever an update/create resource
request is made.
1. Switch to the docker-desktop k8s context
* <code>kubectl config use-context docker-desktop</code>
2. [Optional] You can create a custom namespace for testing if desired or just use the default namespace.
3. Configure the cluster to call the local webhook:
* Setup TLS secret: <code>kubectl apply -f webhook-secret.yaml</code>
* Apply the webhook configuration: <code>kubectl apply -f webhook-configuration.yaml</code>
<h3>Invoking Webhook Endpoints</h3>
At this point all the setup and configuration is complete and the webhook is ready to be tested:
1. Start a local instance of the webhook (substitute the command with the command to start up your webhook)
* <code>datadex webhook --port=443 --keystore-path=key.jks --keystore-pass=123456 --truststore-path=trust.jks --truststore-pass=123456</code>
2. In the docker-desktop cluster environment you previously setup, create or update the resource that the webhook is listening to:
* <code>kubectl apply -f &lt;yaml file&gt;</code>
That&#39;s it! At this point, the Kubernetes API Server in docker-desktop will have called your local webhook instance and
the behaviour of your webhook in an actual live kubernetes cluster can be tested. You can actively develop and debug
with instant feedback.
<h3>Alternate Testing Options</h3>
If you do not want to test your webhook being called from the actual Kubernetes API Server, an alternative is to use
cURL as a quick and easy way to manually call the endpoints of your Webhook:
<ol>
<li>Generate the TLS certificates. See the &quot;Generating Local TLS Certificate Files&quot; section above</li>
<li>Run the webhook service from the command line:</li>
</ol>
<div class="highlight"><pre><code class="language-shell" data-lang="shell"> mvn package -DskipTests
datadex webhook --port=443 --keystore-path=key.jks --keystore-pass=123456 --truststore-path=trust.jks --truststore-pass=123456
</code></pre></div>
<ol>
<li>Use cURL to hit webhook endpoint:</li>
</ol>
<div class="highlight"><pre><code class="language-shell" data-lang="shell"> # Pinging a GET endpoint with no Payload (eg: health endpoint)
 #
 curl -k https://localhost:8443/health
 # Hitting an endpoint expecting data:
 # 
 # The --data option is the payload the the k8s API server will provide when it calls the webhook
 # See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#webhook-request-and-response for details
 #
 curl --data &quot;@&lt;Request JSON File Path&gt;&quot; -k https://localhost:443/&lt;endpoint&gt;
</code></pre></div></description>
</item>
<item>
<title>PostgreSQL Queries on JSONB Fields with Ecto</title>
<link>https://techblog.thescore.com/2022/06/23/embedded-schema-queries-with-ecto/</link>
<pubDate>Thu, 23 Jun 2022 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2022/06/23/embedded-schema-queries-with-ecto</guid>
<description>The Promotions team at theScore is responsible for enabling dynamic marketing campaigns within <a href="https://thescore.bet/">theScore Bet</a> app. Each promotional campaign allows us to specify an audience of users who will be available to participate in the campaign, by comparing on past actions that a given user has taken against a set of dynamic rules. For example, when creating a new promotion, we can specify that its audience can only include patrons who have previously deposited a specific amount of money, or placed a specific quantity of bets.
Disclaimer: The specific scenario, and format of the data shown in the examples throughout, is contrived entirely for the purposes of this article.
<h2>An overview of the PostgreSQL schema</h2>
The Promotions service depends on the ability to read past events placed by a patron. These events are received asynchronously from other internal services, via Kafka, and then stored internally in the Promotions database as a generic &quot;user action&quot;. These actions are used for a variety of purposes within our Promotions service, but in this case, we are exploring looking at the past actions of a given user to see if they fulfill the audience criteria of a given promotional campaign. The table which they get stored within has been created with a migration that resembled the code sample below.
<div class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule Promotions.Repo.Migrations.CreateUserActionsTable do
 use Ecto.Migration
 def change do
 create table(:user_actions) do
add :action_id, :string, null: false
add :action_type, :action_type, null: false
add :occurred_at, :utc_datetime_usec, null: false
add :region, :region, null: false
add :user_id, :binary_id, null: false
add :attributes, :jsonb, null: false
 timestamps()
 end
 end
end
</code></pre></div>
The corresponding Ecto schema for the &quot;user_actions&quot; table looks like the following.
<div class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule Promotions.UserAction do
 use Ecto.Schema
 alias Promotions.UserAction.CasinoWagerAttributes
 alias Promotions.UserAction.DepositAttributes
 alias Promotions.UserAction.RegistrationAttributes
 alias Promotions.UserAction.SportsbookBetAttributes
 schema &quot;user_actions&quot; do
 field :action_id, :string
field :action_type, Ecto.Enum, values: [:registration, :deposit, :sportsbook_bet, :casino_wager]
 field :occurred_at, :utc_datetime_usec
field :region, Ecto.Enum, values: [:usa_new_jersey, :usa_colorado, :usa_indiana, :usa_iowa, :can_ontario]
 field :user_id, Ecto.UUID
embeds_one :casino_wager_attributes, CasinoWagerAttributes, source: :attributes
embeds_one :deposit_attributes, DepositAttributes, source: :attributes
embeds_one :registration_attributes, RegistrationAttributes, source: :attributes
embeds_one :sportsbook_bet_attributes, SportsbookBetAttributes, source: :attributes
 timestamps()
 end
end
</code></pre></div>
Each of the various embedded schemas to populate the <code>attributes</code> field is represented in a separate Ecto schema of its own. For the purposes of this article, only a truncated sample of the <code>SportsbookBetAttributes</code> embedded schema will be shown below as well.
<div class="highlight"><pre><code class="language-elixir" data-lang="elixir">defmodule Promotions.UserAction.SportsbookBetAttributes do
 use Ecto.Schema
 @primary_key false
 embedded_schema do
 field :amount_cents_wagered, :integer
 field :amount_cents_payout, :integer
field :bet_type, Ecto.Enum, values: [:straight, :parlay]
 field :odds, :integer
 embeds_many :legs, SportsbookBetLegAttributes
 end
end
defmodule Promotions.UserAction.SportsbookBetLegAttributes do
 use Ecto.Schema
 @primary_key false
 embedded_schema do
 field :event_id, :string
 field :event_type, :string
 field :market_id, :string
 field :market_selection_id, :string
 field :market_type, :string
 field :in_play, :boolean
 field :odds, :integer
 end
end
</code></pre></div>
<h2>Example Criteria to Turn into Database Queries</h2>
The audience criteria of a promotional campaign can be based on any type of action that we record (ie. deposit, sportsbook bet, etc) and the various attributes of the action. The following are examples of such an audience rule.
<ul>
<li> The patron registered their account in January 2022</li>
<li> The patron has placed at least 3 deposits from within New Jersey</li>
<li> The patron placed at least one sportsbook bet on the last years Super Bowl</li>
</ul>
For the first two examples, the data is stored within more fields that are of a more primitive data type within the <code>user_actions</code> table of the database, and are standard to query. For the last example, it requires querying the JSONB <code>attributes</code> field to determine if any of the &quot;legs&quot; on any of the users previous sportsbook bets contain the <code>event_id</code> associated with last years Super Bowl.
Similarly, the audience criteria also needs to support scenarios where &quot;The patron has placed a sportsbook bet that was not on last years Super Bowl&quot; as well. This would require looking for a sportsbook bet that does not have an <code>event_id</code> corresponding to the Super Bowl for the previous year in any of its legs.
The queries we need to write need to work for both &quot;at least one leg of a sportsbook contains&quot; and &quot;none of the legs of a sportsbook bet can contain&quot; scenarios. To do so, we need to investigate how to make precise queries on a set of data within a JSONB field in PostgreSQL, and subsequently how we can turn those queries into composable Ecto queries within our Elixir application as well.
Notably, we want to be able to support operators of &quot;in&quot; and &quot;not in&quot; when querying for data, as opposed to operators such as &quot;equals&quot;, &quot;not equals&quot;, &quot;greater than or equals&quot;, etc.
<h2>Query of a JSONB Field</h2>
The first results our team came across while looking into this was using PostgreSQL native <a href="%5Bhttps://www.postgresql.org/docs/current/functions-json.html%5D(https://www.postgresql.org/docs/current/functions-json.html)">functions and operators</a>. We quickly realized that these would not fit all of our use-cases. It would be challenging to turn it into a reusable Ecto query that would work for a variety of fields and would be an inelegant solution for deeply nested data.
The next option we came across was to use the function <code>jsonb_to_recordset</code> to create a queryable schema of records from an array within a JSONB field (ie. the <code>embeds_many :legs</code> section of our bet attributes).
In the example below, we’ll try to find if the user has at least one bet with the following attributes.
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml">user_id: &quot;fcca980b-8edc-4848-91cc-4e00ff47019c&quot;
region: :can_ontario
amount_cents_wagered: at least 1500 (ie. $15)
event_id: (not in any of the following)
 - &quot;e8a95b17-972c-4421-9090-8bf66c0804a9&quot;
 - &quot;5fbef652-e772-4bae-84f5-31255fe2fadd&quot;
 - &quot;dd92d86c-fe6d-43a7-8765-a3e1026145db&quot;
</code></pre></div>
<h2>Using <code>jsonb_to_recordset</code> with SQL</h2>
The first step to transforming this into a usable query within our Elixir app was to try to get it working with a query made directly against the database to validate that it is the right tool for the job.
After some initial research, we composed the following SQL query:
<div class="highlight"><pre><code class="language-sql" data-lang="sql">SELECT
 public.user_actions AS action,
jsonb_to_recordset(action.attributes-&gt;&#39;legs&#39;) AS leg(&quot;event_id&quot; varchar)
WHERE
action.user_id = &#39;f0bb75d3-2cce-4328-9177-02349fa99de6&#39; AND
action.region = &#39;ontario&#39; AND
(action.attributes-&gt;&#39;amount_cents_wagered&#39;)::integer &gt;= 1500 AND
 leg.event_id NOT IN (
 &#39;e8a95b17-972c-4421-9090-8bf66c0804a9&#39;,
 &#39;5fbef652-e772-4bae-84f5-31255fe2fadd&#39;,
 &#39;dd92d86c-fe6d-43a7-8765-a3e1026145db&#39;
 );
</code></pre></div>
While this allows us to use the operators we want (ie. <code>&gt;=</code> and <code>NOT IN</code>) on data within the &quot;attributes&quot; JSONB field, and gets us the exact records we want, it would be a challenge to try to make it fit into an Ecto query since Ecto queries only allow for one entity to be in the <code>from</code> clause of a given query. For example, we are permitted to begin a query with <code>from UserAction</code>, but are unable to create a query that selects from two data sources such as <code>from UserAction, fragment(&quot;jsonb_to_recordset(...)&quot;)</code>. If there is a way around that, we were unable to find a solution.
The next option we looked to was modifying the first query by utilizing a lateral-join to try to make a &quot;user action&quot; such that it has a one-to-many relationship with its own &quot;legs&quot; nested within the JSONB &quot;attributes&quot; field. By doing so, we eliminate the need to use multiple clauses in the <code>FROM</code> section of the query, which helps gets us closer to a valid Ecto query translation. The query then looks like this:
<div class="highlight"><pre><code class="language-sql" data-lang="sql">SELECT
 action.*
FROM
 user_actions AS action
INNER JOIN LATERAL (
 SELECT *
FROM jsonb_to_recordset(action.attributes-&gt;&#39;legs&#39;)
 AS leg(&quot;event_id&quot; varchar)
) AS leg ON true
WHERE
action.user_id = &#39;f0bb75d3-2cce-4328-9177-02349fa99de6&#39; AND
action.region = &#39;can_ontario&#39; AND
(action.attributes-&gt;&#39;amount_cents_wagered&#39;)::integer &gt;= 1500 AND
 leg.event_id NOT IN (
 &#39;e8a95b17-972c-4421-9090-8bf66c0804a9&#39;,
 &#39;5fbef652-e772-4bae-84f5-31255fe2fadd&#39;,
 &#39;dd92d86c-fe6d-43a7-8765-a3e1026145db&#39;
 )
</code></pre></div>
The problem we quickly saw with this query is that will look to see if any of the &quot;legs&quot; of a specific bet adhere to the condition. Instead, we want to make sure that the all of the legs within the bet are not on those listed events (ie. cannot have any of the specified &quot;event id&quot; values). As such, we found that putting the inner join in a subquery helped to achieve the goal, wrapping it in a &quot;NOT EXISTS&quot; condition.
<div class="highlight"><pre><code class="language-sql" data-lang="sql">SELECT
 action.*
FROM
 user_actions AS action
WHERE
action.user_id = &#39;f0bb75d3-2cce-4328-9177-02349fa99de6&#39; AND
action.region = &#39;ontario&#39; AND
(action.attributes-&gt;&#39;amount_cents_wagered&#39;)::integer &gt;= 1500 AND
 NOT EXISTS (
 SELECT
 *
 FROM
 user_actions AS lateral_action
 INNER JOIN LATERAL (
 SELECT *
FROM jsonb_to_recordset(lateral_action.attributes-&gt;&#39;legs&#39;)
 AS leg(&quot;event_id&quot; varchar)
 ) AS leg ON true
 WHERE
action.id = lateral_action.id AND
 leg.event_id IN (
 &#39;e8a95b17-972c-4421-9090-8bf66c0804a9&#39;,
 &#39;5fbef652-e772-4bae-84f5-31255fe2fadd&#39;,
 &#39;dd92d86c-fe6d-43a7-8765-a3e1026145db&#39;
 )
 );
</code></pre></div>
This query seems to meet all of the goals that we were looking to achieve, so the next step was to take it and turn it into a composable query within Ecto.
<h2>Translating SQL query to an Ecto Query</h2>
Many parts of our query are pretty quick and easy to make into an <code>Ecto.Query</code> structure. We can begin with everything before the subquery. For this example, we can assume all of the values were passed in and the operators for each part of the expression are known:
<div class="highlight"><pre><code class="language-elixir" data-lang="elixir">def query(user_id, region, amount_cents, event_ids)
 from action in Promotions.UserAction,
 as: :user_action,
where: action.user_id == ^user_id,
where: action.region == ^region,
where: fragment(&quot;(?-&gt;?)::integer&quot;, action.sportsbook_bet_attributes, &quot;amount_cents_wagered&quot;) &gt;= ^amount_cents
end
</code></pre></div>
We also need to build a query around the <code>jsonb_to_recordset</code> data that we created in the lateral join seen earlier.
This was a bit challenging, as <code>jsonb_to_recordset</code> is a native function of PostgreSQL and there are no functions built into Ecto to work with it directly. It also requires us to define an <code>AS ... ON ...</code> set of clauses to use with it. Since Ecto will create its own <code>AS</code> alias with each &quot;from&quot; or &quot;join&quot;, that presents challenges as well.
After some research on this issue on the elixir forums, we were pointed in the right direction to wrap the subquery in parentheses, so that we could use our own <code>AS</code> within it, and it would not conflict with the one Ecto adds on.
<div class="highlight"><pre><code class="language-elixir" data-lang="elixir">def sportsbook_bet_legs_query do
 from action in Promotions.UserAction,
 as: :leg_user_action,
 inner_lateral_join:
 legs in fragment(
 &quot;(SELECT * FROM jsonb_to_recordset(?-&gt;&#39;legs&#39;) AS record(event_id varchar)&quot;,
 field(action, :sportsbook_bet_attributes)
 )
 on: true
 as: :sportsbook_bet_legs
end
</code></pre></div>
To utilize both queries together, we need to include the <code>sportsbook_bet_legs_query</code> function as a subquery within the first, and wrap it in a <code>where not exists</code> clause of the outer query.
Since the subquery uses a lateral join, we need to remember to join it back to the outer query using Ecto’s <code>parent_as/1</code> function. Another example of this can be found in Ecto’s documentation for <code>exists/1</code> as well (<a href="https://hexdocs.pm/ecto/Ecto.Query.API.html#exists/1">https://hexdocs.pm/ecto/Ecto.Query.API.html#exists/1</a>)
<div class="highlight"><pre><code class="language-elixir" data-lang="elixir">def sportsbook_bet_query(user_id, region, amount_cents_wagered, event_ids) do
 from action in Promotions.UserAction,
 as: :action,
where: action.user_id == ^user_id,
where: action.region == ^region,
where: fragment(&quot;(?-&gt;?)::integer&quot;, action.sportsbook_bet_attributes, &quot;amount_cents_wagered&quot;) &gt;= ^amount_cents_wagered,
 where:
 not exists(
 from [
 leg_user_action: action_subquery,
 sportsbook_bet_legs: sportsbook_bet_legs
 ] in event_ids_query(),
where: parent_as(:action).id == action_subquery.id,
where: sportsbook_bet_legs.event_id in ^event_ids
 )
end
</code></pre></div>
With the function now complete, it can be called by simply taking the earlier example and passing in the arguments to the reusable function:
<div class="highlight"><pre><code class="language-elixir" data-lang="elixir">user_id = &quot;fcca980b-8edc-4848-91cc-4e00ff47019c&quot;
region = :can_ontario
amount_cents_wagered = 1_500
event_ids = [
 &quot;e8a95b17-972c-4421-9090-8bf66c0804a9&quot;,
 &quot;5fbef652-e772-4bae-84f5-31255fe2fadd&quot;,
 &quot;dd92d86c-fe6d-43a7-8765-a3e1026145db&quot;
]
ecto_query = sportsbook_bet_query(user_id, region, amount_cents_wagered, event_id)
matching_bets = Repo.all(ecto_query)
if not Enum.empty?(matching_bets) do
 # user qualified for promotion
end
</code></pre></div>
As a reminder, one of the goals of our function was to find a &quot;user action&quot; record that did not contain any of the values of the list of <code>event_id</code>’s from the attributes of any of its array of &quot;leg&quot; data points. One benefit of structuring the query in the way that has been shown here is that if we want to flip it to find if one (or more) legs does include a particular <code>event_id</code>, we can simply remove the <code>not</code> from the <code>not exists</code> condition without any other modifications.
The queries that we need in our application are more robust, including changing up the operators that we use instead of statically choosing &quot;equals&quot;, &quot;greater than or equals&quot; and &quot;not in&quot; as outlined in our example, but from this template we were able to build out a library of robust and composable queries to scope results based on attributes both in more primitive data types within our database schema, as well as JSONB fields too.
I&#39;ve set up a small-scale Elixir application which can be used to quickly scaffold a database with records, as seen in the format throughout this blog post, so you can test the queries seen within this article and tweak them as needed to experiment. <a href="https://github.com/michaelfich/ecto_jsonb_example">Github Repository: Ecto JSONB Example</a>.
</description>
</item>
<item>
<title>Migrating PostgreSQL to Aurora with near Zero-Downtime</title>
<link>https://techblog.thescore.com/2018/06/29/migrating-postgresql-to-aurora-with-near-zero-downtime/</link>
<pubDate>Fri, 29 Jun 2018 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2018/06/29/migrating-postgresql-to-aurora-with-near-zero-downtime</guid>
<description><img src="/assets/elephant-migration.jpg" alt="">
<a href="https://flickr.com/photos/jeaneeem/22658750486" title="15_CBP_JMP-39">15CBPJMP-39</a> by <a href="https://flickr.com/people/jeaneeem">jeaneeem</a> is licensed under <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/">CC BY-NC-ND</a>
<h1>Migrating PostgreSQL to Aurora with near Zero-Downtime</h1>
The goal here is to migrate an Amazon Web Services (AWS) Relational Database Service (RDS) PostgreSQL Database to Aurora with as little downtime as possible. The following documents the initial strategy that was considered using read replica promotion, followed by the end result using AWS&#39;s Database Migration Service (DMS).
<h1>Why Aurora</h1>
From the <a href="https://aws.amazon.com/rds/aurora/details/">Aurora Details Page</a>:
<blockquote>
Amazon Aurora is designed to offer greater than 99.99% availability, increasing MySQL and PostgreSQL performance and availability by tightly integrating the database engine with an SSD-backed virtualized storage layer purpose-built for database workloads. Amazon Aurora&#39;s storage is fault-tolerant and self-healing and disk failures are repaired in the background without loss of database availability. Amazon Aurora is designed to automatically detect database crashes and restart without the need for crash recovery or to rebuild the database cache. If the entire instance fails, Amazon Aurora will automatically fail over to one of up to 15 read replicas.
</blockquote>
Essentially, better performance, reliability, and the ability to scale.
Our configuration changes become easier. Aurora acts as a cluster, so we have two endpoints to deal with, and the replicas underneath will adjust dynamically. We use <a href="https://github.com/taskrabbit/makara">Makara</a> to spread our database queries between our master and slaves. With RDS PostgreSQL we would have to enumerate all our slave databases so we could take advantage of the replicas. If we added a new replica, we would have to adjust our configuration to take advantage of it. With Aurora, we have two endpoints to deal with now (primary/writer and a read-only URL). These endpoints will dynamically rotate between the available instances that fall under that cluster URL. This is great as it handles a failover to a different primary database, or changing the number of replicas. Effectively, we end up with less configuration, but we still need Makara to direct our write and read queries to the two cluster URLs.
<h1>Migration Approaches</h1>
As mentioned earlier, there are two approaches to doing this migration. First, we&#39;ll cover the recommended approach that AWS suggested as we&#39;re on the RDS platform. As we&#39;ll cover, it is easy and quick but it does incur downtime during the migration. The second approach uses DMS and was brought up to us after opening a support ticket with AWS on how to do our migration with zero downtime.
<h2>Read Replica Promotion</h2>
This is the recommended approach for migrating from PostgreSQL to Aurora. It is easy and has a little risk (during the migration), although it has unavoidable downtime.
<ul>
<li>Create an Aurora read replica off of your master database.</li>
<li>Pause the master PostgreSQL database.
<ul>
<li>This is done so that the no additional write changes are occurring (as it won&#39;t be replicated to the Aurora read replica during the promotion process).</li>
</ul></li>
<li>Promote the Aurora read replica.
<ul>
<li>This takes about 10-20 minutes (at least in our experience).</li>
</ul></li>
<li>Deploy app configuration changes to use new Aurora cluster for write/reads.</li>
<li>Immediately start building additional read replicas to match existing number.
<ul>
<li>Each one takes about 5-10 minutes to create (at least in our experience).</li>
</ul></li>
<li>When happy with the new setup, remove/delete the old databases.</li>
</ul>
It is recommended to do a dry run of the promotion (and just throw it away afterwards) to gauge how long that process takes.
The issue here is that you have:
<ul>
<li>Downtime due to the promotion process. You need to prevent writes from happening on the master. Replication between the PostgreSQL master and the Aurora read replica breaks when the promotion process starts.</li>
<li>Immediately after you have your new Aurora master, you need to start building read replicas up.
<ul>
<li>In our case, we would want to match our existing number of replicas. So this would take additional time. Otherwise, you possibly run the risk of overwhelming the Database without those read replicas up.</li>
</ul></li>
</ul>
One good thing is that even with the downtime the old read replicas will continue to serve up <code>GET</code> requests. In addition, any CDN/<a href="https://varnish-cache.org/">Varnish</a> caching will also help serve stale data during the downtime.
<h2>Full Load and Ongoing Replication with DMS</h2>
This approach is much more involved and finicky, but it does offer a near zero-downtime migration strategy. The general process here is:
<ul>
<li>Create an Aurora read replica (this is so we can keep the DB schema)</li>
<li>Promote Aurora read replica so it is on its own</li>
<li>Tweak settings on PostgreSQL DB</li>
<li>Setup DMS Endpoints for the original PostgreSQL and Aurora databases</li>
<li>Setup DMS Replication Instance to assist in the migration</li>
<li>Setup DMS Task to use the endpoints and the replication instance</li>
<li>Truncate Aurora data (we want to wipe the data, but we keep the schema)</li>
<li>Initiate the migration
<ul>
<li>The first phase is a full load of the tables from the source to the target</li>
<li>The second phase is an on-going replication of changes from the source to the target</li>
</ul></li>
<li>Both DBs are in-sync now</li>
<li>Create read replicas off of the Aurora master</li>
<li>Change app configuration to point to Aurora cluster</li>
<li>Delete old DBs</li>
</ul>
This is a multi-part process, so the following sections will walk through each part.
<h3>Setting up Aurora Read Replica</h3>
<ul>
<li>Create an Aurora read replica of the PostgreSQL database
<ul>
<li>Name it something like <code>&lt;service&gt;—&lt;environment&gt;-aurora</code></li>
</ul></li>
<li>By creating an Aurora instance, we also get an Aurora cluster (i.e., <code>&lt;service&gt;—&lt;environment&gt;-aurora-cluster</code>)
<ul>
<li>Ensure the subnet group and the security groups are set to the same values as defined for the PostgreSQL DB.</li>
</ul></li>
<li>You don&#39;t need to have multi-AZ here, Aurora will handle fast failover if we have a replica</li>
<li>Additional read replicas can then be added later using <code>&lt;service&gt;—&lt;environment&gt;-aurora-&lt;##&gt;</code></li>
</ul>
At this point, we will have an Aurora read replica that is receiving replicated data changes off the master.
<h3>Aurora Read Replica Promotion</h3>
To isolate our newly created Aurora read replica so it can be its own master, we&#39;re going to promote the replica. This process allows the instance to receive writes, and be the master for other replicas. We need to do this as otherwise we are limited to only one Aurora read replica off of a PostgreSQL master database. By starting the promotion process the on-going replication is broken between the PostgreSQL and Aurora databases.
<h3>Tweak the Database Parameters</h3>
To enable the ability for PostgreSQL to replicate to Aurora we need to make sure we have certain database parameters set.
<h4>Configure PostgreSQL for Replication</h4>
The first thing you need to do is change the <code>rds.logical_replication</code> parameter to 1. As described in <a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html#CHAP_Source.PostgreSQL.RDSPostgreSQL">the documentation</a> this parameter actually configures additional related parameters that allow the replication to process:
<blockquote>
As part of applying this parameter, AWS DMS sets the <code>wal_level</code>, <code>max_wal_senders</code>, <code>max_replication_slots</code>, and <code>max_connections</code> parameters.
</blockquote>
Another parameter configuration that <a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html#CHAP_Source.PostgreSQL.Prerequisites">Amazon recommends</a> is to set <code>wal_sender_timeout</code> to 0.
<blockquote>
The <code>wal_sender_timeout</code> parameter terminates replication connections that are inactive longer than the specified number of milliseconds. Although the default is 60 seconds, we recommend that you set this parameter to zero, which disables the timeout mechanism.
</blockquote>
Without setting the timeout parameter, the streaming of Write-Ahead Logging (WAL) files can be terminated if the sender is under load and does not send a WAL file within the timeout period. To prevent any risk of this it is best to disable it during the on-going replication phase of the migration process.
There are additional <code>wal</code> settings available, but we shouldn&#39;t need to configure them.
<h4>Configure Aurora for Replication</h4>
On the Aurora instance, we want to limit any foreign key constraints from triggering as the full loads for tables are occurring. Fortunately, if we change the <code>session_replication_role</code> parameter to <code>replica</code>, it only keeps replica related constraint triggers active. This effectively disables foreign key constraints for us.
<h3>Create Replication Instance</h3>
DMS uses a dedicated replication instance to help facilitate the migration process. In a PostgreSQL to Aurora migration, this machine is responsible for connecting to both the source and target databases and transforming and transferring the data.
The following steps expand on what is needed to set up a replication instance for the migration.
<ul>
<li>Create a replication instance under DMS
<ul>
<li>Name it as specific as possible for its purpose (i.e., production-api-replication-instance)</li>
<li>The Instance Size might need to be tweaked based on various factors (<a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.html#CHAP_ReplicationInstance.InDepth">documentation on instance sizes</a>)</li>
<li>For example, in our staging we used <code>dms.t2.large</code>, while for production we used <code>dms.c4.large</code></li>
<li>Set the VPC to the same as the source and target databases</li>
<li>Enable Multi-AZ as it is needed for on-going replication</li>
<li>The instance doesn’t need to be Publicly Accessible (as the internal connections are done within the VPC)</li>
<li>Ensure the VPC Security Group under Advance is set to <code>default</code></li>
<li>This allows for connections between the databases and the replication instance</li>
</ul></li>
</ul>
<h3>Create Source and Target Endpoints</h3>
We need to define the endpoints of our databases in DMS. One to represent the source database (PostgreSQL) and another to represent the target database (Aurora). The following steps are to be applied to each endpoint.
<ul>
<li>The Server Name is the DB’s endpoint (i.e., the connection URL)
<ul>
<li>You cannot use a read replica as your source (it does not support on-going replication)</li>
</ul></li>
<li>The User and Password are for the master user account of the database
<ul>
<li>The account needs to be the master account. Otherwise, you need to follow the additional instructions listed in the <a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html#CHAP_Source.PostgreSQL.RDSPostgreSQL">documentation</a></li>
<li>The Password cannot have certain characters (<code>: ; + %</code>)</li>
</ul></li>
<li>Test the endpoint
<ul>
<li>After a success ensure you refresh the schema</li>
</ul></li>
</ul>
<h3>Create Task</h3>
The DMS task is where we actually configure and use the endpoints and replication instance to accomplish the migration. There are a couple of options and considerations to be made while creating a task. The following steps outline how to create a task, and the settings to configure:
<ul>
<li>Use the source and target endpoints along with the replica instance that was created for this migration task</li>
<li>Set the Migration type to <code>Migrate existing data and replicate ongoing changes</code>
<ul>
<li>This ensures that we are doing the 2 phase approach:</li>
<li>Doing a full-load of the table&#39;s data</li>
<li>Doing on-going replication of changes on the source database</li>
</ul></li>
<li>Target table preparation mode should be <code>Do nothing</code>
<ul>
<li>The reason here is that we want to preserve the table&#39;s metadata (i.e., indexes, defaults, constraints)</li>
<li><code>Truncate</code> could work, but is likely to fail due to constraints in the database</li>
<li>Remember that by using <code>Do nothing</code> the target database needs to be truncated manually prior to running the task</li>
</ul></li>
<li>Stop task after full load completes should be set to <code>Don&#39;t stop</code>.
<ul>
<li>This ensures the on-going replication process starts immediately when a table is fully loaded</li>
</ul></li>
<li>For the table mappings, we want to migrate all tables (i.e., <code>where schema name is like &#39;public&#39; and table name is like &#39;%&#39;, include</code>)</li>
<li>It is useful to enable the validation and logging to see how things are progressing during the migration</li>
</ul>
Include LOB columns in replication is an interesting setting, LOBs are Large Objects that exist during the migration. As DMS is possibly migrating to a different database type a transformation of data types occurs. To understand what we&#39;re dealing with you need to look up the source database&#39;s supported types in the documentation. For example, using the DMS documentation you can see what <a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html#CHAP_Source.PostgreSQL.DataTypes">PostgreSQL data types</a> end up being LOBs.
The following section will dig a little deeper into LOBs and the considerations that need to be considered.
<h4>Full or Limited LOB Mode</h4>
There are two options for handling LOBs during the migration: Full or Limited LOB Mode. LOBs are potentially massive objects that reside in the database, and they normally don&#39;t have a fixed size in the column. The following <a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.LOBSupport.html">documentation</a> describes the two options:
DMS provides you with the Full LOB Mode:
<blockquote>
In full LOB mode, AWS DMS migrates all LOBs from source to target regardless of size. In this configuration, AWS DMS has no information about the maximum size of LOBs to expect. Thus, LOBs are migrated one at a time, piece by piece. Full LOB mode can be quite slow.
</blockquote>
DMS also provides the Limited LOB Mode:
<blockquote>
In limited LOB mode, you set a maximum size LOB that AWS DMS should accept. Doing so allows AWS DMS to pre-allocate memory and load the LOB data in bulk. LOBs that exceed the maximum LOB size are truncated and a warning is issued to the log file. In limited LOB mode, you get significant performance gains over full LOB mode. We recommend that you use limited LOB mode whenever possible.
</blockquote>
Initially, it makes sense to just use Full LOB Mode as it&#39;ll preserve data by migrating LOBs regardless of size. The big issue with this choice is speed. In our migration, we achieve full migration in 2 hours using Limited LOB Mode and we estimated about 90 hours using Full LOB Mode. The issue with using Limited LOB Mode is that you can essentially lose data if the LOB&#39;s size is less than the <code>Max LOB Size</code> parameter.
To work around this concern, the plan is to determine the max LOB size in the database and set the <code>Max LOB Size</code> to slightly bigger than that value (i.e., multiply it by 2). By using a larger <code>Max LOB Size</code> than the largest LOB we have in the database, we are ensured to have the full data migrated without any data loss.
A manual way to check for the size of a LOB column is to use the following query: <code>SELECT max(pg_column_size(column_name)) FROM table_name;</code>. This will return the max number of bytes used in that column. This is the value you&#39;ll want to be larger than. The following is a Rails rake task that walks through every table&#39;s column and identifies the max sizes for any LOB column (for PostgreSQL):
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">namespace :scripts do
 desc &#39;Print out the max size of LOBs in the database. Usage: bin/rake scripts:max_lob_size&#39;
 task max_lob_size: :environment do
 output = {}
 LOB_TYPES = %w(
 hstore
 composite
 array
 jsonb
 json
 polygon
 path
 xml
 tsquery
 tsvector
 bytea
 text
 ).freeze
ActiveRecord::Base.connection.tables.each do |table_name|
next if table_name == &#39;schema_migrations&#39; || table_name.starts_with?(&#39;awsdms_&#39;)
 columns_sql = &lt;&lt;~HEREDOC
 SELECT *
 FROM information_schema.columns
 WHERE table_schema = &#39;public&#39;
 AND table_name = &#39;#{table_name}&#39;
 HEREDOC
columns_results = ActiveRecord::Base.connection.execute(columns_sql)
 columns_results.each do |row|
 column_name = row[&#39;column_name&#39;]
 column_type = row[&#39;data_type&#39;]
 if LOB_TYPES.include?(column_type)
 size_sql = &lt;&lt;~HEREDOC
 SELECT max(pg_column_size(#{column_name})) FROM #{table_name};
 HEREDOC
size_results = ActiveRecord::Base.connection.execute(size_sql)
table_hash = output[table_name.to_sym] = {}
column_hash = table_hash[column_name.to_sym] = {}
 column_hash[:column_type] = column_type
column_hash[:max_bytes_size] = size_results[0][&#39;max&#39;].to_i
 end
 print &#39;.&#39;
 end
 end
 puts &quot;\nRaw Dump&quot;
 pp output
max_bytes = output.values.map { |x| x.values.first[:max_bytes_size] }.compact.max
 puts &quot;Max bytes found in a LOB column is #{max_bytes}&quot;
 end
end
</code></pre></div>
<h4>LOB Nullability Constraints</h4>
There are some concerns with <code>NOT NULL</code> database constraints on columns that become LOBs during the migration. DMS has a certain process for dealing with tables with LOBs:
<ul>
<li>The data for each column in a row is added, except for LOB columns</li>
<li>LOB columns are left with a <code>null</code> placeholder</li>
<li>Afterwards, LOB data replaces the <code>null</code> placeholders</li>
</ul>
This two-step process of dealing with LOB data types is where we have to be concerned with <code>NOT NULL</code> constraints. The migration will fail if any LOB columns have a <code>NOT NULL</code> constraint. In PostgreSQL, a common case of LOBs is <code>jsonb</code> and <code>array</code> columns. So we have to remove these constraints on the target database just until the full table load phase is done. For example, the following statement would do the trick, <code>ALTER TABLE my_table ALTER COLUMN lob_column DROP NOT NULL;</code>. Do not forget to add the <code>NOT NULL</code> constraints back after the full table load phase is done.
<h3>Migration Time</h3>
At this point, the migration process is pretty much ready! Let&#39;s break the process up into before/during/after migration.
<h4>Before Migration</h4>
Make sure that the:
<ul>
<li>The DMS endpoints, task and replication instance are present and configured</li>
<li>The Aurora instance is ready:
<ul>
<li>Truncated data</li>
<li>Disabled LOB nullability constraints</li>
</ul></li>
</ul>
One thing that is suggested is to run <code>Assess</code> on the task so you can get a report of potential issues. In our case, there were a couple of Partially supported datatypes : float8 on a few columns. This ended up changing the rounding of floats (i.e., 1.4999999998 --&gt; 1.5). It is worth noting that these differences occur after the migration process is completed and changes are being done on the new database type.
For the migration, ideally, it is done during a low activity period. In addition, if possible stop any background jobs just before the migration, and wait for the current jobs to finish. The jobs can resume processing after the migration, this is to reduce risk.
<h4>During Migration</h4>
<ul>
<li>Keep an eye on the source database&#39;s health
<ul>
<li>There will be additional load placed on it during the migration</li>
<li>If needed you could always lower the number of tables loaded in parallel during the full load phase (under the advanced settings in the Task)</li>
</ul></li>
<li>Monitor the task&#39;s Table Statistics tab to make sure tables are progressing well</li>
<li>Monitor the task&#39;s Task Monitoring tab to make sure the on-going replication is keeping up
<ul>
<li>The <code>CDCIncomingChanges</code> should be as close to 0 as possible</li>
<li>During the full load phase, the <code>CDCIncomingChanges</code> will climb as the on-going replication changes are stored until the tables have fully loaded into the target database</li>
</ul></li>
<li>Monitor the replication instance&#39;s Monitoring tab to make sure the <code>FreeStorageSpace</code> and <code>FreeableMemory</code> are not dropping too low
<ul>
<li>If any of these are too low then the migration can fail</li>
</ul></li>
</ul>
The first phase is full table load, where all the source data is dumped into the target. Be aware that large tables can take some time. After a table has been fully loaded into the target, the on-going replication phase starts for that table.
The on-going replication phase is where inserts/deletes/updates are replicated from the source to the target. After all the tables are in this phase (all tables in a Table Completed state), it is now safe to re-enable the LOB nullability constraints that were disabled earlier.
<h5>Validations</h5>
If you have validations enabled for the Task then the validation columns in the Table Statistics will update during the migration. These validations put additional load on the source and target database as the row data is compared after that row has been migrated. It is an on-going validation process.
Personally, I found the validation to be very flaky. It is either really slow and it also reports validation errors that are not actual errors. In our case, we didn&#39;t pay much attention to the validation failures as spot checking proved that the data was fine. There were minor cases where time columns were slightly off. I am unsure how the validation actually works (i.e., when it does the checks), as the on-going replication could be lagging behind. According to the <a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html#CHAP_Validating.Limitations">documentation</a>, there are some limitations that might prevent it from being that useful:
<ul>
<li>If one or more rows are being continuously modified during the validation, then AWS DMS can&#39;t validate those rows. However, you can validate those busy rows manually, after the task completes.</li>
<li>If AWS DMS detects more than 10,000 failed or suspended records, it will stop the validation. Before you proceed further, resolve any underlying problems with the data.</li>
</ul>
In either case under the target&#39;s database, there is a new table called <code>awsdms_validation_failures_v1</code> that contains information on the failures. The <code>KEY</code> and <code>TABLE_NAME</code> columns can be used to identify the record in question. It is then possible to check the source and target record and see if there are any issues. One other problem we had with spot-checking validation is that our PostgreSQL database used <code>UUID</code>s for primary keys, this resulted in the <code>KEY</code> column having truncated data on the <code>UUID</code>.
<h4>After Migration</h4>
Hopefully, the migration went successfully, and both the source and target database are in-sync. At this point, the zero-downtime migration can occur -- simply point the application at the new database. It is advisable to wait for any replication lag or queued up <code>CDCIncomingChanges</code> to drain before proceeding. You might have to wait a little bit for the connections of your application to cycle over to the new database, but you can monitor this in CloudWatch, or force the cycle (i.e., restart Rails Unicorn servers). Don&#39;t forget to resume any background queues. After sufficient time, you can decommission the old database resources.
In the event that the migration doesn&#39;t go as planned, it is possible to revert back to the old database. The only issue is that any write data that occurred during the migration process (i.e., only on the Aurora database) wouldn&#39;t be present on the old database (i.e., PostgreSQL database). This is simply an issue that cannot be worked around, and hopefully, the low activity period reduces the amount of data loss.
</description>
</item>
<item>
<title>Make the most of your Chat Channels</title>
<link>https://techblog.thescore.com/2017/06/30/make-the-most-of-your-chat-channels/</link>
<pubDate>Fri, 30 Jun 2017 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2017/06/30/make-the-most-of-your-chat-channels</guid>
<description><center>
![](/assets/communication-tower.jpg)
_[Communication Tower](https://flic.kr/p/finMno) by [Cocoy Pusung](https://www.flickr.com/photos/95876508@N02/), on Flickr_
</center>
Organizations tend to use some instant messaging platform in addition to email. Ideally, the chosen platform permeates throughout the ranks of the organization instead of having fractured communication between members. The real-time and group collaborative nature of these messaging platforms are quite appealing.
I have personally used a couple of messaging platform so far (i.e., Hipchat, Gitter, Discord, Slack). Overall, they all offer the concept of channels. Channels are simply persisted chat rooms in which members can communicate asynchronously. Topics and names of channels are left up to the members who use them.
My goal is to outline some steps to better foster an effective use of channels in your messaging platform of choice. I am currently using Slack at theScore, and so the context of this post will be within Slack.
<h1>The Cleanse</h1>
<blockquote>
Macaitis recommends that if no one has used a channel for sixty days, it’s best to archive it. -- <a href="https://expand.openviewpartners.com/former-slack-cmo-bill-macaitis-on-how-slack-uses-slack-868ffb495b71">Former Slack CMO, Bill Macaitis, on How Slack Uses Slack</a>
</blockquote>
An organization that uses chat channels for communication will accrue an abundance of such. Some of these channels are used more frequently than others. To keep things slightly under control, you should routinely clean the channels. As to who should be doing this, I would say everyone should try to keep their communication platform clean. There are a few benefits to this:
<ol>
<li>Keeps a more narrow focus within the organization</li>
<li>Easier for users to keep a handle on their channels</li>
<li>Newcomers feel less overwhelmed at the number of channels</li>
<li>Less ambiguity on where conversations/information should be put</li>
</ol>
Archiving channels doesn&#39;t have to be perfect, although there will be clear choices. If the need of the channel arises again, it can be recreated/unarchived as needed.
<h1>Types of Channels</h1>
Channels tend to fall under specific categories that define their behaviour or purpose. For example, <a href="https://slack.global.ssl.fastly.net/5ccb/pdfs/admins_guide.pdf">Slack provides a guideline</a> on what these channel types are:
<h2>Global</h2>
<blockquote>
i.e., #general, #announcements, #everyone
</blockquote>
A global channel is one that everyone is apart of. Normally this would be a general channel, however productivity can be limited depending on the number of members -- it&#39;s more of a social channel. A common use case for using a global channel is for organizational announcements.
<h2>Location</h2>
<blockquote>
i.e., #toronto, #ontario, #canada
</blockquote>
An organization might be spread across many locations. These channels offer a way to group conversations that pertain to specific locations. How to name these channels depends on how your organization is structured geographically -- it might be based on cities, regions, or even offices within a city.
<h2>Team</h2>
<blockquote>
i.e., #engineering, #engineering-ios, #sales, #designers
</blockquote>
These channels are rather important in facilitating communication within teams. Realistically you would have high-level groups such as #engineering that all your engineers are apart of. It would also be ideal to create sub-teams to accommodate specializations such as #engineering-android and #engineering-web. By following this naming convention for teams and sub-teams, the channels are grouped and sorted through the naming convention.
<h2>Project</h2>
<blockquote>
i.e., #sports-app, #chat-bot, #squadup
</blockquote>
Often projects are underway. To help communication within that project, which might include individuals across different teams, a project channel is useful. As projects come and go, it might be worth archiving channels for projects that are no longer current.
<h2>Topical</h2>
<blockquote>
i.e., #soccer, #javascript, #anime
</blockquote>
To me, these are the interesting channels! The previously mentioned channels types were more geared towards business communication, and generally are easily formed around the business needs. Topical channels could be on anything that interests a group of people. Generally, these channels are organically formed within an organization.
<h2>Temporary</h2>
<blockquote>
i.e., #xmas-party, #offsite-retreat, #brainstorm-session
</blockquote>
These channels are short-lived and generally used for time-sensitive events. They have a specific purpose, and rarely offer much value after that purpose is completed. These channels could be deleted or archived when they are no longer needed. Channels are cheap to create, and so people shouldn&#39;t be worried about making one-off channels to help them accomplish specific tasks.
<h1>Organizing Channels</h1>
As we saw there are many categories of channels. To help with the organization of the multiple channels, a naming convention could be used.
One suggestion is to prefix all team channels with <code>team-</code> (i.e., #team-engineering). The same could be done with project channels using <code>project-</code> (i.e., #project-esports). The actual prefix doesn&#39;t matter as long as it is unique (enough) and consistent. A benefit is that channels are ordered alphanumerically, making it easier to browse active teams/projects. If you wanted, you could even take the same approach with topics (i.e., #topic-soccer).
Another idea is to have a #meta channel where you can talk about improving the use of the platform itself. For example, new topic channels can be posted there, same with renames and such. Ensure everyone is a part of the channel so information travels. If there is support for it, you could also pin/sticky some guidelines on the platform (i.e., link to this article).
<h1>Topical Channels (at theScore)</h1>
As previously mentioned, topical channels are the interesting channels that I want to touch on more here. I&#39;ll just say that this is coming from an software organization&#39;s perspective, as I am a part of theScore engineering team.
Here are a few of our topical channels:
<ul>
<li>#ping-pong</li>
<li>#overwatch</li>
<li>#pokemon-go</li>
<li>#podcasts</li>
<li>#board-games</li>
<li>#food</li>
</ul>
While these channels do provide topical information on their specific topics, we didn&#39;t really have engineering topic channels.
<h2>Birth of Engineering Topical Channels</h2>
We were doing some recent work with <a href="https://facebook.github.io/react/">React</a> and <a href="http://graphql.org/">GraphQL</a>, and we saw the birth of two engineering topical channels:
<ul>
<li>#graphql</li>
<li>#react</li>
</ul>
Before this work, we did not really have engineering topical channels. This caused slight inefficiencies in the flow of knowledge through our engineering team. We made these new channels known at the engineering level, and also invited people who are actively working in the areas. This approach injected people who cared about the topic into the channel. As we go about our day and we find something interesting related to one of these topics, it becomes easy to drop that information into the channel. In addition, more specific questions can be asked in these channels as their topic is fairly narrow.
Eventually, you end up seeing people who are not directly working in that topic area, but are still in the channel. To me this suggests people have an interest.
<h2>Educational Value of Topical Communication</h2>
These specific topical channels on programming languages/frameworks/concepts are extremely beneficial to individuals, and the organization.
<ul>
<li>Provides a place for focused conversations to take place. For example, if someone had a general GraphQL question where do you go to ask it? Possibly in one our engineering channels, or maybe a specific team that is using it? In this case, having a dedicated topical channel for #graphql would be beneficial.</li>
<li>Promotes people to widen their interests, as they can simply join a channel and slowly absorb information. For example, if I was interested in #machine-learning, I could join the channel and occasionally I&#39;ll see people post links to articles, conference talks, and just general conversation on the topic.</li>
</ul>
There is great educational value in these channels, especially in larger organizations where there are many teams and projects. These channels offer a place to share findings and communication to prevent knowledge silos.
<h2>Effective Slack Plan</h2>
Now at theScore, this is our plan in improving our usage of Slack channels:
<ol>
<li>Archive channels with no activity in the last 60 days</li>
<li>Rename team channels with the <code>team-</code> prefix</li>
<li>Rename project channels with the <code>proj-</code> prefix</li>
<li>Create a 2-level hierarchy for teams (i.e., #eng, #eng-ios)</li>
<li>Create engineering topical channels (i.e., #rails, #android, #swift)</li>
<li>Create a #meta channel (along with a note to some channel conventions)</li>
</ol>
With routine maintenance, we will keep our Slack channels focused and organized. Hopefully we will see the benefit to education/knowledge sharing with the increased organization of our chat channels.
</description>
</item>
<item>
<title>Cut Code Review Noise on GitHub</title>
<link>https://techblog.thescore.com/2017/03/28/cut-code-review-noise-on-github/</link>
<pubDate>Tue, 28 Mar 2017 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2017/03/28/cut-code-review-noise-on-github</guid>
<description><center>
![](/assets/scissors.jpg)
_[Painting scissors with light 4 by Zechariah Judy, on Flickr](https://www.flickr.com/photos/9918311@N02/4268582634/in/photostream/)_
</center>
For this post I&#39;m not going to detail my preferred approach for working through a source code change using GitHub&#39;s pull requests -- another post will likely contain this. I&#39;m instead going to touch on GitHub&#39;s review requests feature, and present an effective and efficient way to handle code reviews with GitHub&#39;s interface.
<h2>GitHub Review Requests and Reactions</h2>
GitHub, in late 2016, introduced <a href="https://github.com/blog/2291-introducing-review-requests">review requests</a> to their platform. This was a welcome addition to code reviewing pull requests. It exposed a mechanism to request reviews without using @mention or chat to get people&#39;s attention.
GitHub made it easier to identify pull requests that require your attention by <a href="https://github.com/blog/2306-filter-pull-request-reviews-and-review-requests">filtering review requests</a>. Additionally, GitHub also allowed review requests to integrate with their <a href="https://github.com/blog/2051-protected-branches-and-required-status-checks">protected branches</a> feature. It became possible to prevent merging a pull request until at least one reviewer approved that pull request.
Last year GitHub released <a href="https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments">reactions</a> to simply reduce noise in large issues and pull requests. With this now you can simply add an emoji reaction instead of a new comment.
Overall, these are all incremental steps in the right direction towards the goal of a collaborative environment within GitHub. More and more, I find myself using review requests to communicate with other developers. Requesting a review now sends the appropriate notification to the individual. In the past, I would have pasted a link to the pull request in our chat room, or made an @mention comment. By moving away from the old style of getting reviews, I find there is less chatter and noise for coordinating code reviews.
<h2>Committed Reviewers</h2>
I&#39;m going to introduce the concept of committed reviewers:
<blockquote>
A reviewer who is committed to the overall quality and correctness of the pull request.
</blockquote>
This implies that committed reviewers are individuals who give the final approval for the pull request before merging.
This concept might not be applicable for various projects or organizations. I do recommend it, however, as it tends to promote higher quality code ending up in your codebase.
With GitHub&#39;s request reviewers feature, this means that if you are requested, you are now a committed reviewer. The pull request cannot be merged until all committed reviewers approve it. This prevents a scenario where a reviewer could still be working through a pull request when it&#39;s merged, wasting time and potentially missing issues. I have seen this happen when authors request numerous reviewers just to expedite the process.
<h2>Coordinating Code Reviews</h2>
Imagine we&#39;re on a team of 4 people. You just created a pull request in GitHub. Now you want some eyes on the new changes you are proposing to put into the codebase.
You recall the old ways of using an @mention just to get reviewers, same with pasting the link in our chat. You don&#39;t want to disturb the team with unnecessary noise. Instead, it is time to use the new request reviewers interface in GitHub.
<center>
![](/assets/reviewers-state-1.png)
</center>
You decide to directly request reviews from Jane and Bob, as you know they are familiar with this part of the system. You decided to request only two reviews as we have an informal rule of requiring two approvals before merging in any pull request.
<center>
![](/assets/reviewers-state-2.png)
</center>
Bob leaves a request for changes inquiring on one aspect of your code. In one of his comments he indicates that Mary had encountered a similar problem and her solution was slightly different from yours.
<center>
![](/assets/reviewers-state-3.png)
</center>
You read over the requested changes from Bob and make the corrections.
Mary chimes in regarding that comment and leaves some insight there for you. You read it over and leave a GitHub Reaction to express your thanks, which indicates that you acknowledged her comment.
<center>
![](/assets/reviewers-state-4.png)
</center>
You now need to signal to Bob that you addressed his concerns so he can look at the new changes you added. Instead of getting his attention via chat or a @mention comment, you can remove and request him again for a review.
<blockquote>
Unfortunately, GitHub does not provide a one button click to request a review again from someone who is already a reviewer.
</blockquote>
Bob receives a notification that he has been requested for a review and looks over the changes, finally approving it.
Jane gets back and also approves your pull request.
<center>
![](/assets/reviewers-state-5.png)
</center>
At this specific point you have the two approvals that we as a team decided is required before merging a pull requests. Mary&#39;s in a comment state, which is fine as it indicates that she is not a committed reviewer.
You finally get around to merging in your pull request when you notice that Mary has added herself as a reviewer. She decided to give your pull request a complete review now as a committed reviewer. The intent was communicated through the request review.
<center>
![](/assets/reviewers-state-6.png)
</center>
Even though you got two approvals, you know that Mary is committed to reviewing your pull requests. At this point you wait for her results as to not waste her time, or to potentially miss any issues she might raise.
<center>
![](/assets/reviewers-state-7.png)
</center>
Everything looks great. You have approvals across the board, so hit that merge button!
<h3>Less Noise and Wasted Time</h3>
A couple of things you might have noticed as we ran through that code review scenario:
No unnecessary commenting on the pull request to indicate that an individual has acknowledged something. GitHub Reactions provide an unobtrusive way for an individual to express themselves. Often reactions are replacing low-value comments (i.e,. +1, LGTM, awesome!).
No unnecessary @mention comments to indicate that someone should review the pull request. In the past, we would have @mention possible reviewers in the description or as comments in the pull request. The reviewers are now clearly indicated in the reviewers section of the pull request.
No unnecessary @mention is needed to indicate a reviewer&#39;s request for change has been addressed. Previously, it was common to @mention a reviewer when their concerns were addressed, thus causing noise with comments.
With the above points on @mention you could also factor in that this communication could have been in chat (public channels or privately).
Another scenario that we have encountered in the past is that the reviewer would come back preemptively to review the pull request as they noticed new commits. The problem is that sometimes new changes are still being worked on and pushed up incrementally. This can potentially waste the reviewer&#39;s time as they have to look over more changes soon after.
As a reviewer, you know that you will receive a notification via GitHub when your attention is needed for a pull request. It also becomes easier to scan pull requests for where your action is required.
<h2>Dealing with Notifications</h2>
There are a couple of options when dealing with notifications surrounding GitHub:
<ul>
<li>Built-in web notification</li>
<li>Email notifications</li>
<li>Third-party application (depends on operating system)</li>
</ul>
I personally never got much benefit from the web notification, however, this could just be how I consume information. Without an actual notification appearing in my notification center (macOS), information doesn&#39;t reach me well.
Email notifications are currently my preferred approach, as I can receive request review notifications via email. In addition, with email you can get fancy with filters to further reduce notification to only what you want.
I use <a href="https://ptsochantaris.github.io/trailer/">Trailer.app</a> in addition to email. With Trailer I am able to target specific repositories for native notification. Prior to review requests, this would have been my ideal approach for dealing with notifications at my work machine. There is active development to support the recent addition of GitHub review requests and reactions.
Regardless of delivery mechanism, take some time to figure how to deal with notifications. Each user will have different needs. For my case, I&#39;m really only interested in emails about Comments on Issues and Pull Requests and Pull Request reviews on my Participated Conversations (configured via <a href="https://github.com/settings/notifications">GitHub&#39;s Notifications</a>).
<h2>Keeping it all in GitHub</h2>
Going back to the above scenario, no use of direct form of communication was used to facilitate the code review, everything was kept within GitHub. Of course, deeper discussions should use those mediums, but the key is that the coordination of the code review was kept entirely in the GitHub platform. This reduces chatter and noise that we receive throughout the day, and keeps the GitHub pull request succinct.
Using review requests clearly states the next action for the reviewer and author. With both parties actively using the provided utilities in GitHub, code review collaboration becomes much more manageable.
<hr>
<h2>TL;DR</h2>
<ul>
<li>The idea is to reduce the unnecessary noise within a pull request using GitHub&#39;s features.</li>
<li>GitHub review requests keep state of each reviewer during the code review process.</li>
<li>Make use of re-requesting a review when you have addressed a reviewer&#39;s concerns.</li>
<li>Make use of reactions to acknowledge a comment if possible.</li>
<li>Be aware of committed reviewers and ensure everyone knows of their roles.</li>
<li>Ensure that all committed reviewers have approved the pull request before merging.</li>
</ul>
</description>
</item>
<item>
<title>Keep on Learning, but don't Forget to Remember</title>
<link>https://techblog.thescore.com/2016/06/07/keep-on-learning-but-dont-forget-to-remember/</link>
<pubDate>Tue, 07 Jun 2016 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2016/06/07/keep-on-learning-but-dont-forget-to-remember</guid>
<description>Learning is a way of life for a developer. Frequently we are exposed to techniques and tips such as: keybindings, shell commands, utilities, new functions/methods, new applications. While learning, we want to keep on remembering the previous techniques and tips that we&#39;ve learned so that we can keep them fresh and on our tool belt. It is not uncommon to forget some learned knowledge if you don&#39;t use it often.
I would like to present my own problem and solution surrounding this idea.
<h2>Problem: Customization of vim/zsh</h2>
Vim and zsh are my editor and shell of choice, both decisions I made early in my career. I have since spent countless hours using both and each are indispensable to my development process. As typical of any developer I am always looking for ways to improve my process. Near the beginning of my career I discovered that both <a href="http://vimawesome.com/">vim</a> and <a href="https://github.com/unixorn/awesome-zsh-plugins">zsh</a> have thriving plugin communities.
I saw that there were <a href="https://github.com/spf13/spf13-vim">popular</a> vim <a href="https://github.com/carlhuda/janus">distributions</a> that come pre-packaged with opinionated set of keybindings, plugins and themes. I originally used one of these and immediately felt overwhelmed with the added functionality. I eventually came to terms with the error of my ways, and went back to vanilla vim. Learning from my mistake I decided to slowly incorporate features I felt were useful as I came to need them.
I continue to stumble upon new plugins that simplifies tasks. I add custom keybindings to perform certain motions. I create custom vim functions. While observing colleagues I take mental notes of vim motions that I have yet to take advantage of. I occasionally look through my setup and notice features and keybindings that I have not used in a while. Sometimes I remove them, but other times I&#39;ll keep it and make a try and use it.
My use of zsh is nearly an identical story to that of my use with vim. In either case I routinely accumulate and prune tips that I find useful. My largest gripe is forgetting about new tips that I want to make part of my normal tool set. In most cases these have yet to become habit and common knowledge to me through repetitive exposure and use.
<h2>Solution</h2>
I decided to write my accumulation of tips down as one-liners in <code>tips.txt</code> files, while using directories under <code>~/.tips</code> for categorization:
<div class="highlight"><pre><code class="language-text" data-lang="text">~/.tips❯ tree
.
├── ruby
│ └── rspec
│ └── tips.txt
├── vim
│ ├── plugins
│ │ ├── ctrlp
│ │ │ └── tips.txt
│ │ └── nerdtree
│ │ └── tips.txt
│ └── vanilla
│ └── tips.txt
└── zsh
└── tips.txt
</code></pre></div>
Ultimately my goal is to be presented with a random tip when I open a new shell. To accomplish this the following code snippet is placed near the start of my <code>.zshrc</code> (also works in a <code>.bashrc</code>):
<div class="highlight"><pre><code class="language-text" data-lang="text"># Displays a random tip from the .tips directory when opening the shell
# Requires gshuf (brew install coreutils)
(
TIP_PATH=$(find ~/.tips -type f -name tips.txt | gshuf -n1) # Pick a random tips.txt file
TIP_TILE=${TIP_PATH#&quot;$HOME/.tips/&quot;} # i.e., ~/.tips/vim/vanilla/tips.txt ---&gt; vim/vanilla/tips.txt
echo &quot;From ${TIP_TILE%.txt}:&quot; # i.e., &quot;From vim/vanilla/tips:&quot;
gshuf -n1 &lt; &quot;$TIP_PATH&quot; # Displays a random line from the tip file
)
</code></pre></div>
Now throughout the day when I open many new shells, I am presented with a random recorded tip:
<div class="highlight"><pre><code class="language-text" data-lang="text">From vim/plugins/nerdtree/tips:
`&lt;f2&gt;` toggles nerdtree open/close
~/.tips❯
</code></pre></div><div class="highlight"><pre><code class="language-text" data-lang="text">From zsh/tips:
`j` allows you to jump to marked directories (via `jump`)
~/.tips❯
</code></pre></div>
<h2>Don&#39;t Forget to Remember</h2>
As I continue to learn new things I&#39;ll be added them to the appropriate <code>tips.txt</code> within my <code>.tips</code> directory. If I am presented with a tip that is no longer relevant I can simply remove it. Overall this seems like a great solution for recollecting tips.
</description>
</item>
<item>
<title>Using State Machines to Handle Workflows</title>
<link>https://techblog.thescore.com/2016/04/19/using-state-machines-to-handle-workflows/</link>
<pubDate>Tue, 19 Apr 2016 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2016/04/19/using-state-machines-to-handle-workflows</guid>
<description>Have you registered for a new user account in an app before? How about adding your payment info for buying something online? Those are two examples of what a “workflow” can be. Workflows are usually associated with a set of screens that appear in different orders depending on what the user selects. For example, registering with Facebook vs. email can change the sequence and number of screens presented to the user. We will demonstrate how state machines can be used to handle this logic in an iOS app.
<h2>What are state machines?</h2>
State machines contain a set of states with transitions between them. Each transition corresponds to an event, and when the event occurs, the state machine moves from one state to the next. For handling workflows, we will be using a finite state machine. Below is an abstract example.
<img src="http://i.imgur.com/4B4cs7g.png" alt="Imgur">
The left arrow pointing into State A denotes the initial state. When moving between states, it should be easy to determine where to go because transitions are uniquely defined. For example, State A cannot have another transition labelled T1 going to State C. 
Note: I could have labelled all the transitions differently (T1, T2, T3, T4), but it&#39;s not necessary if we&#39;re careful about reuse!
<h2>Example Scenario</h2>
Users are asked to register an account with us before they begin using the app. Our goal is to collect some basic information for the user, but also give them the ability to register later. We are given some screen designs by our designer to use for the registration process. 
<img src="http://i.imgur.com/PQoAOb5.png" alt="Imgur">
We can begin by creating our list of states for the registration process. Let’s start by marking each of the screens as a state. We’ll also need to add a finished state even though it doesn’t correspond to a screen so that we can link our transitions properly.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">enum RegistrationStates {
case Login
case CreateProfile
case FacebookProfile
case SelectAvatar
case TermsAndConditions
case Finished
}
</code></pre></div>
Events will be used to determine which screen we display next. In our registration process, the next screen will be shown when the user taps on a button, so we can use the list of user actions for our events.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">enum RegistrationActions {
case Login
case Next
case Back
case RegisterLater
case FacebookRegister
}
</code></pre></div>
Now we can create our registration state machine based on our designs and screen flow.
<img src="http://i.imgur.com/Qnn5E8d.png" alt="Imgur">
<h3>Bonus</h3>
Since there are multiple transitions entering Select Avatar. Some people might want the back button in Select Avatar to return to either Facebook Profile or Create Profile, depending on where the user came from. 
<img src="http://i.imgur.com/QNXcDym.png" alt="Imgur">
This situation can be handled by storing logic outside of the state machine. One way you can do this is by setting a flag outside of the state machine (perhaps right before we navigate to that screen). Then, when the user taps back from select avatar, we would check the flag to determine which event to fire.
<h2>Implementation</h2>
Right now, our state machine is written as a Swift generic. Following the definition of a finite state machine, this is what it expects and how it behaves:
<ul>
<li>Contains a set of states</li>
<li>Contains a set of events</li>
<li>Contains transition functions, that accepts a state and event, then returns a target state </li>
<li>Initial state</li>
<li>Final state</li>
</ul>
To use the state machine for our registration flow, we provide it with our list of states, events, and initial state.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">var registrationStateMachine = StateMachine&lt;RegistrationStates, RegistrationActions&gt;(initialState: .Login)
</code></pre></div>
Now we need to connect our states by providing possible transitions (based on user actions) for each state. We’ve utilized subscripting to make this assignment more readable. To make it easier to visualize, we can refer to the state machine diagram we created earlier. I’ve included it here again for reference. 
<img src="http://i.imgur.com/MVM5cRF.png" alt="Imgur">
<div class="highlight"><pre><code class="language-swift" data-lang="swift">registrationStateMachine[.Login] = [
.Next : .CreateProfile,
.Login : .Finished
]
registrationStateMachine[.CreateProfile] = [
.Next : .SelectAvatar,
.FacebookRegister : .FacebookProfile,
.RegisterLater : .TermsAndConditions
]
registrationStateMachine[.FacebookProfile] = [
.Next : .SelectAvatar,
.RegisterLater : .TermsAndConditions
]
registrationStateMachine[.SelectAvatar] = [
.Next : .TermsAndConditions,
.Back : .CreateProfile
]
registrationStateMachine[.TermsAndConditions] = [
.Next : .Finished
]
</code></pre></div>
<h3>Usage</h3>
Since our state machine is separate from the UI logic of our registration flow, we will need to send events to our state machine to determine what to display next. We’ve mapped events to user actions, so we can trigger events in our button responder methods. I’ve added a protocol to handle sending events to the state machine.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">protocol RegistrationActionProtocol {
func notifyStateMachine(source: UIViewController, _ event: RegistrationActions)
}
</code></pre></div><div class="highlight"><pre><code class="language-swift" data-lang="swift">@IBAction func didTapEmailRegister(sender: AnyObject) {
delegate?.notifyStateMachine(self, .Next)
}
@IBAction func didTapSkip(sender: AnyObject) {
delegate?.notifyStateMachine(self, .RegisterLater)
}
</code></pre></div>
Next, we’ve used <code>if let</code> to unwrap <code>nextState</code> because the state machine will return <code>nil</code> if there is no valid state for the given event. When a non-nil state is returned, we can map it to a screen with a switch statement, then push the view controller onto our navigation controller.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">func notifyStateMachine(source: UIViewController, _ event: RegistrationActions) {
if let nextState = registrationStateMachine.transition(event) {
switch nextState {
case .CreateProfile:
self.navigationController?.pushViewController(createProfileViewController, animated: true)
case .FacebookProfile:
self.navigationController?.pushViewController(facebookProfileViewController, animated: true)
...
}
}
}
</code></pre></div>
That’s it! You now have a state machine that handles registration flow. By decoupling it from the UI logic, you can freely modify screen display order without affecting other registration components.
You can find the sample project <a href="https://github.com/JessicaIp/SimpleStateMachine.git">here</a>.
<h2>Summary</h2>
State machines can be used to represent a variety of workflows in an app with minimal code duplication. They are particularly useful when the flow is complex or constantly changing.
</description>
</item>
<item>
<title>UITableView & UICollectionView: Killing stringly typed cells with Swift</title>
<link>https://techblog.thescore.com/2016/04/04/Typed-UITableView-&-UICollectionView-Dequeuing-in-Swift/</link>
<pubDate>Mon, 04 Apr 2016 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2016/04/04/Typed-UITableView-&-UICollectionView-Dequeuing-in-Swift</guid>
<description>UITableView &amp; UICollectionView are the bread and butter of many iOS applications. However, dequeuing cells with string identifiers can result in brittle code that doesn&#39;t scale as table complexity increases. We&#39;re going to look at using Swift&#39;s type system to get rid of the ad-hoc typing we get from using string identifiers and eliminate the need for type casts in the process.
<h2>Introduction</h2>
It&#39;s helpful to look at the original context in which an API was used in order to understand the problems it solves. To many of us the following code will be all too familiar.
<div class="highlight"><pre><code class="language-objective-c" data-lang="objective-c">- (UITableViewCell *)tableView:(UITableView *)tableView cellForRowAtIndexPath:(NSIndexPath *)indexPath
{
static NSString *CellIdentifier = @&quot;MyCellIdentifier&quot;;
MyCell *cell = (MyCell *)[tableView dequeueReusableCellWithIdentifier: CellIdentifier];
if (cell == nil) {
cell = [[MyCell alloc] initWithStyle: UITableViewCellStyleDefault reuseIdentifier:CellIdentifier];
 }
 // Do stuff with cell
 return cell; 
}
</code></pre></div>
This convention has not conceptually changed since its introduction. The code is highly localized: the definition of the identifier, the dequeuing of the cell, and the allocation of a cell with a given identifier are all in one place, making it easy to reason about. This also makes the casting of the cell&#39;s type seem innocuous.
As apps become increasingly complex, <code>UITableViewDelegate</code>s often started displaying multiple data types, leading to patterns like the following:
<div class="highlight"><pre><code class="language-objective-c" data-lang="objective-c">- (UITableViewCell *)tableView:(UITableView *)tableView cellForRowAtIndexPath:(NSIndexPath *)indexPath
{
 UITableViewCell *cell;
if (indexPath.section == 0) {
static NSString *CellIdentifier = @&quot;MyCellIdentifier&quot;;
MyCell *myCell = (MyCell *)[tableView dequeueReusableCellWithIdentifier: CellIdentifier];
if (myCell == nil) {
myCell = [[MyCell alloc] initWithStyle: UITableViewCellStyleDefault reuseIdentifier: CellIdentifier];
 }
 // Do stuff with cell
 cell = myCell
 }
 else {
static NSString *OtherCellIdentifier = @&quot;MyOtherCellIdentifier&quot;;
MyOtherCell *otherCell = (MyOtherCell *)[tableView dequeueReusableCellWithIdentifier: OtherCellIdentifier];
if (otherCell == nil) {
otherCell = [[MyOtherCell alloc] initWithStyle: UITableViewCellStyleDefault reuseIdentifier: OtherCellIdentifier];
 }
 // Do stuff with other cell
 cell = otherCell;
 }
 return cell; 
}
</code></pre></div>
Most developers are still pretty comfortable with this. Locality is high and the code is fairly easy to reason about, but that cast is there again. I copied the else case and changed the bits I needed but I&#39;m still using strings as a type system to discern the type of cell that should be dequeued. This could become a problem as my table grows in complexity.
iOS 5 added prototype cells, which sticks the reuse identifier in a storyboard. It also added <code>- (void)registerNib:(UINib *)nibforCellReuseIdentifier:(NSString *)identifier</code> and iOS 6 gave us <code>- (void)registerClass:(Class)cellClass forCellReuseIdentifier:(NSString *)identifier</code> which is often called from <code>-(void)viewDidLoad</code>. These methods allow us to neatly separate out the concerns of cell configuration and instantiation/registration, but now it&#39;s spread all over our class, and our stringly typed cells are likely to bite us at some point in the future.
Luckily, Swift gives us some tools to help deal with this.
<h2>Providing a consistent method of determining identifers</h2>
Identifiers are how we determine type information for cells in <code>UITableView</code> and <code>UICollectionView</code>. They work well for small applications but degrade as the complexity of the <code>UITableView</code> (and thus the number of types) increases. Providing a consistent method of determining identifiers will go a long way towards making life easier.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">extension UITableViewCell {
public class func defaultIdentifier() -&gt; String {
return NSStringFromClass(self)
}
}
extension UITableViewHeaderFooterView {
public class func defaultIdentifier() -&gt; String {
return NSStringFromClass(self)
}
}
</code></pre></div>
Now we have an easy way of producing an identifier based on the type. Additionally, subclasses can provide their own implementations if the default implementations don&#39;t suffice. This approach isn&#39;t uncommon, and you could do a lot worse than getting here and calling it a day.
Note that we use <code>NSStringFromClass(self)</code> instead of <code>String(self)</code> for the default implementation. This is because NSStringFromClass includes the module name which can be helpful for differentiating from the <code>MyApp.MyCell</code> and <code>SomeThirdPartyDependecy.MyCell</code> in the event that the two are used in the same tableview at some point in the future. 
<h2>Using Types for Registration</h2>
But why would we stop there? Now that every type has a method of producing an identifier why not just use the type itself for registration?
<div class="highlight"><pre><code class="language-swift" data-lang="swift">// Cells
extension UITableView {
public func register&lt;T: UITableViewCell&gt;(cellClass `class`: T.Type) {
registerClass(`class`, forCellReuseIdentifier: `class`.defaultIdentifier())
}
public func register&lt;T: UITableViewCell&gt;(nib: UINib, forClass `class`: T.Type) {
registerNib(nib, forCellReuseIdentifier: `class`.defaultIdentifier())
}
}
// Header / Footer
extension UITableView {
public func register&lt;T: UITableViewHeaderFooterView&gt;(headerFooterClass `class`: T.Type) {
registerClass(`class`, forHeaderFooterViewReuseIdentifier: `class`.defaultIdentifier())
}
public func register&lt;T: UITableViewHeaderFooterView&gt;(nib: UINib, forHeaderFooterClass `class`: T.Type) {
registerNib(nib, forHeaderFooterViewReuseIdentifier: `class`.defaultIdentifier())
}
}
</code></pre></div>
This is starting to look pretty good, we have strong types and can use the types themselves as registration rather than providing a potentially error prone <code>String -&gt; Type mapping</code>. This is also pretty trivial, the nice bit here is that generics give us a bit of type safety around what we can register.
<h2>Using Types to Dequeue</h2>
We had been using the above for quite a while and it took us a bit longer for us to put together then next piece. Which uses generics and the type system to get an appropriate type back out.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">extension UITableView {
func dequeueReusableCell&lt;T: UITableViewCell&gt;(withClass `class`: T.Type) -&gt; T? {
return dequeueReusableCellWithIdentifier(`class`.defaultIdentifier()) as? T
}
func dequeueReusableCell&lt;T: UITableViewCell&gt;(withClass `class`: T.Type, forIndexPath indexPath: NSIndexPath) -&gt; T {
guard let cell = dequeueReusableCellWithIdentifier(`class`.defaultIdentifier(), forIndexPath: indexPath) as? T else {
fatalError(&quot;Error: cell with identifier: \(`class`.defaultIdentifier()) for index path: \(indexPath) is not \(T.self)&quot;)
}
return cell
}
func dequeueResuableHeaderFooterView&lt;T: UITableViewHeaderFooterView&gt;(withClass `class`: T.Type) -&gt; T? {
return dequeueReusableHeaderFooterViewWithIdentifier(`class`.defaultIdentifier()) as? T
}
}
</code></pre></div>
Some of you may be looking at that <code>fatalError()</code> and getting cagey but I assure you it`s all right. Way back in June, <a href="https://www.natashatherobot.com">Natasha</a> wrote a <a href="https://www.natashatherobot.com/ios-using-the-wrong-dequeuereusablecellwithidentifier/">post</a> about using the right dequeue method. Let&#39;s have a look at that type signature again.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">func dequeueReusableCellWithIdentifier(identifier: String, forIndexPath indexPath: NSIndexPath) -&gt; UITableViewCell
</code></pre></div>
There&#39;s no <code>?</code> at the end but that string as an identifier means one of two things. Either UIKit is lying to us or somewhere deep in it&#39;s bowels it&#39;s going to have a fit and crash on you. Turns out it&#39;s the latter and you get a <code>NSInternalConsistencyException</code>. The docs say: 
<blockquote>
IMPORTANT 
You must register a class or nib file using the registerNib:forCellReuseIdentifier: or registerClass:forCellReuseIdentifier: method before
calling this method.
</blockquote>
This makes sense, how can <code>UITableView</code> guarantee that there is going to be a cell unless that identifier is used to register a cell class? If the identifier isn&#39;t registered then there&#39;s been an error, possibly something as simple as a typo on the part of the programmer.
The motivation for Natasha&#39;s article is to get away from having to unwrap optional cells. Unfortunately if we have custom cell subclasses we still end up with either:
<div class="highlight"><pre><code class="language-swift" data-lang="swift">if let cell = tableView.dequeueReusableCellWithIdentifier(&quot;MyCellIdentifier&quot;, forIndexPath: indexPath) as? MyCell {
// Do stuff with cell
}
</code></pre></div>
or
<div class="highlight"><pre><code class="language-swift" data-lang="swift">let cell = tableView.dequeueReusableCellWithIdentifier(&quot;MyCellIdentifier&quot;, forIndexPath: indexPath) as! MyCell
// Do stuff with cell
</code></pre></div>
Neither of these are particularily nice looking: we either introduce a different <code>if let</code> to check that the type is the one expected or we perform a force cast to the type and hope that there isn&#39;t a mapping mis-match between cell and identifier. Both solutions will still crash if the identifier has never been registered and they litter your code with a bunch of explicit type information. Our new dequeue method looks much cleaner.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">let cell = tableView.dequeueReusableCell(withClass: MyCell.self, forIndexPath: indexPath)
// Do stuff with cell
</code></pre></div>
Putting it all together we get:
<div class="highlight"><pre><code class="language-swift" data-lang="swift">class MyTableViewController: UITableViewController {
override func viewDidLoad() {
super.viewDidLoad()
tableView.register(cellClass: MyCell.self)
}
override func tableView(tableView: UITableView, numberOfRowsInSection section: Int) -&gt; Int {
return 1
}
override func tableView(tableView: UITableView, cellForRowAtIndexPath indexPath: NSIndexPath) -&gt; UITableViewCell {
let cell = tableView.dequeueReusableCell(withClass: MyCell.self, forIndexPath: indexPath)
// Do stuff with cell
return cell
}
}
</code></pre></div>
This looks pretty good. We no longer need to rely on strings for registration and dequeuing. We don&#39;t have to add type annotations or casting to use a custom cell. And if we get a fatal error the problem is easy to debug as it simply means that the cell you are trying to use was never registered with this class. 
You can grab the complete code <a href="https://gist.github.com/tapi/60a6ee1cbfeabd12180ad88d9714591f">here</a>.
</description>
</item>
<item>
<title>Defined Methods in Rake Tasks; You're Gonna Have a Bad Time</title>
<link>https://techblog.thescore.com/2016/03/17/defined_methods-in-rake-tasks-you-re-gonna-have-a-bad-time/</link>
<pubDate>Thu, 17 Mar 2016 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2016/03/17/defined_methods-in-rake-tasks-you-re-gonna-have-a-bad-time</guid>
<description>Rake tasks provide a nice way to handle common tasks surrounding a ruby project. Within Rails projects they are nearly unavoidable and even have their own directory from which they are autoloaded. Eventually a project will grow in size and complexity to warrant multiple task files for better separation of concerns. This alone is nothing to be worried about, but it&#39;s when you start using methods in your task files where you&#39;re gonna have a bad time.
Let&#39;s setup a dummy Rails project that has a task file that calculates and saves blog metrics.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># lib/tasks/blog_metrics_task.rake
desc &#39;Calculate and save blog metrics&#39;
task :blog_metrics do
 metrics = calculate_blog_metrics
 save(metrics)
end
def calculate_blog_metrics
 puts &quot;Calculating blog metrics&quot;
end
def save(metrics)
 puts &quot;Saving blog metrics&quot;
end
</code></pre></div>
When we run our task it does exactly what we wanted and expected it to do.
<div class="highlight"><pre><code class="language-text" data-lang="text">$ rake blog_metrics
Calculating blog metrics
Saving blog Metrics
</code></pre></div>
No problem! Now lets fast-forward in time to when we want to add another task that creates a new blog post.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># lib/tasks/create_blog_post_task.rake
desc &#39;Create and save a new blog post&#39;
task :create_blog_post do
 blog_post = generate_default_blog_post
 save(blog_post)
end
def generate_default_blog_post
 puts &quot;Generating a default blog post&quot;
end
def save(blog_post)
 puts &quot;Saving defualt blog post&quot;
end
</code></pre></div>
When we run our new task it does exactly what we wanted and expected it to do.
<div class="highlight"><pre><code class="language-text" data-lang="text">$ rake create_blog_post
Generating a default blog post
Saving default blog post
</code></pre></div>
Another success! Now here is where things get interesting. Let&#39;s go back and run the first correctly working task.
<div class="highlight"><pre><code class="language-text" data-lang="text">$ rake blog_metrics
Calculating blog metrics
Saving default blog post
</code></pre></div>
Woah... it&#39;s the <code>#save</code> that was defined in the other task file -- <code>create_blog_post_task.rake</code>.
This is kind of shocking and might have caught you off guard. Rails automatically loads all rake tasks (i.e., requires their file) when executing any rake task. The gotcha here is that the defined methods in the loaded tasks files end up defined on the global namespace. These defined methods are therefore accessible across rake files, so it is possible for methods to clash and be redefined if their signatures match.
To better illustrate the order of events:
<ol>
<li><code>rake blog_metrics</code></li>
<li>Rails autoloads all rake tasks in alphanumeric order</li>
<li><code>lib/tasks/blog_metrics_task.rake</code> is loaded and defines <code>#calculate_blog_metrics</code> and <code>#save</code></li>
<li><code>lib/tasks/create_blog_post_task.rake</code> is loaded and defines <code># generate_default_blog_post</code> and redefines <code>#save</code></li>
<li><code>blog_metrics</code> task is executed using last defined <code>#save</code>, which was defined in the other task file</li>
</ol>
No worries right? Rake provides a &#39;namespace&#39; DSL. So we can modify our tasks to use this.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">namespace :blog_metrics do
 desc &#39;Calculate and save blog metrics&#39;
 task :run do
 metrics = calculate_blog_metrics
 save(metrics)
 end
 def calculate_blog_metrics
 puts &quot;Calculating blog metrics&quot;
 end
 def save(metrics)
 puts &quot;Saving blog metrics&quot;
 end
end
</code></pre></div>
We should be in the clear now.
<div class="highlight"><pre><code class="language-text" data-lang="text">$ rake blog_metrics:run
Calculating blog metrics
Saving default blog post
</code></pre></div>
Nope! The namespace DSL does nothing for the defined methods. So we still have the same problem.
There are a couple of solutions to this problem:
<ol>
<li>Rename the methods, and ensure all future methods are uniquely named</li>
<li>Inline the contents of the defined methods</li>
<li>Extract the methods into a module/class and use that</li>
</ol>
<h2>Solution #1 - Uniquely Named Methods</h2>
It is possible to simply ensure that we uniquely name our methods so that they do no clash and end up redefining each other.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># lib/tasks/blog_metrics_task.rake
desc &#39;Calculate and save blog metrics&#39;
task :blog_metrics do
 metrics = calculate_blog_metrics
 save(metrics)
end
def calculate_blog_metrics
 puts &quot;Calculating blog metrics&quot;
end
def save_blog_metrics(metrics)
 puts &quot;Saving blog metrics&quot;
end
</code></pre></div><div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># lib/tasks/create_blog_post_task.rake
desc &#39;Create and save a new blog post&#39;
task :create_blog_post do
 blog_post = generate_default_blog_post
 save(blog_post)
end
def generate_default_blog_post
 puts &quot;Generating a default blog post&quot;
end
def save_default_blog_post(blog_post)
 puts &quot;Saving defualt blog post&quot;
end
</code></pre></div>
This works and is a quick fix, although it is not exactly sustainable and requires you to be conscientious when naming new methods.
<h2>Solution #2 - Inline Method Contents</h2>
To ensure that method redefinition doesn&#39;t occur we can simply remove the methods and inline their content.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># lib/tasks/blog_metrics_task.rake
desc &#39;Calculate and save blog metrics&#39;
task :blog_metrics do
 puts &quot;Calculating blog metrics&quot;
 metrics = # Inline calculating work
 puts &quot;Saving blog metrics&quot;
 # Inline saving work
end
</code></pre></div><div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># lib/tasks/create_blog_post_task.rake
desc &#39;Create and save a new blog post&#39;
task :create_blog_post do
 puts &quot;Generating a default blog post&quot;
 blog_post = # Inline blog post generation work
 puts &quot;Saving default blog post&quot;
 # Inline saving work
end
</code></pre></div>
This is also a quick fix, and might be optimal depending on the size, complexity, and reuse of the method&#39;s content.
<h2>Solution #3 - Extract Methods into Module/Class</h2>
Removing the methods from the rake files themselves is another valid solution. The methods can be extracted into their own class or module and used within the task files.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># lib/blog_metric_calculator.rb
class BlogMetricCalculator
 def metrics
 puts &quot;Calculating blog metrics&quot;
 end
 def save(metrics)
 puts &quot;Saving blog metrics&quot;
 end
end
# lib/tasks/blog_metrics_task.rake
require &#39;lib/blog_metric_calculator&#39;
desc &#39;Calculate and save blog metrics&#39;
task :blog_metrics do
 calculator = BlogMetricCalculator.new
 calculator.save(calculator.metrics)
end
</code></pre></div><div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># lib/blog_post_creator.rb
module BlogPostCreator
 def self.create_default_blog_post
 puts &quot;Generating a default blog post&quot;
 end
 def self.save(post)
 puts &quot;Saving default blog post&quot;
 end
end
# lib/tasks/create_blog_post_task.rake
require &#39;lib/blog_post_creator&#39;
desc &#39;Create and save a new blog post&#39;
task :create_blog_post do
 blog_post = BlogPostCreator.create_default_blog_post
 BlogPostCreator.save(blog_post)
end
</code></pre></div>
This is the preferred method if there is sufficient complexity involved. By extracting the methods you begin to build up a set of related concerns within a module/class. By having an external entity outside of the rake tasks themselves you can now test the defined functionality!
</description>
</item>
<item>
<title>Introduction to Fisticuffs</title>
<link>https://techblog.thescore.com/2015/12/18/fisticuffs/</link>
<pubDate>Fri, 18 Dec 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/12/18/fisticuffs</guid>
<description><a href="https://github.com/scoremedia/Fisticuffs">Fisticuffs</a> is a compact Swift framework for view-model bindings on iOS, inspired by <a href="http://knockoutjs.com">KnockoutJS</a>. It lets developers quickly set up responsive applications without needing to add intermediate view-updating logic.
<h2>Background</h2>
The Model-view-viewmodel (MVVM) pattern has gained a lot of traction in the iOS community over the past couple years, and for good reason. Separating out the display logic (&quot;view model&quot;) from the actual display (&quot;view&quot;) has great benefits for code organization and testability.
Fisticuffs simplifies the cycle of updating views based on view-model changes by extending UIKit components for data binding. By encapsulating presentation logic in view-models we can create testable view code. Adding a binding layer like Fisticuffs lets us quickly tie this presentation logic to a view without having to write a lot of wiring.
<h2>Overview</h2>
At the core of Fisticuffs are 2 classes:
<ul>
<li><code>Observable</code> - Wraps a value, notifying any subscribers when its value changes. Think of <code>Observable</code> as a read-write property.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">let name: Observable&lt;String&gt; = Observable(&quot;Johnny Appleseed&quot;)
name.value = &quot;John Smith&quot;
</code></pre></div></li>
<li><code>Computed</code> - Read-only value computed from one or more <code>Observable</code>s. Additionally, since <code>Computed</code> has knowledge of which <code>Observable</code>s it depends on, it automatically notifies its subscribers when any of its dependencies are changed. Think of <code>Computed</code> as a computed property.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">let greeting: Computed&lt;String&gt; = Computed {
return &quot;Hello, \(name.value)&quot;
}
print(greeting.value) // &quot;Hello, John Smith&quot;
</code></pre></div></li>
</ul>
<h3>UI Binding</h3>
Fisticuffs extends many UIKit classes to support binding <code>Observable</code> &amp; <code>Computed</code>&#39;s to their properties. Building on the previous snippets, we could do something like this:
<div class="highlight"><pre><code class="language-swift" data-lang="swift">let nameTextField: UITextField = ...
let greetingLabel: UILabel = ...
nameTextField.b_text.bind(name)
greetingLabel.b_text.bind(greeting)
</code></pre></div>
In this case, the value of <code>greetingLabel.text</code> will automatically update as the user enters text in the <code>nameTextField</code>.
<h2>Example</h2>
Traditionally, without a bindings framework, one might implement a login page like so:
<div class="highlight"><pre><code class="language-swift" data-lang="swift">class LoginViewController: UIViewController {
@IBOutlet var usernameField: UITextField!
@IBOutlet var passwordField: UITextField!
@IBOutlet var submitButton: UIButton!
private var isLoading = false
override func viewDidLoad() {
super.viewDidLoad()
updateSubmitButtonState()
}
func updateSubmitButtonState() {
submitButton.enabled = isLoading == false &amp;&amp;
usernameField.text.isEmpty == false &amp;&amp;
password.text.isEmpty == false
}
@IBAction usernameFieldEditingChanged(sender: AnyObject?) {
updateSubmitButtonState()
}
@IBAction passwordFieldEditingChanged(sender: AnyObject?) {
updateSubmitButtonState()
}
@IBAction submitButtonTouchUpInside(sender: AnyObject?) {
let username = usernameField.text
let password = passwordField.text
// ... do login with `username` / `password` ...
isLoading = true
updateSubmitButtonState()
}
}
</code></pre></div>
While this approach works, there are a few pain points:
<ul>
<li>It&#39;s somewhat fragile. Any maintainer has to know when to call <code>updateSubmitButtonState()</code>, and it may be challenging to figure out where we are missing an <code>updateSubmitButtonState()</code> call if the <code>submitButton.enabled</code> property is not being updated.</li>
<li>The display and business logic are deeply intertwined, making it difficult to test.</li>
</ul>
Using an MVVM approach with Fisticuffs, we can solve both those problems:
<div class="highlight"><pre><code class="language-swift" data-lang="swift">class LoginViewModel {
let username: Observable&lt;String&gt; = Observable(&quot;&quot;)
let password: Observable&lt;String&gt; = Observable(&quot;&quot;)
let loading: Observable&lt;Bool&gt; = Observable(false)
lazy var canSubmit: Computed&lt;Bool&gt; = Computed { [username = self.username, password = self.password, loading = self.loading] in
return username.value.isEmpty == false &amp;&amp;
password.value.isEmpty == false &amp;&amp;
loading.value == false
}
func doLogin() {
loading.value = true
// ... do login here with `self.username` &amp; `self.password`
}
}
class LoginViewController: UIViewController {
@IBOutlet var usernameField: UITextField!
@IBOutlet var passwordField: UITextField!
@IBOutlet var submitButton: UIButton!
let viewModel = LoginViewModel()
override func viewDidLoad() {
super.viewDidLoad()
// Bind our text fields to the corresponding properties on our view model
usernameField.b_text.bind(viewModel.username)
passwordField.b_text.bind(viewModel.password)
// Bind the `enabled` state of the title button to the `canSubmit` field
submitButton.b_enabled.bind(viewModel.canSubmit)
// Call the `doLogin` function when the `submitButton` is tapped
submitButton.b_onTap.subscribe(viewModel.doLogin)
}
}
</code></pre></div>
We set up a view model (<code>LoginViewModel</code>) that encapsulates all our business logic. Since it has no dependencies on UIKit, it makes it trivial to write tests.
Additionally, because Fisticuffs automatically handles tracking dependencies, it&#39;ll automatically update the submit button&#39;s <code>enabled</code> property any time any of the dependencies changes. This frees the initial developer and any maintainers from tracking those updates manually.
<center>
![In action](/assets/animation.gif)
</center>
<h2>Summary</h2>
Fisticuffs is a great little tool for quickly binding view-models to views. We hope you find it useful, and please check out the project <a href="https://github.com/scoremedia/Fisticuffs">on GitHub</a>!
</description>
</item>
<item>
<title>Sorting by Relative Popularity</title>
<link>https://techblog.thescore.com/2015/12/11/sorting-players/</link>
<pubDate>Fri, 11 Dec 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/12/11/sorting-players</guid>
<description>To drive up user engagement at theScore, we recently introduced an onboarding screen that&#39;s shown when you open the app for the first time. First, you get a list of sports teams that are popular in your area, and the option to subscribe to some of them:
<img src="/assets/follow_teams.png" alt="">
Now based on the teams you choose, it would be nice to also recommend some players for you to follow. So how would one go about choosing which players to recommend out of all those teams?
Looks like I&#39;m gonna have to sort user content again! <a href="https://noamswebsite.wordpress.com/2014/01/23/sorting-posts-by-user-engagement-level-with-elasticsearch-implementation/">last time</a>, I sorted user posts by &quot;interestingness&quot;; this time around I&#39;ll be sorting players from a set of sports teams. Once again we&#39;ll look at why sorting things based on popularity alone is a bad idea, we&#39;ll get a primer on standard deviation, and finally a bit of scripting to put it all together.
Let&#39;s assume we have three six-player teams, where each player has the following number of subscribers:
<div class="highlight"><pre><code class="language-text" data-lang="text">big_team_1:
1. 100 000
2. 110 000
3. 90 000
4. 80 500
5. 140 000
6. 140 500
big_team_2:
1. 120 000
2. 250 000
3. 180 000
4. 135 000
5. 157 000
6. 202 000
small_team:
1. 3 000
2. 100
3. 234
4. 301
5. 250
6. 400
</code></pre></div>
Now let&#39;s consider some properties we want from our algorithm:
<ul>
<li>A compute-once, static value: we don&#39;t want to run our algorithm on every user. We want to give each player in our database a static <code>recommendation_score</code>; a numeric value that is cheap to index.</li>
<li>Simple: the algorithm should be simple and use elementary methods. Recommending players is a small part of the app; there should be little code maintenance involved.</li>
<li>Variety: The purpose of the onboarding process is to get you engaged, so we want to recommend a variety of players from different sports and leagues.</li>
</ul>
<h3>The Naive Approach</h3>
<blockquote>
NOTE: Even though our actual script was written in Ruby, I will be using the <a href="http://julialang.org/" target="_blank">Julia programming language</a> for all my examples. You can find the complete implementation at the bottom of this post.
</blockquote>
The first obvious solution is to simply sort players by their <code>subscription_count</code>. The more popular the player, the more recommendable he is. Here is our naive sorting function:
<div class="highlight"><pre><code class="language-julia" data-lang="julia">function naive_sort(teams)
 Base.sort(
[[team.players for team in teams]...],
by=x -&gt; x.subscribers,
 rev=true
 )[1:5]
end
</code></pre></div>
Which yields:
<div class="highlight"><pre><code class="language-text" data-lang="text">5-element Array{PlayerRecommender.Player,1}:
Big Team 2 Player | subscribers: 250000
Big Team 2 Player | subscribers: 202000
Big Team 2 Player | subscribers: 180000
Big Team 2 Player | subscribers: 157000
Big Team 1 Player | subscribers: 140500
</code></pre></div>
The problem with this approach is that we only get results from the most popular teams. So if you&#39;re a fan of both a very popular NFL team and another team that is not as popular (your local basketball team, perhaps), even the least popular player from the NFL team will be recommended to you, whereas the most popular player from your favorite local basketball team will not show up in your list at all!
<h3>A Better Approach</h3>
What are we really looking for?
Well, I think the players we want to recommend are not necessarily the ones who are most famous, but rather, the ones who are most popular compared to the rest of their team, regardless of how popular that team is (we already know you&#39;re a fan of the team or you wouldn&#39;t have subscribed to it in the first place).
In other words, we want an algorithm that answers the following question:
<blockquote>
Which players deviate the most in popularity from the rest of their team?
</blockquote>
Luckily, there&#39;s a mathematical tool for figuring out just this: <a href="https://en.wikipedia.org/wiki/Standard_deviation" target="_blank">standard deviation</a>.
<hr>
<h4>Standard Deviation tl;dr</h4>
Standard deviation is a pretty straight-forward concept: Take a set of values, and figure out the average value for that set (also known as the <a href="https://en.wikipedia.org/wiki/Arithmetic_mean" target="_blank">arithmetic mean</a>). The standard deviation simply tells us by how much the value of the typical element in our set deviates from the average.
The mathematical representation of this calculation is:
$$ s_N = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (x_i - \overline{x})^2} $$
Where $N$ is the population size, and $\overline{x}$ is the arithmetic mean, which itself is represented by:
$$ \overline{x}_N=\frac{1}{N}\sum_{i=1}^{N} a_i $$
For example, the following two sets have the same average value, but clearly the values are spread out differently:
<div class="highlight"><pre><code class="language-julia" data-lang="julia">close_to_average = [11,8,9,12]
spread_out = [0,1,19,20]
average(close_to_average) == 10
average(spread_out) == 10
standard_deviation(close_to_average) == 1.59
standard_deviation(spread_out) == 9.51
</code></pre></div>
<hr>
Thus we arrive at our simple scoring function. All we need to do is find players whose deviation from the average is substantially higher than that of their teammates:
<div class="highlight"><pre><code class="language-julia" data-lang="julia">function recommend(player, team)
player_dev = player.subscribers - team.average
 if player_dev == 0
 player.recommendation_score = 0
 else
player.recommendation_score = player_dev / team.std_dev
 end
end
</code></pre></div>
Using this scoring function on our original teams, we get the following results:
<div class="highlight"><pre><code class="language-text" data-lang="text">5-element Array{PlayerRecommender.Player,1}:
Small Team Player | subscribers: 3000 | score: 2.2276261544470644
Big Team 2 Player | subscribers: 250000 | score: 1.749555170961297
Big Team 1 Player | subscribers: 140500 | score: 1.3134034577576315
Big Team 1 Player | subscribers: 140000 | score: 1.291753950212176
Big Team 2 Player | subscribers: 202000 | score: 0.6445729577225832
</code></pre></div>
Much better. This time around, our top player actually has the least number of subscribers, but this makes sense, because even though he belongs to a team that is not very popular, his subscription count is tenfold that of his teammates; clearly someone to keep an eye on! (Perhaps a rising star in a college league? Certainly wouldn&#39;t want our recommendation script to ignore that one.)
<h3>Limitations</h3>
This algorithm has many limitations.
<ul>
<li>New players won&#39;t be very well represented (they will by nature have low subscription counts).</li>
<li>All-star teams might result in nobody being particularly recommendable. Though this one might be less of a problem: thanks to our good ol&#39; friend the <a href="https://en.wikipedia.org/wiki/Gaussian_function">bell curve</a>, even among rare anomalies, there are <a href="http://fivethirtyeight.com/features/lionel-messi-is-impossible/" target="_blank">rare anomalies</a>.</li>
</ul>
While there are ways to address these limitations and improve the accuracy of the algorithm (for example, taking into account the rate of change in <code>subscription_count</code>), one has to remember the purpose of this feature: to drive up user engagement during onboarding. Is the added complexity of such changes worth the minimal improvement in the recommendations?
Point is, it&#39;s Friday night and I should go out for a beer now. I&#39;m also looking forward to testing out the enormous Chinese fermentation jug I bought yesterday. It looks something like this, but a LOT bigger:
<img src="/assets/ferm_crock.jpeg" alt="">
And it was only $30. What a bargain.
<hr />
Here is the code used in these examples (working as of Julia 0.4.1). Our actual code at theScore is in Ruby.
<div class="highlight"><pre><code class="language-julia" data-lang="julia">module PlayerRecommender
export Team, Player, big_team_1, big_team_2, small_team
export naive_sort, sort
type Player
 subscribers::Int
 team::AbstractString
 recommendation_score::Float64
 function Player(subscribers)
new(subscribers, &quot;&quot;, 0.0)
 end
end
# Pretty print Player
function Base.show(io::IO, p::Player)
print(io, &quot;$(p.team) $(typeof(p)) | subscribers: $(p.subscribers) | score: $(p.recommendation_score)&quot;)
end
type Team
 players::Array{Player}
 std_dev::Float64
 average::Float64
 name::AbstractString
 function Team(name, players)
 for p in players
 p.team = name
 end
new(players, 0.0, 0.0, name)
 end
end
function naive_sort(teams)
 Base.sort(
[[team.players for team in teams]...],
by=x -&gt; x.subscribers,
 rev=true
 )[1:5]
end
function sort(teams)
 map(set_subscriptions_avg, teams)
 map(std_dev, teams)
 map(recommend, teams)
 Base.sort(
[[team.players for team in teams]...],
by=x -&gt; x.recommendation_score,
 rev=true
 )[1:5]
end
# Private
function average(xs)
reduce(+, xs) / length(xs)
end
function dev_from_average(xs, average)
map(x -&gt; x - average, xs)
end
function recommend(team)
map( x -&gt; recommend(x, team), team.players)
end
function recommend(player, team)
player_dev = player.subscribers - team.average
 if player_dev == 0
 player.recommendation_score = 0
 else
player.recommendation_score = player_dev / team.std_dev
 end
end
function set_subscriptions_avg(team)
team.average = average([p.subscribers for p in team.players])
end
function squares(xs)
map(x -&gt; x * x, xs)
end
function std_dev(team)
team.std_dev = std_dev([p.subscribers for p in team.players], team.average)
end
function std_dev(xs, average)
sqrt(reduce(+, squares(dev_from_average(xs, average))) / length(xs))
end
function subscriptions_avg(team)
team.average = average(float([x.subscribers for x in team.players]))
end
end
</code></pre></div></description>
</item>
<item>
<title>Jbuilder to AMS - A tested journey</title>
<link>https://techblog.thescore.com/2015/11/27/jbuilder-to-ams-a-tested-journey/</link>
<pubDate>Fri, 27 Nov 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/11/27/jbuilder-to-ams-a-tested-journey</guid>
<description><a href="https://blog.codeship.com/building-a-json-api-with-rails-5/">Rails 5</a> is coming and ships with the <code>--api</code> option. This will generate a fresh but slimmed down Rails app whose job is to serve as a JSON API. With this option you&#39;ll get <a href="https://github.com/rails-api/active_model_serializers">Active Model Serializers - AMS</a> installed by default. This is a great library that is only getting better with all of the work that the AMS is doing on it.
Not everyone is creating a new Rails app though... most of us have existing ones, which may use RABL, Jbuilder, or a number of other options to generate JSON responses.
<h2>Knock knock, who&#39;s there? Legacy code.</h2>
Let&#39;s say we have a JSON API written in <a href="https://github.com/rails/jbuilder">Jbuilder</a> (or RABL) and we want to switch it over to using AMS... what&#39;s the first step? And what about after that? In this article I&#39;ll be walking us through converting our app to use AMS through a series of steps to make sure that the actual JSON response doesn&#39;t change.
<h2>Our App</h2>
Our app for this demo will consist of 3 models: League, Team, Player. We&#39;ll be looking at the <code>teams#show</code> endpoint which includes information about the team, the league it is apart of, along with a list of that team&#39;s players.
Our <code>Team</code> model looks like this:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">class Team &lt; ActiveRecord::Base
 belongs_to :league
 has_many :players
end
</code></pre></div>
Our controller is pretty bare bones right now but it does the trick:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">class TeamsController &lt; ApplicationController
 def show
@team = Team.find(params[:id])
 end
end
</code></pre></div>
Now that we&#39;ve got the model and controller out of the way, let&#39;s visit the path <code>/teams/1</code> to take a look at the JSON that is produced:
<div class="highlight"><pre><code class="language-javascript" data-lang="javascript">{
 &quot;id&quot;: 1,
 &quot;name&quot;: &quot;Toronto FC&quot;,
 &quot;updated_at&quot;: &quot;2015-11-27T19:01:49.846Z&quot;,
 &quot;players&quot;: [
 {
 &quot;id&quot;: 1,
 &quot;name&quot;: &quot;Sebastian Giovinco&quot;,
 &quot;position&quot;: &quot;Forward&quot;,
 &quot;nickname&quot;: &quot;Atomic Ant&quot;,
 &quot;jersey_number&quot;: &quot;10&quot;,
 &quot;nationality&quot;: &quot;Italian&quot;,
 &quot;updated_at&quot;: &quot;2015-11-27T19:01:49.852Z&quot;
 },
 {
 &quot;id&quot;: 2,
 &quot;name&quot;: &quot;Sebastian Giovinco&quot;,
 &quot;position&quot;: &quot;Forward&quot;,
 &quot;nickname&quot;: &quot;Atomic Ant&quot;,
 &quot;jersey_number&quot;: &quot;10&quot;,
 &quot;nationality&quot;: &quot;Italian&quot;,
 &quot;updated_at&quot;: &quot;2015-11-27T19:01:49.853Z&quot;
 }
 ],
 &quot;league&quot;: {
 &quot;id&quot;: 1,
 &quot;name&quot;: &quot;MLS&quot;,
 &quot;slug&quot;: &quot;mls&quot;,
 &quot;updated_at&quot;: &quot;2015-11-27T19:01:49.835Z&quot;
 }
}
</code></pre></div>
Yes there are 2 Giovincos... it&#39;s a test app! Here is what our current Jbuilder view looks like to generate that response:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">json.(@team, :id, :name, :updated_at)
json.league do
 json.id @team.league.id
 json.name @team.league.name
 json.slug @team.league.slug
 json.updated_at @team.league.updated_at
end
json.players @team.players do |player|
json.(player, :id, :name, :nationality, :nickname, :jersey_number, :position, :updated_at)
end
</code></pre></div>
<h2>Testing the endpoints</h2>
There will be some serious refactoring involved to migrate from Jbuilder to AMS.
<blockquote>
Code refactoring is the process of restructuring existing computer code - changing the factoring - without changing its external behavior.
</blockquote>
According to that definition that I took off of Wikipedia, refactoring involves changing the code, but not the external behavour. So what is the external behaviour in the case of an endpoint? It is the JSON that the endpoint returns. You could also argue that the speed in which it generates that JSON is also its external behaviour, but for today we&#39;ll be focusing on just making sure the JSON remains the same.
To do this we&#39;ll write some tests! There are many ways to ensure that our JSON responses remain the same, but this is a similar approach to how we&#39;ve been testing our endpoints at theScore. What we&#39;ll do is hit the endpoint, record the result, and from then on test to ensure that what the API is returning is equal to the saved response we have.
Our test will create a few database objects, call the endpoint, and then verify the response:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">require &#39;rails_helper&#39;
RSpec.describe TeamsController, type: :request do
 it &#39;returns correct output for single team&#39; do
 team = create(:team)
create(:player, team: team)
create(:player, team: team)
verify_endpoint &quot;/teams/#{team.id}&quot;
 end
end
</code></pre></div>
The <code>verify_endpoint</code> method isn&#39;t one that comes with rspec... it comes from a small module I wrote to help automate recording the endpoint responses and comparing pre-recorded responses with the current response.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">module RequestHelpers
 FILTERED_FIELDS = [&#39;updated_at&#39;].freeze
 CAPTURE_BASE_PATH = &#39;spec/captures&#39;.freeze
 def verify_endpoint(path)
 get path
 raw_body = response.body
raw_expected = read_or_write_capture(path, raw_body)
 compare_response(raw_body, raw_expected)
 end
 private
 def read_or_write_capture(path, raw_body)
file_name = &quot;#{path.parameterize}.json&quot;
 begin
 read_capture(file_name)
 rescue Errno::ENOENT
 write_capture(file_name, raw_body)
 end
 end
 def read_capture(file_name)
File.read(File.join(CAPTURE_BASE_PATH, file_name))
 end
 def write_capture(file_name, raw_body)
parsed_body = JSON.parse(raw_body)
JSON.pretty_generate(parsed_body).tap do |pretty_json|
File.write(File.join(CAPTURE_BASE_PATH, file_name), pretty_json)
 end
 end
 def compare_response(raw_body, raw_expected)
parsed_body = filter(JSON.parse(raw_body))
parsed_expected = filter(JSON.parse(raw_expected))
expect(parsed_body).to eq(parsed_expected)
 end
 # Remove filtered fields and order keys to make comparisons easier
 def filter(hash)
hash.keys.sort.each_with_object({}) do |key, hsh|
 value = hash[key]
hsh[key] = if value.is_a?(Hash)
 filter(value)
 elsif value.is_a?(Array)
 value.map do |elem|
 filter(elem)
 end
 elsif FILTERED_FIELDS.include?(key)
 :filtered
 else
 value
 end
 end
 end
end
</code></pre></div>
The most complicated method here is the <code>filter</code> method, whose job is basically to filter out fields which will end up changing from test to test, like the <code>updated_at</code> field.
Just to make sure I&#39;m explaining all the code in this example, I&#39;m using <a href="https://github.com/thoughtbot/factory_girl">FactoryGirl</a> to generate the data used in this test. The factories look like this:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">FactoryGirl.define do
 factory :league do
 name &#39;MLS&#39;
 slug &#39;mls&#39;
 end
end
FactoryGirl.define do
 factory :player do
 name &#39;Sebastian Giovinco&#39;
 nickname &#39;Atomic Ant&#39;
 jersey_number &#39;10&#39;
 nationality &#39;Italian&#39;
 position &#39;Forward&#39;
 team
 end
end
FactoryGirl.define do
 factory :team do
 name &#39;Toronto FC&#39;
 league
 end
end
</code></pre></div>
<h2>Time for AMS</h2>
Now that we can verify the responses that our endpoints give, we can create AMS classes. Here are the 3 serializers I have, which should produce the same JSON result.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">class LeagueSerializer &lt; ActiveModel::Serializer
 attributes :id,
 :name,
 :updated_at
end
class PlayerSerializer &lt; ActiveModel::Serializer
 attributes :id,
 :name,
 :position,
 :nickname,
 :jersey_number,
 :nationality,
 :updated_at
end
class TeamSerializer &lt; ActiveModel::Serializer
 attributes :id,
 :name,
 :updated_at
 has_many :players
 belongs_to :league
end
</code></pre></div>
We&#39;ll have to make a small change to the controller so that it uses the AMS classes instead of attempting to find a view.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">class TeamsController &lt; ApplicationController
 def show
@team = Team.find(params[:id])
render json: @team, serializer: TeamSerializer
 end
end
</code></pre></div>
Now it&#39;s time to run the test <code>bundle exec rspec</code>, and to verify that they continue to pass after making our change. It&#39;s easy to forget a field... so let&#39;s leave one out on purpose just to make sure that the tests are working as expected. Plus there is something quite satisfying about making the test turn from red to green.
<div class="highlight"><pre><code class="language-diff" data-lang="diff"> -&quot;league&quot; =&gt; {&quot;id&quot;=&gt;1, &quot;name&quot;=&gt;&quot;MLS&quot;, &quot;slug&quot;=&gt;&quot;mls&quot;, &quot;updated_at&quot;=&gt;:filtered},
 +&quot;league&quot; =&gt; {&quot;id&quot;=&gt;1, &quot;name&quot;=&gt;&quot;MLS&quot;, &quot;updated_at&quot;=&gt;:filtered},
</code></pre></div>
We can see that we forgot the <code>slug</code> field on the league... so we&#39;ll add that to the <code>LeagueSerializer</code> and now the test is passing again.
<h2>Preparing for failure</h2>
We now basically have 2 working ways to generate the JSON in our API... we will eventually get rid of the Jbuilder code but for now let&#39;s leave it. If AMS code gives us problems that we didn&#39;t plan for (performance for example), all it takes is changing our controller to switch back to the previous version while we work out the kinks. After it&#39;s been running in production for a few days and you&#39;re sure there are no issues, feel free to go back and remove the Jbuilder view that is no longer needed.
<h2>Concluding thoughts</h2>
In this post we changed our code from using Jbuilder to AMS, but that wasn&#39;t really the main point. The main point I wanted to get across was both how to test your JSON endpoints, but also how to approach code refactoring in general. Without tests, how do you know the code you refactored still works as expected?
You can get the entire source code used in this article at <a href="https://github.com/leighhalliday/soccer_app">https://github.com/leighhalliday/soccer_app</a>
</description>
</item>
<item>
<title>Go – from playing to production</title>
<link>https://techblog.thescore.com/2015/11/09/go-from-playing-to-production/</link>
<pubDate>Mon, 09 Nov 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/11/09/go-from-playing-to-production</guid>
<description><h2>Proposition</h2>
<ul>
<li>There are so many new languages popping up that it&#39;s hard to keep up with all of them.</li>
<li>It&#39;s nice to be able to quickly evaluate a language by prototyping a small project or two without investing too much time learning it.</li>
<li>Accustomisation to a certain setup/workflow; for example, being able to describe and install dependencies (like Ruby&#39;s <a href="http://bundler.io/gemfile.html">Gemfile</a> for <a href="http://bundler.io">Bundler</a>)</li>
<li>Start simple, evolve to production.</li>
</ul>
If the above holds for you then here&#39;s a practical guide on how to get started with <a href="http://golang.org">Go</a>.
Warning: this is an opinionated approach.
<h2>1. Play</h2>
<a href="https://play.golang.org/">Play web interface</a> offers the fastest way to experiment with <a href="http://golang.org">Go</a>.
It lets you run commands and save your gist for sharing and later use. Useful for both beginners and experienced users.
<h4>Exercises</h4>
<ul>
<li>go to the <a href="https://play.golang.org/">Play web interface</a></li>
<li>press <code>Run</code> to execute</li>
<li>press <code>Share</code> to save your code</li>
</ul>
<h2>2. Tour</h2>
<a href="https://tour.golang.org">Tour</a> walks you through exercises describing facets of <a href="http://golang.org">Go</a>.
Offers solid starting point for a learner.
<h4>Exercises</h4>
<ul>
<li>complete <a href="https://tour.golang.org">Tour</a> &nbsp; (at your convenience)</li>
</ul>
<h2>3. <code>go run</code></h2>
Getting the language running on your machine is the next step once the above options are no longer satisfying.
Given the content of <code>hello_world.go</code>:
<div class="highlight"><pre><code class="language-go" data-lang="go">// hello_world.go
package main
import &quot;fmt&quot;
func main() {
 fmt.Println(&quot;Hello go world!&quot;)
}
</code></pre></div>
Continue with exercises, but before that make sure to clone the <a href="https://github.com/maryanhratson/go-from-playing-to-production">example repo</a>
<h4>Exercises</h4>
<ul>
<li><a href="https://golang.org/doc/install">install Go</a></li>
<li><code>export GOPATH=~/golang &amp;&amp; export PATH=$PATH:$GOPATH/bin &amp;&amp; mkdir $GOPATH</code> set up required environment</li>
<li><code>cd 03-go-run/</code> <a href="https://github.com/maryanhratson/go-from-playing-to-production/tree/e494e7483972971ea2b60ac0791430e6913f9f63/03-go-run">(code)</a></li>
<li>run <code>go run hello_world.go</code> to get the <code>&quot;hello go world&quot;</code> output</li>
<li>run <code>go fmt hello_world.go</code> to auto-format the code</li>
</ul>
<h2>4. <code>go test</code></h2>
Tests help evolve code and ensure it works.
So in order to test that the <code>hello</code> string is generated properly let&#39;s refactor the example and add a test.
<div class="highlight"><pre><code class="language-go" data-lang="go">package main
import &quot;fmt&quot;
func main() {
 fmt.Println(hello(&quot;go&quot;))
}
func hello(one string) string {
return fmt.Sprintf(&quot;hello %s world&quot;, one)
}
</code></pre></div>
and test
<div class="highlight"><pre><code class="language-go" data-lang="go">package main
import &quot;testing&quot;
func TestHello(t *testing.T) {
 exp := &quot;hello go world&quot;
 got := hello(&quot;Go&quot;)
 if exp != got {
t.Errorf(&quot;\nExp: %s\nGot: %s&quot;, exp, got)
 }
}
</code></pre></div>
<h4>Exercises</h4>
<ul>
<li><code>cd 04-go-test</code> <a href="https://github.com/maryanhratson/go-from-playing-to-production/tree/e494e7483972971ea2b60ac0791430e6913f9f63/04-go-test">(code)</a></li>
<li><code>go fmt</code> both files</li>
<li>run <code>go test</code>, notice the failing test</li>
<li>do not fix the test yet</li>
</ul>
Failing test output:
<div class="highlight"><pre><code class="language-console" data-lang="console">--- FAIL: TestHello (0.00s)
 hello_world_test.go:11:
 Exp: hello go world
 Got: hello Go world
FAIL
</code></pre></div>
<h2>5. Continuous testing and Makefile</h2>
<code>Makefile</code> lets you describe automation for routine tasks and save some typing.
In this case we want to have project rebuilt every time a file changes.
<div class="highlight"><pre><code class="language-make" data-lang="make">test:
 go test
re-test:
 reflex -r &#39;\.go&#39; $(MAKE) test
dev-deps:
 go get github.com/cespare/reflex
</code></pre></div>
<h4>Exercise:</h4>
<ul>
<li><code>cd 05-ContinuousTestingAndMakefile</code> <a href="https://github.com/maryanhratson/go-from-playing-to-production/tree/e494e7483972971ea2b60ac0791430e6913f9f63/05-ContinuousTestingAndMakefile">(code)</a></li>
<li>run <code>make dev-deps</code> to install development dependencies</li>
<li>run <code>make re-test</code> and watch the test fail</li>
<li>fix the failing test and see it <code>PASS</code> immediately!</li>
</ul>
<h2>6. Packages, Project Layout</h2>
To demonstrate use of packages and basic project layout, our <code>helloworld</code> will be converted into a library.
NOTE: Currently there&#39;s no official and convenient way to manage dependencies.
<a href="https://getgb.io">Gb</a> was chosen as a tool to govern project layout, building and handling dependencies.
<a href="https://github.com/maryanhratson/go-from-playing-to-production/tree/e494e7483972971ea2b60ac0791430e6913f9f63/06-PackagesAndProjectLayout">New layout</a> now looks like this:
<div class="highlight"><pre><code class="language-console" data-lang="console">tree
.
├── Makefile
├── bin
│ ├── helloworld
│ └── helloworld-server
└── src
 ├── cmd
 │ ├── helloworld
 │ │ └── main.go
 │ └── helloworld-server
 │ └── main.go
 └── helloworld
 ├── hello_world.go
 └── hello_world_test.go
</code></pre></div>
<h4>Changes:</h4>
<ul>
<li>the <a href="https://github.com/maryanhratson/go-from-playing-to-production/blob/e494e7483972971ea2b60ac0791430e6913f9f63/05-ContinuousTestingAndMakefile/hello_world.go#L10-L12"><code>hello</code> function</a> has been renamed to <a href="https://github.com/maryanhratson/go-from-playing-to-production/blob/e494e7483972971ea2b60ac0791430e6913f9f63/06-PackagesAndProjectLayout/src/helloworld/hello_world.go#L6-L8"><code>Hello</code> to be exported</a> and accessible from outside the package</li>
<li><code>helloworld</code> is now a package (<a href="https://github.com/maryanhratson/go-from-playing-to-production/blob/e494e7483972971ea2b60ac0791430e6913f9f63/06-PackagesAndProjectLayout/src/helloworld/hello_world.go#L2">note <code>package helloworld</code></a> instead <code>package main</code> in <code>src/helloworld/*.go</code>)</li>
<li>there are now separate &quot;entry points&quot; (or executables) source of which located under <code>src/cmd</code></li>
<li><code>Makefile</code> now has a <code>build</code> rule that builds the entry points/executables to <code>bin</code> directory</li>
<li><a href="https://getgb.io">Gb</a> is our new <a href="https://github.com/maryanhratson/go-from-playing-to-production/blob/128a56179c6a51a79a069e1e77a0418d16a366a1/06-PackagesAndProjectLayout/Makefile#L14">development dependency</a></li>
</ul>
<h4>Exercise:</h4>
<ul>
<li><code>cd 06-PackagesAndProjectLayout</code> <a href="https://github.com/maryanhratson/go-from-playing-to-production/tree/e494e7483972971ea2b60ac0791430e6913f9f63/06-PackagesAndProjectLayout">(code)</a></li>
<li>run <code>make dev-deps</code> to install new dependencies</li>
<li>run <code>make re-test</code>, notice failing test</li>
<li>fix the failing test</li>
<li>run <code>make build</code> to get binaries built</li>
<li>run <code>bin/helloworld</code> ensure it prints <code>hello go world</code></li>
<li>run <code>bin/helloworld-server</code> and point your browser to <a href="http://localhost:8080/?name=Go">http://localhost:8080/?name=Go</a> to get a web &quot;hello Go world&quot;</li>
</ul>
<h2>7. Dependencies</h2>
Let&#39;s now add a dependency and describe the process.
In this example we&#39;re going to use <a href="https://github.com/julienschmidt/httprouter">httprouter</a> to add a nicer route for our web server hello example.
Adding a dependency with <a href="https://getgb.io">Gb</a> is running a command:
<div class="highlight"><pre><code class="language-console" data-lang="console">gb vendor fetch github.com/julienschmidt/httprouter
</code></pre></div>
As result we now have <code>vendor/</code> directory that looks like this:
<div class="highlight"><pre><code class="language-console" data-lang="console">tree -L 4 vendor/
vendor/
├── manifest
└── src
 └── github.com
 └── julienschmidt
 └── httprouter
</code></pre></div>
where <a href="https://github.com/maryanhratson/go-from-playing-to-production/blob/e494e7483972971ea2b60ac0791430e6913f9f63/07-Dependencies/vendor/manifest"><code>vendor/manifest</code></a> is the file which tracks dependencies added with <code>gb vendor fetch</code> command.
Check <a href="https://github.com/maryanhratson/go-from-playing-to-production/blob/e494e7483972971ea2b60ac0791430e6913f9f63/07-Dependencies/vendor/manifest"><code>vendor/manifest</code></a> into your <a href="https://en.wikipedia.org/wiki/Version_control">SCM</a> to be able to retrieve dependencies later with <code>gb vendor restore</code> (without having to check them in to <a href="https://en.wikipedia.org/wiki/Version_control">SCM</a>).
Now that that dependency is ready to be used let&#39;s update the <a href="https://github.com/maryanhratson/go-from-playing-to-production/blob/e494e7483972971ea2b60ac0791430e6913f9f63/07-Dependencies/src/cmd/helloworld-server/main.go">server code</a> to look like:
<div class="highlight"><pre><code class="language-go" data-lang="go">package main
import (
 &quot;fmt&quot;
 &quot;helloworld&quot;
 &quot;net/http&quot;
 &quot;github.com/julienschmidt/httprouter&quot;
)
func main() {
 router := httprouter.New()
 router.GET(&quot;/&quot;, index)
router.GET(&quot;/:name&quot;, index)
http.ListenAndServe(&quot;:8080&quot;, router)
}
func index(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
name := ps.ByName(&quot;name&quot;)
 if name = &quot;&quot; {
 name = &quot;Unknown&quot;
 }
fmt.Fprint(w, helloworld.Hello(name))
}
</code></pre></div>
<h4>Changes:</h4>
<ul>
<li>import section now includes vendored dependency <code>&quot;github.com/julienschmidt/httprouter&quot;</code></li>
<li>new router <code>router := httprouter.New()</code></li>
<li>new route with named parameter <code>/:name</code></li>
<li><code>index</code> function signature change(as required by the router API)</li>
</ul>
<h4>Exercise:</h4>
<ul>
<li>re-build and run <code>bin/helloworld-server</code></li>
<li>point your browser to <a href="http://localhost:8080/Go">http://localhost:8080/Go</a> and notice &quot;Hello Go World&quot; response</li>
<li>add a <code>Makefile</code> rule that builds and runs the server</li>
<li>add a <code>Makefile</code> rule that re-builds server and runs on a file change</li>
<li>check-in <code>vendor/manifest</code> file to your <a href="https://en.wikipedia.org/wiki/Version_control">SCM</a></li>
<li>write <code>Makefile</code> rule to restore dependencies</li>
</ul>
<h2>8. Deployment</h2>
Deploys are just copying the binary to a server and launching it once it&#39;s built for the platform it&#39;s going to be running on.
Building could be done through <a href="http://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5">cross-compilation</a> or on a build machine.
But it&#39;s left as an exercise to the reader ;)
<h2>References</h2>
<ul>
<li><a href="https://github.com/maryanhratson/go-from-playing-to-production">repo with example code</a></li>
<li>Official <a href="https://golang.org/doc/code.html#GOPATH">GOPATH</a> description</li>
<li>Official <a href="https://golang.org/doc/install">Getting Started Guide</a></li>
<li>Official <a href="https://golang.org/doc/code.html">How to Write Go Code</a></li>
<li><a href="https://golang.org/pkg/testing/">testing package documentation</a></li>
<li>&quot;<a href="http://blog.golang.org/6years">an official package vendoring mechanism</a>&quot; should be announced with Go 1.6</li>
<li><a href="https://golang.org/doc/">more docs</a></li>
</ul>
Thanks for reading and <a href="http://blog.golang.org/6years">happy birthday to <code>Go</code></a>!
</description>
</item>
<item>
<title>Timezones in Python</title>
<link>https://techblog.thescore.com/2015/11/03/timezones-in-python/</link>
<pubDate>Tue, 03 Nov 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/11/03/timezones-in-python</guid>
<description>After struggling for a while to determine how to convert from one timezone to another in Python I thought I&#39;d write a quick little guide on it. Even if the only person it helps is me a few months from now I&#39;ll consider it a success!
It all started when I received the error <code>ValueError: Not naive datetime (tzinfo is already set)</code>. A little bit later I received the error <code>ValueError: astimezone() cannot be applied to a naive datetime</code>. It was about this time that I realized I needed to learn what was actually going on.
<h2>The naive datetime</h2>
Datetimes in Python can either be <code>Naive</code> or <code>Aware</code>. A naive datetime is a datetime that doesn&#39;t know its own timezone. It could be in <code>UTC</code>, it could be in <code>US/Eastern</code>, but it doesn&#39;t know. All it knows is which date and time that it has.
<div class="highlight"><pre><code class="language-python" data-lang="python">import datetime
date = datetime.datetime(2015, 11, 11, 11, 0)
date.tzinfo # None
date.strftime(&quot;%a %b %d %H:%M:%S %Y&quot;) # Wed Nov 11 13:00:00 2015
</code></pre></div>
At this point we don&#39;t know if it is 11AM EST, UTC... it is up for the programmer to keep track of which timezone they are working in.
<h2>Making it aware</h2>
To make our naive datetime aware of its timezone, we can use the <code>pytz</code> library. Let&#39;s make our date be in UTC time. We&#39;ll do this by using the <code>localize</code> method of a timezone.
<div class="highlight"><pre><code class="language-python" data-lang="python">import pytz
utc = pytz.timezone(&#39;UTC&#39;)
aware_date = utc.localize(date)
aware_date.tzinfo # &lt;UTC&gt;
aware_date.strftime(&quot;%a %b %d %H:%M:%S %Y&quot;) # Wed Nov 11 13:00:00 2015
</code></pre></div>
<h2>Converting to another timezone</h2>
Now let&#39;s take our aware date (in UTC) and convert it to a date in the US/Eastern timezone. To do this we can again use <code>pytz</code> along with a method of <code>datetime</code> called <code>astimezone</code>.
<div class="highlight"><pre><code class="language-python" data-lang="python">eastern = pytz.timezone(&#39;US/Eastern&#39;)
eastern_date = aware_date.astimezone(eastern)
eastern_date.tzinfo # &lt;DstTzInfo &#39;US/Eastern&#39; EST-1 day, 19:00:00 STD&gt;
eastern_date.strftime(&quot;%a %b %d %H:%M:%S %Y&quot;) # Wed Nov 11 08:00:00 2015
</code></pre></div>
<h2>Wrapping things up</h2>
We started with a naive date which was unaware of which timezone it was in. We made it aware by localizing it to the UTC timezone, and then we took that aware date and converted it into the US/Eastern timezone.
</description>
</item>
<item>
<title>Respect HTTP Caching</title>
<link>https://techblog.thescore.com/2015/09/24/respect-http-caching/</link>
<pubDate>Thu, 24 Sep 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/09/24/respect-http-caching</guid>
<description>API developers place plenty of consideration on how to handle scaling. A few techniques and approaches: Application caching, sharding, larger/more servers, background queues, HTTP caching, and Content Delivery Networks (CDNs). A well architected API employs a combination of these solutions to achieve the desired scalability. Of the scaling solutions, very few are actually exposed to the consumers of an API. One strategy in which the end-users actively participate in is HTTP Caching.
By respecting HTTP caching on a client&#39;s application, network communications complete quicker and in some cases are unnecessary. Network response payloads are sometimes omitted due to a 304 HTTP response. Finally if you use HTTP caching, application servers will love you.
<h2>HTTP Caching</h2>
If an endpoint is to be hit repeatedly, HTTP caching can dramatically reduce the load on the server. There are a number of ways one can use HTTP caching. Similar to other scaling techniques some of these can be used in tandem.
<h3>Cache-Control</h3>
On an HTTP response a <code>cache-control</code> header may present. This header provides the client insight on how they should approach making similar requests. For example:
<div class="highlight"><pre><code class="language-text" data-lang="text">Cache-Control: max-age=60, must-revalidate
</code></pre></div>
The <code>max-age=60</code> specifies that the request should be cached and used locally for a maximum duration of one minute. The <code>must-revalidate</code> forces subsequent requests to still communicate to the server to respect the freshness of the response (i.e., information which indicates that the content of the response has updated). These two simple cache-control headers provide both non-stale responses with reduced payload, provided the endpoint is respecting the max-age (i.e., not changing frequently).
To better understand cache-controls, our coworker at theScore, Thuva Tharam, has an <a href="http://techblog.thescore.com/2014/11/19/are-your-cache-control-directives-doing-what-they-are-supposed-to-do/">excellent blog post</a> written on the subject. The individual cache-control directives and their effects are described. Suggestions and examples are provided to illustrate how to use cache-control effectively.
<h3>Last-Modified</h3>
As previously mentioned, it is possible for a cached request to <code>re-validate</code> with the server to check for freshness. Subsequent requests have an additional <code>If-Modified-Since</code> which contains the value of the response&#39;s <code>Last-Modified</code> header. This allows the server to compare the provided time on subsequent requests against the responses and return the full response or simply a 304 status code.
When a server returns a full response it takes more time to compute/collect the necessary information along with rendering time for the response. With the correct <code>cache-control</code> along with a <code>Last-Modified</code>, a server spends less time constructing the response. A 304 status code response is quick, and has a small payload size compared to a normal full response.
<h3>ETags</h3>
Even when the <code>Last-Modified</code> value has changed, the actual content might still be the same. If the server takes advantage of ETags then there will be an <code>ETag</code> header on the response. The benefit of using ETags shines on requests that do not change often. ETag caching does not expire based on time like the previous two caching mechanisms, and is instead a function of the response&#39;s content. This form of caching is ideal as it provides caching while still allowing for freshness of updates to the content.
When a subsequent request is made a <code>If-None-Match</code> header is sent along with the previous request&#39;s ETag. The server will then use the ETag to determine freshness. If the ETag of the request match the that of the server&#39;s response a 304 status code is returned.
<h2>Respecting HTTP Caching for API Consumers</h2>
<blockquote>
API Consumers -- Client applications that makes more than one HTTP request to an external web service
</blockquote>
API Consumers can take on different forms such as a browser or desktop/mobile/web application.
<h3>Browsers</h3>
Modern browsers have no problem handling HTTP caching. It is also easy to verify that they are respecting HTTP caching by using their Developer Tools interface. The following image shows two requests made to the same API endpoint (https://api.github.com/users/kevinjalbert) in Google Chrome&#39;s Network Tab of the Developer Tools:
First API request results in a 200 status code (standard success response):
<img src="/assets/200-status-code-chrome.png" alt="First Request results in a Status Code 200">
Second API request results in a 304 status code (resource has not been modified since the last request, no body is returned in response):
<img src="/assets/304-status-code-chrome.png" alt="Second Request results in a Status Code 304">
<h3>Mobile Applications</h3>
Android and iOS have native mechanisms in place that developers can use to handle HTTP caching on any outbound requests. On iOS there is <a href="https://developer.apple.com/library/mac/documentation/Cocoa/Reference/Foundation/Classes/NSURLCache_Class/index.html#//apple_ref/occ/cl/NSURLCache">NSURLCache</a> and on Android there is <a href="http://developer.android.com/reference/android/net/http/HttpResponseCache.html">HttpResponseCache</a>. If possible all HTTP requests made from mobile devices should be using the caching mechanisms.
<h3>Desktop Applications</h3>
Many programming languages can be used when creating desktop applications. In these cases it is best to check whether there is a built HTTP request caching mechanism akin to what Android and iOS have.
<h3>Web Applications</h3>
It is not uncommon to have web services talking to other web services through a defined API. In these situations there might not be a defined networking layer to push API requests through. For example, within a Ruby/Rails application one might use <a href="https://github.com/lostisland/faraday">Faraday</a> to handle network connections. HTTP caching is not enabled by default on Faraday and is instead an opt-in option.
The following is an example of using Faraday with <a href="https://github.com/plataformatec/faraday-http-cache">Faraday::HttpCache</a> middleware to handle HTTP caching.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">require &#39;active_support&#39;
require &#39;faraday&#39;
require &#39;faraday-http-cache&#39;
module ApiConsumer
 class Requester
 def initialize(cached: true)
@connection = Faraday.new(&#39;https://api.github.com/&#39;) do |conn|
 if cached
 conn.use Faraday::HttpCache,
logger: Logger.new(STDOUT),
 instrumenter: ActiveSupport::Notifications
 end
 conn.adapter Faraday.default_adapter
 end
 end
def get(method, params = {})
 start_time = Time.now
response = @connection.get do |request|
 request.url(method, params)
 end
time_taken_ms = (Time.now - start_time) * 1000
puts &quot;Time Taken: #{time_taken_ms.round(3)} ms&quot;
 response
 end
 end
end
# Subscribes to all Faraday::HttpCache events
ActiveSupport::Notifications.subscribe &quot;http_cache.faraday&quot; do |*args|
event = ActiveSupport::Notifications::Event.new(*args)
puts &quot;Cache Status: #{event.payload[:cache_status]}&quot;
end
puts &quot;Faraday with HTTP Caching&quot;
api_requester = ApiConsumer::Requester.new(cached: true)
5.times do
 api_requester.get(&#39;users/kevinjalbert&#39;)
 sleep 2
end
puts &#39;&#39;
puts &quot;Faraday without HTTP Caching&quot;
api_requester = ApiConsumer::Requester.new(cached: false)
5.times do
 api_requester.get(&#39;users/kevinjalbert&#39;)
 sleep 2
end
</code></pre></div>
Executing the above example produces the following log statements. With caching the time taken per request decreases tremendously on subsequent requests. By respecting the HTTP cache-controls of the API, the subsequent request can be used until the cache-control directives say otherwise (i.e., ETag/Last-Modified changes, or the cache expires exceeding the max-age).
<div class="highlight"><pre><code class="language-text" data-lang="text">Faraday with HTTP Caching
D, [2015-09-20T23:06:27.570751 #15189] DEBUG -- : HTTP Cache: [GET /users/kevinjalbert] miss, store
Cache Status: miss
Time Taken: 1396.139 ms
D, [2015-09-20T23:06:29.577798 #15189] DEBUG -- : HTTP Cache: [GET /users/kevinjalbert] fresh
Cache Status: fresh
Time Taken: 1.857 ms
D, [2015-09-20T23:06:31.580719 #15189] DEBUG -- : HTTP Cache: [GET /users/kevinjalbert] fresh
Cache Status: fresh
Time Taken: 0.971 ms
D, [2015-09-20T23:06:33.585455 #15189] DEBUG -- : HTTP Cache: [GET /users/kevinjalbert] fresh
Cache Status: fresh
Time Taken: 1.139 ms
D, [2015-09-20T23:06:35.587833 #15189] DEBUG -- : HTTP Cache: [GET /users/kevinjalbert] fresh
Cache Status: fresh
Time Taken: 1.133 ms
Faraday without HTTP Caching
Time Taken: 807.486 ms
Time Taken: 277.589 ms
Time Taken: 186.115 ms
Time Taken: 475.249 ms
Time Taken: 1130.691 ms
</code></pre></div></description>
</item>
<item>
<title>Faster RSpec Regression Testing</title>
<link>https://techblog.thescore.com/2015/09/21/faster-rspec-regression-testing/</link>
<pubDate>Mon, 21 Sep 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/09/21/faster-rspec-regression-testing</guid>
<description><h2>Testing Approaches</h2>
<ol>
<li>Immediately on the code that is being developed to help guide development. Generally using a subset of the test suite.</li>
<li>Afterwards on the complete codebase to ensure no regressions appear. This is always done using the complete test suite.</li>
</ol>
In an ideal environment, the complete test suite is run after every meaningful code change. This approach works for a small project, but it can grow into a time consuming process. Developers may only run a subset of tests that exercise the modified source code. Faster test execution allows for tighter feedback loops between testing and developing.
Eventually the active development stops as the feature is completed. A developer wants to have a high-level of confidence in the code that they wrote before committing the changes. By running the complete test suite, one can have a certain level of assurance that the code to be committed will not cause issues. Ideally this second phase of testing is carried out locally by the developer, but also on a continuous integration server to prevent regressions.
<h2>RSpecing <code>it</code> up</h2>
RSpec tests are executed using the <code>it</code> method. The code within the <code>it</code> block is executed after applying context using <code>let</code> and <code>before</code>, which can be seen as the pre-amble of the test. In other words, for every <code>it</code> block the test will execute the required pre-amble before the actual <code>it</code> block.
<h3>Many <code>it</code> blocks</h3>
During development it is useful to use the following technique to understand as much detail of failing tests:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"> # A trivial example to show the verification of
 # multiple conditions using many &#39;it&#39; blocks
 RSpec.describe &#39;test with many it blocks&#39; do
it { expect(&#39;a&#39;.upcase).to eq(&#39;A&#39;) }
it { expect(&#39;c&#39;.upcase).to eq(&#39;B&#39;) }
it { expect(&#39;b&#39;.upcase).to eq(&#39;C&#39;) }
 end
</code></pre></div><div class="highlight"><pre><code class="language-text" data-lang="text">Failures:
1) test with many it blocks should eq &quot;B&quot;
Failure/Error: it { expect(&#39;b&#39;.upcase).to eq(&#39;B&#39;) }
expected: &quot;B&quot;
got: &quot;C&quot;
2) test with many it blocks should eq &quot;C&quot;
Failure/Error: it { expect(&#39;b&#39;.upcase).to eq(&#39;C&#39;) }
expected: &quot;C&quot;
got: &quot;B&quot;
</code></pre></div>
The results of the testing present all of the failures. This approach is important during iterative development as fixing one test might cause others to fail.
<h3>Combining <code>it</code> blocks</h3>
Another approach that can be used with RSpec is to combine the many <code>it</code> blocks into one:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"> # A trivial example to show the verification of
 # multiple conditions using a combined &#39;it&#39; block
 RSpec.describe &#39;test with a combined it block&#39; do
 it do
expect(&#39;a&#39;.upcase).to eq(&#39;A&#39;)
expect(&#39;c&#39;.upcase).to eq(&#39;B&#39;)
expect(&#39;b&#39;.upcase).to eq(&#39;C&#39;)
 end
 end
</code></pre></div><div class="highlight"><pre><code class="language-text" data-lang="text">Failures:
1) test with a combined it block should eq &quot;B&quot;
Failure/Error: expect(&#39;c&#39;.upcase).to eq(&#39;B&#39;)
expected: &quot;B&quot;
got: &quot;C&quot;
</code></pre></div>
This time the results present only the first failure, even though two assertions are incorrect. This approach provides less information than the first example. Although there is a different benefit, it runs less examples (tests and their complete pre-amble). This approach leads to quicker test executions, with the loss of the complete listing of failures. Arguably this approach is better suited when a feature is complete and the tests remain present to detect regressions.
<h2>Performance Impact</h2>
In the previous section, two styles of using RSpec&#39;s <code>it</code> blocks were illustrated. The next example showcases both approaches to understand how the testing performance changes. To simulate some pre-amble (i.e., database interactions, setting up objects) sleeps are used when accessing the values from their <code>let</code> blocks.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">require &#39;rspec&#39;
class AttributeObject
attr_reader :value1, :value2, :value3, :value4
def initialize(value1, value2, value3, value4)
 @value1 = value1
 @value2 = value2
 @value3 = value3
 @value4 = value4
 end
end
RSpec.describe &#39;performance using `it` blocks&#39; do
subject { AttributeObject.new(value1, value2, value3, value4) }
let(:value1) { sleep 0.1; 1 }
let(:value2) { sleep 0.1; 2 }
let(:value3) { sleep 0.1; 3 }
let(:value4) { sleep 0.1; 4 }
context &#39;slow tests due to many `it` blocks&#39; do
it { expect(subject.value1).to eq(value1) }
it { expect(subject.value2).to eq(value2) }
it { expect(subject.value3).to eq(value3) }
it { expect(subject.value4).to eq(value4) }
 end
context &#39;faster tests due to a combined `it` block&#39; do
 it do
expect(subject.value1).to eq(value1)
expect(subject.value2).to eq(value2)
expect(subject.value3).to eq(value3)
expect(subject.value4).to eq(value4)
 end
 end
end
</code></pre></div>
For this example it is possible to use both testing approaches. There are two criteria that must be satisfied to allow this transformation. For each <code>it</code> block:
<ul>
<li>The pre-amble is the same. In the example they are all instantiating the <code>AttributeObject</code> with the four value arguments.</li>
<li>The contents are not mutating the environment in a different way from each other. If an <code>it</code> block mutates the environment then the many <code>it</code> block approach might have unexpected results. In the example each <code>it</code> block contains only a single assertion.</li>
</ul>
<div class="highlight"><pre><code class="language-text" data-lang="text"># Many &#39;it&#39; blocks
Finished in 1.67 seconds (files took 0.09327 seconds to load)
4 examples, 0 failures
# Combined &#39;it&#39; block
Finished in 0.41675 seconds (files took 0.09545 seconds to load)
1 example, 0 failures
</code></pre></div>
It is clear to see that the combined <code>it</code> block executes less examples and therefore requires less time. Both of these approaches are testing the exact same thing, however one does so more quickly than the other. There is still the loss of the precise failure reporting per <code>it</code> block. It can be argued that it is not a huge loss given the code under test has been developed and the tests are for regressions.
<h3>Larger Scale Test Suite</h3>
To better illustrate the benefits of using combined <code>it</code> blocks when possible, I have applied this technique to a larger project. The conversion between the two approaches was straight-forward and the results are satisfying. It was possible to consolidated many <code>it</code> blocks (300 less test examples), reducing the run time by 38%.
<div class="highlight"><pre><code class="language-text" data-lang="text"># Before
Finished in 3 minutes 6.3 seconds (files took 10.06 seconds to load)
1506 examples, 0 failures
# After
Finished in 1 minute 55.27 seconds (files took 10.11 seconds to load)
1206 examples, 0 failures
</code></pre></div>
The results are going to differ between projects and testing styles, although in most cases there will be some performance improvements.
<h2>Suggested Testing Workflow</h2>
The following workflow is the one I follow as it provides the best of both worlds:
<ol>
<li>Iteratively develop feature using many <code>it</code> blocks for feedback locally</li>
<li>Complete feature</li>
<li>Convert many <code>it</code> blocks into a combined <code>it</code> block for regressions</li>
<li>Merge feature into master</li>
</ol>
Eventually some issue may come up and a regression might be found. If more work is involved to fix and understand the issue at hand I might split the combined <code>it</code> block to provide more feedback on failures. At this point I would go back to following the workflow from the start.
</description>
</item>
<item>
<title>One Repo to Rule Them All</title>
<link>https://techblog.thescore.com/2015/09/18/one-repo-to-rule-them-all/</link>
<pubDate>Fri, 18 Sep 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/09/18/one-repo-to-rule-them-all</guid>
<description>This week it&#39;s been a hot topic of discussion about how Google has (almost) all of their code within a single, massive, 2-billion lines of code repository.
There was a quote by <a href="https://twitter.com/searls">Justin Searls</a>, which was most definitely him joking around, but it got me thinking about how we share code at theScore.
There&#39;s the common misconception that you either have a &quot;Monolith&quot; or series of &quot;Microservices&quot;. Or as somebody else put it on Twitter: &quot;A swarm of loosely coupled nanoservices&quot;.
<blockquote class="twitter-tweet" lang="en">Microliths.&mdash; Justin Searls (@searls) <a href="https://twitter.com/searls/status/644646188801724416">September 17, 2015</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
We certainly have a number of repos (iOS app, Android app, website, data and API, user accounts and authentication API, etc...), but the &quot;data and API&quot; repo in particular has a number of related Rails apps along with some local gems.
<h2>Sharing code across Rails apps</h2>
The repository I mentioned has 4 different Rails apps inside of it. It has an app for the API, an app for an internal admin panel, an app for processing incoming data, and an app for handling push alerts.
All of these apps share 2 gems which are also found inside the same repository: a gem for the &quot;data&quot; layer (basically the models and objects for interacting with the database), and a support gem which has a few utility classes that are shared, along with some things like lists of country codes and whatnot.
The repository is structured like this:
<ul>
<li>data_processor/ (rails app)</li>
<li>admin/ (rails app)</li>
<li>api/ (rails app)</li>
<li>pusher/ (rails app)</li>
<li>thescore-data/ (gem)</li>
<li>thescore-support/ (gem)</li>
<li>test (script)</li>
</ul>
Each of the 4 Rails apps can have their own gems and dependencies, two of which being the local data and support gems.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># Gemfile
gem &#39;the_score_support&#39;, path: &#39;../thescore-support&#39;
gem &#39;the_score_data&#39;, path: &#39;../thescore-data&#39;
</code></pre></div>
<h2>Decreasing coupling without increasing complexity</h2>
Monoliths are great in certain respects: They allow you to have all of your code inside a single app. If you need to &quot;communicate&quot; to another part of the system, you can do so via a method call (fast, simple) rather than by communicating over HTTP with a JSON request (slow, complex). There is also no need to attempt to version your API... there is a single version and that is the app itself.
But there are obvious drawbacks as well: You can have seemingly unrelated things all packed into the same place (the API along with the admin panel for example), which both may require a number of gems that the other doesn&#39;t need.
Microservices aim to solve some of the problems brought on by a monolith, but end up causing an increase in complexity by having to coordinate APIs across the different services, versioning them and deploying them at the same time.
By combining related &quot;Microservices&quot; into a single repository, you end up getting the benefits from both approaches, but without the drawbacks that each one provides.
<h2>Benefits from testing</h2>
By having all of your apps in a single service you are also able to easily test all of your applications together. You can either run them individually, because each one has their own set of tests, or you can write a simple script which will test each app along with each gem, so that you know that your changes to the data gem hasn&#39;t broken anything in one of the Rails apps.
<div class="highlight"><pre><code class="language-bash" data-lang="bash">#!/bin/bash
export RAILS_ENV=test
export RACK_ENV=test
trap exit 1 SIGINT SIGTERM
# The BASE_DIR is the absolute path of the directory of this script file
BASE_DIR=$(cd $(dirname ${BASH_SOURCE[0]}) &amp;&amp; pwd)
FAILED=0
for project in admin api thescore-data pusher thescore-support data_processor; do
 cd $BASE_DIR/$project
 echo &quot;Running specs for $project with seed $SEED&quot;
 bin/rspec
if [ $? -ne 0 ]; then
 FAILED=1
 fi
done
exit $FAILED
</code></pre></div>
Better yet, you can set up your continuous integration tools to tets your entire project each time you push to the repository, helping to guarantee that all apps in this repo are functioning correctly.
<h2>A single pull request</h2>
Another benefit of having everything within a single repo is that only a single pull request is needed for a change which might span multiple apps and/or gems.
If you&#39;ve ever had to coordinate multiple PRs which must be approved, merged, and deployed in sync, you know how difficult and annoying it can be. It&#39;s not just difficult for you though, it&#39;s difficult for whoever is having to review the PR as well. The reviewers need to be made aware that there are multiple PRs in different repositories, and keep that in mind when reviewing each one individually.
Combining apps and gems into a single repo helps to solve this, and PRs become a lot easier to manage.
<h2>Final thoughts</h2>
You don&#39;t have to go as far as Google, where everything your organization done is in a single repository. But by combining separate apps in a single repository, you&#39;re able to get some of the advantages of having Microservices while not throwing out all the advantages of a Monolith. Or, as Justin Searls said, Microliths.
</description>
</item>
<item>
<title>Database Transactions With pytest</title>
<link>https://techblog.thescore.com/2015/06/25/database-transactions-with-pytest/</link>
<pubDate>Thu, 25 Jun 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/06/25/database-transactions-with-pytest</guid>
<description>At theScore, most of our projects are built on Rails so we&#39;re used to having testing tools built-in. We primarily use RSpec with Rails and that combo gives us a few things by default:
<ol>
<li>Test discovery under <code>spec/</code> or <code>test/</code></li>
<li>Automatic database transaction support per test</li>
</ol>
We have an old, custom Python project that we&#39;ve been working with and slowly modernizing. The need for unit tests came up and I evaluated a few options. I ended up choosing <a href="https://pytest.org/">pytest</a> as it seemed to be most modern and popular framework.
By default, pytest is configured to discover tests in all directories and subdirectories. So, while it does work, it can be slow (because it crawls everything) and it can discover tests that it shouldn&#39;t (such as a vendored code).
I worked around this issue by using the <code>norecursedirs</code> option. This option tells pytest which directories not to go into (no option exists to tell it to only go into certain directories). Here&#39;s a sample config:
<div class="highlight"><pre><code class="language-ini" data-lang="ini"># setup.cfg
[pytest]
norecursedirs = .git vendor my-project/lib my-project/helpers docs config log tmp\*
</code></pre></div>
Note, if you put <code>vendor</code>, then make sure you don&#39;t have a directory like <code>tests/vendor/</code> because it will be ignored:
<div class="highlight"><pre><code class="language-sh" data-lang="sh">$ cat setup.cfg
[pytest]
norecursedirs = config
$ tree
.
├── config
│ └── file.json
├── file_name.py
├── setup.cfg
└── tests
 ├── config
 │ └── my_test.py
 └── my_test.py
4 directories, 6 files
$ py.test --collect-only
======= test session starts =======
platform darwin -- Python 2.7.9 -- py-1.4.27 -- pytest-2.6.4
collected 1 items
&lt;Module &#39;tests/my_test.py&#39;&gt;
 &lt;Class &#39;Test&#39;&gt;
 &lt;Instance &#39;()&#39;&gt;
 &lt;Function &#39;test_foo&#39;&gt;
</code></pre></div>
Anyway, buyer be warned!
To solve #2 (automatic database transaction support per test), it got... a bit tricky. We utilized standard xUnit style tests, so our tests would look like this:
<div class="highlight"><pre><code class="language-python" data-lang="python">class TestWidget:
def setup_method(self, method):
self.subject = Widget()
def test_can_get_meaning_of_life(self):
assert self.subject.get_meaning_of_life() == 42
</code></pre></div>
Say that instantiating <code>Widget</code> actually wrote to the database. At this point, the widget is going to be saved in the database and subsequent tests could error out because they rely on having no widgets in the database.
Our first solution used inheritance like this:
<div class="highlight"><pre><code class="language-python" data-lang="python"># tests/helper.py
class BaseTest:
def setup_method(self, method):
orm.session.begin()
def teardown_method(self, method):
orm.session.rollback()
# tests/unit/widget_test.py
from tests.helper import BaseTest
class TestWidget(BaseTest):
def setup_method(self, method):
super(BaseTest, self).setup_method(method)
self.subject = Widget()
def test_can_get_meaning_of_life(self):
assert self.subject.get_meaning_of_life() == 42
</code></pre></div>
This works for the most part, except that <code>teardown_method</code> does not get called if something failed in <code>setup_method</code> <a href="https://pytest.org/latest/announce/release-2.4.0.html">by design since pytest 2.4</a> (see &#39;issue322&#39;). This means that <code>orm.session.rollback()</code> might not be called.
<h3>Enter Pytest Fixtures</h3>
While the classic xUnit style doesn&#39;t work so well, we do have an alternative: <a href="https://pytest.org/latest/fixture.html">pytest fixtures</a>.
What are these fixtures? Well, they&#39;re basically dependencies that you can require for your tests. Here&#39;s how our code can look now:
<div class="highlight"><pre><code class="language-python" data-lang="python"># tests/helper.py
@pytest.fixture()
def db_transaction(request):
orm.session.begin()
def fin():
orm.session.rollback()
request.addfinalizer(fin)
return orm.session
# tests/unit/widget_test.py
class TestMyWidget:
def test_my_widget(self, db_transaction):
print &quot;\nin passing test\n&quot;
def test_my_failing_widget(self, db_transaction):
print &quot;\nin failing test\n&quot;
raise Exception()
</code></pre></div>
The extended example and output <a href="https://gist.github.com/Nitrodist/60ced9ca02d9e56cde42">can be found here</a>.
The downside of this technique is that we have to remember to opt-in to every test by specifying <code>db_transaction</code> as one of the arguments. To get around this issue, we tried combining <code>autouse</code> and classes:
<div class="highlight"><pre><code class="language-python" data-lang="python"># tests/helper.py
class BaseTest:
# prefixed with &#39;_&#39; since autouse fixtures are executed alphabetically
# see http://stackoverflow.com/a/28593102/269694
@pytest.fixture(autouse=True)
def _wrap_test_in_transaction(self, request):
orm.session.begin()
def fin():
orm.session.rollback()
request.addfinalizer(fin)
# tests/unit/widget_test.py
from tests.helper import BaseTest
class TestWidget(BaseTest):
@pytest.fixture(autouse=True)
def before(self, request):
self.subject = Widget()
def test_can_get_meaning_of_life(self):
assert self.subject.get_meaning_of_life() == 42
</code></pre></div>
That&#39;s it! No other special tricks.
<h3>Conclusion</h3>
This took a bit of thinking to come up with, but maybe you or someone you know has a better way of writing tests with database transactions in pytest. Let us know what you think and leave a comment below!
</description>
</item>
<item>
<title>Reducing WatchKit Traffic With View Models</title>
<link>https://techblog.thescore.com/2015/05/20/reducing-watchkit-traffic-with-view-models/</link>
<pubDate>Wed, 20 May 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/05/20/reducing-watchkit-traffic-with-view-models</guid>
<description>If you own an Apple Watch, you probably noticed something disappointing: 3rd party apps are SLOW.
If you&#39;ve written a watch app, you&#39;ll know that a big part of the problem lies in the bandwidth limitations of phone to watch communication. Architecting a performant WatchKit extension can be difficult due to the write-only nature of WatchKit UI elements and the stringly-typed nature of controllers.
How can we decrease traffic sent from the phone to the watch while simultaneously reducing the complexity of your app&#39;s architecture? In this post I will describe how these can be achieved by using View Models.
<h2>Apple&#39;s Recommendations</h2>
Apple specifically <a href="https://developer.apple.com/watchkit/tips/">recommends</a> that you should:
<ul>
<li>Minimize traffic</li>
<li>Update only what has changed</li>
<li>Load content lazily</li>
</ul>
This can be difficult with the WatchKit SDK since UI elements exist on the watch but are controlled from the WatchKit extension. We can&#39;t query the state of the app running on the watch, forcing us to maintain UI state in our extension. If we don&#39;t do this we risk sending unneccesary updates to the UI on the watch.
View Models allow us to manage UI state without adding complexity to your WatchKit extension.
<h2>View Models</h2>
A View Model is an object for encapsulating the presentation logic of a model. In MVC we don’t want a view to contain this logic since it results in tight coupling to a specific model. Instead, the controller object is generally responsible for containing the presentation logic and uses it to configure a view. However, this just moves the coupling from the view to the controller, leaving you with a controller that is bloated and not reusable.
View Models sit in-between the view and the controller, allowing you to easily encapsulate the manipulation of a model to provide just what is needed to populate the view. Business logic for representing models can be removed from both the view and the controller, increasing their reusability and testability.
For example, here&#39;s a simple View Model for an Event object that exposes only what is needed to populate a row of this table:
<img src="/assets/watchkit.png" alt="">
<div class="highlight"><pre><code class="language-swift" data-lang="swift">struct EventViewModel {
let firstTeamName : String
let secondTeamName : String
let firstTeamScore : String
let secondTeamScore : String
let firstTeamLogo : ImageViewModel?
let secondTeamLogo : ImageViewModel?
var description : String {
return self.firstTeamName + &quot; @ &quot; + self.secondTeamName
}
init(model: Event?, imageSize: CGSize) {
self.firstTeamName = model?.awayTeam?.name ?? &quot;0&quot;
self.secondTeamName = model?.homeTeam?.name ?? &quot;0&quot;
self.firstTeamScore = model?.awayScore?.description ?? &quot;0&quot;
self.secondTeamScore = model?.homeScore?.description ?? &quot;0&quot;
self.firstTeamLogo = model?.awayTeam?.logo != nil ? ImageViewModel(url: model?.awayTeam?.logo.doubleResolutionURL, size: imageSize, imageQuality: 1) : nil
self.secondTeamLogo = model?.homeTeam?.logo != nil ? ImageViewModel(url: model?.homeTeam?.logo.doubleResolutionURL, size: imageSize, imageQuality: 1) : nil
}
}
</code></pre></div>
<h2>Minimizing Traffic &amp; Updating Only What Has Changed</h2>
WatchKit interface elements are write-only, meaning that they can’t be queried to decide whether or not to send an update. Using View Models adds the benefit of holding the current state of the view, allowing the extension to determine if an interface element’s value needs to be updated.
To easily add this ‘diffing’ functionality, we created a simple protocol that all WKInterface elements can implement in an extension (or category, in Objective-C).
<div class="highlight"><pre><code class="language-swift" data-lang="swift">protocol Updatable {
typealias T
func updateFrom(oldValue : T?, to newValue : T?)
}
</code></pre></div>
The implementation of this on WKInterface label looks like this:
<div class="highlight"><pre><code class="language-swift" data-lang="swift">extension WKInterfaceLabel : Updatable {
func updateFrom(oldValue : String?, to newValue : String?) {
if newValue != oldValue {
self.setText(newValue)
}
}
}
</code></pre></div>
Now instead of calling <code>myLabel.setText(“newValue”)</code> we can call <code>myLabel.updateFrom(“oldValue”, to: “newValue”)</code> and <code>setText</code> will only be called if the arguments differ.
Since all of our images are loaded remotely and then sized to the dimensions required by the WKInterfaceImage, we have to compare two different properties of the image. We can do this while still conforming to the Updatable protocol by making a simple ImageViewModel:
<div class="highlight"><pre><code class="language-swift" data-lang="swift">struct ImageViewModel : Equatable {
let url: NSURL?
let size: CGSize
}
func == (lhs: ImageViewModel, rhs: ImageViewModel) -&gt; Bool {
return lhs.url == rhs.url &amp;&amp; lhs.size == rhs.size
}
</code></pre></div>
Now the extension on WKInterfaceImage looks like this:
<div class="highlight"><pre><code class="language-swift" data-lang="swift">extension WKInterfaceImage : Updatable {
func updateFrom(oldValue : ImageViewModel?, to newValue : ImageViewModel?) {
if newValue != nil &amp;&amp; newValue != oldValue {
self.setImage(URL: newValue!.url!, size: newValue!.size)
}
}
}
</code></pre></div>
Here we call a function that we’ve written that asynchronously downloads an image, resizes it, caches it and sets it on the WKInterfaceImage, so by using the View Model we have potentially saved a lot of network and processing work.
<h2>Tables</h2>
Perhaps the most interesting application of View Models in WatchKit is for WKInterfaceTables. Whereas UITableViews have a datasource and delegate, WKInterfaceTables require the controller to maintain an internal mapping of data to both row controller type and row controller identifier, since row controllers are instantiated by a storyboard identifier and not by alloc’ing an instance of a class.
This mapping can result in lots of logic being stuck inside a big loop where each row controller is populated. Additionally, it makes it much easier to discard every row in the table and recreate it, which Apple strongly discourages. We can combine a View Model for the table with individual View Models for each row type to move the presentation logic and the updating logic out of the controller.
<div class="highlight"><pre><code class="language-swift" data-lang="swift">protocol TableViewModel {
var rowTypes : [String] { get }
func rowViewModelAtIndex(index: Int) -&gt; Any?
func table(table: WKInterfaceTable, updateFrom oldRowViewModel:Any?, to newRowViewModel:Any?, atIndex index: Int)
}
extension WKInterfaceTable : Updatable {
func updateFrom(oldValue: TableViewModel?, to newValue: TableViewModel?) {
if let oldTableModel = oldValue, newTableModel = newValue {
// only update if necessary
if oldTableModel.rowTypes.count == 0 {
self.setRowTypes(newTableModel.rowTypes)
}
else if newTableModel.rowTypes != oldTableModel.rowTypes {
// swap old row types for new row types
for (index, newRowType) in enumerate(newTableModel.rowTypes) {
var swap = true
if index &lt; oldTableModel.rowTypes.count {
let oldRowType = oldTableModel.rowTypes[index]
if oldRowType == newRowType {
swap = false
}
else {
self.removeRowsAtIndexes(NSIndexSet(index:index))
}
}
if swap {
self.insertRowsAtIndexes(NSIndexSet(index:index), withRowType: newRowType)
}
}
// remove extraneous rows
let numRowsToRemove = oldTableModel.rowTypes.count - newTableModel.rowTypes.count
if numRowsToRemove &gt; 0 {
self.removeRowsAtIndexes(NSIndexSet(indexesInRange: NSMakeRange(newTableModel.rowTypes.count, numRowsToRemove)))
}
}
// populate each row
for i in 0..&lt;self.numberOfRows {
newTableModel.table(self, updateFrom: oldTableModel.rowViewModelAtIndex(i), to: newTableModel.rowViewModelAtIndex(i), atIndex: i)
}
}
else if let newTableModel = newValue {
self.setRowTypes(newTableModel.rowTypes)
for i in 0..&lt;self.numberOfRows {
newTableModel.table(self, updateFrom: nil, to: newTableModel.rowViewModelAtIndex(i), atIndex: i)
}
}
}
}
</code></pre></div>
Now if we load a data set, do a network fetch and get the same data set, zero updates are sent to the watch. No wasted bandwidth or rendering time. If one label in one row changes, only that change is sent. If we insert a different row type in the middle of the table, only the affected rows are modified.
We swizzled the setters for WKInterfaceLabel and WKInterfaceTable to log their calls, demonstrating the improvements:
Initial loading:
<ul>
<li>1 setRowTypes: call</li>
<li>17 setText: calls</li>
</ul>
Reloading same data:
<ul>
<li>no calls</li>
</ul>
Inserting a new row type and deleting a row from the end:
<ul>
<li>1 insertRowsAtIndexes:withRowType: call</li>
<li>1 removeRowsAtIndexes: call</li>
<li>3 setText: calls</li>
</ul>
Even more useful is that we can easily adhere to Apple’s recommendation:
<blockquote>
“To optimize the launch time of your WatchKit app and make your app feel more responsive, load the content below the initially visible area of a controller after the controller has displayed to the user.”
</blockquote>
If we do a network fetch for the first 3 rows of data, update the table, then do a subsequent fetch for 10 rows of data, we only have to call <code>myTable.updateFrom(_, to:)</code> and don’t have to worry about computing row offsets. Only the rows that have been added to the bottom of the table will be sent to the watch.
<h2>Code Simplification</h2>
Prior to using View Models, our table updating code looked something like this:
<div class="highlight"><pre><code class="language-swift" data-lang="swift">table.setRowTypes(“DateRow”, “EventRow”, “EventRow”)
func updateTableWithData(rowData:[NSObject]) {
for i in 0..&lt;rowData.count {
let data = rowData[i]
switch data {
case is Event:
let rowController = table.rowControllerAtIndex(i) as! EventController
// set all the labels and images
rowController.populate(data)
break
case is Date:
let rowController = table.rowControllerAtIndex(i) as! DateController
// set all the labels and images
rowController.populate(data)
break
default:
break
}
}
}
</code></pre></div>
Now that we’ve moved all of our logic out of the interface controller, an update call to the table looks like this:
<div class="highlight"><pre><code class="language-swift" data-lang="swift">var previousTableViewModel : EventsTableViewModel?
func updateTableWithData(data:[NSObject]) {
let newTableViewModel = EventsTableViewModel(data:data)
self.table.updateFromOldValue(previousTableViewModel, toNewValue: newTableViewModel)
previousTableViewModel = newTableViewModel
}
</code></pre></div>
<h2>Testing</h2>
View Models also have the benefit of adding basic testing functionality to your WatchKit extension. Because we can’t query WatchKit interface elements, we have no way of knowing if they correctly represent the data we are interested in. Now if we assume that the content of the view is accurately represented by the View Model, it allows us to perform business logic tests on the View Model.
<h2>Profiling</h2>
While we can demonstrate that the number of setter calls has been reduced and our code has been simplified, we are unable to profile and measure the performance improvements. The Apple Watch is essentially a black box, with several confounding variables (latency, watch state) that can’t be eliminated when testing. However, our methods follow Apple’s recommendations so we can hope that by reducing traffic and updating only what has changed we improve the responsiveness of our WatchKit Extension.
<h2>Conclusion</h2>
By using View Models the developer can create wrappers around their model objects, simplifying their extension architecture while at the same time reducing the network traffic sent from the phone to the watch. Give this approach a try when you re-architect your WatchKit extension and let us know if you see any performance improvements!
</description>
</item>
<item>
<title>Share Your Bag of Tricks</title>
<link>https://techblog.thescore.com/2015/05/01/share-your-bag-of-tricks/</link>
<pubDate>Fri, 01 May 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/05/01/share-your-bag-of-tricks</guid>
<description>Everyone has specific tips and shortcuts that they have picked up from various sources, including the following:
<ul>
<li>Trial &amp; error or self-discovery</li>
<li>From a friend or colleague</li>
<li>Reading books, blog posts, articles</li>
<li>Watching videos, conferences</li>
<li>Formal education</li>
</ul>
Regardless of how one has gained these skills, it is obvious that they increase their effectiveness and efficiency at tasks. With enough exposure and time, one can amass an impressive assortment of skills, tips and shortcuts. We tend to call these people veterans/experts/experienced/seniors, amongst other words of praise and respect.
This post is not directed towards a specific group of individuals, as the overarching message applies to everyone. It is common to find large visible gaps with respect to skills and/or experience. For example, within a workplace environment a more experienced colleague might have customized and automated parts of their daily workflow to increase their effectiveness and efficiency.
<h2>The Bag of Tricks</h2>
In general the following is true:
<blockquote>
Experienced individuals have something which sets them apart from the less-experienced.
</blockquote>
Let us ignore the common qualities that account for uniqueness such as personality, and focus solely on ones skills and experience. Then, it&#39;s possible to identify their bag of tricks.
A bag of tricks is really a simple concept. Often it is a set of small tips and shortcuts used to make certain tasks easier than they normally would be.
<blockquote>
Bag of Tricks | bag əv triks |
noun
1 informal a set of ingenious plans, techniques, or resources: I better pull something out of my bag of tricks to finish this task quickly
</blockquote>
In some cases the holder of the bag of tricks clearly knows what resides in their bag. In other cases they might be forgetting some items, which occurs over time as the holder has completely integrated it into their normal repertoire.
<h2>Sharing the Bag</h2>
Often in workplaces we have various levels of experience such as: Junior, Intermediate, Senior. In our societies we have accepted the following model:
<blockquote>
Experienced individuals can mentor the less experienced pupils
</blockquote>
This is a very simple learning model, and it works. It can be seen in many industries and professions as they promote formal mentorship/apprenticeship programs. These formal programs often have a cost associated with them, although that typically correlates with the quality received. Informal approaches are often executed between colleagues with little structure.
Everyone has something worth showing. Someone will be interested in learning it, or improving upon it. Levels of experience does not guarantee one will always be teaching or learning. In a symbiotic relationship everyone will be sharing their bag of tricks.
<h3>Pupil&#39;s Perspective</h3>
Getting a peek at someone&#39;s bag of tricks is of great benefit for the pupil. If someone offers advice, there should be no hesitation to accept it. A pupil, or anyone for that matter, should always accept the offer of advice since there is no downside to hearing it out. Even if the pupil has already known about the provided advice, the act of hearing it again reinforces it. It might also be possible for the pupil to give back immediately to the mentor.
The best thing a pupil can do is to provide feedback and appreciation. By questioning the mentor and asking for additional details, it forces the mentor to inspect and reflect on their bag of tricks. In some situation the tides can change and the mentor will learn from the pupil.
<h3>Mentor&#39;s Perspective</h3>
Mentors are encouraged to share their bag of tricks, and most are willing to do so. Sharing knowledge has many benefits:
<ul>
<li>Joy in helping someone</li>
<li>Gained respect from pupil and others</li>
<li>Self-improvement by going over specifics</li>
<li>Constructive criticism</li>
<li>Encouraging similar behaviour</li>
</ul>
Sharing and helping can bring much joy to a mentor, which in turn leads to gaining respect from the pupil and others. As previously mentioned, a mentor may not necessarily know of the content of their bag of tricks. A pupil can probe and inquire on specifics to the point where new light is shone on the bag of tricks. By mentoring a pupil, it encourages others to do the same when they have amassed their own bag of tricks.
<h2>To Share, or Not to Share?</h2>
There might be situations where an individual does not want to share their bag of tricks. One strong reason for this is that they would lose their edge over their colleagues. This is understandable, although if the environment is amenable to learning then there is no real reason to hold back.
If everyone in an environment is holding back on knowledge, it does not sound like a place of growth. Furthermore, if everyone is scared of losing their edge, there are probably greater issues in that environment. If there is no sharing of knowledge between colleagues, everyone is at a disadvantage compared to the alternative.
<h2>Means of Sharing</h2>
It is clear that sharing is a win-win situation. In a workplace environment sharing knowledge is simple and commonplace. Either the pupil asks for help, or the mentor decides to impart knowledge. If the environment is pro-learning then this behaviour will be encouraged and valued. There might even be an explicit mentoring process in place or less informal opportunities such as lunch and learns.
As mentioned before, anyone can mentor regardless of their skill and experience levels. Everyone should take every opportunity to share something from their bag of tricks. Not only will this immediately benefit others, but what goes around comes around. Sharing can take many forms, including the following:
<ul>
<li>Blog Posts, Public Notes, Documentation</li>
<li>Mentoring, Pairing Sessions</li>
<li>Talks, Presentations, Demonstrations</li>
<li>Video/Audio Recordings</li>
</ul>
The best way is to reach a broad audience, so that everyone can benefit from it. Thankfully the Internet is an amazing medium to communicate to the masses.
Take pride and share your bag of tricks!
</description>
</item>
<item>
<title>No Excuses: Verifying RSpec Test Doubles</title>
<link>https://techblog.thescore.com/2015/04/28/no-excuses-verifying-rspec-test-doubles/</link>
<pubDate>Tue, 28 Apr 2015 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2015/04/28/no-excuses-verifying-rspec-test-doubles</guid>
<description>Tests which utilize external services or interact with the database are typically the culprits of long-running tests. We want to keep our tests quick. It is possible to mock/stub out long running database and/or external services calls. This reduces the time a test suite takes to execute.
<h2>Unsheathe the Double</h2>
In Ruby, one approach to mocking is by completely replacing the object of interest with a lightweight <a href="http://www.rubydoc.info/gems/rspec-mocks/frames#Test_Doubles">double</a> using <a href="http://rspec.info/">RSpec</a>. Proper usage of a double can prevent tests from interacting with external services, such as a database (i.e., <code>ActiveRecord</code>).
With respect to RSpec, a double is created by providing a classname or object, along with a hash of messages and their responses. A double can only respond using the provided responses to their defined messages (technically there are other messages that a double can respond to, but for our purpose we do not have to worry about them).
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">dog = double(&#39;Dog&#39;, talk: &#39;Woof&#39;)
dog.talk #=&gt; &quot;Woof&quot;
dog.walk #=&gt; Double &quot;Dog&quot; received unexpected message :walk with (no args)
</code></pre></div>
In the above example, a <code>&#39;Dog&#39;</code> double is created that only knows <code>#talk</code>. When it receives an unknown message like <code>#walk</code> an appropriate exception is raised.
<h2>Double-Edge Double</h2>
A double aims to abstract away from the concrete concepts that they are standing in for (i.e., defined classes/methods/attributes/associations and their implementations). This can simplify tests by only dealing with the immediate concerns in a restricted scope. Using doubles has its perks, but a problem can arise if changes occur to the underlying concrete concepts.
Lets examine the following scenario:
<blockquote>
We have an existing class with multiple defined methods. This class and its methods primarily interact with external services. This class and its methods are used within our test suite, in where we have mocked it out using a double.
We decided to modify a method definition on the described class. Our tests continue to pass.
</blockquote>
Given this scenario we would hope that by modifying the method definition that our existing tests which depend on said method definition to fail. Our tests are using a double in which the method definition it has defined is still valid, even when the actual method definition has been altered.
The consequences of not seeing any failing tests can be serious. Even with minor cosmetic changes, that do not alter functionality, it is still a misleading test. Identifying these tests after the fact can be challenging, as they initially appear to be in fine working order.
It is important to always remember to check the usage of doubles whose underlying concepts are changed. A gem called <a href="https://github.com/xaviershay/rspec-fire">rspec-fire</a> was created to alleviate this task. This gem would verify that a double is actually mocking an actual method defined on the concrete object. As of RSpec 3.0, rspec-fire is now obsolete as RSpec has a set of new <a href="https://relishapp.com/rspec/rspec-mocks/v/3-0/docs/verifying-doubles">verifying doubles</a>.
<h2>Dance of the Double</h2>
A simple example best illustrates the downside of using the original RSpec doubles. In this example we also show how to replace the double with the new and improved verifying doubles, along with their benefits.
For this example we are not dealing with a database, although the idea is easily extendable. We can imagine that our test is creating actual entries in the database, thus incurring the performance hit.
<hr>
First we define an <code>Owner</code> that has a <code>Dog</code>. The <code>Dog</code> responds to <code>#talk</code>. In addition we have a corresponding test that uses a double to mock out the <code>Dog</code> which responds to <code>#talk</code>.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">class Owner
 attr_reader :dog
 def initialize(dog)
 @dog = dog
 end
end
class Dog
 def talk
 &#39;Woof&#39;
 end
end
RSpec.describe Owner do
subject { Owner.new(dog) }
let(:dog) { double(&#39;Dog&#39;, talk: &#39;Fake Woof&#39;) }
it { expect(subject.dog.talk).to eq(&#39;Fake Woof&#39;) }
end
# 1 example, 0 failures
</code></pre></div>
<hr>
We decide to change <code>Dog#talk</code> to <code>Dog#bark</code>, but forget to update the test.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">class Owner
 attr_reader :dog
 def initialize(dog)
 @dog = dog
 end
end
class Dog
 def bark
 &#39;Woof&#39;
 end
end
RSpec.describe Owner do
subject { Owner.new(dog) }
let(:dog) { double(&#39;Dog&#39;, talk: &#39;Fake Woof&#39;) }
it { expect(subject.dog.talk).to eq(&#39;Fake Woof&#39;) }
end
# 1 example, 0 failures
</code></pre></div>
From our test&#39;s perspective everything is still fine, since our double still responds to <code>#talk</code>. Ideally we would want our test here to fail since it is not accurately matching the interface defined by an actual <code>Dog</code> instance.
This is a problem. It is possible to completely forget about fixing the test, since it technically passed.
<hr>
This time we use a verifying double that RSpec provides such as an <code>instance_double</code>. This ensures that the messages the double receives are verified against the interface defined by the concrete object.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">require &#39;rspec&#39;
class Owner
 attr_reader :dog
 def initialize(dog)
 @dog = dog
 end
end
class Dog
 def bark
 &#39;Woof&#39;
 end
end
RSpec.describe Owner do
subject { Owner.new(dog) }
let(:dog) { instance_double(&#39;Dog&#39;, talk: &#39;Fake Woof&#39;) }
it { expect(subject.dog.talk).to eq(&#39;Fake Woof&#39;) }
end
# Failure/Error: let(:dog) { instance_double(&#39;Dog&#39;, talk: &#39;Fake Woof&#39;) }
# Dog does not implement: talk
# 1 example, 1 failure
</code></pre></div>
Now we have a failing test! This is what we expect to happen as our double is attempting to using an unimplemented method. This feedback allows us to take the corrective action on our tests to ensure that they are not forgotten and invalid.
<hr>
We now decide to change the number of arguments on <code>Dog#bark</code>. Again with the old non-verifying double our test would again simply pass. The verifying doubles also check that the number of arguments match the defined interface&#39;s number of arguments.
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">require &#39;rspec&#39;
class Owner
 attr_reader :dog
 def initialize(dog)
 @dog = dog
 end
end
class Dog
 def talk(loud)
loud ? &#39;Woof&#39; : &#39;Woof!!&#39;
 end
end
RSpec.describe Owner do
subject { Owner.new(dog) }
let(:dog) { instance_double(&#39;Dog&#39;, talk: &#39;Fake Woof&#39;) }
it { expect(subject.dog.talk).to eq(&#39;Fake Woof&#39;) }
end
# Failure/Error: let(:dog) { instance_double(&#39;Dog&#39;, talk: &#39;Fake Woof&#39;) }
# Wrong number of arguments. Expected 1, got 0.
# 1 example, 1 failure
</code></pre></div>
<h2>Double Development</h2>
If the underlying class is loaded <code>instance_double</code> will do the verifying on the class. In the situation where the class is not loaded than it acts as a normal double.
During development an <code>instance_double</code> allows one to develop in isolation if the class you are mocking out does not yet exist. Eventually when the test is more-or-less complete, it is possible to simply load the class and the <code>instance_double</code> will start to verify on the loaded class.
In addition, during development you can use <a href="https://github.com/nevir/rubocop-rspec/blob/master/lib/rubocop/cop/rspec/verified_doubles.rb"><code>rubocop-rspec</code></a> to ensure you always verify your doubles.
<h2>Concluding Double</h2>
TL;DR -- There are no excuses, verify your RSpec test doubles.
Verifying your RSpec test doubles can, and will, save you from many headaches down the road. In most cases the change required to use verifying doubles is relatively easy. The benefits are clear, worthwhile and your test suite will thank you.
</description>
</item>
<item>
<title>Mutate your Rack middleware's env!</title>
<link>https://techblog.thescore.com/2014/12/04/modify-your-racks-env-hash/</link>
<pubDate>Thu, 04 Dec 2014 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2014/12/04/modify-your-racks-env-hash</guid>
<description>If you&#39;ve been a Ruby developer for a while, chances are that you&#39;ve written some kind of Rack middleware. Rack is a pretty big part of web development with Rails, and here at theScore we&#39;ve developed a few, small pieces of middleware to add additional data to our requests so that our Rails applications can use it.
If you&#39;re not familiar with Rack or Rack middleware, I would recommend <a href="http://southdesign.de/blog/rack.html">this article on Rack</a> and <a href="http://stackoverflow.com/questions/2256569/what-is-rack-middleware">this stackoverflow question on the middleware</a>.
<h2>Background and Problem</h2>
While looking to integrate <a href="http://yellerapp.com/">Yeller</a>, we ran into a bug in our middleware. The bug came up because Yeller was looking for Rails specific data that was being attached to the request&#39;s <code>env</code> hash when an error was raised. In this case, during a request, <code>ActionController::Metal</code> sets <code>action_controller.instance</code> on the <code>env</code> hash:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">def dispatch(name, request) #:nodoc:
 @_request = request
 @_env = request.env
 @_env[&#39;action_controller.instance&#39;] = self
 process(name)
 to_a
end
</code></pre></div>
(<a href="https://github.com/rails/rails/blob/7c4bfe1c954ef90acf4f790e46fcbbd07d85af3e/actionpack/lib/action_controller/metal.rb#L195">source pinned to rails 4.1.4</a>)
In Yeller, the error reporting code extracts that information to give us a better idea of what controller and action was involved when the error occurred:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">def render_exception_with_yeller(env, exception)
 # ...
 controller = env[&#39;action_controller.instance&#39;]
 # ...
 if controller
 # ...
location = &quot;#{controller.class.to_s}##{params[:action]}&quot;
 else
 # code without location information available
 end
end
</code></pre></div>
(<a href="https://github.com/tcrayford/yeller_ruby/blob/355cb6b874c6ddf0b3ee1d3d5012b9db16b7e0c0/lib/yeller/rails.rb#L58">source pinned to yeller_ruby 0.2.2</a>)
When we ran Yeller&#39;s verification Rake task, our errors were being sent to Yeller&#39;s servers, but the Rake task also told us that we were missing that Rails specific information (the location, for example). So, we scratched our heads a bit. Error reporting worked in new Rails applications, so it had to be something in our code.
I started by removing our custom middleware -- we had three of them and I removed all of them. The rake task succeeded! So I started adding them in one-by-one until I added one and the rake task failed. At this point, I had narrowed the code down to about 3 lines and they looked something like this:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">def call(env)
api_version = env[&#39;HTTP_X_API_VERSION&#39;].presence || default_api_version
api_version = api_version.to_s.split(&#39;,&#39;).first # Handle bad API versions with commas in them
@app.call(env.merge(&#39;the_score.api_version&#39; =&gt; ApiVersionDecorator.new(api_version)))
end
</code></pre></div>
<h2>The Solution</h2>
After showing Yeller&#39;s team the middleware, they saw the problem! <code>env.merge</code> returns a new hash. The &#39;gotcha&#39; in this case is this:
<blockquote>
The scope of the <code>env</code> hash exists after calls to <code>@app.call</code> finish. Because of this, <code>@app.call</code> should be invoked with the same <code>env</code> hash passed into the method.
</blockquote>
Middleware that rely on subsequent middleware (or the app) to set information (e.g. <code>action_controller.instance</code>) on the <code>env</code> hash also rely on the fact that the subsequent middleware won&#39;t pass a different <code>env</code> hash to the subsequent middleware. If the subsequent middleware do pass a different hash, here&#39;s what happens:
<ol>
<li>The new <code>env</code> is only used from that point forward in the middleware stack</li>
<li>Needless memory is allocated because you&#39;ve just duplicated every request&#39;s information in a separate <code>env</code></li>
</ol>
The fix and proper way to do this is to instead mutate your hash through <code>Hash#[]</code>, <code>Hash#merge!</code>, <code>Hash#store</code>, and so forth. A new hash isn&#39;t created when you use these methods. Here&#39;s the fixed code:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">def call(env)
api_version = env[&#39;HTTP_X_API_VERSION&#39;].presence || default_api_version
api_version = api_version.to_s.split(&#39;,&#39;).first # Handle bad API versions with commas in them
@app.call(env.merge!(&#39;the_score.api_version&#39; =&gt; ApiVersionDecorator.new(api_version)))
end
</code></pre></div>
If you want to see the problem happening in your console with a &#39;real&#39; Rack request (OK, not really, it uses <code>Rack::MockRequest</code> in the tests), you can clone <a href="https://github.com/Nitrodist/rack-middleware-gotcha">this project</a> that I&#39;ve created, <code>bundle install</code>, and run <code>rspec</code>.
<h2>Bonus: How to Test It</h2>
We had a suite of tests using rpsec around these simple pieces of middleware, so we added something like this to every test:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">class MyMiddleware
 def initialize(app)
 @app = app
 end
 def call(env)
 @app.call(env)
 end
end
describe MyMiddleware do
let(:app) { double(:app) }
let(:env) { double(:env) }
subject { MyMiddleware.new(app) }
 it &quot;calls app with the same env hash&quot; do
expect(app).to receive(:call).with(env)
 subject.call(env)
 end
end
</code></pre></div>
<h2>Conclusion</h2>
Rack is fairly straightforward, but there definitely are some &#39;gotchas&#39; that you can run into. This is just one example and there are a few more that you need to consider when creating Rack middleware. In a future post, I&#39;ll explain another &#39;gotcha&#39; with regards to multithreading and Rack middleware.
Another interesting point is that this is a &#39;bug&#39; that has been in our code base for a little over two years without being noticed. It just goes to show that every once in a while, you should take a look at the pieces of middleware that your application depends on: they may be misbehaving like ours was!
</description>
</item>
<item>
<title>Are Your Cache-Control Directives Doing What They Are Supposed to Do?</title>
<link>https://techblog.thescore.com/2014/11/19/are-your-cache-control-directives-doing-what-they-are-supposed-to-do/</link>
<pubDate>Wed, 19 Nov 2014 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2014/11/19/are-your-cache-control-directives-doing-what-they-are-supposed-to-do</guid>
<description>HTTP offers very powerful support for <a href="http://tools.ietf.org/html/rfc2616#section-13">caching</a>:
<blockquote>
The goal of caching in HTTP/1.1 is to eliminate the need to send requests in many cases,
and to eliminate the need to send full responses in many other cases. The former reduces the
number of network round-trips required for many operations; we use an &quot;expiration&quot; mechanism
for this purpose (see section 13.2). The latter reduces network bandwidth requirements; we
use a &quot;validation&quot; mechanism for this purpose (see section 13.3).
</blockquote>
If you&#39;re not familiar with how the &quot;expiration&quot; and &quot;validation&quot; mechanisms work in different
kinds of HTTP caches, you should start with <a href="http://tomayko.com/writings/things-caches-do">Things Caches Do</a>
by Ryan Tomayko. For a more in-depth look at HTTP caching, check out
<a href="https://www.mnot.net/cache_docs">Caching Tutorial for Web Authors and Webmasters</a> by Mark
Nottingham.
<h2>Private and Shared Caches</h2>
Private caches can only serve a single user, whereas shared caches can serve any number
of users. To take full advantage of HTTP caching, you need to have a private cache on your
clients, and a shared gateway cache (also known as reverse proxy cache or surrogate cache)
just in front of your backend.
The classic example of a private client cache is the browser cache. <a href="http://developer.android.com/reference/android/net/http/HttpResponseCache.html">HttpResponseCache</a>
and <a href="http://nshipster.com/nsurlcache">NSURLCache</a> are examples of private client caches for
Android and iOS respectively.
<a href="https://www.varnish-cache.org">Varnish</a> and <a href="http://www.squid-cache.org">Squid</a> are examples
of gateway caches. CDNs such as <a href="http://www.akamai.com">Akamai</a> and <a href="http://www.fastly.com">Fastly</a>
are essentially geo-distributed gateway caches.
<h2><code>Cache-Control</code> Header</h2>
<code>Cache-Control</code> header is used to control how the caches between your user and application
behave. It can be used as a request or response header (similar to <code>Content-Type</code>). But, for
the purposes of this blog post, we&#39;re going to explore using <code>Cache-Control</code> as a response
header from your backend application.
Note that there can be any number of shared caches between the private cache on the client,
and the gateway cache right in front of your backend. If you use HTTPS, you don&#39;t have to
worry about how these intermediate caches (i.e. proxy cache installed on a company network)
would behave. At theScore, we leverage HTTP caching in all of our APIs. Furthermore, all of
our APIs use HTTPS as such we don&#39;t have to worry about intermediate caches, and this blog
post will do the same for the most part.
<h2><code>Cache-Control</code> Directives</h2>
If you use <code>Cache-Control</code> as a response header, there are only 9
<a href="https://tools.ietf.org/html/rfc7234#section-5.2.2">directives</a> to worry about. That&#39;s it!
Here&#39;s a summary of the directives from <a href="https://www.mnot.net/cache_docs">Caching Tutorial for Web Authors and Webmasters</a>
by Mark Nottingham:
<blockquote>
Useful <code>Cache-Control</code> response headers include:
<ul>
<li><code>max-age=[seconds]</code> - specifies the maximum amount of time that a representation will be
considered fresh. Similar to <code>Expires</code>, this directive is relative to the time of the request,
rather than absolute. <code>[seconds]</code> is the number of seconds from the time of the request you
wish the representation to be fresh for.</li>
<li><code>s-maxage=[seconds]</code> - similar to <code>max-age</code>, except that it only applies to shared
(e.g., proxy) caches.</li>
<li><code>public</code> - marks authenticated responses as cacheable; normally, if HTTP authentication
is required, responses are automatically private.</li>
<li><code>private</code> - allows caches that are specific to one user (e.g., in a browser) to store the
response; shared caches (e.g., in a proxy) may not.</li>
<li><code>no-cache</code> - forces caches to submit the request to the origin server for validation
before releasing a cached copy, every time. This is useful to assure that authentication
is respected (in combination with public), or to maintain rigid freshness, without
sacrificing all of the benefits of caching.</li>
<li><code>no-store</code> - instructs caches not to keep a copy of the representation under any conditions.</li>
<li><code>must-revalidate</code> - tells caches that they must obey any freshness information you give
them about a representation. HTTP allows caches to serve stale representations under special
conditions; by specifying this header, you’re telling the cache that you want it to strictly
follow your rules.</li>
<li><code>proxy-revalidate</code> — similar to <code>must-revalidate</code>, except that it only applies to proxy caches.</li>
</ul>
</blockquote>
You can mix and match these directives in a number of different combinations:
<ul>
<li><code>Cache-Control: private, max-age=10</code></li>
<li><code>Cache-Control: max-age=10, s-maxage=300</code></li>
<li><code>Cache-Control: max-age=10, s-maxage=300, proxy-revalidate</code></li>
</ul>
As you can see, the directives are pretty straightforward to understand. They&#39;re easy to use as
well if you assume that all the caches between your end user and application correctly implement
the spec. Unfortunately, as with any spec, you can&#39;t make that assumption. You need to be aware
of any spec misinterpretations in the implementation of the caches that you&#39;re using, and
properly account for them.
<h2>Getting <code>Cache-Control</code> Right</h2>
Let&#39;s look at a scenario, in which we consider most of the <code>Cache-Control</code> directives from
above in the process of arriving at the final combination.
Suppose you have a mobile application that consumes a REST API, and one of the API end-points
has <code>Cache-Control: max-age=7200</code> (the response is considered fresh for 2 hours). Further suppose
that due to a new bug in the API, the response for this end-point changes in a way that causes
the app to crash. You can fix the bug and purge the cache for this end-point on your gateway
cache (gateway caches generally give you this kind of control). But, this won&#39;t stop the clients
that have the bad response cached from crashing. In the worst case, these clients will continue
to crash for up to 2 hours. You essentially get stuck with stale content, and just have to
wait it out.
To avoid getting stuck with stale content on a client, we want to make sure that the client
always revalidates the content before serving it from the client cache. We would still save
bandwidth with this approach. Having said that, we still want the original behaviour (the response
is considered fresh for 2 hours) on the gateway cache.
It looks like <code>no-cache</code> directive fits the bill perfectly as it allows you &quot;to maintain rigid
freshness, without sacrificing all of the benefits of caching.&quot; Note that <code>no-cache</code> does not
mean &quot;not to cache&quot; (you use <code>no-store</code> for that). But, sooner than later, we would discover
that some popular client caches (certain versions of IE and Firefox) treat <code>no-cache</code> as
<code>no-store</code>. So, let&#39;s abandon this option and start with the following instead:
<div class="highlight"><pre><code class="language-text" data-lang="text">Cache-Control: max-age=0
</code></pre></div>
With this in place, the response expires right away, which will force revalidation. However,
this will force revalidation on the gateway cache as well. In other words, all the requests
will end up hitting the backend. This can be fixed by using <code>s-maxage</code>:
<div class="highlight"><pre><code class="language-text" data-lang="text">Cache-Control: max-age=0, s-maxage=7200
</code></pre></div>
Caches can be configured to serve stale content (i.e. in case of network error during
revalidation), but we can enforce that stale content is not served even with such
configuration or under any other circumstances:
<div class="highlight"><pre><code class="language-text" data-lang="text">Cache-Control: max-age=0, s-maxage=7200, must-revalidate
</code></pre></div>
On a second look, we might decide it&#39;s best to serve stale content from client cache since
revalidation can fail due to network errors that can happen more often than not in a
mobile application. However, we still don&#39;t want our gateway cache to serve stale content under
any circumstances. We can achieve this by using <code>proxy-revalidate</code> instead of <code>must-revalidate</code>:
<div class="highlight"><pre><code class="language-text" data-lang="text">Cache-Control: max-age=0, s-maxage=7200, proxy-revalidate
</code></pre></div>
This is the final combination of directives we need to handle the scenario we originally
described at the beginning of this section.
<h2>Beyond <code>Cache-Control</code></h2>
The solution we came up above will not work if we need to worry about any intermediate
shared (proxy) caches between the client cache and gateway cache (this is the case if we
don&#39;t use HTTPS). This is because <code>s-maxage</code> applies to those caches as well, and we have no
control over purging cached content on them. We need a way to specify directives that
only apply to our gateway cache. Enter <a href="http://tools.ietf.org/html/draft-nottingham-surrogates-00#section-3.6.2">Surrogate-Control</a>.
<code>Surrogate-Control</code> is like <code>Cache-Control</code> except it only applies to gateway caches (also
known as surrogate caches). It is part of a draft spec, but some popular gateway caches
(i.e. Squid and Fastly) already support it. The gateway caches that don&#39;t support
<code>Surrogate-Control</code> usually have their own custom header that works the same way (i.e.
Akamai has <code>Edge-Control</code>).
Now you know about <code>Surrogate-Control</code>, the final solution to handle the scenario described
in the last section is to use both <code>Cache-Control</code> and <code>Surrogate-Control</code>:
<div class="highlight"><pre><code class="language-text" data-lang="text">Cache-Control: max-age=0
Surrogate-Control: max-age=7200, must-revalidate
</code></pre></div>
<h2>Conclusion</h2>
HTTP caching is powerful, but severely underused. Even when it is used, people rarely
leverage it to its maximum potential. Take the time to read all of <a href="http://tools.ietf.org/html/rfc2616#section-13">Caching in HTTP</a>
section of HTTP/1.1 RFC now (even if you&#39;ve already read it before). And, don&#39;t forget
to pay close attention to the subsection on <code>Cache-Control</code>.
</description>
</item>
<item>
<title>The Toiling Programmer: Succeeding as a Developer</title>
<link>https://techblog.thescore.com/2014/10/22/succeeding-as-a-developer/</link>
<pubDate>Wed, 22 Oct 2014 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2014/10/22/succeeding-as-a-developer</guid>
<description>Not sure if you&#39;re a toiling programmer? Ask yourself these questions:
<ul>
<li>Do you feel like you&#39;re drowning, that there&#39;s no end in sight?</li>
<li>Do you get stuck on long running problems (1 or more days long)?</li>
<li>Are you unsure what needs to be done on a <a href="http://en.wikipedia.org/wiki/Issue_tracking_system#Issues">ticket</a>, and yet you continue to work on it?</li>
<li>Do you often get feedback that functionality is missing after you&#39;ve shipped the work?</li>
</ul>
That feeling of dread is cancerous towards your work, your relationships at work and your well being. I&#39;ve personally experienced it many times during my short career -- it&#39;s not a fun experience and that&#39;s why I am writing about it.
Work doesn&#39;t have be like that. You can stop the cycle and feel way better about yourself!
<h2>Manage expectations of other people</h2>
For me, one of the feelings of dread comes from thinking of what other people are thinking of myself -- whether or not I&#39;m doing something quickly enough, or thorough enough. Things of that nature.
Part of that thinking comes from not talking to those people often enough. People are quite understanding of someone doing work because that&#39;s what most people are doing at work!
Talk to the people about the project that you&#39;re working on. Talk about the hardships. Talk about the small victories, the frustrations; people who care about your work want to know that you&#39;re interested in it and that you&#39;re making progress! Even the &#39;failures&#39; can be celebrated!
Being up-front and honest will save you tons of time and frustration. Talk to the person who knows that part of the system the best -- they will have great insight into how it works and the &#39;gotchas&#39; in the system. If you can spend 8 hours solving a problem without help and 30 minutes talking to a colleague about it to get the same result (and the added benefit of having trust between you and your colleague), surely it&#39;s worth it to reach out and talk about your problems.
Admit to your colleagues that you don&#39;t know what you&#39;re doing (one of my favourite phrases) and seek their advice. It&#39;s very liberating! Here are the upsides:
<ul>
<li>You solve the problem quickly with the help of your colleagues</li>
<li>You gain the trust of your colleagues (and boss?)</li>
<li>The problem actually is non-trivial and you did need the help (think of the hours/days you just saved of banging your head against a wall)</li>
</ul>
Here are the downsides:
<ul>
<li>You expose yourself as a fraud (hint: you&#39;re not a fraud)</li>
<li>Loss of pride</li>
<li>Other people think less of you</li>
</ul>
On that last note, I don&#39;t think that&#39;s a valid concern. If someone thinks less of you in this case, then that relationship is toxic. Not being able to rely on the expertise, wisdom, and experience of your co-workers at work is a terrible burden to carry. If someone comes out and exposes themselves as a person who would do that to you, then at least you now know what kind of person they are (net gain for you!).
Learn to rely on your co-workers and to confide in them.
<h3>Who to talk to</h3>
Talk to your boss. Your boss has the most control over your future at your company. Besides the obvious personal reasons, your boss needs to know how things are going because they are often involved with planning for the future with regards to you, your team, and the rest of the company.
Talk to your project manager. If you have a PM, then they are going to need to know how long something is going to take. Periodic feedback on tasks is just as important as estimates because things often take longer than expected! Keep him up to date on things to maintain that relationship!
Talk to your colleagues. There may be a lot of expertise within your team on the problem that you&#39;re working on. Draw upon them and you&#39;ll be a happier person. Regular stand-ups usually allow for this communication to happen.
Talk to stakeholders. The people that you&#39;re doing the work for are most likely interested in how the work is going! They also can give you early feedback on what you have so far and the casual conversations that you have with them can lead you to new information or even just motivation to do the job well.
<h2>Specific Techniques</h2>
While I think the most important aspect is communication, being able to apply specific techniques can be very useful. I&#39;ve come across a few techniques that I think you can apply to combat the feeling of dread.
<h3>Perspectives as a tool</h3>
Does the problem have you stumped? Try thinking about it from a different perspective. Put yourself in their shoes and start asking yourself what&#39;s important and what&#39;s not important with regards to the problem at hand. You can use this technique to come up with missing functionality and to come up with unique solutions.
A great example of this is a problem like refactoring code. From a product owner perspective, refactoring code has no direct benefit, and yet they know that there are future benefits to refactoring the code into something more easy to work with. Conversely, your team would love someone to give the code some TLC because eventually they&#39;ll have to change some behaviour in the affected code.
What if you thought about this from the perspective of someone who wasn&#39;t a programmer? You may find simple solutions to the problems that are presented to you. For example, the person might have ended up going with a simple spreadsheet in Excel. That ends up being a lot more maintainable than a MySQL database + Rails + Web Server. Even if it needs to be online and concurrently accessed, Google Docs Spreadsheets fills this niche!
Think about the perspectives of these people:
<ul>
<li>Users</li>
<li>Product owner</li>
<li>Your team</li>
<li>Legal</li>
<li>Competitors</li>
</ul>
<h3>Talk to the person who made the ticket</h3>
This seems obvious to some people, but it&#39;s not obvious to others. Talk to the person who made the ticket! Maybe they didn&#39;t have all the facts. Maybe they&#39;ve changed their minds or perhaps the work isn&#39;t needed anymore. If you do the work for the ticket without talking to the person responsible for creating it, then that&#39;s a red flag.
Say a task will take you 3 days to do the way it&#39;s written in the ticket. After talking with the ticket reporter, you find out that the part that would take 2 days to do is more of a nice-to-have, and now the ticket will only take 1 day.
That doesn&#39;t mean that the additional work shouldn&#39;t be done, though. It can live on in another ticket where, if it ends up actually being that important, the work will be done. Simply put, delaying a feature until it&#39;s absolutely necessary allows your team and business to focus on the pain points of what you&#39;re trying to solve.
<h3>Do something different</h3>
Working on something else while you think about the problem can really help you come up with a solution to the problem. Take another small ticket on while you think about the larger ticket and a solution may just come to mind.
Or, just take a walk! Or get a coffee, the options are pretty much endless and each person has their own way of taking their mind off the problem.
Another small point here is that the people who you consulted earlier for a solution may come back to you with a solution as well.
<h2>Conclusion</h2>
Always keep an open mind while you&#39;re developing. You may not realize it, but you probably have a support network around you that you can rely on!
</description>
</item>
<item>
<title>Background Processing: Use Einhorn to Spawn and Manage Worker Processes</title>
<link>https://techblog.thescore.com/2014/10/20/background-processing-use-einhorn-to-spawn-and-manage-workers/</link>
<pubDate>Mon, 20 Oct 2014 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2014/10/20/background-processing-use-einhorn-to-spawn-and-manage-workers</guid>
<description>Most modern web applications have a background processing subsystem. In other words,
whenever something does not need to be done in the web request (i.e. sending a
welcome email when a new user signs up), it can be queued up and processed
later outside of the request-response cycle. This allows the web requests to
be fast and responsive.
<h2>Anatomy of a Background Processing System</h2>
A background processing system usually consists of the following:
Queue is where the &quot;jobs&quot; are stored till they&#39;re processed. A job contains
the data necessary for the task at hand (i.e. we need the <code>user_id</code> to look up the
user&#39;s email in our database before sending a welcome email), as well as some
meta data (i.e. the time the job entered into the queue).
Producers put jobs into the queue. For example, the request that handles
user sign-ups can produce a new job for each successful sign-up so a welcome email
will be sent to the user who has just signed up.
Consumers know how to process one or more kinds of jobs. For example, the
consumer responsible for sending welcome emails would know how to look up the
user, build the welcome message, and deliver it to the user&#39;s email address. In
other words, the consumers implement your business logic.
Workers poll the queue for new jobs. When a job becomes available, they
pop it off the queue, and hand it off to the consumer that knows how to process
that job.
<h2>Putting the Pieces Together</h2>
In most languages, you can find a library that addresses all of the pieces above, and
presents you with a complete solution. In Ruby, you can find several libraries such as
<a href="https://github.com/resque/resque">resque</a>, <a href="https://github.com/nesquena/backburner">backburner</a>,
and <a href="https://github.com/collectiveidea/delayed_job">delayed_job</a> that do this.
Suppose you want to put together a custom background processing system. The most
important pieces you need to worry about are the queue and workers. What you choose
as a queue or how you implement your workers will determine the performance and
scalability of the system. How you should choose a queue is outside the scope of
this blog post so let&#39;s move on to talking about the workers.
<h2>The Workers</h2>
As described earlier, workers sit in a loop, and wait for new jobs. When a job
becomes available, they pop it off the queue and process it by delegating to
the appropriate consumer. Here is a sample worker in Ruby that uses
<a href="http://kr.github.io/beanstalkd">beanstalk</a> as the queue:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">#!/usr/bin/env ruby
class Worker
 def self.run(consumers)
 # REF https://github.com/beanstalkd/beaneater
queue = Beaneater::Pool.new([&#39;localhost:11300&#39;])
 # Assume keys map to beanstalk tubes
 queue.watch!(*consumers.keys)
 loop do
 # Wait for 10 seconds to get a job
job = queue.tubes.reserve(10)
 # Assume that we originally enqueued the job with a JSON body
job_data = JSON.parse(job.body)
consumers.fetch(job.tube).perform(job_data)
 end
 end
end
# Assume that we have AlertDetector and DeviceRegistrar defined
# in our app as below:
#
# class AlertDetector
# def self.perform(job_data)
# # Detect an alert (i.e. new goal is scored)
# end
# end
#
# class DeviceRegistrar
# def self.perform(job_data)
# # Register the device for push notifications
# end
# end
Worker.run(
 &#39;alert_detector&#39; =&gt; AlertDetector,
 &#39;device_registrar&#39; =&gt; DeviceRegistrar
)
</code></pre></div>
Given that you put the code above in <code>bin/work</code> inside your application, you
can run it as <code>./bin/work</code> from the root of the application.
The worker above can process only one job at a time. This won&#39;t scale as the
number of jobs increase. You need to be able to process many jobs in parallel.
You can use threads to do this, but threads are difficult to use, and even more
difficult to debug. In addition, if your worker is written in Ruby or Python,
you don&#39;t have true threads anyway.
The alternative is to run many copies of the worker process, which is a
reliable and popular way to achieve concurrency in unix. Now, you need to
worry about how you spin up and manage multiple worker processes. This is
where most people reinvent the the wheel, and write custom code. Don&#39;t
reinvent the wheel.
<h2>Enter Einhorn</h2>
<a href="https://github.com/stripe/einhorn">Einhorn</a> bills itself as the
language-independent shared socket manager. But, shared socket management is
just one of its many features:
<ul>
<li>Einhorn lets you spin up any number of worker processes (the number can be
adjusted on the fly)</li>
<li>Einhorn can spawn a new pool of workers and gracefully kill off the old ones,
allowing seamless upgrades to new versions of your code.</li>
<li>Einhorn gets out of your application&#39;s way</li>
</ul>
This means we can take the worker we&#39;ve written earlier, and run 20 copies of
it without modifying the original code in any way: <code>einhorn -n 20 ./bin/work</code>. That&#39;s
it. Einhorn does indeed get out of your application&#39;s way.
If you use Ruby, Einhorn can also preload your application, which means that it
loads everything prior to forking so that your code is only stored in memory once.
<h2>Conclusion</h2>
At theScore, we&#39;ve started using Einhorn to spin up and manage worker processes
in our background processing system that is responsible for generating and delivering
push alerts. We process millions of messages through that system every day. Einhorn
turned out to be simpler and much more resilient than the custom code we used to have.
</description>
</item>
<item>
<title>3 Ways to Create Classes in Ruby</title>
<link>https://techblog.thescore.com/2014/07/19/3-ways-to-create-classes-in-ruby/</link>
<pubDate>Sat, 19 Jul 2014 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2014/07/19/3-ways-to-create-classes-in-ruby</guid>
<description>Classes are <a href="http://en.wikipedia.org/wiki/First-class_citizen">first-class objects</a> in Ruby.
All classes happen to be instances of <code>Class</code>. In other words, classes are <code>Class</code> objects
just like <code>&#39;hello&#39;</code> and <code>&#39;world&#39;</code> are <code>String</code> objects.
Out of the box, there are 3 ways to create classes in Ruby.
<h2>Use the <code>class</code> Keyword</h2>
In the vast majority of the cases, you would use the <code>class</code> keyword to create a class.
This is usually called a class definition:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">class Customer
attr_reader :full_name, :card_type, :card_number
def initialize(full_name, card_type, card_number)
 @full_name = full_name
 @card_type = card_type
 @card_number = card_number
 end
end
# Creating and using a Customer object:
customer = Customer.new(&#39;Thuva Tharma&#39;, &#39;visa&#39;, 1234)
customer.full_name #=&gt; Thuva Tharma
customer.card_type #=&gt; visa
customer.card_number #=&gt; 1234
</code></pre></div>
If you&#39;re a Rubyist, you&#39;ve seen and done this so many times. So, let&#39;s move on.
<h2>Use <code>Class.new</code></h2>
Just like how we created a customer with <code>Customer.new</code> in the example above, we can create
a class with <code>Class.new</code>:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">Customer = Class.new do
attr_reader :full_name, :card_type, :card_number
def initialize(full_name, card_type, card_number)
 @full_name = full_name
 @card_type = card_type
 @card_number = card_number
 end
end
</code></pre></div>
This is functionally equivalent to regular class definition, but more explicit.
By convention, classes are referenced by constants in Ruby. This is why we&#39;re assigning
the object returned by <code>Class.new</code> to <code>Customer</code> constant, but we can assign it to
a regular variable as well. On the other hand, if you don&#39;t use a constant in a
regular class definition, you will get an error:
<div class="highlight"><pre><code class="language-text" data-lang="text">SyntaxError: class/module name must be CONSTANT
</code></pre></div>
Now, why would you ever want to use <code>Class.new</code> over regular class definition? Let&#39;s
look at creating a custom error class (for good reason, custom error classes should
inherit from <code>StandardError</code>):
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">class CustomerError &lt; StandardError
end
# This usually gets shortened as:
class CustomerError &lt; StandardError; end
</code></pre></div>
You can pass in an argument to <code>Class.new</code> to specify the super class for the class
you&#39;re creating. With this in mind, you can define <code>CustomerError</code> more elegantly:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">CustomerError = Class.new(StandardError)
</code></pre></div>
This is such a minor improvement, but a good use-case nonetheless. Another use-case
for <code>Class.new</code> might be in testing. If you need to define classes only to be used
by your tests, you don&#39;t want to pollute the global namespace with the test class
constants. You can use <code>Class.new</code> to store the test classes in local or instance
variables.
<h2>Use <code>Struct</code></h2>
<code>Struct</code> is available as part of Ruby&#39;s standard library, and you don&#39;t need to
<code>require</code> anything to use it. <code>Struct</code> lets you create classes for data container
objects without any boilerplate:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">Customer = Struct.new(:full_name, :card_type, :card_number)
</code></pre></div>
This has all the features of <code>Customer</code> class we created with regular class
definition as well as using <code>Class.new</code> above. In addition, the customer objects
will have equality as per <a href="http://en.wikipedia.org/wiki/Value_object">value object</a>
semantics:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">customer1 = Customer.new(&#39;Thuva Tharma&#39;, &#39;visa&#39;, 1234)
customer2 = Customer.new(&#39;Thuva Tharma&#39;, &#39;visa&#39;, 1234)
customer3 = Customer.new(&#39;Thuva Tharma&#39;, &#39;amex&#39;, 1234)
customer1 == customer2 #=&gt; true
customer1 == customer3 #=&gt; false
</code></pre></div>
On the other hand, we lose arity checking for arguments passed in to <code>Customer.new</code>.
In other words, you don&#39;t get <code>ArgumentError</code> if you pass in less arguments:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># full_name, card_type, card_number are all nil
Customer.new
# card_type, card_number are nil
Customer.new(&#39;Thuva Tharma&#39;)
# card_number is nil
Customer.new(&#39;Thuva Tharma&#39;, &#39;visa&#39;)
# full_name, card_type, card_number are all present
Customer.new(&#39;Thuva Tharma&#39;, &#39;visa&#39;, 1234)
</code></pre></div>
This is not good. But, if you limit using <code>Struct</code> to create inner classes for
value objects, it&#39;s still safe and useful:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">class Customer
 # Internal: Class to create credit card value objects
CreditCard = Struct.new(:holder, :type, :number)
 attr_reader :full_name, :card
def initialize(full_name, card_type, card_number)
 @full_name = full_name
@card = CreditCard.new(full_name, card_type, card_number)
 end
end
</code></pre></div>
<h2>Conclusion</h2>
In Ruby, you can create classes using the <code>class</code> keyword (class definition),
<code>Class.new</code>, and <code>Struct</code>. Although you would use regular class definition
in the vast majority of cases, <code>Class.new</code> and <code>Struct</code> would sometimes come
in handy if you know how to use them.
</description>
</item>
<item>
<title>Select and Map Are Good</title>
<link>https://techblog.thescore.com/2014/06/25/select-and-map-are-good/</link>
<pubDate>Wed, 25 Jun 2014 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2014/06/25/select-and-map-are-good</guid>
<description>This article argues that when able to one should break down
iteration operations over an array into <code>#map</code> and <code>#select</code> as opposed to
operating on the enumerable through an <code>#each</code>.
<h2>The Examples</h2>
Throughout this article I will refer to the following, contrived, example:
You have an array of numbers <code>[1, 2, 3, 4, 5]</code> and you want to subtract 3
from each of the items and then remove all items that are 0.
A) Using <code>#each</code> you could express this as:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">the_array = [1, 2, 3, 4, 5]
new_array = []
the_array.each do |item|
 new_item = item - 3
 if new_item != 0
 new_array &lt;&lt; new_item
 end
end
</code></pre></div>
B) Using <code>#select</code> and <code>#map</code> you could express this as:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">the_array = [1, 2, 3, 4, 5]
new_array = the_array.map { |item| item - 3 }
new_array.select! { |item| item != 0 }
</code></pre></div>
<h2>The Arguments</h2>
<h3>Better Seperation of Logic</h3>
Example A is doing two things in one block whereas example B is
doing just one thing in each of the two blocks. In general, the less
there is in a block the easier that block is to understand. Breaking a
problem down into map and select means that you have broken the problem
up into two distinct parts.
<h3>Clarity</h3>
Which brings us to what those two parts do. They are actually named.
If I am reading example B and I am trying to find the part where items
are removed, then I look in the select block. If I am looking for the part where the
items are changed then I look in the map block. The method names tell how the
block is to be used.
When reading example A, I have to read all of the <code>#each</code> block if I am
looking for where the values are changed or when the items are
removed.
<h2>Potential Counter Arguments</h2>
I think some people may argue that speed is a big issue. The idea is that
you are iterating over the enumerable twice so it is using more time.
Okay let us assume the time to to process <code>item - 3</code> and assign it to
variable/add it to the array is <code>a</code> and that time to check the
<code>new_item/item != 0</code> and add it the array is <code>b</code>, and the time to setup
each iteration of the array is <code>c</code>. We will also assume <code>i</code> is the number
of iterations to travese the array.
So example A will take <code>i(a + b + c)</code> time and example B will take
<code>i(a + c) + i(b + c)</code> . The difference between these two ends up being
B - A so <code>i(a + b) + 2ic - (i(a + b) + ic)</code> <code>= ic</code>. The difference
is going to be the time to setup the iterations.
Let us check the actual difference in time with a much bigger array:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">require &#39;benchmark&#39;
def with_each(the_array)
 new_array = []
 the_array.each do |item|
 new_item = item - 3
 if new_item != 0
 new_array &lt;&lt; new_item
 end
 end
 new_array
end
def with_map_and_select(the_array)
new_array = the_array.map { |item| item - 3 }
new_array.select! { |item| item != 0 }
end
the_array = (-10000000..10000000).to_a
Benchmark.bmbm do |x|
x.report(&quot;With #each&quot;) { with_each(the_array) }
x.report(&quot;With #map and #select&quot;) { with_map_and_select(the_array) }
end
</code></pre></div>
The output on my computer is:
<div class="highlight"><pre><code class="language-text" data-lang="text"> user system total real
With #each 1.830000 0.040000 1.870000 ( 1.924606)
With #map and #select 2.440000 0.030000 2.470000 ( 2.468480)
</code></pre></div>
So the time difference is about +25% for example B and this is a
really simple example. The difference as a percentage will fall as the
complexity of the operations increases.
So the time difference does exists but I don&#39;t think this is going to be a
huge factor in most cases.
</description>
</item>
<item>
<title>Simple Ways to Protect an API: HTTP Basic Authentication and HTTP Token Authentication</title>
<link>https://techblog.thescore.com/2014/06/25/http-basic-authentication-and-http-token-authentication/</link>
<pubDate>Wed, 25 Jun 2014 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2014/06/25/http-basic-authentication-and-http-token-authentication</guid>
<description>The backend for theScore mobile apps consists of many different web services (Core
Sports, Personalization, Push Alerts, etc.). Each service provides its own RESTful API.
Some of these services are internal. As such, they only have a very limited number of
consumers. Having a limited number of consumers for an API lets you keep the authentication
really simple for that API.
For the purposes of this blog post, suppose that we have two services. One that is responsible
for detecting alerts (i.e. new goal is scored), and another that is responsible for actually
sending those alerts to mobile devices. Let&#39;s call these <code>Detector</code> and <code>Sender</code> respectively.
Also, suppose that <code>Sender</code> is a web service, and <code>Detector</code> is a background processing system.
Further, whenever <code>Detector</code> detects an alert, it uses the API provided by <code>Sender</code> to deliver
the alerts.
The internals of the services described above would look like the following in Ruby/Rails:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># Detector Service
class GoalDetector
 def perform(*args)
 # Send the alert upon detection of new alert
 # REF https://github.com/lostisland/faraday
client = Faraday.new(&#39;https://sender.abc.com&#39;)
client.post(&#39;/alerts&#39;, message: &#39;Spain Goal&#39;)
 end
end
# Sender Service (https://sender.abc.com)
class AlertsController &lt; ActionController::Base
 # POST /alerts
 def create
 # Send an alert with message in params[:message]
 end
end
</code></pre></div>
All is good except that our <code>Sender</code> API is public. Anyone who happens to stumble upon
that API can discover this great power to send push alerts to our users. So, we need
a way to protect the API from unauthorized access.
<h2>HTTP Basic Authentication</h2>
This is generally the go-to solution for this problem. Our example above would look like
the following with Basic Authentication support:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># Detector Service
class GoalDetector
 def perform(*args)
 # Send the alert upon detection of new alert
 # REF https://github.com/lostisland/faraday
client = Faraday.new(&#39;https://sender.abc.com&#39;) do |c|
c.basic_auth(&#39;username&#39;, &#39;password&#39;)
 end
client.post(&#39;/alerts&#39;, message: &#39;Spain Goal&#39;)
 end
end
# Sender Service (https://sender.abc.com)
class AlertsController &lt; ActionController::Base
 before_action :authenticate
 # POST /alerts
 def create
 # Send an alert with message in params[:message]
 end
 private
 def authenticate
authenticate_or_request_with_http_basic do |username, password|
username == &#39;username&#39; &amp;&amp; password == &#39;password&#39;
 end
 end
end
# NOTE: In real applications, we will not have the
# authentication credentials lying around in code.
# We will store them in external configuration.
</code></pre></div>
Now, the <code>POST</code> request to <code>https://sender.abc.com/alerts</code> would require that an
<code>Authorization</code> header is present with the credentials encoded correctly in the value.
The header would look like:
<div class="highlight"><pre><code class="language-text" data-lang="text"># dXNlcm5hbWU6cGFzc3dvcmQ=\n is Base64-encoded value
# of the string username:password
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=\n
</code></pre></div>
If you don&#39;t have the credentials, you won&#39;t be able to generate the <code>Authorization</code>
header correctly. If you send the wrong value for the header in your request, you will
get <code>401 Unauthorized</code> response back. So, we&#39;ve now effectively protected the <code>Sender</code>
service from unauthorized access.
<h2>HTTP Token Authentication</h2>
With Basic Authentication, you need to configure both username and password
in <code>Sender</code> and <code>Detector</code> services. But, if the password is really strong, then it
is a secure credential by itself. In other words, you can think of the password as a
secure token. In this case, the username becomes redundant. This is where Token
Authentication comes in.
Our code example would look like the following with Token Authentication support:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># Detector Service
class GoalDetector
 def perform(*args)
 # Send the alert upon detection of new alert
 # REF https://github.com/lostisland/faraday
client = Faraday.new(&#39;https://sender.abc.com&#39;) do |c|
 c.token_auth(&#39;WCZZYjnOQFUYfJIN2ShH1iD24UHo58A6TI&#39;)
 end
client.post(&#39;/alerts&#39;, message: &#39;Spain Goal&#39;)
 end
end
# Sender Service (https://sender.abc.com)
class AlertsController &lt; ActionController::Base
 before_action :authenticate
 # POST /alerts
 def create
 # Send an alert with message in params[:message]
 end
 private
 def authenticate
authenticate_or_request_with_http_token do |token, options|
 token == &#39;WCZZYjnOQFUYfJIN2ShH1iD24UHo58A6TI&#39;
 end
 end
end
# NOTE: In real applications, we will not have the
# authentication credentials lying around in code.
# We will store them in external configuration.
</code></pre></div>
Now, the <code>POST</code> request to <code>https://sender.abc.com/alerts</code> would require that that an
<code>Authorization</code> header is present with the correct token in the value. The header
would look like:
<div class="highlight"><pre><code class="language-text" data-lang="text">Authorization: Token token=&quot;WCZZYjnOQFUYfJIN2ShH1iD24UHo58A6TI&quot;
</code></pre></div>
If you send the wrong token in the <code>Authorization</code> header, you will get
<code>401 Unauthorized</code> response back. Again, we&#39;ve protected the API from unauthorized
access.
<h2>Conclusion</h2>
Both HTTP Basic Authentication and HTTP Token Authentication offer really simple
solutions to protect an API from unauthorized access. But, with Token Authentication,
you will have one less thing to configure in your services and consumers. Because of
that, I prefer using Token Authentication.
Having said that, unlike Basic Authentication which is an approved spec, Token
Authentication is a draft spec:
<ul>
<li><a href="http://tools.ietf.org/html/rfc2617">HTTP Authentication: Basic and Digest Access Authentication</a></li>
<li><a href="http://tools.ietf.org/html/draft-hammer-http-token-auth-01">HTTP Authentication: Token Access Authentication</a></li>
</ul>
Regardless of being a draft spec, Token Authentication is well-supported and really
easy-to-use. I don&#39;t see any practical reasons not to use it.
NOTE: Both Basic Authentication and Token Authentication are insecure unless used
over <code>HTTPS</code>. All the code examples above use <code>HTTPS</code> to make this explicit. We use
<code>HTTPS</code> on all of our APIs at theScore.
</description>
</item>
<item>
<title>How You Nest Modules Matters in Ruby</title>
<link>https://techblog.thescore.com/2014/05/28/how-you-nest-modules-matters-in-ruby/</link>
<pubDate>Wed, 28 May 2014 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2014/05/28/how-you-nest-modules-matters-in-ruby</guid>
<description>Ruby provides two different syntaxes to nest modules (and classes):
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"># Syntax #1
module API
 module V1
 end
end
# Syntax #2
module API::V1
end
</code></pre></div>
Syntax #2 requires that <code>API</code> module already exists. Most Rubyists know that.
Besides this difference, they think that these two syntaxes are interchangeable.
With that line of thinking, what syntax to use for nesting modules turns out to
be a matter of preference.
But, the syntax you choose matters, which I will demonstrate shortly with
examples that relate to how you would organize a versioned REST API.
Modules are just constants in Ruby, as such, regular constant look-up rules apply.
The very first rule relates to how modules are nested. For the purpose of this
blog post, we will ignore regular constants and just focus on modules.
When you reference a constant in a nested context, Ruby looks it up inside
out:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">module API
 class Responder
 end
 module V1
 class Controller
 def action
 Responder.respond_with(&#39;Hello, World!&#39;)
 end
 end
 end
end
</code></pre></div>
When <code>#action</code> gets called, Ruby will see if <code>Responder</code> is defined inside
<code>API::V1::Controller</code> first, <code>API::V1</code> second, <code>API</code> third, and finally in the
top level. If it wasn&#39;t for this inside-out look-up, referencing <code>Responder</code> as
above would not have worked. Instead, it would have to be referenced as
<code>API::Responder</code>.
You can actually access the nesting information Ruby uses for this
look-up with <code>Module.nesting</code>:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">module API
 class Responder
 end
 module V1
 class Controller
 p Module.nesting #=&gt; [API::V1::Controller, API::V1, API]
 def action
 Responder.respond_with(&#39;Hello, World!&#39;)
 end
 end
 end
end
</code></pre></div>
Now, let&#39;s see how the look-up behavior changes when we mix in syntax #2:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">module API
 class Responder
 end
end
module API::V1
 class Controller
 p Module.nesting #=&gt; [API::V1::Controller, API::V1]
 def action
 Responder.respond_with(&#39;Hello, World!&#39;)
 end
 end
end
</code></pre></div>
When <code>#action</code> gets called, you will now get a <code>NameError</code>:
<div class="highlight"><pre><code class="language-text" data-lang="text">NameError: uninitialized constant API::V1::Controller::Responder
</code></pre></div>
This is because some nesting information gets lost when you use syntax #2.
You can see this from what <code>Module.nesting</code> returns in the example above.
As per that example, Ruby now looks for <code>Responder</code> inside <code>API::V1::Controller</code>
and <code>API::V1</code>. It no longer looks inside <code>API</code>, where <code>Responder</code> is actually
defined hence the error.
<h2>Conclusion</h2>
The syntax you use for nesting modules in Ruby should not be a matter of
preference. The syntax you use could be the difference between a working
program, and the one that throws an error. So choose wisely. Having said
that, I prefer syntax #1.
By the way, we&#39;ve explored just one aspect of how constant look-up works
in Ruby in general. If you use Rails, the matters gets even more complicated
with the autoloading magic. The following references will help you learn more:
<ul>
<li><a href="http://cirw.in/blog/constant-lookup">Everything you ever wanted to know about constant lookup in Ruby</a></li>
<li><a href="http://urbanautomaton.com/blog/2013/08/27/rails-autoloading-hell">Rails autoloading — how it works, and when it doesn&#39;t</a></li>
</ul>
</description>
</item>
<item>
<title>Benchmarking JSON Generation in Ruby</title>
<link>https://techblog.thescore.com/2014/05/23/benchmarking-json-generation-in-ruby/</link>
<pubDate>Fri, 23 May 2014 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2014/05/23/benchmarking-json-generation-in-ruby</guid>
<description>At theScore, we have a big JSON API with hundreds of end-points that expose
sports data. Since sports data is very rich, the JSON representation of most
of the resources on our API tend to be complex. As a result, a lot of time is
spent on generating JSON in our Rails application.
At the time we wrote the API, <a href="https://github.com/nesquena/rabl">RABL</a> was a
great choice. When <a href="https://github.com/rails-api/active_model_serializers">ActiveModel Serializers</a>
came out with an object-oriented approach to generating JSON, we were excited.
We tried it out, and ended up with something that is more maintainable than
the RABL counterparts.
But, migrating the entire JSON generation code from RABL to ActiveModel
Serializers (AMS) would require massive effort. Since RABL is still
relatively easy to use and maintain, more ease of use and maintenance
aspects of AMS is not a good enough reason to do the migration. On the
other hand, if we can significantly increase the performance of JSON
generation, then the effort to do this migration is justified.
Having said that, we had been using plain Ruby presenters to generate JSON
with great success in some of our smaller API projects. If AMS turn out to
be much slower than using presenters, then we might want to explore presenters
a little more as a possible approach for our main API.
So that&#39;s enough background as to why we&#39;re doing this benchmark.
<h2>The Setup</h2>
For the benchmark, we extracted real RABL code from our API that generates
JSON for basketball teams and events, and simplified a bit. Then, we implemented
AMS and plain Ruby presenters to generate the same JSON. We have 3 different
cases that we want to benchmark.
<h3>Case 1: Ultra Simple</h3>
The simplest case in which we generate JSON for a single team object:
<div class="highlight"><pre><code class="language-json" data-lang="json">{
 &quot;abbreviation&quot;: &quot;TOR&quot;,
 &quot;full_name&quot;: &quot;Toronto Raptors&quot;,
 &quot;location&quot;: &quot;Toronto&quot;
}
</code></pre></div>
<h3>Case 2: Simple</h3>
Slightly more complex than above in which we embed team objects into
the main event object:
<div class="highlight"><pre><code class="language-json" data-lang="json">{
 &quot;game_date&quot;: &quot;2014-05-21&quot;,
 &quot;game_type&quot;: &quot;Regular Season&quot;,
 &quot;status&quot;: &quot;Final&quot;,
 &quot;away_team&quot;: {
 &quot;abbreviation&quot;: &quot;MIA&quot;,
 &quot;full_name&quot;: &quot;Miami Heat&quot;,
 &quot;location&quot;: &quot;Miami&quot;,
 &quot;medium_name&quot;: null,
 &quot;short_name&quot;: &quot;Heat&quot;
 },
 &quot;home_team&quot;: {
 &quot;abbreviation&quot;: &quot;TOR&quot;,
 &quot;full_name&quot;: &quot;Toronto Raptors&quot;,
 &quot;location&quot;: &quot;Toronto&quot;,
 &quot;medium_name&quot;: &quot;Toronto&quot;,
 &quot;short_name&quot;: &quot;Raptors&quot;
 }
}
</code></pre></div>
<h3>Case 3: Complex</h3>
The most complex case in which we not only embed team objects and
a box score object into the main event object, but the box score
in turn also embed a last play object. Also, there are more
attributes on the event object.
<div class="highlight"><pre><code class="language-json" data-lang="json">{
 &quot;game_date&quot;: &quot;2014-05-21&quot;,
 &quot;game_type&quot;: &quot;Regular Season&quot;,
 &quot;status&quot;: &quot;Final&quot;,
 &quot;share_url&quot;: &quot;http://thesco.re/123&quot;,
 &quot;sport_name&quot;: &quot;basketball&quot;,
 &quot;away_ranking&quot;: 10,
 &quot;away_region&quot;: &quot;Pac-12&quot;,
 &quot;home_ranking&quot;: 15,
 &quot;home_region&quot;: &quot;Top 25&quot;,
 &quot;important&quot;: true,
 &quot;location&quot;: &quot;Washington, DC&quot;,
 &quot;away_team&quot;: {
 &quot;abbreviation&quot;: &quot;MIA&quot;,
 &quot;full_name&quot;: &quot;Miami Heat&quot;,
 &quot;location&quot;: &quot;Miami&quot;,
 &quot;medium_name&quot;: null,
 &quot;short_name&quot;: &quot;Heat&quot;
 },
 &quot;home_team&quot;: {
 &quot;abbreviation&quot;: &quot;TOR&quot;,
 &quot;full_name&quot;: &quot;Toronto Raptors&quot;,
 &quot;location&quot;: &quot;Toronto&quot;,
 &quot;medium_name&quot;: &quot;Toronto&quot;,
 &quot;short_name&quot;: &quot;Raptors&quot;
 },
 &quot;box_score&quot;: {
 &quot;has_statistics&quot;: true,
 &quot;progress&quot;: &quot;11:23 2nd&quot;,
 &quot;attendance&quot;: &quot;21,307&quot;,
 &quot;referees&quot;: &quot;Thuva, Nate, Roel&quot;,
 &quot;last_play&quot;: {
 &quot;points_type&quot;: &quot;Field Goal&quot;,
 &quot;player_fouls&quot;: 10,
 &quot;player_score&quot;: 15,
 &quot;record_type&quot;: &quot;Postseason&quot;,
 &quot;seconds&quot;: 100
 }
 }
}
</code></pre></div>
We will also benchmark generating JSON for a collection of 100 objects
for each case above.
Further, the benchmark is run under Ruby 2.1.1, and with the latest versions
of RABL and AMS (0.9.3 and 0.8.1 respectively) at the time of writing this.
The machine in which the benchmark is run is irrelevant because we&#39;re only
interested in relative performance of RABL, AMS, and presenters. You can
checkout the setup <a href="https://github.com/thuva/json_serialization_benchmark">here</a>.
We will be running the benchmark twice. You will see why shortly.
<h2>First Run</h2>
As you can see below, AMS is about 2-3X slower than presenters. Since we&#39;re not going for
pure speed, it&#39;s still worth using AMS as it provides many features that we will likely to
implement ourselves for presenters. So we&#39;re going to stop comparing them now.
On the other hand, RABL is significantly slower than AMS. It is about 20-25X slower. This
difference remains the same when we deal with a collection of objects.
<div class="highlight"><pre><code class="language-text" data-lang="text">The times you see is the total for 10,000 iterations. Refer to &quot;The Setup&quot; section
above to understand the different cases: Ultra Simple, Simple, and Complex.
user system total real
RABL Ultra Simple 1.610000 0.420000 2.030000 ( 2.029277)
AMS Ultra Simple 0.080000 0.000000 0.080000 ( 0.083180)
Presenters Ultra Simple 0.030000 0.000000 0.030000 ( 0.029341)
--------------------------------------------------------------------------------------
RABL Simple 9.080000 2.250000 11.330000 ( 11.342106)
AMS Simple 0.510000 0.000000 0.510000 ( 0.505133)
Presenters Simple 0.140000 0.000000 0.140000 ( 0.140356)
--------------------------------------------------------------------------------------
RABL Complex 19.030000 4.490000 23.520000 ( 23.529937)
AMS Complex 1.010000 0.000000 1.010000 ( 1.007729)
Presenters Complex 0.320000 0.010000 0.330000 ( 0.320893)
user system total real
RABL Ultra Simple: Collection 1.400000 0.420000 1.820000 ( 1.829971)
AMS Ultra Simple: Collection 0.060000 0.000000 0.060000 ( 0.062879)
Presenters Ultra Simple: Collection 0.020000 0.000000 0.020000 ( 0.017285)
--------------------------------------------------------------------------------------
RABL Simple: Collection 8.420000 2.180000 10.600000 ( 10.590213)
AMS Simple: Collection 0.440000 0.000000 0.440000 ( 0.443547)
Presenters Simple: Collection 0.110000 0.000000 0.110000 ( 0.105298)
--------------------------------------------------------------------------------------
RABL Complex: Collection 18.370000 4.370000 22.740000 ( 22.744374)
AMS Complex: Collection 0.940000 0.000000 0.940000 ( 0.944308)
Presenters Complex: Collection 0.270000 0.000000 0.270000 ( 0.272239)
</code></pre></div>
<h2>Second Run</h2>
As per the results of the first benchmark run, RABL is much slower compared to AMS.
There is no good reason why it has to be. So we did a bit of investigation, and discovered
that RABL spends a lot of time on template lookup. We can configure RABL to cache this lookup:
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">Rabl.configure do |config|
 config.cache_sources = true
end
</code></pre></div>
Below, you can see the results of the benchmark with template lookup caching enabled for RABL.
Although performance doubles for RABL, it is still much slower. It is now about 12X
slower compared to AMS.
<div class="highlight"><pre><code class="language-text" data-lang="text"> user system total real
RABL Ultra Simple 1.050000 0.000000 1.050000 ( 1.051644)
AMS Ultra Simple 0.080000 0.000000 0.080000 ( 0.084204)
Presenters Ultra Simple 0.030000 0.000000 0.030000 ( 0.028589)
--------------------------------------------------------------------------------------
RABL Simple 5.820000 0.020000 5.840000 ( 5.834718)
AMS Simple 0.490000 0.000000 0.490000 ( 0.492227)
Presenters Simple 0.130000 0.000000 0.130000 ( 0.128998)
--------------------------------------------------------------------------------------
RABL Complex 12.790000 0.020000 12.810000 ( 12.810967)
AMS Complex 0.990000 0.010000 1.000000 ( 0.998783)
Presenters Complex 0.330000 0.000000 0.330000 ( 0.322893)
user system total real
RABL Ultra Simple: Collection 0.830000 0.000000 0.830000 ( 0.832709)
AMS Ultra Simple: Collection 0.060000 0.000000 0.060000 ( 0.062520)
Presenters Ultra Simple: Collection 0.020000 0.000000 0.020000 ( 0.018377)
--------------------------------------------------------------------------------------
RABL Simple: Collection 5.420000 0.010000 5.430000 ( 5.428974)
AMS Simple: Collection 0.440000 0.000000 0.440000 ( 0.444187)
Presenters Simple: Collection 0.100000 0.000000 0.100000 ( 0.103999)
--------------------------------------------------------------------------------------
RABL Complex: Collection 12.350000 0.010000 12.360000 ( 12.364715)
AMS Complex: Collection 0.940000 0.010000 0.950000 ( 0.942812)
Presenters Complex: Collection 0.270000 0.000000 0.270000 ( 0.263692)
</code></pre></div>
<h2>Conclusion</h2>
To reiterate, AMS is significantly faster compared RABL. Migrating to AMS from RABL
not only gives us greater ease of use and maintenance, but it also gives us great
performance gains. So the effort to do the migration is rightly justified.
Benchmarking is a tricky business. There might be some big holes in the benchmarking
above. Checkout out the setup <a href="https://github.com/thuva/json_serialization_benchmark">here</a>,
and feel free to comment if you find any issues with it.
Update: When the post was originally written, the assumption was that
<a href="https://github.com/ohler55/oj">Oj</a> was being used as the JSON encoding engine by all
of RABL, AMS, and presenters However, it turned out not to be the case.
In fact, it was only being used by RABL as it is the engine RABL uses by default. In
othewords, the benchmark was unfair to AMS and presenters. With the Oj boost to AMS and
presenters, the relative performance of RABL gets even worse. This post has been updated
to include the latest benchmark results. You can find the original benchmark results
<a href="https://gist.github.com/thuva/69f2cccee857e4fdc4f0">here</a>.
</description>
</item>
<item>
<title>Analyzing S3 and CloudFront Access Logs with AWS RedShift</title>
<link>https://techblog.thescore.com/2014/05/14/analyzing-s3-and-cloudfront-access-logs-with-redshift/</link>
<pubDate>Wed, 14 May 2014 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2014/05/14/analyzing-s3-and-cloudfront-access-logs-with-redshift</guid>
<description>It&#39;s not really a surprise that we deal with fairly high volumes of user
traffic at theScore. We use AWS for pretty much all of our hosting needs and
their two object distribution services - CloudFront and S3 - are two of the
most heavily used offerings in our environment. S3 provides file storage for
essentially unlimited data, split across as many &quot;buckets&quot; as you would like
to create. Separate from that is CloudFront, which is a content distribution
network similar to Akamai or Fastly. CloudFront points at one or more origins,
and adheres to standard HTTP caching headers. Typically we will deploy S3
buckets or application servers behind CloudFront &quot;distributions&quot; - this allows
the majority of HTTP requests to our properties be soaked up by CloudFront
edge servers.
On a daily basis this works very well for us. The challenge comes though when
we need to perform any analysis on these logs. The S3 and CloudFront logs for
us are easily terabytes of data per year, and traditional log parsers tend to
not handle that size of data. There are some providers out there that specialize
in parsing these logs, and none of them have been able to provide reports on a
dataset of this size unfortunately.
<h2>Columnar Databases</h2>
So enter RedShift. RedShift is a fairly new offering from AWS. It&#39;s a columnar
database built off of PostgreSQL. The scope of this post doesn&#39;t go into the
details of columnar data stores, but the TL;DR is that instead of arranging data
on disk by records (or rows) the data is stored by each column. So say you have
a classic &quot;users&quot; table with username, e-mail, and password columns. In MySQL on
disk the data will be stored in exactly that order first for user ID 1, then 2,
then 3 and so on. With a columnar store all the usernames are first stored, and
then afterwards all of the e-mail addresses, and then all of the passwords.
This drives a couple properties that make RedShift and others very fast when used
for data analysis. Due to the columns being grouped together physically, running
reports across specific columns of your data is very, very fast. For the above
contrived &quot;users&quot; example, this isn&#39;t very relevant. Instead however think about
querying invoices in this type of story - very often you&#39;d like to run reports
that are grouped by the company responsible for the invoice, the payment method,
and perhaps summary information on the dollar amount of the invoice.
There are of course several downsides that would stop you from using RedShift or
its brethren for online transaction processing. Namely pretty much none of them are
ACID compliant, which is a big deal if you actually value your data (or at least,
it&#39;s a big deal when you need ACID compliance). The other primary disadvantage is
that since the data is essentially being stored in a non-natural form, inserts and
updates are much more costly than a standard database and updates to the data
definition are usually impossible without recreating tables.
With these constrains though these databases are still a fantastic tool when paired
with a more traditional SQL database. Usually you will feed the data from your OLTP
system into your data warehouse on a scheduled basis.
<h2>RedShift for log data</h2>
Log data is an interesting case for RedShift. In our environment as mentioned
previously we have so much log data from our CloudFront and S3 usage that nobody
could conceivably work with those datasets using standard text tools such as grep
or tail. Many people load their access logs into databases, but we have not found
this to be feasible using MySQL or PostgreSQL due to the fact that ad-hoc queries
run against sets with billions of rows can take hours. Once imported into RedShift
the same queries take minutes at the most.
<h2>Creating a RedShift instance</h2>
Make sure you have the latest version of the <a href="https://aws.amazon.com/cli">AWS CLI tools</a>.
You can install this if you already have Python and pip by simply running
<code>pip install awscli</code>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">~$ aws redshift create-cluster\
 --cluster-identifier logdata \
 --cluster-type single-node \
 --node-type dw1.xlarge \
 --master-username luke \
 --master-user-password password \
 --cluster-security-groups default
</code></pre></div>
Wait for the cluster to boot - you can check the status at any time using:
<div class="highlight"><pre><code class="language-bash" data-lang="bash">~$ aws redshift describe-clusters --cluster-identifier logdata
</code></pre></div>
This should output quite a bit of useful data about your cluster:
<div class="highlight"><pre><code class="language-json" data-lang="json">{
 &quot;Clusters&quot;: [
 {
 &quot;PubliclyAccessible&quot;: true,
 &quot;MasterUsername&quot;: &quot;luke&quot;,
 &quot;VpcSecurityGroups&quot;: [],
 &quot;NumberOfNodes&quot;: 1,
 &quot;PendingModifiedValues&quot;: {},
 &quot;ClusterVersion&quot;: &quot;1.0&quot;,
 &quot;AutomatedSnapshotRetentionPeriod&quot;: 1,
 &quot;ClusterParameterGroups&quot;: [
 {
 &quot;ParameterGroupName&quot;: &quot;default.redshift-1.0&quot;,
 &quot;ParameterApplyStatus&quot;: &quot;in-sync&quot;
 }
 ],
 &quot;DBName&quot;: &quot;dev&quot;,
 &quot;PreferredMaintenanceWindow&quot;: &quot;wed:03:00-wed:03:30&quot;,
 &quot;Endpoint&quot;: {
 &quot;Port&quot;: 5439,
 &quot;Address&quot;: &quot;your-endpoint.us-east-1.redshift.amazonaws.com&quot;
 },
 &quot;AllowVersionUpgrade&quot;: true,
 &quot;ClusterCreateTime&quot;: &quot;2014-05-12T17:13:26.280Z&quot;,
 &quot;ClusterSecurityGroups&quot;: [
 {
 &quot;Status&quot;: &quot;active&quot;,
 &quot;ClusterSecurityGroupName&quot;: &quot;default&quot;
 }
 ],
 &quot;ClusterIdentifier&quot;: &quot;logdata&quot;,
 &quot;ClusterNodes&quot;: [
 {
 &quot;NodeRole&quot;: &quot;SHARED&quot;,
 &quot;PrivateIPAddress&quot;: &quot;snip&quot;,
 &quot;PublicIPAddress&quot;: &quot;snip&quot;
 }
 ],
 &quot;AvailabilityZone&quot;: &quot;us-east-1d&quot;,
 &quot;NodeType&quot;: &quot;dw1.xlarge&quot;,
 &quot;Encrypted&quot;: false,
 &quot;ClusterRevisionNumber&quot;: &quot;786&quot;,
 &quot;ClusterStatus&quot;: &quot;available&quot;
 }
 ]
}
</code></pre></div>
You can see that the cluster status is set to &quot;active&quot;. Also shown is the
endpoint that you should use to connect to the RedShift cluster; the standard
Postgres command line tools should work just fine:
<div class="highlight"><pre><code class="language-bash" data-lang="bash">~$ psql -h your-endpoint.us-east-1.redshift.amazonaws.com -p 5439 -U luke -W
</code></pre></div>
<h2>Creating the initial schema</h2>
Note that we will be creating separate tables for S3 and CloudFront logs, simply
because the format differs in some small ways. Common columns share their names so
that any queries you end up writing against the data will work in either.
Run the following creation statements through the PostgreSQL client:
<div class="highlight"><pre><code class="language-sql" data-lang="sql">CREATE TABLE s3_logentries (
creator VARCHAR(MAX) ENCODE RUNLENGTH,
bucket VARCHAR(255) ENCODE LZO,
logdate VARCHAR(MAX) SORTKEY ENCODE LZO,
logtime VARCHAR(MAX) ENCODE LZO,
cip VARCHAR(50) ENCODE LZO,
requestor VARCHAR(MAX) ENCODE LZO,
requestid VARCHAR(MAX) ENCODE LZO,
operation VARCHAR(MAX) ENCODE LZO,
requestkey VARCHAR(MAX) ENCODE LZO,
uri VARCHAR(MAX) DISTKEY ENCODE LZO,
status VARCHAR(MAX) ENCODE LZO,
errorcode VARCHAR(MAX) ENCODE LZO,
bytessent VARCHAR(MAX) ENCODE LZO,
objectsize VARCHAR(MAX) ENCODE LZO,
totaltime VARCHAR(MAX) ENCODE LZO,
turnaroundtime VARCHAR(MAX) ENCODE LZO,
referer VARCHAR(MAX) ENCODE LZO,
useragent VARCHAR(MAX) ENCODE LZO,
versionid VARCHAR(10) ENCODE LZO
);
CREATE TABLE cf_logentries (
 logdate VARCHAR(MAX) SORTKEY,
logtime VARCHAR(MAX) ENCODE LZO,
edge VARCHAR(40) ENCODE LZO,
 bytessent INT ENCODE LZO,
cip VARCHAR(50) ENCODE LZO,
method VARCHAR(50) ENCODE LZO,
host VARCHAR(MAX) ENCODE LZO,
uri VARCHAR(MAX) DISTKEY ENCODE LZO,
status VARCHAR(20) ENCODE LZO,
creferrer VARCHAR(MAX) ENCODE LZO,
useragent VARCHAR(MAX) ENCODE LZO,
cs_uri_query VARCHAR(MAX) ENCODE LZO,
cookie VARCHAR(MAX) ENCODE LZO,
x_edge_result_type VARCHAR(MAX) ENCODE LZO,
x_edge_request_id VARCHAR(MAX) ENCODE LZO,
x_host_header VARCHAR(MAX) ENCODE LZO,
protocol VARCHAR(10) ENCODE LZO,
 cs_bytes INT ENCODE LZO,
time_taken VARCHAR(MAX) ENCODE LZO
);
</code></pre></div>
There are a few things to call out here. You&#39;ll note that in most cases there&#39;s not
really much thought given to the data type - instead of limiting the <code>VARCHAR</code>s and what
not, it&#39;s just relaxed with <code>MAX</code> size constraints for each column. Typically you&#39;d want
to size things appropriately; however for our simple examples this really has no effect
on the performance of our queries.
Absent are any specifications for index creation and foreign key constraints. RedShift
(like most databases of it&#39;s type) don&#39;t require or even support index specifications as
they are not used by the data storage backend or the query planner. Foreign key
constraints, since they depend on traditional indexes, are also unsupported although
RedShift will accept their creation for informational purposes only.
Additionally there&#39;s some new keywords here. The first is the <code>ENCODE</code> keyword. RedShift
will compress all data that you import, and each column can have a separate compression
definition. In practice if you leave off the compression stanza then RedShift will wait
for a couple hundred thousand rows to be inserted and then begin analyzing the columns
to select an appropriate compression algorithm. Once you&#39;ve worked with similar datasets
though, you can capture these compression specifications and re-use them in your table
creation.
Last are the <code>SORTKEY</code> and <code>DISTKEY</code> options on columns. <code>SORTKEY</code> specifies how the data in
a table is stored on disk, and will dictate the natural order. If most of your reports
are constrained or ordered by a single column then that would be the best candidate for
this key (in our example the date is an obvious one).
<code>DISTKEY</code> needs a little more explaining. RedShift is a multiple node system, and can have
anywhere from 1 to 100 database nodes participating in your instance. This test instance,
and the one we actually use right now at theScore, are both single instance. However, when
you branch out into a multi-instance topology then RedShift will begin sharding your data
across the nodes using the chosen <code>DISTKEY</code>. The <a href="http://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-best-dist-key.html">documentation</a>
recommends that you select a distribution key that will be used frequently in <code>JOIN</code>s, which
is why we pick the URI. We don&#39;t actually use multiple nodes here though, so it&#39;s just
for future reference.
<h2>Load some data!</h2>
Now that you actually have the schema up and running, let&#39;s actually load some data.
CloudFront distributions and S3 buckets both can be configured to log to a specific bucket
and key; obviously that has to be working for these examples. Let&#39;s assume that we have
a distribution in CloudFront with the ID of <code>E1DHT7QI9H0ZOB</code>, and that is&#39; logging to a
bucket named <code>cloudfront-logs</code>.
RedShift adds a <code>COPY</code> command to the Postgres language that allows you to import logs from
that bucket directly. It&#39;s quite flexible and allows you to specify delimiters, error
thresholds, compression schemas and so on. For our simple example though, we&#39;ll just load
one month of logs from just one of our CloudFront distributions:
<div class="highlight"><pre><code class="language-sql" data-lang="sql">COPY cf_logentries
 FROM &#39;s3://cloudfront-logs/E1DHT7QI9H0ZOB.2014-04-&#39;
 CREDENTIALS &#39;aws_access_key_id=;aws_secret_access_key=&#39;
DELIMITER &#39;\t&#39; MAXERROR 200 FILLRECORD IGNOREHEADER 2 gzip;
</code></pre></div>
We&#39;re loading standard, tab-delimited gzip-compressed log files with the headers ignored.
Additionally we&#39;re using the <code>FILLRECORD</code> parameter. This causes the import process to
pad out missing records at the end of a line with <code>NULL</code> values. This is pretty useful as
periodically the CloudFront and S3 log formats change, and they add new columns. Finally,
we&#39;re telling the command to abort the copy if 200 errors are found.
This could take some - in my example the full load took just over 3 hours:
<div class="highlight"><pre><code class="language-text" data-lang="text">INFO: Load into table &#39;cf_logentries&#39; completed, 137647904 record(s) loaded successfully.
COPY
</code></pre></div>
You can use the same <code>COPY</code> command to load your S3 logs as long as you specify the correct
table as the first parameter to the command.
<h2>Running Queries</h2>
That&#39;s pretty much it for the setup. Now you can just run arbitrary queries against the log
tables. Here some fairly trivial examples that have come in handy:
<div class="highlight"><pre><code class="language-sql" data-lang="sql">SELECT uri, count(*) as hits
FROM cf_logentries
GROUP BY uri ORDER BY hits DESC;
</code></pre></div>
Dead simple; this takes all the records and just spits out the unique URIs by popularity:
<div class="highlight"><pre><code class="language-text" data-lang="text"> uri | hits
----------------------------------------------------------+----------
/api/v1/news | 17017442
/api/v1/news.json | 9923266
/api/v1/rivers/nhl/news/pinned | 6901556
/api/v1/rivers/nhl/news | 6280842
/api/v1/rivers/nba/news/pinned | 5227507
/api/v1/rivers/top_news/news | 5051595
/api/v1/rivers/nba/news | 4705110
/api/v1/rivers/nfl/news/pinned | 4565983
/api/v1//news.json | 4342518
/api/v1/rivers/nfl/news.json | 4206018
/api/v1/rivers/nfl/news | 3379180
/api/v1/nhl | 2994995
/api/v1/rivers/mlb/news/pinned | 2992940
/api/v1/rivers/mlb/news | 2649503
/api/v1/rivers/top_news/news/pinned | 2516112
/api/v1/rivers/nhl/news.json | 2493204
/api/v1/rivers/nba/news.json | 2481048
/api/v1/rivers/1/news.json | 2278832
</code></pre></div>
As simple as it is, the query is looking at 135 million rows and executes in
approximately 10 seconds using the base RedShift instance type. You can modify
the query a bit and see requests per day like so:
<div class="highlight"><pre><code class="language-sql" data-lang="sql">SELECT COUNT(*),
LEFT(logdate, 12) as day
FROM cf_logentries
GROUP BY day
ORDER BY day;
</code></pre></div><div class="highlight"><pre><code class="language-text" data-lang="text"> count | day
---------+------------
4180128 | 2014-04-01
4874722 | 2014-04-02
4402991 | 2014-04-03
4781033 | 2014-04-04
4676276 | 2014-04-05
4781785 | 2014-04-06
3863464 | 2014-04-07
4368877 | 2014-04-08
5306479 | 2014-04-09
4870124 | 2014-04-10
4141415 | 2014-04-11
4896455 | 2014-04-12
5314497 | 2014-04-13
4695349 | 2014-04-14
4315279 | 2014-04-15
4569367 | 2014-04-16
4509744 | 2014-04-17
3657470 | 2014-04-18
4069335 | 2014-04-19
4127096 | 2014-04-20
4445855 | 2014-04-21
5043469 | 2014-04-22
4803794 | 2014-04-23
4538804 | 2014-04-24
3929871 | 2014-04-25
5056818 | 2014-04-26
4710418 | 2014-04-27
4640340 | 2014-04-28
5389785 | 2014-04-29
4686864 | 2014-04-30
</code></pre></div>
With CloudFront you really should care about your cache hit ratio - maybe it&#39;s
obvious, but the load on your origin systems decrease as your content becomes
easier to cache. This query will look at the most used URLs and give you a cache
hit ratio:
<div class="highlight"><pre><code class="language-sql" data-lang="sql">WITH cache_hits AS (
 SELECT uri, cs_uri_query,
 COUNT(*) AS hits
 FROM cf_logentries
 WHERE x_edge_result_type
 IN (&#39;Hit&#39;, &#39;RefreshHit&#39;)
 GROUP BY uri, cs_uri_query
),
top_uris AS (
SELECT uri, cs_uri_query, count(*)
 AS total
 FROM cf_logentries
 GROUP BY uri, cs_uri_query
ORDER BY total DESC LIMIT 1000
)
SELECT top_uris.uri AS uri,
LEFT(top_uris.cs_uri_query,20) AS qs,
 top_uris.total AS requests,
 cache_hits.hits AS cachehits,
LEFT(((cachehits::float / requests) * 100), 5) AS hitrate
FROM top_uris
JOIN cache_hits ON top_uris.uri = cache_hits.uri
AND top_uris.cs_uri_query = cache_hits.cs_uri_query
ORDER BY top_uris.total DESC;
</code></pre></div>
One minute later, I get these results.
<div class="highlight"><pre><code class="language-text" data-lang="text"> uri | qs | requests | cachehits | hitrate
-------------------------------------+----------------------+----------+-----------+---------
/api/v1/rivers/nhl/news/pinned | limit=20 | 5301492 | 5136042 | 96.87
/api/v1/rivers/nba/news/pinned | limit=20 | 3750216 | 3553899 | 94.76
/api/v1/rivers/nfl/news/pinned | limit=20 | 2540830 | 2385556 | 93.88
/api/v1/rivers/top_news/news/pinned | limit=10 | 2516105 | 2353308 | 93.52
/api/v1/rivers/mlb/news/pinned | limit=20 | 2102154 | 1952020 | 92.85
/api/v1/rivers/nfl/news/pinned | - | 2025153 | 1845123 | 91.11
/api/v1/rivers/nhl/news | limit=14 | 2010704 | 1933622 | 96.16
/api/v1/rivers/nfl/news.json | limit=10 | 1869261 | 1696612 | 90.76
/api/v1/rivers/nhl/news | limit=15 | 1845157 | 1776506 | 96.27
/api/v1/rivers/nhl/news/pinned | - | 1600064 | 1454696 | 90.91
/api/v1/rivers/nhl/news.json | limit=10 | 1515039 | 1379522 | 91.05
/api/v1/rivers/nba/news | limit=14 | 1480116 | 1396547 | 94.35
/api/v1/rivers/nba/news/pinned | - | 1477291 | 1291877 | 87.44
/api/v1/rivers/nba/news.json | limit=10 | 1371061 | 1194216 | 87.10
/api/v1/rivers/top_news/news | limit=20&amp;embed_ids=t | 1289425 | 1139939 | 88.40
/api/v1/rivers/1/news/pinned | - | 1167849 | 1000635 | 85.68
/api/v1/rivers/top_news/news | limit=4 | 1122927 | 1040506 | 92.66
/api/v1/rivers/nba/news | limit=15 | 1114807 | 1051494 | 94.32
/api/v1/rivers/1/news.json | limit=10 | 1072153 | 909073 | 84.78
/api/v1/rivers/mlb/news | limit=14 | 933096 | 862023 | 92.38
/api/v1/rivers/mlb/news/pinned | - | 890785 | 755674 | 84.83
/api/v1/rivers/nfl/news | limit=16 | 868015 | 807376 | 93.01
/api/v1/rivers/mlb/news.json | limit=10 | 820678 | 696375 | 84.85
/api/v1/rivers/mlb/news | limit=15 | 805442 | 743153 | 92.26
/api/v1/rivers/top_news/news | limit=5 | 739409 | 690218 | 93.34
/api/v1/rivers/nfl/news | limit=17 | 623315 | 581964 | 93.36
/api/v1/rivers/nba/news | limit=16 | 582077 | 551094 | 94.67
/api/v1/rivers/nhl/news | limit=16 | 572628 | 552693 | 96.51
/api/v1/rivers/nfl/news | limit=15 | 454028 | 425197 | 93.64
/api/v1/rivers/nhl/news | limit=13 | 438636 | 424338 | 96.74
/api/v1/rivers/ncaab/news/pinned | limit=20 | 418286 | 333987 | 79.84
/api/v1/rivers/ncaab/news/pinned | - | 376280 | 287165 | 76.31
</code></pre></div>
These hit rates are decent for most things, but you can definitely see right away
how the query parameters can be tuned a bit to provide an ever faster experience for
clients consuming these endpoints.
<h2>Conclusion</h2>
That&#39;s pretty much it for now. There&#39;s a lot more you can do with this, as the sky is the
limit when it comes to the queries you can write against your data. Hopefully this was a
beneficial introduction to RedShift with a novel use case. If I got anything wrong let
me know!
There are other obvious related things you&#39;d want to do if you use a setup like this
(such as automating the data load) that we may cover in a future post.
</description>
</item>
<item>
<title>Using Airbrake with Rake and not Rails</title>
<link>https://techblog.thescore.com/2014/05/12/using-airbrake-with-rake-and-not-rails/</link>
<pubDate>Mon, 12 May 2014 00:00:00 +0000</pubDate>
<author></author>
<guid>https://techblog.thescore.com/2014/05/12/using-airbrake-with-rake-and-not-rails</guid>
<description>We use <a href="http://airbrake.io">Airbrake</a> for our error reporting. An error happens,
the error gets sent to Airbrake and then we&#39;re notified via email. We can then
investigate all the goodness from their website. Fantastic!
Most Ruby apps are centered around some kind of web service (mainly Ruby on Rails).
As such, Airbrake has really good Rails integration – even on their rake tasks!
Airbrake is in a lot of our projects, some of which that do not use Rails, however.
A problem that I noticed while debugging why a rake task hadn&#39;t reported errors to Airbrake was two fold:
<ol>
<li>Airbrake is not configured to rescue from rake exceptions by default</li>
<li>Even if you have it enabled, it only works if you have Rails loaded</li>
</ol>
Isn&#39;t that crazy? <a href="https://github.com/airbrake/airbrake/blob/51bf71ba517e9dbb1adb26f52eb6fa73a2f21c86/lib/airbrake/railtie.rb#L9">Here&#39;s the (insane) code that necessitates it</a>.
It definitely wasn&#39;t at the forefront of my mind to realize that enabling
<code>rescue_rake_exceptions</code> wouldn&#39;t rescue from...rake exceptions.
The solution is to follow their <a href="https://github.com/airbrake/airbrake/wiki/Using-Airbrake-with-Rake">wiki entry</a>
(which I had to edit to make it correct):
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby">require &#39;airbrake/rake_handler&#39; # require this if you don&#39;t have rails loaded
Airbrake.configure do |config|
 config.rescue_rake_exceptions = true
end
</code></pre></div>
I&#39;ve <a href="https://github.com/airbrake/airbrake/issues/292">raised the issue</a> on
Airbrake&#39;s github. Feel free to visit the link and weigh in!
Update: we <a href="https://github.com/airbrake/airbrake/pull/294">have a solution</a>
that will be included in the next release of the airbrake gem!
</description>
</item>
</channel>
</rss>

Home · About · News · Docs · Terms