Congratulations!

[Valid Atom 1.0] This is a valid Atom 1.0 feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

Source: http://www.tbray.org/ongoing/ongoing.atom

  1. <?xml version='1.0' encoding='UTF-8'?>
  2. <feed xmlns='http://www.w3.org/2005/Atom'
  3.      xmlns:thr='http://purl.org/syndication/thread/1.0'
  4.      xml:lang='en-us'>
  5. <title>ongoing by Tim Bray</title>
  6. <link rel='hub' href='http://pubsubhubbub.appspot.com/' />
  7. <id>https://www.tbray.org/ongoing/</id>
  8. <link href='https://www.tbray.org/ongoing/' />
  9. <link rel='self' href='https://www.tbray.org/ongoing/ongoing.atom' />
  10. <link rel='replies'       thr:count='101'       href='https://www.tbray.org/ongoing/comments.atom' />
  11. <logo>rsslogo.jpg</logo>
  12. <icon>/favicon.ico</icon>
  13. <updated>2020-05-25T00:33:02-07:00</updated>
  14. <author><name>Tim Bray</name></author>
  15. <subtitle>ongoing fragmented essay by Tim Bray</subtitle>
  16. <rights>All content written by Tim Bray and photos by Tim Bray Copyright Tim Bray, some rights reserved, see /ongoing/misc/Copyright</rights>
  17. <generator uri='/misc/Colophon'>Generated from XML source code using Perl, Expat, Emacs, Mysql, Ruby, Java, and ImageMagick.  Industrial-strength technology, baby.</generator>
  18.  
  19. <entry>
  20. <title>Topfew Fun</title>
  21. <link href='https://www.tbray.org/ongoing/When/202x/2020/05/18/TopFew' />
  22. <link rel='replies'        thr:count='7'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/05/18/TopFew#comments' />
  23. <id>https://www.tbray.org/ongoing/When/202x/2020/05/18/TopFew</id>
  24. <published>2020-05-18T12:00:00-07:00</published>
  25. <updated>2020-05-19T00:01:03-07:00</updated>
  26. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Software' />
  27. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology' />
  28. <category scheme='https://www.tbray.org/ongoing/What/' term='Software' />
  29. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>This was a   <a href='https://en.wikipedia.org/wiki/Victoria_Day'>long weekend in Canada</a>; since I’m unemployed and have no workaday cares,   I should have plenty of time to do family stuff. And I did. But I also got interested in a small programming problem and, over the   course of the weekend, built   a tiny tool called   <a href='https://github.com/timbray/topfew'>topfew</a>. It does a thing you can already do, only faster, which is what I   wanted. But I remain puzzled</div></summary>
  30. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  31.  <p>This was a
  32.  <a href="https://en.wikipedia.org/wiki/Victoria_Day">long weekend in Canada</a>; since I’m unemployed and have no workaday cares,
  33.  I should have plenty of time to do family stuff. And I did. But I also got interested in a small programming problem and, over the
  34.  course of the weekend, built
  35.  a tiny tool called
  36.  <a href="https://github.com/timbray/topfew">topfew</a>. It does a thing you can already do, only faster, which is what I
  37.  wanted. But I remain puzzled.</p>
  38.  <p><i>[To those who are interested in Amazon or politics or photography, feel free to skip this. I’ll come back to those
  39.  things.]</i></p>
  40.  <h2 id='p-1'>What it does</h2>
  41.  <p>Back in 2016 I tweeted
  42.  <a href="https://twitter.com/timbray/status/715306244886560768">Hello "sort | uniq -c | sort -rn | head" my old friend</a>, and got
  43.  a few smiles. If you feed that incantation a large number of lines of text it produces a short list showing
  44.  which among them appear most often, and how many times each of those appears.</p>
  45.  <p>It’s a beautiful expression of the old Unix craft of building simple tools that you can combine to produce surprising and
  46.  useful results.  But when the data gets big, it can also be damn slow.
  47.  I have a little script on the <span class='o'>ongoing</span> Web server that uses this technique to tell me which of my pieces are
  48.  most popular, and how many times each of those was read.</p>
  49.  <p>In the last couple of weeks, my logfiles have gotten really big,
  50.  and so my script has gotten really slow. As in, it takes the best part of a minute to run and, while it’s running, it sucks the
  51.  life out that poor little machine.</p>
  52.  <p>It turns out to be a pretty common problem.  Anyone who’s operating an online service finds themselves processing logfiles to
  53.  diagnose problems or weirdness, and you regularly need to answer questions like “who are the top requestors?” and “what are the
  54.  most common API calls?”</p>
  55.  <h2 id='p-2'>What it is</h2>
  56.  <p>Topfew is a simple-minded little Go program, less than 500 lines of code and almost half of that tests.  If you have an Apache
  57.  server log named <code>access_log</code>, and you wanted to find out what the top twelve IP addresses were hitting your server
  58.  hardest, you could say:</p>
  59.  <p><code>awk '{print $1}' access_log | sort | uniq -c | sort -rn | head -12</code></p>
  60.  <p>With topfew, you’d say:</p>
  61.  <p><code>tf -fields 1 -few 12 access_log</code> (“tf” is short for “topfew”.)</p>
  62.  <p>The two would produce almost the same output, except for topfew would be faster. How much faster?  Well, that’s complicated
  63.  (see below)… but really quite a bit.</p>
  64.  <p>I don’t think there’s anything terribly clever about it; My bet is that if you asked a dozen senior developers to write this,
  65.  they’d all end up with nearly identical approaches, and they’d all be pretty fast.  Anyhow, as the links here make obvious, it’s
  66.  <a href="https://github.com/timbray/topfew">on GitHub</a>. If anyone out there needs such a thing and hasn’t written their own in a
  67.  weekend project, feel free to use and
  68.  improve.</p>
  69.  <p>Since I haven’t done a Go project from scratch in a long time, I’m not only out of practice but out of touch with current best
  70.  practices, so it’s probably egregiously unidiomatic in lots of ways. Oh well.</p>
  71.  <h2 id='p-3'>How fast?</h2>
  72.  <p>Well, that’s interesting.  I ran a few tests on my Mac and it was many many times faster than the sort/uniq incantation, so I
  73.  smiled. I thought “Let’s try bigger data!” and built a Linux version (saying <code>GOOS=linux go build</code> still feels like
  74.  living in the future to me), hopped over to my Web server, and ran exactly those queries from the last section on a
  75.  recent Apache logfile (8,699,657 lines, 2.18G). Topfew was faster, but not <em>many times faster</em> like on my
  76.  Mac. So I ran it on the Mac too.  Here’s a summary of the results. All the times are in seconds.</p>
  77.  <table>
  78.    <tr>
  79.      <td class="empty"></td><th>cp<br/>time</th><th>sort/uniq<br/>time</th><th>topfew<br/>time</th><th>speedup<br/>net of I/O</th>
  80.    </tr>
  81.    <tr align="right"><td>Linux</td><td>17.29</td><td>37.66</td><td>26.75</td><td>2.15x</td></tr>
  82.    <tr align="right"><td>Mac</td><td>1.40</td><td>137.46</td><td>8.81</td><td>18.36x</td></tr>
  83.  </table>
  84.  <p>The first column (“cp time”) is the number of seconds it takes to copy the big file from one directory to another.
  85.  The last column is produced by subtracting the “cp” time from each of the other two, leaving something like the amount of extra latency
  86.  introduced by the computing in excess of the time spent just copying data, then comparing the results.</p>
  87.  <p>Anyhow, those are some puzzling numbers. Stray thoughts and observations that I’m not going to dignify with the term
  88.  “hypotheses”:</p>
  89.  <ol>
  90.    <li><p>The Mac has a damn fast SSD.</p></li>
  91.    <li><p>The Linux box’s disk is sluggish.</p></li>
  92.    <li><p>Perhaps the basic tools like <code>sort</code> and <code>uniq</code> are surprisingly fast on Linux or the Go runtime is
  93.    surprisingly slow.  Or, you know, the mirror image of that statement.</p></li>
  94.    <li><p>The sort/uniq incantation is inherently parallel while topfew is 100% single-threaded. Maybe the Mac is dumb about
  95.    scheduling processes in a shell pipeline across its eight cores?</p></li>
  96.  </ol>
  97.  <h2 id='p-4'>Giggling</h2>
  98.  <p>I broke out the Go profiler (Go is batteries-included in ways that other popular languages just aren’t) and ran that
  99.  command against the Apache logfile again. I was poking around the output and Lauren said “I hope you’re laughing at the computer,
  100.  not at me talking to the cat.”</p>
  101.  <p>The profiler claimed that 73% of the time was spent in <code>syscall.syscall</code> reading the input data, and the first trace
  102.  of actual application code was 2.27% of the time being spent in the regexp library doing field splitting (on
  103.  <code>\s+</code>).</p>
  104.  <p>The
  105.  <a href="https://github.com/timbray/topfew/blob/master/internal/counter.go#L31">core code that actually accumulates counts</a> for
  106.  keys was consuming 1.54% of the elapsed time, and 93% of <em>that</em> was updating the total in a <code>map[string]uint64</code>.</p>
  107.  <h2 id='p-5'>Next steps?</h2>
  108.  <p>I’m probably curious enough to poke around a little more. And maybe someone reading this will kindly explain to me in words of one
  109.  syllable what’s going on here.  But anyhow, the tool is useful enough as it is to save me a noticeable amount of time.</p>
  110. </div></content></entry>
  111.  
  112. <entry>
  113. <title>Meta Blog</title>
  114. <link href='https://www.tbray.org/ongoing/When/202x/2020/05/13/Meta-Blog' />
  115. <link rel='replies'        thr:count='3'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/05/13/Meta-Blog#comments' />
  116. <id>https://www.tbray.org/ongoing/When/202x/2020/05/13/Meta-Blog</id>
  117. <published>2020-05-13T12:00:00-07:00</published>
  118. <updated>2020-05-17T10:18:15-07:00</updated>
  119. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Publishing' />
  120. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology' />
  121. <category scheme='https://www.tbray.org/ongoing/What/' term='Publishing' />
  122. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Web' />
  123. <category scheme='https://www.tbray.org/ongoing/What/' term='Web' />
  124. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>My recent     <a href='/ongoing/When/202x/2020/04/29/Leaving-Amazon'>Amazon-exit piece</a> got an order of magnitude more traffic then even the     post popular outings here normally do. Which turned my mind to thoughts of blogging in 2020, the why and how of the thing.     Here they are, along with      hit-counts and referer data from last week.  Probably skip this unless you’re interested in social-media dynamics and/or     publishing technology</div></summary>
  125. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  126.    <p>My recent
  127.    <a href="/ongoing/When/202x/2020/04/29/Leaving-Amazon">Amazon-exit piece</a> got an order of magnitude more traffic then even the
  128.    post popular outings here normally do. Which turned my mind to thoughts of blogging in 2020, the why and how of the thing.
  129.    Here they are, along with
  130.    hit-counts and referer data from last week.  Probably skip this unless you’re interested in social-media dynamics and/or
  131.    publishing technology.</p>
  132.    <h2 id='p-1'>Numbers</h2>
  133.    <p>In the first week after publication,
  134.    <a href="/ongoing/When/202x/2020/04/29/Leaving-Amazon">Bye, Amazon</a> was read somewhat more than 614,669 times by a human
  135.    (notes on the uncertainty below).</p>
  136.    <p>The referers with more than 1% of that total were Twitter (18.13%), Hacker News (17.09%), Facebook (10.55%), Google (4.29%),
  137.    <a href="https://www.vice.com/en_us/article/z3bjpj/amazon-vp-tim-bray-resigns-calls-company-chickenshit-for-firing-protesting-workers">Vice</a> (3.40%),
  138.    Reddit(1.66%), and
  139.    <a href="https://www.cnbc.com/2020/05/04/amazon-engineer-resigns-over-companys-treatment-of-workers.html">CNBC</a> (1.09%).
  140.    I think <cite>Vice</cite> was helped by getting there first. I genuinely honestly have no idea how the piece got mainlined so
  141.    fast into the media’s arteries.</p>
  142.    <p>For comparison, my
  143.    <a href="https://twitter.com/timbray/status/1257326798565564421">first Twitter post</a> is up to 1.015M impressions as of now and,
  144.    last time I checked,
  145.    <a href="https://twitter.com/emahlee/status/1249968430465175552">Emily’s</a> was quite a ways ahead.</p>
  146.    <p>It’s hard for me to know exactly how many people actually read any one <span class='o'>ongoing</span> piece because I publish a
  147.    full-content RSS/Atom feed. Last time I checked, I estimated 20K or so subscribers, but nobody knows how many actually
  148.    read any given piece. If I cared, I’d put in some kind of a tracker.  That 614K number above comes from a script
  149.    that reads the log and counts the number of fetches executed by a little JavaScript fragment included in each page. Not a
  150.    perfect measure of human-with-a-browser visits but not terrible.</p>
  151.    <h2 id='p-2'>But aren’t blogs dead?</h2>
  152.    <p>Um, nope. For every discipline-with-depth that I care about (software/Internet, politics, energy economics, physics), if
  153.    you want to find out what’s happening and you want to find out from first-person practitioners, you end up reading a blog.</p>
  154.    <p>They’re pretty hard to monetize, which means that the people who write them usually aren’t primarily bloggers, they’re
  155.    primarily professional economists or physicists or oil analysts or
  156.    Internet geeks.  Since most of us don’t even try to
  157.    monetize ’em, they’re pretty ad-free and thus a snappy reading
  158.    experience.</p>
  159.    <p>Dense information from real experts, delivered fast. Why would you want any other kind?</p>
  160.    <h2 id='p-3'>Static FTW</h2>
  161.    <p><span class='o'>ongoing</span> ran slow but anyone who was willing to wait fifteen seconds or so got to read that blog.
  162.    One reason is that the site is “static” which is to say all the payload is in a bunch of ordinary files in ordinary directories on
  163.    a Linux box, so the web server just has to read ’em out of the disk (where by “disk” I mean in-memory filesystem cache when
  164.    things are running hot) and
  165.    push ’em out over the wire.
  166.    (The bits and pieces are described and linked to from the
  167.    <a href="/ongoing/misc/Colophon">Colophon</a>.)</p>
  168.    <p>It turns out that at the very hottest moments, the Linux box never got much above 10% CPU, but the
  169.    100M virt NIC was saturated.</p>
  170.    <p>I’ve never regretted writing my own blogging system from the ground up. I’m pretty sure I’ve learned things about the
  171.    patterns of traffic and attention on the Internet that I couldn’t have learned any other way.</p>
  172.    <p>If I were going to rewrite this, since everything’s static, I’d just run it out of an S3 bucket and move the publishing script
  173.    into a Lambda function. It’d be absurdly cheap and it’d laugh at blog storms like last week’s.</p>
  174.    <p>It’s not a top priority.</p>
  175. </div></content></entry>
  176.  
  177. <entry>
  178. <title>Responses</title>
  179. <link href='https://www.tbray.org/ongoing/When/202x/2020/05/06/Answers' />
  180. <link rel='replies'        thr:count='6'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/05/06/Answers#comments' />
  181. <id>https://www.tbray.org/ongoing/When/202x/2020/05/06/Answers</id>
  182. <published>2020-05-06T12:00:00-07:00</published>
  183. <updated>2020-05-07T23:37:55-07:00</updated>
  184. <category scheme='https://www.tbray.org/ongoing/What/' term='Business/Internet' />
  185. <category scheme='https://www.tbray.org/ongoing/What/' term='Business' />
  186. <category scheme='https://www.tbray.org/ongoing/What/' term='Internet' />
  187. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>Boy, when your   <a href='/ongoing/When/202x/2020/04/29/Leaving-Amazon'>I’m-outta-here essay</a> goes viral, do you ever get a lot of input.  A few responses came up often enough    to be worth sharing.   This was via email, Twitter DMs, blog comments, and LinkedIn messages. All of which went completely batshit</div></summary>
  188. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  189.  <p>Boy, when your
  190.  <a href="/ongoing/When/202x/2020/04/29/Leaving-Amazon">I’m-outta-here essay</a> goes viral, do you ever get a lot of input.  A few responses came up often enough
  191.  to be worth sharing.
  192.  This was via email, Twitter DMs, blog comments, and LinkedIn messages. All of which went completely batshit.</p>
  193.  <p>So, some answers. But first…</p>
  194.  <h2 id='p-1'>Thanks for the kind words!</h2>
  195.  <p>I had no notion how the world might react to a cranky old overpaid geek’s public temper tantrum.
  196.  The world’s been astonishingly warm and welcoming. Apparently I hit a hot button I didn’t know existed. The
  197.  crankiest geek on the planet couldn’t fail to have their heart warmed. So in a huge number of cases, simply “Thanks for the kind
  198.  words” was the right thing to say.</p>
  199.  <p>Dear readers: Yes, some of the answers described in this piece were kept handy in editor buffers and delivered by
  200.  cut-and-paste. But I did read a lot of the messages, and all the ones I actually responded to.</p>
  201.  <h2 id='p-2'>“Can you come on our TV show?”</h2>
  202.  <p>You name it: ABC, BBC, Bloomberg, CBC, CBS, CTV, CNBC, CNN, NBC, NPR, and a whole lot of cool blogs and indies.  Also Anderson
  203.  Cooper’s people!</p>
  204.  <h2 id='p-3'>“Can we get on the phone so I can ask you some questions?”</h2>
  205.  <p>A variation on the theme, from non-broadcast organizations.</p>
  206.  <p>They all got the same answer: “Hmm, I'm not that interesting, just a grumpy old one-percenter white-guy
  207.  engineer with a social media presence. If you want to go live with this story you should do it with the actual people who got
  208.  fired, who are young, fresh-faced, passionate, and really at the center of the news story. I recommend reaching out to Emily
  209.  Cunningham (contact info redacted) or Maren Costa (same) or Chris Smalls (same).”</p>
  210.  <h2 id='p-4'>“OK, we talked to them. Now will you talk to us live?”</h2>
  211.  <p>These people were nice and just trying to do their job.  I agreed to answer a couple of email questions in a couple of cases,
  212.  but mostly just said “For complicated reasons, I don’t want to be the public face of this story. Sorry.”</p>
  213.  <h2 id='p-5'>“Complicated reasons?”</h2>
  214.  <p>Yeah, the story is about firing whistleblowers, not about a random Canadian Distinguished Engineer’s reaction to it.  So news
  215.  organizations should follow the primary sources, not me.</p>
  216.  <p>There’s more. I put a lot of thought into what I should say, and then really a whole lot of work into writing that blog
  217.  piece. I had help with style and fact-checking. (Thanks, Lauren. Thanks, Emily.) It is very, very, unlikely that anything I’d
  218.  improvise on a phone-call or TV
  219.  interview would be better.  I’ve also had bitter first-hand experience with the
  220.  <a href="https://www.goodreads.com/quotes/65213-briefly-stated-the-gell-mann-amnesia-effect-is-as-follows-you">Gell-Mann amnesia
  221.  effect</a>.</p>
  222.  <p>I think it worked. The news coverage, lacking alternatives, quoted heavily from the blog, and I thought
  223.  basically all of it came out fair and accurate. Let’s acknowledge that this tactic is only available to someone who’s near the
  224.  center of a news story, is a competent writer, and has a good place to publish.</p>
  225.  <p>I’ve no interest in becoming some sort of full-time anti-Amazon activist. I just don’t want to work for an
  226.  organization that fires whistleblowers.  I said so. It looks like the message got through.</p>
  227.  <h2 id='p-6'>“What response did you expect?”</h2>
  228.  <p>I have seventeen years of blogging scars, so I speak from experience in saying: No idea. I’ve had blogs that I considered
  229.  mightily important and labored over for days sink like a stone with no trace. Then I’ll toss off some three-paragraph squib that I
  230.  wrote while watching TV and drinking gin, and it goes white-hot.  Neither outcome would have surprised me.</p>
  231.  <h2 id='p-8'>“What were you trying to accomplish?”</h2>
  232.  <p>I’m a blogger. I’ve been writing the story of my life here for seventeen years. Enough people read it and respond to give me
  233.  hope that it’s at least intermittently interesting, and perhaps even useful. I’m a writer, I can’t not write.
  234.  This is a major turning point in my life. I totally
  235.  couldn’t not write it. That’s all.  That’s <em>really, really</em> why.</p>
  236.  <h2 id='p-10'>“What about Brad’s piece?”</h2>
  237.  <p>They’re asking about
  238.  <a href="https://www.linkedin.com/pulse/response-tim-brays-departure-brad-porter/">Response to Tim Bray’s departure</a> by
  239.  <a href="https://www.linkedin.com/in/brad-porter-1a989/">Brad Porter</a>. Since he has the same “VP/Distinguished Engineer”
  240.  title I did you’d think he’d be a peer. Actually he’s way more important and influential than I used to be, partly because he’s been
  241.  there since the early days and is directly involved with the retail operation.</p>
  242.  <p>I believe that (as Brad says) Amazon retail is working intensely and intelligently to make the
  243.  warehouses safer.  But I also believe the workers. And anyhow, firing whistleblowers is just way, way out of
  244.  bounds.</p>
  245.  <p>While it’s sort of a sideshow to the real issue here (firing whistleblowers), Brad also wrote:</p>
  246.  <blockquote>
  247.    <p>Ultimately though, Tim Bray is simply wrong when he says “It’s that Amazon treats the humans in the warehouses as fungible
  248.    units of pick-and-pack potential.” I find that deeply offensive to the core.</p>
  249.  </blockquote>
  250.  <p>We’ll have to agree to disagree. If you run an organization with hundreds of thousands of line workers and tens of thousands of
  251.  managers, and where turnover is typically significant, you need processes where the staff are fungible. Two things can be true at
  252.  once: You work hard to preserve your employees’ health, and your administrative culture treats them as fungible units.</p>
  253.  <p>I actually found the patterns emerging in the responses to Brad’s piece more culturally interesting than his original
  254.  post.</p>
  255.  <p>And hey, bonus: There’s another Amazon voice in the conversation:
  256.  <a href="https://www.linkedin.com/in/antonok/">Anton Okmyanskiy</a>, who’s a “Principal Engineer”, which is to say regarded as a
  257.  world-class technologist, wrote
  258.  <a href="https://www.linkedin.com/pulse/tim-bray-quit-amazon-my-opinion-anton-okmyanskiy/?trackingId=GaNNzAG%2FQY%2ByI9IEKicP8g%3D%3D">Tim Bray quit Amazon.
  259.  My thoughts...</a>. It contains the remarkable sentence “Amazon should stay ahead of anger-driven regulatory enforcement by becoming
  260.  a leader on social justice issues.”</p>
  261.  <p>It’d be great if a few more Amazonian voices weighed in. But I’m not holding my breath.</p>
  262.  <h2 id='p-11'>“Any regrets?”</h2>
  263.  <p>Yes, I regret intensely that I didn’t link to Emily Cunningham’s original
  264.  <a href="https://twitter.com/emahlee/status/1249968430465175552">“Amazon fired me” tweet thread</a>, which is exquisite (you have to click “show this
  265.  thread”).</p>
  266.  <h2 id='p-9'>Favorite response?</h2>
  267.  <p>Note, header not in quotes because nobody asked, but I’ll answer anyhow. I could drop a dozen portentous media-heavyweight
  268.  names and yeah, pretty well everyone weighed in. But it’s not close, my fave was Wonkette:
  269.  <a href="https://www.wonkette.com/amazon-vp-tim-bray-resigns-in-protest">Amazon VP VIP Tim Bray Quitfires Self Over 'Chickensh*t'
  270.  Activist Quitfirings</a>. It says, of yr humble scrivener, “Come the revolution, let's remember not to eat this one” and “Class
  271.  Traitor of the Day”. These lodged in what I thought was a thoroughly lucid and spirited take on the situation.</p>
  272.  <h2 id='p-12'>Once again</h2>
  273.  <p>Thanks for the kind words!</p>
  274.  
  275.  <!--
  276.  <h2 id='p-7'>“What result did you get?”</h2>
  277.  <p>Coverage in
  278.  
  279.    <a href="https://techcrunch.com/2020/05/04/aws-engineer-tim-bray-resigns-from-amazon-following-worker-firings/">TechCrunch</a>,
  280.  
  281.    <a href="https://www.smartcompany.com.au/coronavirus/coronavirus-update-for-business-cost-crisis-elon-musk-slack-weekend/">Smart
  282.    Company (Australia)</a>,
  283.    <a href="https://www.scmp.com/tech/big-tech/article/3082902/amazon-engineer-quits-forfeits-1-million-salary-protest-firings">South
  284.    China Morning Post</a>,
  285.    <a href="https://www.cnbc.com/2020/05/04/amazon-engineer-resigns-over-companys-treatment-of-workers.html">CNBC</a>,
  286.    <a
  287. href="https://www.vice.com/en_us/article/z3bjpj/amazon-vp-tim-bray-resigns-calls-company-chickenshit-for-firing-protesting-workers">Vice</a>,
  288.    <a href="https://www.vox.com/recode/2020/5/4/21246361/amazon-vp-tim-bray-resigns-activist-firing-aws">Vox</a>,
  289.    <a href="https://gizmodo.com/amazon-vp-rips-companys-chickenshit-labor-practices-in-1843240153">Gizmodo</a>,
  290.    <a href="https://www.cnet.com/news/amazon-vp-resigns-in-protest-over-firings-of-activist-employees/">CNET</a>,
  291.    <a href="https://www.latimes.com/business/technology/story/2020-05-04/amazon-coronavirus-tim-bray-resigns">LA Times</a>,
  292.    <a
  293. href="https://siliconangle.com/2020/05/04/aws-vice-president-senior-engineer-tim-bray-resigns-worker-firings/">SiliconAngle</a>,
  294.    <a href="https://www.cnn.com/2020/05/04/tech/amazon-vp-tim-bray-quits/">CNN</a>,
  295.    <a href="https://www.zdnet.com/article/top-aws-engineer-tim-bray-quits-1m-plus-job-over-amazon-firing-employees/">ZDNet</a>,
  296.    <a href="https://fortune.com/2020/05/04/amazon-engineer-quits-forfeits-1-million-protest-firings/">Fortune</a>,
  297.    <a href="https://www.businessinsider.com/amazon-engineer-resigned-treatment-warehouse-whistleblowers-2020-5?op=1">Business Insider</a>,
  298.    <a href="https://www.nytimes.com/2020/05/04/business/amazon-tim-bray-resigns.html">NY Times</a>,
  299.    <a href="https://www.bbc.com/news/business-52534567">BBC</a>,
  300.    <a href="https://www.axios.com/top-amazon-web-services-engineer-resigns-worker-firings-fc737327-6baf-44e0-8b28-3c36e384f57a.html">Axios</a>,
  301.  
  302.    <a href="https://www.washingtonpost.com/technology/2020/05/04/top-amazon-executive-quits-over-firings-warehouse-workers-climate-activists/">Washington
  303.    Post</a>,
  304.    <a href="https://qz.com/1850893/amazon-vp-tim-bray-quits-over-amazon-firing-activist-employees/">Quartz</a>,
  305.    <a href="https://www.theverge.com/2020/5/4/21246284/amazon-vp-resign-whistleblower-firings-warehouse-workers">The Verge</a>,
  306.    <a href="https://arstechnica.com/tech-policy/2020/05/amazon-vp-resigns-over-companys-chickenst-firing-of-whistleblowers/">Ars Technica</a>,
  307.  
  308.    <a href="https://nypost.com/2020/05/04/amazon-vp-quits-calls-company-chickens-t-for-firing-protesters/">New York Post</a>,
  309.    <a href="https://mashable.com/article/amazon-vp-resigns-warehouse-workers-coronavirus/">Mashable</a>,
  310.    <a href="https://www.engadget.com/amazon-vp-quits-over-whistleblower-firings-152305288.html">Engadget</a>,
  311.    <a href="https://www.theglobeandmail.com/business/article-canadian-senior-executive-quits-amazon-over-firing-of-employees/">Globe
  312.    and Mail</a>,
  313.    <a href="https://www.wonkette.com/amazon-vp-tim-bray-resigns-in-protest">Wonkette</a>,
  314.  yadda yadda yadda.</p>
  315.  
  316.  
  317.  
  318.  <p>Notes: 10K twitter
  319.  
  320.  Facebook
  321.  LinkedIn
  322.  </p>
  323.  -->
  324. </div></content></entry>
  325.  
  326. <entry>
  327. <title>Bye, Amazon</title>
  328. <link href='https://www.tbray.org/ongoing/When/202x/2020/04/29/Leaving-Amazon' />
  329. <link rel='replies'        thr:count='638'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/04/29/Leaving-Amazon#comments' />
  330. <id>https://www.tbray.org/ongoing/When/202x/2020/04/29/Leaving-Amazon</id>
  331. <published>2020-04-29T12:00:00-07:00</published>
  332. <updated>2020-05-04T16:02:15-07:00</updated>
  333. <category scheme='https://www.tbray.org/ongoing/What/' term='Business/Internet' />
  334. <category scheme='https://www.tbray.org/ongoing/What/' term='Business' />
  335. <category scheme='https://www.tbray.org/ongoing/What/' term='Internet' />
  336. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>May 1<sup>st</sup> was my last day as a VP and Distinguished Engineer at Amazon Web Services, after five years and five months   of rewarding fun.   I quit in dismay at Amazon firing whistleblowers who were making noise about warehouse employees frightened of Covid-19</div></summary>
  337. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  338.  <p>May 1<sup>st</sup> was my last day as a VP and Distinguished Engineer at Amazon Web Services, after five years and five months
  339.  of rewarding fun.
  340.  I quit in dismay at Amazon firing whistleblowers who were making noise about warehouse employees frightened of Covid-19.</p>
  341.  <p>What with big-tech salaries and share vestings, this will probably cost me over a
  342.  million (pre-tax) dollars, not to mention the best job I’ve ever had, working with awfully good people. So I’m pretty blue.</p>
  343.  <h2 id='p-1'>What happened</h2>
  344.  <p>Last year, Amazonians on the tech side banded together as
  345.  <a href="https://medium.com/@amazonemployeesclimatejustice">Amazon Employees for Climate Justice</a> (AECJ), first coming to the
  346.  world’s notice with an
  347.  <a href="https://medium.com/@amazonemployeesclimatejustice/public-letter-to-jeff-bezos-and-the-amazon-board-of-directors-82a8405f5e38">open
  348.  letter</a> promoting a shareholders’ resolution calling for dramatic action and leadership from Amazon on the global climate
  349.  emergency. I was one of its 8,702 signatories.</p>
  350.  <p>While the resolution got a lot of votes, it didn’t pass. Four months later, 3,000 Amazon tech workers from around the world
  351.  <a href="https://twitter.com/AMZNforClimate/status/1184500462873010177">joined in the Global Climate Strike walkout</a>. The day
  352.  before the walkout, Amazon announced
  353.  <a href="https://www.cnbc.com/2019/09/19/jeff-bezos-speaks-about-amazon-sustainability-in-washington-dc.html">a large-scale
  354.  plan</a> aimed at making the company part of the climate-crisis solution.  It’s not as though the activists were acknowledged by
  355.  their employer for being forward-thinking; in fact, leaders
  356.  were
  357.  <a href="https://www.theguardian.com/technology/2020/jan/02/amazon-threatened-fire-employees-speaking-out-climate-change-workers-say">threatened
  358.  with dismissal</a>.</p>
  359.  <p>Fast-forward to the Covid-19 era. Stories surfaced of unrest in Amazon warehouses, workers raising alarms about
  360.  being uninformed, unprotected, and frightened. Official statements claimed every possible safety precaution
  361.  was being taken. Then a worker organizing for better safety conditions was fired, and
  362.  <a href="https://www.vice.com/en_us/article/5dm8bx/leaked-amazon-memo-details-plan-to-smear-fired-warehouse-organizer-hes-not-smart-or-articulate">brutally
  363.  insensitive remark</a>s appeared in
  364.  leaked executive meeting notes where the focus was on defending Amazon “talking points”.</p>
  365.  <p>Warehouse workers reached out to AECJ for support. They responded by internally promoting
  366.  <a href="https://docs.google.com/forms/d/e/1FAIpQLSfNxRiOtKkPDoCi3iUe6XPyM0rzYzXezaSc4soA-3jX9SGpOw/viewform">a petition</a> and
  367.  organizing a video call for Thursday April 16 featuring warehouse workers from around the world, with guest activist
  368.  <a href="https://en.wikipedia.org/wiki/Naomi_Klein">Naomi Klein</a>.  An announcement sent to internal mailing
  369.  lists on Friday April 10th was apparently the
  370.  flashpoint. Emily Cunningham and Maren Costa, two visible AECJ leaders, were fired on the spot that day.  The justifications
  371.  were laughable; it was clear to any reasonable observer that they were turfed for whistleblowing.</p>
  372.  <p>Management could have objected to the event, or demanded that outsiders be excluded, or that leadership be represented, or any
  373.  number of other things; there was plenty of time. Instead, they just fired the activists.</p>
  374.  <h2 id='p-3'>Snap!</h2>
  375.  <p>At that point I snapped. VPs shouldn’t go publicly rogue, so I escalated through the proper channels and by the
  376.  book. I’m not at liberty to disclose those discussions, but I made many of the arguments appearing in this essay.  I
  377.  think I made them to the appropriate people.</p>
  378.  <p>That done, remaining an Amazon VP would have meant, in effect, signing off on actions I despised.
  379.  So I resigned.</p>
  380.  <p>The victims weren’t abstract entities but real people; here are some of their names:
  381.  Courtney Bowden, Gerald Bryson, Maren Costa, Emily Cunningham, Bashir Mohammed, and Chris Smalls. </p>
  382.  <p>I’m sure it’s a coincidence that every one of them is a
  383.  person of color, a woman, or both. Right?</p>
  384.  <p>Let’s give one of those names a voice.
  385.  <a href="https://www.cnn.com/2020/04/22/tech/amazon-warehouse-bashir-mohamed/index.html">Bashir Mohamed said</a> “They fired me to
  386.  make others scared.” Do you disagree?</p>
  387.  <p><i>[There used to be a list of adjectives here, but voices I
  388.  respect told me it was mean-spirited and I decided it didn’t add
  389.  anything so I took it out.]</i></p>
  390.  <h2 id='p-2'>What about the warehouses?</h2>
  391.  <p>It’s a matter of fact that workers are saying they’re at risk in the warehouses.  I don’t think
  392.  the media’s done a terribly good job of telling their stories.  I went to the video chat that got Maren and Emily fired, and
  393.  found listening to them moving.  You can listen too if you’d like.  
  394.  <a href="https://www.youtube.com/watch?v=5s5Aup66EZU">Up on YouTube</a> is another full-day videochat; it’s nine hours long, but
  395.  there’s a table of contents, you can decide whether you want to hear people from Poland, Germany, France, or multiple places in
  396.  the USA.
  397.  Here’s
  398.  <a href="https://www.nytimes.com/2020/04/05/technology/coronavirus-amazon-workers.html">more reportage from the NY Times</a>.</p>
  399.  <p>It’s not just workers who are upset.  Here are
  400.  <a href="https://www.seattletimes.com/business/amazon/amazon-workers-say-coronavirus-safety-measures-are-missing-despite-what-company-says/">Attorneys-general
  401.  from 14 states speaking out</a>. Here’s the
  402.  <a href="https://www.npr.org/2020/04/27/846438983/amazon-warehouse-safety-inadequate-n-y-attorney-general-s-office-says">New York
  403.  State Attorney-general</a> with more detailed complaints. Here’s
  404.  <a href="https://www.cnn.com/2020/04/24/tech/amazon-france-appeal/index.html">Amazon losing in French courts, twice</a>.</p>
  405.  <p>On the other hand, Amazon’s messaging has been urgent that they are prioritizing this issue and putting massive efforts into
  406.  warehouse safety. I actually believe this: I have heard detailed descriptions from people I trust of the intense work and huge
  407.  investments. Good for them; and let’s grant that you don’t turn a supertanker on a dime.</p>
  408.  <p>But I believe the worker testimony too.  And at the end of the day, the big problem isn’t the specifics of Covid-19 response.
  409.  It’s that Amazon treats the humans in the warehouses as fungible units of   pick-and-pack potential.
  410.  Only that’s not just Amazon, it’s how 21st-century capitalism is done.</p>
  411.  <p>Amazon is exceptionally well-managed and has demonstrated
  412.  great skill at spotting opportunities and building repeatable processes for exploiting them.  It has a corresponding lack of
  413.  vision about the human costs of the relentless growth and accumulation of wealth and power.
  414.  If we don’t like certain things Amazon is
  415.  doing, we need to put legal guardrails in place to stop those things.  We don’t need to invent anything new; a
  416.  combination of antitrust and living-wage and worker-empowerment legislation, rigorously enforced, offers a clear path
  417.  forward.</p>
  418.  <p>Don’t say it can’t be done, because  <a href="https://www.bbc.com/news/world-europe-52346451">France is doing it</a>.</p>
  419.  <h2 id='p-7'>Poison</h2>
  420.  <p>Firing whistleblowers isn’t just a side-effect of macroeconomic forces, nor is it intrinsic to the function of free
  421.  markets. It’s evidence of a vein of toxicity running through the company culture. I choose neither to serve nor drink that poison.</p>
  422.  <h2 id='p-4'>What about AWS?</h2>
  423.  <p>Amazon Web Services (the “Cloud Computing” arm of the company), where I worked, is a different story.  It treats its
  424.  workers humanely, strives for work/life balance, struggles to move the diversity needle (and mostly fails, but so does everyone
  425.  else), and is by and large an ethical organization. I genuinely admire its leadership.</p>
  426.  <p>Of course, its workers have power. The average pay is very high, and anyone who’s unhappy can
  427.  walk across the street and get another job paying the same or better.</p>
  428.  <h2 id='p-5'>Spot a pattern?</h2>
  429.  <p>At the end of the day, it’s all about power balances. The warehouse workers are weak and getting weaker, what with mass
  430.  unemployment and (in the US) job-linked health insurance. So they’re gonna get treated like crap, because capitalism.  Any
  431.  plausible solution has to start with increasing their collective strength.</p>
  432.  <h2 id='p-6'>What’s next?</h2>
  433.  <p>For me?  I don’t know, genuinely haven’t taken time to think about it. I’m sad,  but I’m
  434.  breathing more freely.</p>
  435. </div></content></entry>
  436.  
  437. <entry>
  438. <title>Mac Migration Pain</title>
  439. <link href='https://www.tbray.org/ongoing/When/202x/2020/04/25/Mac-Migration' />
  440. <link rel='replies'        thr:count='3'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/04/25/Mac-Migration#comments' />
  441. <id>https://www.tbray.org/ongoing/When/202x/2020/04/25/Mac-Migration</id>
  442. <published>2020-04-25T12:00:00-07:00</published>
  443. <updated>2020-05-03T12:12:53-07:00</updated>
  444. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Mac OS X' />
  445. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology' />
  446. <category scheme='https://www.tbray.org/ongoing/What/' term='Mac OS X' />
  447. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>What happened was, my backpack got stolen with my work and personal Macs inside.  The work machine migrated effortlessly but I just   finished up multiple days of anguish getting the personal machine going and thought I’d blog it while it was fresh in my mind.   This might be helpful to others, either now or maybe later at the end of a Web search.  But hey, I’m typing this   <span class="o">ongoing</span> fragment on it, so the story had a happy ending.  And to compensate for its length and sadness,   I’ll give it a happy beginning; two, in fact</div></summary>
  448. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  449.  <p>What happened was, my backpack got stolen with my work and personal Macs inside.  The work machine migrated effortlessly but I just
  450.  finished up multiple days of anguish getting the personal machine going and thought I’d blog it while it was fresh in my mind.
  451.  This might be helpful to others, either now or maybe later at the end of a Web search.  But hey, I’m typing this
  452.  <span class='o'>ongoing</span> fragment on it, so the story had a happy ending.  And to compensate for its length and sadness,
  453.  I’ll give it a happy beginning; two, in fact.</p>
  454.  <h2 id='p-1'>Happy corporate</h2>
  455.  <p>The IT guys were excellent, they’re in quarantine mode too but somehow arranged on two hours’ notice that I could drop by the
  456.  spookily-empty office and there it was on a shelf outside the IT room. I’d backed everything that mattered up straight to S3, it
  457.  restored fast and painlessly, and all I had to do was re-install a couple of apps.</p>
  458.  <p>As
  459.  <a href="/ongoing/When/202x/2020/02/04/Bye-Bye-Time-Machine">I’ve said before</a>, Arq worked well.  But as you will soon learn,
  460.  there are pointy corners and boy did I ever get bruises.</p>
  461.  <h2 id='p-10'>Happy Ukelele</h2>
  462.  <p>When I’m writing text for humans to read, I like to use typographical quotes (“foo” rather than "foo"), apostrophes (it’s not
  463.  it's), and ellipses (… is one character not three). These are included in every font you’re likely to have, so why not use them?
  464.  The default Mac keystrokes to get them are awkward and counter-intuitive.</p>
  465.  <p>Since (blush) I hadn’t backed up the keyboard layout I’d been running for years, I went and got
  466.  <a href="https://software.sil.org/ukelele/#about">Ukelele</a>, a nifty little software package that makes it absurdly easy to
  467.  modify your Mac keyboard layout.  Here’s a screenshot.</p>
  468.  <img src="ukelele.png" alt="Ukelele" />
  469.  <p>I produced a layout called “Typographic Quotes” which remaps four keys:</p>
  470.  <ol>
  471.    <li><p>Opt-L is “</p></li>
  472.    <li><p>Opt-; is ”</p></li>
  473.    <li><p>Opt-' is ’</p></li>
  474.    <li><p>Opt-. is …</p></li>
  475.  </ol>
  476.  <p>Anyhow, <a href="typoquotes.tgz">here it is</a> (with a nice icon even), download
  477.  and expand it and go looking on the Web for instructions how to add it on your version of MacOS, it’s not rocket science.</p>
  478.  <p>Another nice thing about Ukelele: It’s ships with maybe the best How-To tutorial I’ve ever encountered in my lifetime of
  479.  reading software how-tos.</p>
  480.  <h2 id='p-2'>Who am I?</h2>
  481.  <p>When I set up the new machine, it asked me if I wanted to transfer from an older one and I said no; so it unilaterally decided
  482.  that my Mac username (and home directory) would be “/Users/timothybray” as opposed to the previous “twbray”.  Sort of dumb and
  483.  clumsy but you wouldn’t think it’d hurt that much.  Hah.</p>
  484.  <h2 id='p-3'>Arq hell</h2>
  485.  <p>My backup, made with a copy of Arq 5 that I paid for, is on
  486.  <a href="/ongoing/When/201x/2016/12/20/Network-backup">our Synology NAS</a>.  I had to fumble around a bit to remember how to turn
  487.  it into an AFP mount, then I went to
  488.  <a href="https://www.arqbackup.com/">arqbackup.com</a> and hit the first Download button I saw.
  489.  Once I fired up Arq I started, at a moderate pace and step by step, to be
  490.  driven mad. Because Arq flatly refused to see the network filesystem.  I tried endless tricks and helpful-looking suggestions I
  491.  found on the Web and got nowhere.</p>
  492.  <p>Here’s a clue: I’d typed in my license code and Arq had said “Not an Arq 6 license code”, which baffled me, so I went ahead in
  493.  free-trial mode to get the restore running.  Wait a minute… all the backups were with Arq <em>5</em>.  OK then. So I went and dug
  494.  up an Arq 5 download.</p>
  495.  <p>It could see the NAS mounts just fine but things didn’t get better.  The backup directory looked like
  496.  <code>/Volumes/home/tim/Backup</code> and there were two different save-sets there, a special-purpose one-off and my “real”
  497.  backup, charmingly named “C1E0FFBA-D68B-4B51-870A-F817FC0DF092”. So I pointed Arq at C1E0… and hit the “Restore” button
  498.  and it… did nothing. Spun a pinwheel for like a
  499.  half-second and stopped and left nothing clickable on the screen.  Nor was there any useful information in any log I could find.</p>
  500.  <p>Rumor has it that I impressed my children with the depth and expressiveness of my self-pitying moans.</p>
  501.  <p>So I went poking around to look at the backup save-set that Arq was refusing to read, and noticed that its modification date
  502.  was toeday. Suddenly it seemed perfectly likely that I’d somehow corrupted it and blackness was beginning to close in at edges of
  503.  my vision.</p>
  504.  <p>[Yes, dear reader, the Synology in turn backs itself up to S3 but I’d never actually tried to use that so my faith that it’d
  505.  even work was only modest.]</p>
  506.  <p>I finally got it to go.
  507.  Protip: Don’t point Arq at your saveset, point it at the directory that contains them.  Why, of course!</p>
  508.  <p>So I launched the restore and, being a nervous kinda guy, kept an eye on the progress window.  Which started to emit a series
  509.  of messages about “File already exists”. Huh?  Somehow I’d fat-fingeredly started two Arq restores in parallel and they were
  510.  racing to put everything back. So I stopped one but it still kept hitting collisions for a long, long time. Anyhow it seemed to
  511.  work.</p>
  512.  <p>Now, it turns out that when Arq started up it pointed out it was going to restore to /Users/twbray where they’d come from, so I
  513.  pointed it at “timothybray” instead.</p>
  514.  <h2 id='p-4'>Lightroom hell</h2>
  515.  <p>I really care about my pictures a lot.  So I fired up Lightroom Classic, it grunted something about problems wth my catalog,
  516.  and limped up in pile-of-rubble condition.  I have a directory called “Current” that all the pictures I’m working on live in
  517.  before they get filed away into the long-term date hierarchy, and it seemed to contain hundreds of pictures from years back, and
  518.  my date tree didn’t have anything after September 2017, and all the pictures in Current that I <em>could</em> see were from my
  519.  phone, none from any of the actual camera cameras.</p>
  520.  <p>I tried restoring one of the catalog backups that Lightroom helpfuly generates but that wasn’t helpful.</p>
  521.  <p>I may once again have made more loud self-pitying noises than strictly necessary.  I poked around, trying to
  522.  understand what was where and absolutely couldn’t make any sense of things.  Scrolling back and forth suggested the Fuji camera
  523.  photos were there, but Lightroom couldn’t locate them.</p>
  524.  <p>Eventually it dawned on me that Lightroom expects you to have your photos splashed around across lots of disks and so in its
  525.  catalog it puts absolute file pathnames for everything, nothing relative. So the photos were all under “timothybray” and the
  526.  catalog pointers were all into “twbray”. So <em>how the freaking hell did Lightroom find all the Pixel photos!?!?</em>.</p>
  527.  <p>Since I’d made probably-destructive changes while thrashing around, I just Arq-restored everything to /Users/twbray and left it
  528.  there and now Lightroom’s happy.</p>
  529.  <p>Was my trail of tears over?  If only.</p>
  530.  <h2 id='p-5'>Blogging hell</h2>
  531.  <p>The software that generates the deathless HTML you are now reading was written in 2002, a much simpler and more innocent time,
  532.  in Perl; in fact it comprises one file named <code>ong.pl</code> which has 2,796 lines of pure software beauty, let me
  533.  tell ya.</p>
  534.  <p>It uses Mysql to keep track of what’s where, and Perl connects to databases via “DBD” packages, in this case
  535.  <a href="https://metacpan.org/pod/DBD::mysql">DBD::mysql</a>.  Which comes all ready to install courtesy of
  536.  <a href="https://www.cpan.org/">CPAN</a>, Perl’s nice package manager.</p>
  537.  <p>You get all this built right into MacOS. Isn’t that great?  No, it isn’t, because it plays by Apple rules which means that
  538.  relies on something called
  539.  <a href="https://blog.krzyzanowskim.com/2018/12/05/rpath-what/">@rpath</a>.  Which can go burn in hell, I say. The effect
  540.  of it being there is that things that load dynamically refuse to load from anywhere but <code>/usr/lib</code>, and your modern
  541.  MacOS versions make it completely physically impossible to update anything on that filesystem.  So in effect it’s impossible to
  542.  install DBD::mysql using the system-provided standard CPAN.  There’s this thing called
  543.  <a href="https://medium.com/@donblas/fun-with-rpath-otool-and-install-name-tool-e3e41ae86172">install_name_tool</a> that looks
  544.  like it should in theory help, but the instructions include poking around inside object libraries and life’s too short.</p>
  545.  <h2 id='p-6'>SQL hell</h2>
  546.  <p>You might well ask “Why on earth are you using mysql anyhow in 2020? Especially when it’s a dinky blogging system with a few
  547.  thousand records?”  My only answer is “It was 2002” and it’s not a good one.  So I thought maybe I could wire in another engine,
  548.  and the obvious candidate was
  549.  <a href="https://www.sqlite.org/index.html">SQLite</a>, software that I’ve had nothing but good experience with: Fast,
  550.  stripped-down, robust, and simple.</p>
  551.  <p>DBD::SQLite installed with no trouble, so I handcrafted my database init script to fix some
  552.  trifles SQLite complained about, and fired up the blogging system. Boom crash splat fart bleed. It turns out that the language
  553.  mysql calls “SQL” is only very loosely related to the dialect spoken by SQLite. Which one’s right? I don’t fucking care, OK?  But it
  554.  would have been a major system rewrite.</p>
  555.  <h2 id='p-7'>Docker hell</h2>
  556.  <p>So I took pathetically to Twitter and whined at the world for advice. The most common suggestion was Docker; just make an
  557.  image with mysql and Perl and the other pieces and let ’er rip.  Now I’ve only ever fooled around unseriously with Docker and
  558.  quickly learned that I was just not up to this challenge. Yep, I could make an image with Perl and DBD::mysql, and I could make one
  559.  with mysql, but could I get one to talk to each other? Could I even connect to the Dockerized mysql server? I could not, and all
  560.  the helpful notes around the Net assume you already have an M.Sc. in Dockerology.</p>
  561.  <p>So this particular hell was really my fault as much as any piece of software’s.</p>
  562.  <img src="https://www.tbray.org/ongoing/When/202x/2020/04/25/js.png" alt="John Siracusa" class="inline" />
  563.  <h2 id='p-8'>Deus ex machina!</h2>
  564.  <p>Among the empathetic chatter I got back on Twitter, exactly one person said “I have DBD::mysql working on Catalina.”
  565.  That was
  566.  <a href="http://hypercritical.co/">John Siracusa</a>. I instantly wrote back “how?” and the answer was simple: Don’t use the Apple
  567.  flavors of perl, build your own from scratch and use that. All of a sudden everything got very easy.</p>
  568.  <ol>
  569.    <li><p>Download the source from Perl.org.  Since I’d filled up /usr/local with Homebrew stuff, I worked out of /usr/local/tim.</p></li>
  570.    <li><p><code>./Configure -des -Dprefix=/usr/local/tim</code></p></li>
  571.    <li><p><code>make</code></p></li>
  572.    <li><p><code>make test</code></p></li>
  573.    <li><p><code>make install</code></p></li>
  574.    <li><p><code>PATH=/usr/local/tim/bin:$PATH</code></p></li>
  575.    <li><p><code>cpan install DBI</code></p></li>
  576.    <li><p><code>cpan install DBD::mysql</code></p></li>
  577.  </ol>
  578.  <p>Then everything just worked.</p>
  579.  <p>Now, I have to say that watching those Configure and make commands in action is a humbling experience.  In 2020 it’s
  580.  fashionable to diss Perl, but watching endless screens-full of i-dotting and t-crossing and meticulous testing flow past reminded me
  581.  that those of us doing software engineering and craftsmanship need a healthy respect for the past.</p>
  582.  <h2 id='p-9'>Was I done?</h2>
  583.  <p>Of course not.  A whole bunch of other bits and pieces needed to be installed from Homebrew and CPAN and Ruby Gems and so on
  584.  and so forth, but that’s not a thing that leaves you frustrated or bad.</p>
  585.  <p>Boy, next time I do this, it’ll go better. Rght?</p>
  586. </div></content></entry>
  587.  
  588. <entry>
  589. <title>Early 2020 Mac Upgrades</title>
  590. <link href='https://www.tbray.org/ongoing/When/202x/2020/04/05/New-Mac-Stuff' />
  591. <link rel='replies'        thr:count='2'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/04/05/New-Mac-Stuff#comments' />
  592. <id>https://www.tbray.org/ongoing/When/202x/2020/04/05/New-Mac-Stuff</id>
  593. <published>2020-04-05T12:00:00-07:00</published>
  594. <updated>2020-04-25T20:18:24-07:00</updated>
  595. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Mac OS X' />
  596. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology' />
  597. <category scheme='https://www.tbray.org/ongoing/What/' term='Mac OS X' />
  598. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>With notes on microphones, Apple’s Magic Keyboard, laptop stands, and my <i>[gasp!]</i> new email client,     <a href='https://freron.com/'>MailMate</a></div></summary>
  599. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  600.    <p>With notes on microphones, Apple’s Magic Keyboard, laptop stands, and my <i>[gasp!]</i> new email client,
  601.    <a href="https://freron.com/">MailMate</a>.</p>
  602.    <p>Suddenly everyone’s working from home. Ours isn’t that big<span class='dashes'> —</span> in particular, only one
  603.    office<span class='dashes'> —</span> and there are three other people working or
  604.    studenting from it, so instead I’ve been using
  605.    <a href="/ongoing/When/201x/2019/08/06/Jeanneau-795">our boat</a>.  It’s cramped but scenic.  As the number of weeks that I’ll
  606.    be doing this stretches out into the unknowable future, there’s a chance for geek accessorization!  I believe
  607.    that in these days those of us who still have incomes should make a little effort to spread it around, especially when it might
  608.    boost our productivity a bit, or anyhow our morale.</p>
  609.    <h2 id='p-1'>Audio</h2>
  610.    <p>My job involves meetings every day. I viscerally loathe headphones after the first hour or so; in my tiny boxy office at
  611.    work I have decent audio and a big ugly
  612.    <a href="https://amzn.to/2UI643I">Shure MV51</a> mike that I don’t love; its aesthetic is brutalist and it
  613.    doesn’t work well with
  614.    <a href="https://aws.amazon.com/chime/">Amazon Chime</a> echo cancellation, so I often have to do the fast-finger unmute/mute when I
  615.    want to talk.</p>
  616.    <img src="https://www.tbray.org/ongoing/When/202x/2020/04/05/mv5.png" alt="Shure MV5" class="inline" />
  617.    <p>For the boat I got an
  618.    <a href="https://amzn.to/2ULuh9y">MV5</a> which is smaller, prettier, and plays nicer with Chime. Highly
  619.    recommended.</p>
  620.    <p>The boat has a stereo that while lousy includes Bluetooth, so I send the Mac audio there. The speakers in the
  621.    cabin ceiling are shitty but still better than those on the ancient 2015 MBPro I’m nursing along until corporate starts shipping
  622.    the new 16" with the good keyboard. Speaking of which….</p>
  623.    <h2 id='p-2'>Keyboard</h2>
  624.    <p>I have a decent little
  625.    <a href="https://amzn.to/2xNj9zY">Dell 24" 4K</a> outboard monitor and I was just parking the laptop in front of it and using
  626.    its keyboard and trackpad.  And this wasn’t terrible but I was really short on desk space for tea and snacks and phone and
  627.    notepad.</p>
  628.    <p>I’ve been using and liking Apple outboard keyboards forever so I got a
  629.    <a href="https://amzn.to/2XerMxY">“space grey” Apple Magic Keyboard</a> and wow, it was good before but it’s better now.
  630.    There’s no longer a distinction between wired and Bluetooth, they’re all wireless now, but come with a USB for charging.</p>
  631.    <p>To set it up you plug in the USB and by the time you look at the screen there’s a popup saying “Bluetooth keyboard paired
  632.    kthsby.”</p>
  633.    <p>The key travel and feel are perfection, and it’s quiet, which I like and is basic courtesy in a person who lives in
  634.    videoconferences.</p>
  635.    <img src="https://www.tbray.org/ongoing/When/202x/2020/04/05/keyboard.png" alt="Apple black Magic Keyboard" />
  636.    <p>There’s one awful flaw though, which I
  637.    <a href="/ongoing/When/200x/2008/07/22/On-Keyboards">discussed in 2008</a>: The keyboard intelligently includes big phat arrow
  638.    keys and the Home/End/Delete cluster, but stupidly insists on having the numeric keyboard, of interest only to
  639.    accountants. I regret the waste of scarce desk surface.</p>
  640.    <h2 id='p-3'>Stand</h2>
  641.    <p>I have two problems with the Mac being on the desktop. First, chronic neck pain from a lifetime of looking down
  642.    at laptops. Second, everybody I teleconference with gets a view featuring my grizzled neck and chin. So I went looking for a
  643.    stand and, on the Wirecutter’s recommendation, bought a
  644.    <a href="https://amzn.to/2X8U25b">Roost</a>.  It’s a really slick piece of design.</p>
  645.    <p>But take a moment and look at
  646.    the enclosed instructions. I first unfolded them late at night in dim lamplight and thought they were glyphs scrawled in
  647.    blackened blood by a
  648.    giant tentacle on the vast walls of R’lyeh; words no human throat could form, the sound of which would rip men’s minds to bleeding
  649.    shreds.</p>
  650.    <img src="https://www.tbray.org/ongoing/When/202x/2020/04/05/IMG_20200404_141553.png" alt="Roost laptop stand instructions" />
  651.    <p>I’d failed to notice the instructions until I had figured out the device, which is easy enough. Just as well, my R’lyehian is
  652.    weak.</p>
  653.    <h2 id='p-4'>Email</h2>
  654.    <p>At work, the server is (I think) Exchange, and you can use more or less any client that can talk to it. I started with
  655.    Outlook but quickly found the Office aesthetic soul-crushing. Then I used Mail.app for a while, but it’s just slooooow.
  656.    Then I moved over to Outlook Web App and yeah, that Outlook flavor, but it’s freakishly
  657.    fast. (There’s this thing called a Web browser where all you need to refresh is what the human is looking at, and OWA
  658.    takes good advantage.) But the composer is lame and I had to refresh too often, losing the speed advantage.</p>
  659.    <img src="https://www.tbray.org/ongoing/When/202x/2020/04/05/mailmate.png" alt="MailMate" class="inline" />
  660.    <p>So
  661.    <a href="https://aws.amazon.com/blogs/security/aws-security-profiles-eric-brandwine-vp-and-distinguished-engineer/">a smart
  662.    person at work</a> recommended
  663.    <a href="https://freron.com/">MailMate</a>, saying it was “Quirky but fast and powerful” which I’d say is fair: As fast as
  664.    OWA. I’m only a few days in but I’m not going back.  Every single time I’ve thought “I want to make it do X” it’s turned out
  665.    that there’s a nice straightforward way to get there.</p>
  666.    <p>One particular feature I love: It’s easy to turn off the red unread count in the Dock icon, which is a productivity
  667.    plus: I want to pick the time, unprompted, when I turn my attention to my email.</p>
  668.    <p>I occasionally get a little lost navigating through deeply nested email threads (when something goes viral on the Amazon
  669.    Principal-Engineers mailing list, you’re way into spaghetti territory). But I assume I’ll get the hang of it.</p>
  670.    <h2 id='p-6'>Boat as a Mac accessory</h2>
  671.    <p>It’s not a big boat, just over 24' in length; so I’m cramped; there is exactly one place to sit, with very limited scope for
  672.    squirming.  There’s no HVAC system run by Other People keeping the temperature controlled.</p>
  673.    <img src="https://www.tbray.org/ongoing/When/202x/2020/04/05/IMG_20200406_110224.png" alt="WFB" />
  674.    <img src="https://www.tbray.org/ongoing/When/202x/2020/04/05/IMG_20200406_170309.png" alt="WFB" />
  675.    <p>But; it’s a treat to be down by the water (roughly
  676.    <a href="https://goo.gl/maps/DKe8j1486mEEjaRWA">here</a>).  I’m hyper-aware of each day’s changeful weather story.
  677.    I perceive the passage of seasons with microscopic precision as the sun’s path changes week to
  678.    week and I adjust curtains to dodge the dazzle.</p>
  679.    <p>I have a 25-minute bike commute that puts opening and closing
  680.    punctuation around the day.</p>
  681.    <p>I get visits from seals, cormorants, mallards, crows, and gulls, plus there are two eagles nesting in a nearby tree.</p>
  682.    <p>I have a little burner to make tea on, a tiny fridge for my
  683.    sandwich, and a hard narrow berth for those irresistible nap
  684.    attacks. </p>
  685.    <p>I’m fully aware that all this is down to my decades-long run of good luck, and that I’m
  686.    once again enjoying hyperoverprivilege.</p>
  687. </div></content></entry>
  688.  
  689. <entry>
  690. <title>MacOS Lore, Early 2020</title>
  691. <link href='https://www.tbray.org/ongoing/When/202x/2020/04/04/MacOS-in-2020' />
  692. <link rel='replies'        thr:count='7'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/04/04/MacOS-in-2020#comments' />
  693. <id>https://www.tbray.org/ongoing/When/202x/2020/04/04/MacOS-in-2020</id>
  694. <published>2020-04-04T12:00:00-07:00</published>
  695. <updated>2020-04-05T02:45:33-07:00</updated>
  696. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Mac OS X' />
  697. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology' />
  698. <category scheme='https://www.tbray.org/ongoing/What/' term='Mac OS X' />
  699. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>I’ve used a Mac for   <a href='/ongoing/When/200x/2002/06/21/GettingStarted'>an extremely long time</a>; long enough that this blog’s topic tag is   <a href='/ongoing/What/Technology/Mac%20OS%20X/'>Mac OS X</a>, what macOS used to be called.  Herewith a looking-back   summary</div></summary>
  700. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  701.  <p>I’ve used a Mac for
  702.  <a href="/ongoing/When/200x/2002/06/21/GettingStarted">an extremely long time</a>; long enough that this blog’s topic tag is
  703.  <a href="/ongoing/What/Technology/Mac%20OS%20X/">Mac OS X</a>, what macOS used to be called.  Herewith a looking-back
  704.  summary.</p>
  705.  <h2 id='p-1'>Previously on <cite>Mac Lore</cite>…</h2>
  706.  <p>Clicking on that “Mac OS X” link cost me a half hour rat-holing on my journey over the years. So you don’t have to,
  707.  here’s a compendium of advice I give people about the Right Way To Do It; although some of the recommendations are not
  708.  Mac-specific:</p>
  709.  <ol>
  710.    <li>
  711.    <img src="https://www.tbray.org/ongoing/When/202x/2020/04/04/dock.png" alt="My Dock" class="inline" />
  712.      <p>Your screen is wider than it is high. So put the Dock on the side not the bottom.</p>
  713.    </li>
  714.    <li><p>Dock size and magnification are a matter of taste, but don’t auto-hide it, because…</p></li>
  715.    <li><p>Remove from the Dock every app that you don’t use regularly.  That way, everything there is either running or likely to
  716.    be, and it becomes a useful visual status check.  To the right is a snapshot of mine as I write this.</p></li>
  717.    <li><p>Go spend some quality time in the System Preferences for Trackpad.  Definitely turn on “Tap to click” and “Secondary
  718.    click”.  Then
  719.    <a href="https://apple.stackexchange.com/questions/374138/how-to-enable-double-tap-to-drag-in-mac-os-x-catalina">use the
  720.    accessibility preferences</a> to enable double-tap-and-drag.</p></li>
  721.    <li><p>While you’re in System Preferences, make sure your keyboard repeat rate is turned up to the max; few things are more
  722.    boring than holding down the spacebar or whatever and watching the cursor inch across the screen.</p></li>
  723.    <li><p>Command-space, which brings up Spotlight search, is your friend.  It’s really pretty good. Not enough people know that
  724.    you can highlight things in the result list and type command-I to get a nice little popup with useful information about what you just
  725.    found.</p></li>
  726.    <li><p>Use the
  727.    <a href="/ongoing/When/201x/2012/04/22/Tab-Lore">tab trick</a> in your favorite browser for one-click access to things you
  728.    care about.  (When I wrote that piece it didn’t work in Safari, but now it does.)</p></li>
  729.    <li><p>Keep a couple of browsers around.  Chrome and Safari are both great on Mac, Firefox is OK but recently I’ve found it
  730.    slow.  It’s common to use one for work stuff and the other for personal.  Another option is to be logged into Google in only one
  731.    of them and Google-invisible in the other.  Speaking of which, Safari is starting to have a strong privacy story.</p></li>
  732.    <li>
  733.      <img src="https://www.tbray.org/ongoing/When/202x/2020/04/04/preview.png" alt="Preview" class="inline" />
  734.      <p>Your browser will open PDFs directly and want you to read them there. Don’t. Download ’em and open ’em up in the awesome
  735.    Preview app.  Particularly if they’re big or complicated; Preview laughs at 500-page graphically-complex documents and provides
  736.    a superior read/search/navigate experience.</p></li>
  737.    <li><p>Despite the fact that Preview is great, do <em>not</em> try to use it to fill in legal forms.  It will look like
  738.    it’s trying to work, but it won’t. For that purpose (and that purpose only) go get
  739.    <a href="https://get.adobe.com/reader/">Adobe Acrobat Reader</a>.</p></li>
  740.    <li><p>Related to Preview: Let’s assume you’re a professional who sometimes needs to show off your work. So use the
  741.    command-control-shift-4 gesture to grab a piece of your screen, shift over to Preview, hit
  742.    command-N and it creates a new graphic with what you just captured. The only fly in the ointment is that when you save it, it’ll
  743.    want to use PNG and you almost always want JPG, so you have to toggle that on the Save menu.</p>
  744.    <p>This is how I captured the Dock image above.</p></li>
  745.    <li>
  746.      <img src="https://www.tbray.org/ongoing/When/202x/2020/04/04/keynote.png" alt="Keynote" class="inline" />
  747.      <p>If you have to give a presentation, use Keynote; it and Preview are Apple’s two truly great Mac apps. Do not go near
  748.    PowerPoint, it’s a travesty on Mac.</p></li>
  749.    <li><p>Learn to use the
  750.    <a href="/ongoing/When/201x/2013/06/29/OS-X-Emacs-Keystrokes">control-key navigation tricks</a>. They make editing
  751.    text<span class='dashes'> —</span> any text in almost any app<span class='dashes'> —</span> dramatically faster.</p></li>
  752.    <!--
  753.    <li><p>For extra credit, get comfortable with the Terminal and its command line; there are lots of things that you can do in
  754.    Finder that are immensely faster and easier in the Terminal.  In particular
  755.    <a href="https://scriptingosx.com/2017/02/the-macos-open-command/">the wonderful “open” command</a>.</p></li>
  756.    -->
  757.    <li><p>Turn off all the notifications you possibly can.  You should own your time.  If you have a reasonably active life
  758.    there will <em>always</em> be new things to read in mail and Twitter and Slack and so on; so go read them when you come to a
  759.    stopping point.  The only notification I leave on is the desktop
  760.    <a href="https://signal.org/">Signal</a> app, because you have to know me pretty well
  761.    to reach me there.  And (at work) mentions on
  762.    <a href="https://aws.amazon.com/chime/">Amazon Chime</a>.</p></li>
  763.    <li><p>Inbox Zero is a great idea but unattainable by most of us. Instead, try the
  764.    <a href="/ongoing/When/201x/2012/12/24/Inbox-light">Low-stress Inbox</a> technique.</p></li>
  765.    <li><p><a href="/ongoing/When/201x/2017/07/16/On-Password-Managers">Use a password manager.</a>  Really, <em>please</em> use
  766.    one.</p></li>
  767.  </ol>
  768.  <p>I wrote the first of these in 2002. I wonder how many more are in my future?</p>
  769. </div></content></entry>
  770.  
  771. <entry>
  772. <title>Plague Journal, April 4</title>
  773. <link href='https://www.tbray.org/ongoing/When/202x/2020/04/04/Plague-Journal-04-04' />
  774. <link rel='replies'        thr:count='1'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/04/04/Plague-Journal-04-04#comments' />
  775. <id>https://www.tbray.org/ongoing/When/202x/2020/04/04/Plague-Journal-04-04</id>
  776. <published>2020-04-04T12:00:00-07:00</published>
  777. <updated>2020-04-04T18:16:43-07:00</updated>
  778. <category scheme='https://www.tbray.org/ongoing/What/' term='The World/Plague Journal' />
  779. <category scheme='https://www.tbray.org/ongoing/What/' term='The World' />
  780. <category scheme='https://www.tbray.org/ongoing/What/' term='Plague Journal' />
  781. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>I’m an optimist so I don’t put the year in the title.  Once again: Writing is therapeutic. Open up whatever program you use to   write stuff in, and see what comes out.  Today’s adventure was the Socially Distanced Farmers’ Market</div></summary>
  782. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  783.  <p>I’m an optimist so I don’t put the year in the title.  Once again: Writing is therapeutic. Open up whatever program you use to
  784.  write stuff in, and see what comes out.  Today’s adventure was the Socially Distanced Farmers’ Market.</p>
  785.  <img src="https://www.tbray.org/ongoing/When/202x/2020/04/04/IMG_20200404_114016.png" alt="Socially Distanced Farmers’ Market" />
  786.  <p>They were organized as hell, the market subdivided into three Zones, each with its own line-up, social distance chalked on the
  787.  sidewalk.  The people density was unusually high in the neighborhood and a lot of people have started just walking down the middle
  788.  of the street, screw the motorists. I find this cheering.</p>
  789.  <p>People are so open and friendly! Everyone has a smile for everyone and random conversations break out
  790.  between strangers. This is usually a good thing, but in the Zone 1 lineup I found myself behind a conspiracy-theorist, talking
  791.  about how the “higher ups” were getting ready to impose food rationing.</p>
  792.  <p>I think serious food shortages unlikely in the developed world, but parts of our agricultural industry depend for harvesting
  793.  on poorly-paid abusively-treated migrant laborers.  When they can’t come, we’ll see if agribiz lets the produce rot in the fields or
  794.  raises wages enough to attract unemployed Canadians; and produce prices correspondingly.</p>
  795.  <h2 id='p-1'>Market menu</h2>
  796.  <p>Potatoes. More potatoes.  Damn, really a lot of potatoes.  We have a stash but I bought fingerlings anyhow because they were
  797.  cute.  The first new harvest in mid-spring is rhubarb; I bet there’s some starting next week.</p>
  798.  <p>The shopping list comprised apples, rolled oats, salad greens, dill, chives, cilantro, and a red onion.  I scored about
  799.  50%<span class='dashes'> —</span> by noon-ish when I got there, a lot of vendors’ shelves were looking bare. But also I got
  800.  <a href="http://www.oddsocietyspirits.com/spirits/">artisanal gin</a>, handcrafted chocolate, and wildflower honey.</p>
  801.  <p>We’re having a virtual movie party with a friend this evening;
  802.  <a href="https://en.wikipedia.org/wiki/Turtle_Diary">Turtle Diary</a>, I think.</p>
  803.  <p>Take time to be kind to each other; loved ones and strangers too.</p>
  804. </div></content></entry>
  805.  
  806. <entry>
  807. <title>Plague Journal, April 2</title>
  808. <link href='https://www.tbray.org/ongoing/When/202x/2020/04/02/Plague-Journal-04-02' />
  809. <link rel='replies'        thr:count='2'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/04/02/Plague-Journal-04-02#comments' />
  810. <id>https://www.tbray.org/ongoing/When/202x/2020/04/02/Plague-Journal-04-02</id>
  811. <published>2020-04-02T12:00:00-07:00</published>
  812. <updated>2020-04-03T02:36:24-07:00</updated>
  813. <category scheme='https://www.tbray.org/ongoing/What/' term='The World/Plague Journal' />
  814. <category scheme='https://www.tbray.org/ongoing/What/' term='The World' />
  815. <category scheme='https://www.tbray.org/ongoing/What/' term='Plague Journal' />
  816. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>Hey folks, one decent therapy for times like these when the world’s trying to drive you crazy is to tell your story; doesn’t   matter if anyone’s listening. This is adapted   from an email to the family that got kind of out of control.  Write your own #PlagueJournal entries and I’ll read ’em</div></summary>
  817. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  818.  <p>Hey folks, one decent therapy for times like these when the world’s trying to drive you crazy is to tell your story; doesn’t
  819.  matter if anyone’s listening. This is adapted
  820.  from an email to the family that got kind of out of control.  Write your own #PlagueJournal entries and I’ll read ’em.</p>
  821.  <p>I feel vaguely like I’m setting a bad example as I cycle furiously on empty-ish streets across town each day to
  822.  <a href="/ongoing/When/201x/2019/08/06/Jeanneau-795">the boat</a> and
  823.  back; the smallest office I’ve ever had, but the view is decent.</p>
  824.  <p>The weather remains obstinately wintry, temperature refusing to
  825.  venture out of single digits, which is OK when the sun’s out which it mostly isn’t. Nonetheless everything that can grow flowers is
  826.  already showing them or expanding the buds as fast as possible.  Mom was supposed to come visit us about now and there would have
  827.  been lots to look at.</p>
  828.  <p>Downtown, people gather on their balconies and cheer wildly for three minutes for the caregivers at 7PM when the hospital
  829.  shifts change; we were driving through the neighborhood last week, unsuspecting, as it exploded.  Not a dry cheek in the
  830.  car. Locally we’ve revived and expanded a long-neglected neighborhood mailing list, seeking people who might need some help;
  831.  plenty are offering but everyone seems to be making a go of it.  Anyhow, this very evening we gathered in a socially-distanced way
  832.  at the end of the block to bang pots and drums and tambourines and clap hands for three minutes, lots of smiles and none forced.</p>
  833.  <p>Some people are much more monastic in their isolation: go shopping once every other week, stay inside.  We find the grocery
  834.  stores are sanitizing and social-distancing effectively, so we shop more often. Also we’d really like some of the restaurants to
  835.  survive this thing so we’re getting take-out a couple or three times a week.  Plus we go for lots of walks - there’s a new
  836.  sidewalk courtesy where you make space for each other, stepping into the (empty) roads or on people’s lawns if need be. Very
  837.  Canadian.</p>
  838.  <p>We pick charities and send them money but so much of the population was already living so close to the edge, these wounds will
  839.  take a long time to heal.</p>
  840.  <p>We’re actually keeping in better touch with our friends than in healthy times, via Zoom and Skype and so on.  Unfortunately
  841.  what we talk about mostly is the plague news. I’m kind of tired of talking about it.  One of the best military blogs is entitled
  842.  
  843.  <a href="https://warisboring.com/">War is Boring</a><span class='dashes'> —</span> well, so is Covid-19.</p>
  844.  <p>I, a data-driven numbers guy, find the daily recitation of statistics maddening, although everyone in the province loves our
  845.  chief medic
  846.  <a href="https://en.wikipedia.org/wiki/Bonnie_Henry">Dr Bonnie Henry</a>, who has a silken voice and a Stoic demeanor.  They give
  847.  numbers like “number of cases” which I think
  848.  means “positive tests”, a number that is entirely useless because the testing is (quite reasonably) directed at
  849.  the most vulnerable and critical demographics.  I am beginning to zero in on the number of cases admitted to hospital every day,
  850.  or rather the rate of change in the number admitted<span class='dashes'> —</span> at least there’s clarity in what that
  851.  means<span class='dashes'> —</span> and in BC, the rate of change
  852.  in daily admissions is zero-ish.  Which is not exactly good but not catastrophic. Catastrophic is New York today and
  853.  Florida &amp; Louisiana &amp; Alabama &amp; Texas looking forward four weeks.  I don’t want to think about India and Africa.</p>
  854.  <p>Alberta is doing a little better than us, Ontario worse but not terrible, Quebec worse. But not bad like America, so far.</p>
  855.  <p>At
  856.  <a href="https://aws.amazon.com/">work</a>, we are running hot. Everyone’s stuck at home and using the Internet, which means us.
  857.  Over on the retail side, the
  858.  order spikes are frightening given that a lot of our employees are staying home for excellent reasons and the ones who are coming
  859.  in have to work at half-speed due to constant disinfecting and social distancing.   I see headlines in progressive publications
  860.  saying how we are cruelly ignoring the plague conditions; and internal emails about all the products that are going on
  861.  four-or-more-week
  862.  delivery because they have to run everything extra-slow to protect the staff so they can keep shipping
  863.  groceries and cleaning products. I really honestly don’t know what to think.</p>
  864.  <p>Our side of the company just has to make sure there are enough
  865.  incremental waves of computers available each new day to keep Zoom and Netflix and ambulance-dispatch apps on the air.</p>
  866.  <p>The boy and the girl are both at-school-online right now, Lauren and I at-work-online.  If it weren’t for the boat we’d be in trouble
  867.  since the guest room is under post-asbestos-remediation reconstruction and we’re packed in pretty tight already 9-to-5 without me
  868.  being in the house.</p>
  869.  <p>I’d really rather not be living inside a historically-significant news story.  But we all are, so the only choice is to make
  870.  the best of it.  The virus doesn’t care how brave you are, only how smart you are.</p>
  871.  <p>In closing:
  872.  <a href="https://xkcd.com/2287/">https://xkcd.com/2287/</a>.</p>
  873. </div></content></entry>
  874.  
  875. <entry>
  876. <title>Eventing Facets</title>
  877. <link href='https://www.tbray.org/ongoing/When/202x/2020/03/07/Eventing-Facets' />
  878. <link rel='replies'        thr:count='2'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/03/07/Eventing-Facets#comments' />
  879. <id>https://www.tbray.org/ongoing/When/202x/2020/03/07/Eventing-Facets</id>
  880. <published>2020-03-07T12:00:00-08:00</published>
  881. <updated>2020-04-03T01:43:13-07:00</updated>
  882. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Software/Eventing' />
  883. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology' />
  884. <category scheme='https://www.tbray.org/ongoing/What/' term='Software' />
  885. <category scheme='https://www.tbray.org/ongoing/What/' term='Eventing' />
  886. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>What happened was, at re:Invent 2019 I gave a talk entitled     <a href='https://www.youtube.com/watch?v=h46IquqjF3E'>Moving to Event-Driven Architecture</a>, discussing a list of     characteristics that distinguish eventing and messaging services.  It was a lot of work pulling the     material together and I’ve learned a couple of things since then; thus, welcome to the     <a href='/ongoing/What/Technology/Software/Eventing/'>Eventing Facets</a> blog series, of which this is the hub. It’s going to     take a while to fill this out</div></summary>
  887. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  888.    <p>What happened was, at re:Invent 2019 I gave a talk entitled
  889.    <a href="https://www.youtube.com/watch?v=h46IquqjF3E">Moving to Event-Driven Architecture</a>, discussing a list of
  890.    characteristics that distinguish eventing and messaging services.  It was a lot of work pulling the
  891.    material together and I’ve learned a couple of things since then; thus, welcome to the
  892.    <a href="/ongoing/What/Technology/Software/Eventing/">Eventing Facets</a> blog series, of which this is the hub. It’s going to
  893.    take a while to fill this out.</p>
  894.    <img src="https://www.tbray.org/ongoing/When/202x/2020/03/07/facets.png" alt="On-stage in Vegas" />
  895.    <h2 id='p-1'>Eventing or messaging?</h2>
  896.    <p>They’re just labels. If it’s publish-and-subscribe people usually say “event”; if there’s a designated receiver, “message”.
  897.    But there’s a lot of messy ground in between; and in my experience, the software does more or less the same stuff whichever
  898.    label’s on the cover.  So I may simultaneously talk about eventing and use the word “message” to describe the byte packages
  899.    being shuffled around.</p>
  900.    <h2 id='p-2'>Disclosure</h2>
  901.    <p>I work for AWS and am a member of the “Serverless” group that includes Lambda, SQS, SNS, MQ, and EventBridge.  I
  902.    also support services that make large-scale use of Kinesis. You can expect this to have obvious effects on the perceptions and
  903.    opinions offered herein.  I’m also reasonably well-acquainted with Rabbit/MQ and Kafka, but without any hands-on.</p>
  904.    <h2 id='p-3'>Process</h2>
  905.    <p>For each facet, I’ll introduce it here and write a blog piece or link to an existing one that covers the territory. In the
  906.    process, I’ll build up a table relating how a list of well-known eventing/messaging services support it, or don’t; I premiered a
  907.    version of that table in the talk (see the picture above) and more than one person has asked for a copy.</p>
  908.    <p>OK, here we go.</p>
  909.    <h2 id='p-4'>Facet: FIFO</h2>
  910.    <p>When you inject events into the cloud, can you rely on them coming out in the same order?  For details, see
  911.    <a href="/ongoing/When/202x/2020/03/07/FIFO-Facet">Facet: FIFO</a>.</p>
  912.    <h2 id='p-7'>Facet: Deduplication</h2>
  913.    <p>When you fire an event into the cloud, can you be sure it’ll only come out again once?  See
  914.    <a href="/ongoing/When/202x/2020/03/08/Deduping-Facet">Facet: Deduping</a>.</p>
  915.    <h2 id='p-8'>Facet: Subscribe or Receive</h2>
  916.    <p>When there’s an event in the cloud, how many different receivers can receive it?  Just one? See
  917.    <a href="/ongoing/When/202x/2020/03/09/PubSub-Direct-Facet">Facet:
  918.    Point-to-Point vs. Pub/Sub</a>.</p>
  919.    <h2 id='p-14'>Facet: Serverless or Broker</h2>
  920.    <p>Does your eventing infrastructure look like a cloud service, just an endpoint in the cloud?  Or is it an actual machine, or
  921.    cluster of machines, that you own and operate and connect to?  For details, see
  922.    <a href="/ongoing/When/202x/2020/03/10/Serverless-Facet">Facet: Broker vs Serverless</a>.</p>
  923.    <h2 id='p-15'>Facet: Push or Pull</h2>
  924.    <p>Do you have to poll the cloud for events, or can you get them pushed to you?  For details, see
  925.    <a href="/ongoing/When/202x/2020/03/21/Push-Pull-Facet">Facet: Push vs Pull.</a></p>
  926.    <h2 id='p-16'>Facet: Filtering vs firehose</h2>
  927.    <p>When you reach out into the cloud and use a queue or broker or bus to receive events, the simplest model is “I subscribe and
  928.    you send me everything.” Often this is perfectly appropriate. But not always; I’ve seen software subscribed to
  929.    high-volume sources that starts with really dumb code to look at the messages and decide whether they’re interesting, and
  930.    ends up throwing most of them on the ground.</p>
  931.    <p>In recent years I’ve increasingly noticed the infrastructure software offering some sort of built-in filtering capability, so
  932.    you can provide a query or pattern-match expression that says which events you want to see.</p>
  933.    <p>We built this into CloudWatch Events (now known as
  934.    <a href="https://aws.amazon.com/eventbridge/">EventBridge</a>) back in 2014, and people seemed to like it.  Then a couple of
  935.    years ago, SNS added filters to subscriptions and, since then, I’ve noticed that a steadily-increasing proportion of all
  936.    subscriptions use them.</p>
  937.    <p>So I’m pretty convinced that some sort of filtering capability is a basic feature that eventing and messaging software really
  938.    need.  There’s no standard for how to do this. Some of the filtering expressions look like SQL and some look like regular
  939.    expressions.</p>
  940.    <pre><code>{
  941.  "vendor": [ "vitamix", "instapot" ],
  942.  "product_detail": {
  943.    "price_usd": [
  944.      { "numeric": ["&lt;=", 150] }
  945.    ]
  946.  }
  947. }</code></pre>
  948.    <p>I don’t think this subject deserves its own article, especially since back in December I wrote
  949.    <a href="/ongoing/When/201x/2019/12/18/Content-based-filtering">Content-based Filtering</a>, which describes on particular
  950.    flavor of this technology that looks like the sample above, i.e. not like either SQL or regexes.  I think the piece gives a pretty good feeling for the
  951.    power and utility of event filtering generally.</p>
  952.    <h2 id='p-9'>Still to come</h2>
  953.    <p>Here are the ones I know I’m going to write about, and maybe there’ll be more:
  954.    <cite>Payload Flavors</cite>, <cite>Records vs. Documents</cite>,
  955.    <cite>Uniform vs Heterogeneous Events</cite>, <cite>Replaying and Sidelining</cite>.  Like I said, this will
  956.    take a while.</p>
  957.    <h2 id='p-5'>Summarizing</h2>
  958.    <p>Since this table includes references to software that I’ve never actually used, it’s perfectly possible that my research has
  959.    been imperfect and there are errors.  Please let me know if so and I’ll dig in deeper and correct if necessary.</p>
  960.    <table >
  961.      <tr><td class="empty"></td>
  962.      <th><a href="https://aws.amazon.com/amazon-mq/">ActiveMQ</a></th>
  963.      <th><a href="https://activemq.apache.org/components/artemis/">Artemis</a></th>
  964.      <th><a href="https://aws.amazon.com/eventbridge/">Event<br/>Bridge</a></th>
  965.      <th><a href="https://kafka.apache.org/">Kafka</a></th>
  966.      <th><a href="https://aws.amazon.com/kinesis/">Kinesis</a></th>
  967.      <th><a href="https://www.rabbitmq.com/">RabbitMQ</a></th>
  968.      <th><a href="https://aws.amazon.com/sns">SNS</a></th>
  969.      <th><a href="https://aws.amazon.com/sqs">SQS</a></th>
  970.      <th><a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html">SQS<br/>FIFO</a></th></tr>
  971.      <tr align="center"><th align="right">FIFO?</th><td>Yes</td><td>Yes</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td><td>No</td><td>Yes</td></tr>
  972.      <tr align="center"><th align="right">Dedupe?</th><td>*</td><td>Yes</td><td>No</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>No</td><td>Yes</td></tr>
  973.      <tr align="center"><th align="right">P2P or<br/>Pub/Sub</th><td>Both</td><td>Both</td><td>P/S</td><td>P/S</td><td>P/S</td><td>Both</td><td>P/S</td><td>P2P</td><td>P2P</td></tr>
  974.      <tr align="center"><th align="right">Serverless<br/>or Broker</th><td>B</td><td>B</td><td>S</td><td>B</td><td>†</td><td>B</td><td>S</td><td>S</td><td>S</td></tr>
  975.      <tr align="center"><th align="right">Push<br/>or Pull</th><td>Both</td><td>Both</td><td>Push</td><td>Pull</td><td>‡</td><td>Both</td><td>Push</td><td>Pull</td><td>Pull</td></tr>
  976.      <tr align="center"><th align="right">Filtering?</th><td>Yes</td><td>Yes</td><td>Yes</td><td>*</td><td>No</td><td>No</td><td>Yes</td><td>No</td><td>No</td></tr>
  977.    </table>
  978.    <h2 id='p-6'>Notes</h2><p> </p>
  979.    <p><b><span class="note">*</span></b> It looks like the software can do this but either it isn’t straightforward or doesn’t work out of the box.</p>
  980.    <p><b><span class="note">†</span></b> Kinesis is a cloud service but since you have to provision shards, it kind of acts like a broker too.  On the
  981.    other hand, adding and deleting shards is possible if a bit awkward.</p>
  982.    <p><b><span class="note">‡</span></b> Kinesis is pull-based, but there’s the
  983.    <a href="https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl-v2.html">Kinesis Client Library</a> which
  984.    gives a very push-like feeling.</p>
  985. </div></content></entry>
  986.  
  987. <entry>
  988. <title>Facet: Push vs Pull</title>
  989. <link href='https://www.tbray.org/ongoing/When/202x/2020/03/21/Push-Pull-Facet' />
  990. <link rel='replies'        thr:count='1'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/03/21/Push-Pull-Facet#comments' />
  991. <id>https://www.tbray.org/ongoing/When/202x/2020/03/21/Push-Pull-Facet</id>
  992. <published>2020-03-21T12:00:00-07:00</published>
  993. <updated>2020-03-22T15:56:59-07:00</updated>
  994. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Software/Eventing' />
  995. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology' />
  996. <category scheme='https://www.tbray.org/ongoing/What/' term='Software' />
  997. <category scheme='https://www.tbray.org/ongoing/What/' term='Eventing' />
  998. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>If you want to process events, you can fetch them from the infrastructure or you can have the infrastructure hand them to you.   Neither idea is crazy</div></summary>
  999. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  1000.  <p>If you want to process events, you can fetch them from the infrastructure or you can have the infrastructure hand them to you.
  1001.  Neither idea is crazy.</p>
  1002.  <p><i>[This is part of the
  1003.  <a href="/ongoing/What/Technology/Software/Eventing/">Event Facets</a> series.]</i></p>
  1004.  <p>When you make a request to fetch data, that’s called “pull”.
  1005.  Which is an off-by-one error; the “u” should be an “o” as in “poll”, because that’s how most network stuff
  1006.  works. I’ve heard old farts of my own vintage claim that on the network, polling is really all there is if you dig in deep
  1007.  enough.
  1008.  But we do use the term “push”, in which<span class='dashes'> —</span> maybe it’s just an illusion<span class='dashes'> —</span> the
  1009.  infrastructure reaches out and hands the event to you.</p>
  1010.  <h2 id='p-1'>For example</h2>
  1011.  <p>I’ll use two AWS services that I work nearby.</p>
  1012.  <p>First,
  1013.  <a href="https://aws.amazon.com/sqs/">SQS</a>, which is pure pull. You make one call to receive a batch of messages,
  1014.  and then another to delete them, once processed.  If they’re being pushed in the front end of the service faster than you’re
  1015.  taking them out, they build up inside, which is usually OK.</p>
  1016.  <img src="https://www.tbray.org/ongoing/When/202x/2020/03/21/pull.png" alt="Pull" />
  1017.  <div class='caption'><p>Pull!<br/>Credit: Petar Milošević /
  1018.  <a href="https://creativecommons.org/licenses/by-sa/4.0">CC BY-SA</a></p></div>
  1019.  <img src="https://www.tbray.org/ongoing/When/202x/2020/03/21/push.png" alt="Push" />
  1020.  <div class='caption'><p>Push!<br/>Credit: Rjhartley at English Wikipedia /
  1021.  <a href="https://creativecommons.org/licenses/by-sa/3.0">CC BY-SA</a></p></div>
  1022.  <p>In <a href="https://aws.amazon.com/sns/">SNS</a>, by contrast, events are dropped onto a “Topic” and then you subscribe things to the
  1023.  topic. You can subscribe SQS queues, Lambda functions, mobile notifications, and arbitrary HTTP endpoints.  When
  1024.  someone drops messages onto the topic, SNS “pushes” them to all the subscribed parties.</p>
  1025.  <h2 id='p-2'>On webhooks</h2>
  1026.  <p>“Webhook” is just another word for push.
  1027.  I remember when I first heard the idea in the early days of the Web, I thought it was complete magic:  “I’ll just
  1028.  send you something, and the something will include the address where you HTTP POST back to me.”  Asynchronous, loosely-coupled,
  1029.  what’s not to like?</p>
  1030.  <p>There are a lot of webhooks out there. We talk to the
  1031.  <a href="https://aws.amazon.com/eventbridge/integrations/">EventBridge Partners</a> who are injecting their customers’ events into
  1032.  AWS, and one thing we talk about is good ways to get events from customers back to them. The first answer is usually
  1033.  “we do Webhooks”, which means push.</p>
  1034.  <h2 id='p-3'>On being pushy</h2>
  1035.  <p>On the face of it, push delivery sounds seductive.  Particularly when you can back your webhook with something like a Lambda
  1036.  function. And sometimes it works great.  But there are two big problems.</p>
  1037.  <p>The first is scaling. If you’re putting up an endpoint and inviting whoever to push events to it, you’re implicitly making a
  1038.  commitment to have it available, secured, and ready to accept the traffic.  It’s really easy to get a nasty surprise. Particularly
  1039.  if you’re building a public-facing cloud app and it gets popular.  I can guarantee that we have plenty of services here in AWS
  1040.  that can accidentally crush your webhook like a bug with a traffic surge.</p>
  1041.  <p>Then there’s administration. You want data pushed to you, but you want it done securely.  That means whoever’s doing the
  1042.  pushing has to have credentials, and you have to trust them to manage them, and you have to arrange for rotation and
  1043.  invalidation as appropriate, and you really don’t want to be handling support calls of the form “I didn’t change anything and now
  1044.  my pushes are failing with HTTP 403!”</p>
  1045.  <h2 id='p-4'>On pulling</h2>
  1046.  <p>Polling, on the face of it, is a little more work.  If you care about latency, you’re going to have to have a host post a
  1047.  long-poll against the eventing infrastructure, which means you’re going to have to own and manage a host. (If you’re not
  1048.  latency-sensitive, just arrange to fire a Lambda every minute or so to pick up events, way easier.)</p>
  1049.  <p>On the other hand, you need <em>never</em> have a scaling problem, because you control your own polling rate, so you can
  1050.  arrange to only pick up work that you have capacity to handle.</p>
  1051.  <p>On the admin side, everybody already owns the problem of managing their cloud service provider credentials so presumably
  1052.  that’s not extra work.</p>
  1053.  <p>I’m not saying push is never a good idea; just that grizzled old distributed-systems geeks like me have a sort of warm comfy
  1054.  feeling about polling, especially for the core bread-and-butter workloads.</p>
  1055.  <h2 id='p-5'>A small case study</h2>
  1056.  <p>Ssshhh! I’m going to tell an AWS-internals secret.</p>
  1057.  <p>I talk a lot with the SNS team. They have a front-end fleet, a metadata
  1058.  fleet, and then the fleet that actually does the deliveries to the subscriptions.  It turns out that the software that delivers to
  1059.  HTTP-endpoint subscriptions is big, and maybe the most complex part of the whole service.</p>
  1060.  <p>When I learned that I asked one of the
  1061.  engineers why and she told me “Because those are other people’s endpoints. They make a configuration mistake or forget to rotate
  1062.  credentials or send a flood of traffic that’s like eight times what they can handle. And we have to keep running in all those
  1063.  cases.”</p>
  1064. </div></content></entry>
  1065.  
  1066. <entry>
  1067. <title>Facet: Broker vs Serverless</title>
  1068. <link href='https://www.tbray.org/ongoing/When/202x/2020/03/10/Serverless-Facet' />
  1069. <link rel='replies'        thr:count='2'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/03/10/Serverless-Facet#comments' />
  1070. <id>https://www.tbray.org/ongoing/When/202x/2020/03/10/Serverless-Facet</id>
  1071. <published>2020-03-10T12:00:00-07:00</published>
  1072. <updated>2020-03-21T15:17:05-07:00</updated>
  1073. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Software/Eventing' />
  1074. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology' />
  1075. <category scheme='https://www.tbray.org/ongoing/What/' term='Software' />
  1076. <category scheme='https://www.tbray.org/ongoing/What/' term='Eventing' />
  1077. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>Your event infrastructure might be a service in the cloud or might be an actual computer (or cluster) you connect to.  Both     choices are perfectly sensible. The trade-offs?  It’s complicated</div></summary>
  1078. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  1079.    <p>Your event infrastructure might be a service in the cloud or might be an actual computer (or cluster) you connect to.  Both
  1080.    choices are perfectly sensible. The trade-offs?  It’s complicated.</p>
  1081.    <p><i>[This is part of the
  1082.    <a href="/ongoing/What/Technology/Software/Eventing/">Event Facets</a> series.]</i></p>
  1083.    <p>At AWS where I work, our mainstream home-grown services (Kinesis, SQS, SNS) are all serverless. I mean, there are servers,
  1084.    lots of ’em, but you can’t see ’em, there’s just an endpoint that you connect to, usually with HTTP, to produce and consume
  1085.    events.  Which should be unsurprising, AWS is built on the proposition that everything should be a service. Here are two:</p>
  1086.    <img src="https://www.tbray.org/ongoing/When/202x/2020/03/10/sqs.png" alt="SQS" />
  1087.    <img src="https://www.tbray.org/ongoing/When/202x/2020/03/10/sns.png" alt="SNS" />
  1088.    <p>But there’s another approach, usually referred to as a “message broker”. Brokers are software packages that push around
  1089.    events or messages, and they typically need to be installed and run on servers that you own or rent. Then your app connects to
  1090.    them to send and receive.  Here are two:</p>
  1091.    <img src="https://www.tbray.org/ongoing/When/202x/2020/03/10/active.png" alt="ActiveMQ" />
  1092.    <img src="https://www.tbray.org/ongoing/When/202x/2020/03/10/rabbit.png" alt="RabbitMQ" />
  1093.    <p>The trade-offs, well, they’re complicated.</p>
  1094.    <h2 id='p-1'>Serverless?</h2>
  1095.    <p>These services have one huge advantage in that you don’t have to think about scaling.
  1096.    You can pretty well fire as much traffic at them as you want at them, and they’ll find a way to soak it up.  Load forecasting
  1097.    sucks and the penalties for being wrong are severe, so it’s best just to not do it.  Also, with serverless you don’t have to own
  1098.    and monitor and patch and maintain the servers, which is nice.</p>
  1099.    <h2 id='p-2'>Brokers?</h2>
  1100.    <p>First some background: When you work with a broker, you often don’t use HTTP. You nail up a TCP connection and
  1101.    just pump bytes back and forth. For this to work, you obviously need some sort of session and message framing protocol and boy,
  1102.    are there ever a lot of them, for example
  1103.    <a href="https://en.wikipedia.org/wiki/MQTT">MQTT</a>,
  1104.    <a href="https://en.wikipedia.org/wiki/Streaming_Text_Oriented_Messaging_Protocol">STOMP</a>,
  1105.    <a href="https://activemq.apache.org/openwire">OpenWire</a>, and
  1106.    <a href="https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol">AMQP</a> (watch out for AMQP, it comes in different,
  1107.    incompatible, versions).</p>
  1108.    <p>Often you don’t have to worry about that; you use some combination of a standardized API like
  1109.    <a href="https://en.wikipedia.org/wiki/Java_Message_Service">JMS</a> or a popular library like
  1110.    <a href="https://en.wikipedia.org/wiki/Apache_Camel">Apache Camel</a> and the messaging Just Happens.</p>
  1111.    <p>The fact that you’re using permanent connections mean that you should be able to expect lower latency, because you don’t have
  1112.    the well-known HTTP overheads in setting up and tearing down connections.  On the other hand, it means your software has to deal
  1113.    with the situation when your nailed-up connection breaks, which 100% of network connections eventually do.  (Once again, your
  1114.    library may take care of this for you.)</p>
  1115.    <h2 id='p-3'>Except for…</h2>
  1116.    <p>Let’s start with that scaling advantage that services like SQS and SNS offer.  It’s real. But… maybe you don’t care.  If you
  1117.    load up one of the MQs on a big honkin’ EC2 instance with lots of CPU and memory and threads, you can push an astonishing number
  1118.    of messages/second through it.  Like, maybe ten times the number you’ll ever need to send for the foreseeable future.</p>
  1119.    <p>On the other hand, how about that latency advantage that brokers give you, because of the nailed-up connection?  It’s
  1120.    real. But… on the HTTP side, we have
  1121.    <a href="https://en.wikipedia.org/wiki/Push_technology#Long_polling">HTTP long polling</a>, which can reduce receive latency
  1122.    a lot. And of course an increasing share of HTTP is now
  1123.    <a href="https://en.wikipedia.org/wiki/HTTP/2">HTTP/2</a>, which multiplexes multiple requests on a single long-lived
  1124.    connection.</p>
  1125.    <p>And that’s not all. The next step after HTTP/2 is
  1126.    <a href="https://en.wikipedia.org/wiki/QUIC">QUIC</a>, likely to be rebranded as HTTP/3.  It provides HTTP request semantics
  1127.    using
  1128.    <a href="https://en.wikipedia.org/wiki/User_Datagram_Protocol">UDP</a>, so there are no permanent connections at all, and in
  1129.    principle amazingly low latencies should be possible.</p>
  1130.    <h2 id='p-5'>Reliability?</h2>
  1131.    <p>On the serverless side, the story is excellent.  SNS and SQS use all the usual AWS availability-zone tricks to ensure that
  1132.    even if hosts crash (which by the way they do all the time), data keeps flowing.</p>
  1133.    <p>Brokers have a decent story too, if not quite as rock-solid. ActiveMQ makes it easy to set up broker pairs backed by a single
  1134.    filesystem-based message store, where the backup takes over reasonably quickly on host failure. (Unless of course you’ve built
  1135.    up a huge number of un-received messages, in which case restarts can get very sketchy.)  RabbitMQ runs in clusters, often of
  1136.    size three, and when you lose one you can add a new one back in and it’ll pick up state from the others.  Once again, if you’ve
  1137.    got a huge backlog, you may experience pain.</p>
  1138.    <h2 id='p-4'>In conclusion…</h2>
  1139.    <p>It’s complicated. But you already knew that.</p>
  1140. </div></content></entry>
  1141.  
  1142. <entry>
  1143. <title>Plague Advice</title>
  1144. <link href='https://www.tbray.org/ongoing/When/202x/2020/03/14/Social-Distance' />
  1145. <link rel='replies'        thr:count='2'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/03/14/Social-Distance#comments' />
  1146. <id>https://www.tbray.org/ongoing/When/202x/2020/03/14/Social-Distance</id>
  1147. <published>2020-03-14T12:00:00-07:00</published>
  1148. <updated>2020-03-15T11:29:00-07:00</updated>
  1149. <category scheme='https://www.tbray.org/ongoing/What/' term='The World' />
  1150. <category scheme='https://www.tbray.org/ongoing/What/' term='The World' />
  1151. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>A couple weeks back “Social Distancing” would’ve been a Big Thinker’s title in <cite>The Times</cite>, about the Downside   Of Facebook.  Now it’s a best practice if you care about flattening the COVID-19 curve and saving grandmothers.  I’m a believer;   recently I   tweeted   <a href='https://twitter.com/timbray/status/1237796651978199040'>Cancel Everything</a> and I meant it.  But this shouldn’t mean   that you can’t go outside; or shop; or photograph</div></summary>
  1152. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  1153.  <p>A couple weeks back “Social Distancing” would’ve been a Big Thinker’s title in <cite>The Times</cite>, about the Downside
  1154.  Of Facebook.  Now it’s a best practice if you care about flattening the COVID-19 curve and saving grandmothers.  I’m a believer;
  1155.  recently I   tweeted
  1156.  <a href="https://twitter.com/timbray/status/1237796651978199040">Cancel Everything</a> and I meant it.  But this shouldn’t mean
  1157.  that you can’t go outside; or shop; or photograph.</p>
  1158.  <p>Today we went to the
  1159.  <a href="https://eatlocal.org/markets/riley-park/">Riley Park Winter Farmers Market</a>, eleven blocks away in cold spring
  1160.  sun.</p>
  1161.  <img src="https://www.tbray.org/ongoing/When/202x/2020/03/14/XT302289.png" alt="At the Riley Park Farmers Market" />
  1162.  <div class='caption'><p>It’s way less crowded than the supermarket.<br/>Damn it was cold, 5°C at best. But I bought beets.</p></div>
  1163.  <blockquote>
  1164.    <h2 id='p-6'>Plague Advice</h2>
  1165.    <p>Find a way to support your local merchants to the extent you can while staying safe. Don’t be the one who who didn’t know
  1166.    what you’d got till it was gone.</p>
  1167.  </blockquote>
  1168.  <p>Here in BC we’re not in hard-lockdown where you have to stay inside and order your food. I and many others here put a
  1169.  lot of weight on words from our Provincial Health Officer Dr. Bonnie Henry (no Twitter of her own but she has
  1170.  <a href="https://twitter.com/bonniehenryfans">a fan club)</a>. As of this writing, she says that for now, it’s OK to shop for food
  1171.  and basics, to eat out, and especially to do things that are outside. Farmers markets offer two out of three.</p>
  1172.  <p>Just because that’s OK for us doesn’t mean it’s OK for you.  BC seems to have got lucky with early containment; it helps that
  1173.  anybody with Those Symptoms can have a COVID-19 test for the asking.</p>
  1174.  <p>I suppose the containment will break down
  1175.  at some point and Dr Henry will bring the hard-lockdown hammer.  Because she knows her shit and speaks the truth, she’ll bring the
  1176.  populace along with her.</p>
  1177.  <img src="https://www.tbray.org/ongoing/When/202x/2020/03/14/Bonnie-Harvey.png" alt="" />
  1178.  <div class='caption'><p>Bonnie Henry</p></div>
  1179.  <p>How has Knowing One’s Shit and Speaking Truth become, so often, anything but the default expected behavior?</p>
  1180.  <blockquote>
  1181.    <h2 id='p-1'>Plague Advice</h2>
  1182.    <p>Wherever you are, find your local Dr Henry equivalent and just <em>Do whatever the fuck they say.</em></p>
  1183.  </blockquote>
  1184.  <p>Back to Farmers Markets. There’s more space between you and the staff and the other shoppers. The aisles are wider. There’s no
  1185.  door or doorhandle. No shared shopping carts.  No sharing atmosphere in an enclosed space.</p>
  1186.  <p>Now I see I did one thing wrong: I paid with cash.  These days every greyhaired ponytailed organic-herbs vendor has (in Canada
  1187.  at least) a little goober you can wave your credit card or phone at; no touching required.  Cash is a
  1188.  notorious germ vector at the best of times, which these really totally aren’t.</p>
  1189.  <blockquote>
  1190.    <h2 id='p-3'>Plague Advice</h2>
  1191.    <p>After you’ve been to the market, don’t touch your face till you get home, then wash the hell out of your hands. Where by “the
  1192.    market” I mean anywhere.</p>
  1193.  </blockquote>
  1194.  <p>At the market there was a stringband, and also a dude in a wheelchair with an N95 facemask and electric guitar, soaring chords
  1195.  with snarly echo.  I left donations in both their open instrument cases; I hope they were careful to wash their hands after
  1196.  handling the cash.</p>
  1197.  <img src="https://www.tbray.org/ongoing/When/202x/2020/03/14/XT302285.png" alt="At the Riley Park Farmers Market" />
  1198.  <blockquote>
  1199.    <h2 id='p-5'>Plague Advice</h2>
  1200.    <p>Support performing artists. Because performances aren’t safe so performers don’t get paid. I have tickets to see Martin Barre,
  1201.    the Cowboy Junkies, and Billie Eilish; none of those performances will happen and I’m not asking for my money back. Go visit the
  1202.    Web sites of the musicians you like and find out how they sell music and merch and buy some already.</p>
  1203.  </blockquote>
  1204.  <p>When I read the stories about the hard-lockdown locales, the policy seems to include no going outside. Sure, don’t enter crowded
  1205.  spaces, including public transit. And don’t crawl malls. But if there’s a green space you can walk to and it’s big enough that
  1206.  people can maintain a respectful distance, I don’t see the downside.</p>
  1207.  <p>The upside is you’ll stay saner.</p>
  1208.  <blockquote>
  1209.    <h2 id='p-7'>Plague Advice</h2>
  1210.    <p>Go (carefully) outside!</p>
  1211.  </blockquote>
  1212. </div></content></entry>
  1213.  
  1214. <entry>
  1215. <title>Facet: FIFO</title>
  1216. <link href='https://www.tbray.org/ongoing/When/202x/2020/03/07/FIFO-Facet' />
  1217. <link rel='replies'        thr:count='0'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/03/07/FIFO-Facet#comments' />
  1218. <id>https://www.tbray.org/ongoing/When/202x/2020/03/07/FIFO-Facet</id>
  1219. <published>2020-03-07T12:00:00-08:00</published>
  1220. <updated>2020-03-10T01:52:47-07:00</updated>
  1221. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Software/Eventing' />
  1222. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology' />
  1223. <category scheme='https://www.tbray.org/ongoing/What/' term='Software' />
  1224. <category scheme='https://www.tbray.org/ongoing/What/' term='Eventing' />
  1225. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>When you inject events into the cloud, do you care whether they come out in the same order they went in?  If you do, you’ll be     asking for “FIFO” (rhymes with “Fly, foe!”) (stands for “First In First Out”). Some software has it, some doesn’t</div></summary>
  1226. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  1227.    <p>When you inject events into the cloud, do you care whether they come out in the same order they went in?  If you do, you’ll be
  1228.    asking for “FIFO” (rhymes with “Fly, foe!”) (stands for “First In First Out”). Some software has it, some doesn’t.</p>
  1229.    <p><i>[This is part of the
  1230.    <a href="/ongoing/What/Technology/Software/Eventing/">Event Facets</a> series.]</i></p>
  1231.    <p>If you think about it carefully, FIFO is only meaningful when you have a single-threaded sender and a single-threaded
  1232.    receiver; with more than one, who gets to say which message was sent (or received) first?</p>
  1233.    <img src="https://www.tbray.org/ongoing/When/202x/2020/03/07/Caravan_in_the_desert.png" alt="Caravan in the desert, Morocco." />
  1234.    <div class='caption'><p>Real-world FIFO semantics.<br/>Credit: “Caravan in the desert. Morocco, Sahara.”<br/>
  1235.    ©<a href="https://commons.wikimedia.org/wiki/User:Black_Sickle">Sergey
  1236.    Pesterev</a>/<a href="https://commons.wikimedia.org/wiki/Main_Page">Wikimedia
  1237.    Commons</a>/<a href="https://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA 4.0</a></p></div>
  1238.    <p>In practice, this means that a FIFO event stream can really only run as fast a single computer can send messages. Assuming
  1239.    there’s any real business significance to the messages, this is pretty slow: hundreds of transactions/second is regarded as
  1240.    extremely fast for a FIFO sequence.</p>
  1241.    <p>It turns out that in a high proportion of cases, what you have is a huge fast data stream that’s partitioned into many
  1242.    sub-streams, and the FIFO operates at the sub-stream level.  Think of the clickstream coming off a big Web site.  You might care
  1243.    about the order of events that are part of an individual session (there could be thousands or millions of sessions), but not whether
  1244.    one session’s events are before or after another’s. </p>
  1245.    <p>This is common enough that you’ll hear people use phrases like “session ordering”.</p>
  1246.    <p>That’s why I like the picture above. There are actually three strongly-ordered mini-caravans, which could arrive at the oasis in
  1247.    any old order while locally preserving strict within-caravan camel ordering.</p>
  1248.    <p>If you look at actual real-world software, there’s usually good support for this big-river-of-ordered-sessions model.  For
  1249.    example, both Kinesis and Kafka have session selection arguments and call them
  1250.    <a href="https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#partition-key">Partition Keys</a>. Both expose their
  1251.    shards, which are selected by the partition keys and where the FIFO semantics apply. This makes them shine for
  1252.    huge-scale session-oriented pub/sub streaming.</p>
  1253.    <p>SQS FIFO calls its session identifier
  1254.    <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html">Message
  1255.    Group IDs</a> and hides the internal shards.</p>
  1256.    <h2 id='p-1'>I always want FIFO, right?</h2>
  1257.    <p>Certainly, if you’re applying transactions to a bank account. But maybe not;
  1258.    Consider the case of
  1259.    <a href="https://smile.amazon.com/">Amazon.com</a>. Every time you press “buy”, the Web site generates a message saying what you
  1260.    bought, how you paid for it, where, to ship it, and so on, and drops that into the cloud, where it’ll eventually be picked up by
  1261.    the systems that do acquisition and packaging and shipping and accounting and tax and compliance and so on.  But if you think of it,
  1262.    since each of those orders is independent of all the others, the order doesn’t matter that much.</p>
  1263.    <img src="https://www.tbray.org/ongoing/When/202x/2020/03/07/horse-race.png" alt="2019年のジャパンカップ スタート後の先行争い" />
  1264.    <div class='caption'><p>Not FIFO.<br/>Credit: “2019年のジャパンカップ スタート後の先行争い”,<br/>
  1265.    <a href="https://commons.wikimedia.org/wiki/File:2019_japancup_15horses_running.jpg">by nakashi</a>.</p></div>
  1266.    <h2 id='p-1'>FIFO is hard</h2>
  1267.    <p>If your app is like Amazon.com, you should definitely use a service that doesn’t offer FIFO, because it’ll be cheaper and more
  1268.    elastic. It turns out that FIFO isn’t free<span class='dashes'> —</span> it’s not even cheap<span class='dashes'> —</span>  because
  1269.    it’s hard to build and operate.</p>
  1270.    <p>It’s not hard to understand why. A typical big Web service is commonly sharded, which is to say there’s a front-end fleet of
  1271.    hosts handling a flow of requests from many sources, which it deals out to more hosts in the back-end fleet to process.  Sharding
  1272.    lets you handle really any level of traffic.</p>
  1273.    <p>The way it typically works is that incoming requests end up at some sort of “load balancer” to deal the traffic
  1274.    out. Obviously this means you can’t maintain global FIFO ordering; that’s sort of OK for the reasons described just above.
  1275.    If you want to maintain the more typical and useful “session FIFO” semantics, you need to route the messages associated with each
  1276.    session to the same shard.  This practice is called “affinity” and, as I
  1277.    <a href="/ongoing/When/201x/2019/09/25/On-Sharding#p-1">described here</a>, it’s difficult.</p>
  1278. </div></content></entry>
  1279.  
  1280. <entry>
  1281. <title>Facet: Point-to-Point vs Pub/Sub</title>
  1282. <link href='https://www.tbray.org/ongoing/When/202x/2020/03/09/PubSub-Direct-Facet' />
  1283. <link rel='replies'        thr:count='0'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/03/09/PubSub-Direct-Facet#comments' />
  1284. <id>https://www.tbray.org/ongoing/When/202x/2020/03/09/PubSub-Direct-Facet</id>
  1285. <published>2020-03-09T12:00:00-07:00</published>
  1286. <updated>2020-03-10T01:52:35-07:00</updated>
  1287. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Software/Eventing' />
  1288. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology' />
  1289. <category scheme='https://www.tbray.org/ongoing/What/' term='Software' />
  1290. <category scheme='https://www.tbray.org/ongoing/What/' term='Eventing' />
  1291. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>When there’s an event in the cloud, how many different receivers can receive it?  There are two plausible answers: Just one, or     anyone</div></summary>
  1292. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  1293.    <p>When there’s an event in the cloud, how many different receivers can receive it?  There are two plausible answers: Just one, or
  1294.    anyone.</p>
  1295.    <p><i>[This is part of the
  1296.    <a href="/ongoing/What/Technology/Software/Eventing/">Event Facets</a> series.]</i></p>
  1297.    <img src="https://www.tbray.org/ongoing/When/202x/2020/03/09/postman.png" alt="Postman delivers letter" />
  1298.    <h2 id='p-2'>Point-to-point</h2>
  1299.    <p>The idea is that any given message can only be read by one receiver. You might imagine that, as the old photo above suggests,
  1300.    the sender supplies an address, choosing the receiver. But that’s actually a pretty rare scenario.  What’s much more common is the
  1301.    way that SQS works.</p>
  1302.    <p>In SQS, when you send a message, you have to pick which queue you’re sending it on.  Any number of receivers can be polling the
  1303.    queue, but only one will receive any given message. The API is clever; once you’ve received a message and processed it
  1304.    successfully, you’re supposed to delete it. There’s a time window after you’ve received it during which nobody else can see it. If
  1305.    the message isn’t deleted in time, it’ll pop back into availability. It’s called the “visibility timeout” and there are APIs for
  1306.    setting and adjusting it.</p>
  1307.    <p>Think about a back-end system in one of Amazon.com’s warehouses, picking up order events from the Web site. You want enough
  1308.    event readers in the warehouse to keep it busy, and you want to have each order processed by only one of them; so this flavor of
  1309.    point-to-point is just the ticket.</p>
  1310.    <p>It’s not a rare scenario at all; the Apache Artemis message broker
  1311.    <a href="https://activemq.apache.org/components/artemis/documentation/2.0.0/address-model.html">calls it “anycast”</a>.</p>
  1312.    <img src="https://www.tbray.org/ongoing/When/202x/2020/03/09/subscribers.png" alt="Publish and Subscribe" />
  1313.    <div class='caption'><p>Credit: Terje Skjerdal from Høvåg, Norway /
  1314.    <a href="https://creativecommons.org/licenses/by/2.0">CC BY</a></p></div>
  1315.    <h2 id='p-3'>Pub/Sub</h2>
  1316.    <p>It’s short for “Publish and Subscribe”, which nobody ever says. This is probably the default mode for most of the software that
  1317.    has the word “event” in its name.  The idea is, you publish your events onto a Bus or a Stream or a Topic or a
  1318.    Queue<span class='dashes'> —</span> there’s no standard terminology. Then receivers subscribe to the whatever-it’s-called and the
  1319.    events on it are in principle available to all of them.</p>
  1320.    <p>There are all sorts of variations in how subscriptions work; push vs pull, filter vs firehose; they’re important enough to
  1321.    deserve their own entries in this series.</p>
  1322.    <p>There are loads and loads of apps in every sector of business that are a good fit for pub/sub. I’d be amazed if there are
  1323.    readers in a business of any size that doesn’t have some running.</p>
  1324.    <p>At AWS, we have multiple services that do pub/sub: Kinesis, Managed Kafka, MQ, and then my favorite, SNS, which is used by more
  1325.    or less everyone to do more or less anything.  The way it works is, you publish messages to at Topic, and then you subscribe
  1326.    receivers to it. They can be SQS queues, Lambda functions, mobile-device messaging, and arbitrary HTTP endpoints. And you can have
  1327.    as many subscribers as you want, SNS will do the fan-out.</p>
  1328. </div></content></entry>
  1329.  
  1330. <entry>
  1331. <title>Facet: Deduping</title>
  1332. <link href='https://www.tbray.org/ongoing/When/202x/2020/03/08/Deduping-Facet' />
  1333. <link rel='replies'        thr:count='0'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/03/08/Deduping-Facet#comments' />
  1334. <id>https://www.tbray.org/ongoing/When/202x/2020/03/08/Deduping-Facet</id>
  1335. <published>2020-03-08T12:00:00-07:00</published>
  1336. <updated>2020-03-10T01:50:16-07:00</updated>
  1337. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology/Software/Eventing' />
  1338. <category scheme='https://www.tbray.org/ongoing/What/' term='Technology' />
  1339. <category scheme='https://www.tbray.org/ongoing/What/' term='Software' />
  1340. <category scheme='https://www.tbray.org/ongoing/What/' term='Eventing' />
  1341. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>When you fire an event into the cloud, can you be sure it’ll only come out again once?  It turns out that      sometimes they come out more often than they go in.  This may or may not be a problem in your     application. If it is, there are techniques to help work around it</div></summary>
  1342. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  1343.    <p>When you fire an event into the cloud, can you be sure it’ll only come out again once?  It turns out that
  1344.    sometimes they come out more often than they go in.  This may or may not be a problem in your
  1345.    application. If it is, there are techniques to help work around it.</p>
  1346.    <h2 id='p-1'>“At-least once”</h2>
  1347.    <p>This is a phrase you’ll hear a lot when you hang around with eventing/messaging people, (and cloud people generally).
  1348.    Builders work so hard at making sure everything gets delivered that they can end up doing it more than once.</p>
  1349.    <h2 id='p-2'>How can that happen?</h2>
  1350.    <p>Here’s a scenario: Suppose your software wants to retrieve and process events
  1351.    from a bus or topic or stream or whatever the service is called.  And suppose you retrieve one, then something goes
  1352.    wrong before you can acknowledge it. For
  1353.    example, the host your code’s running on might have failed. Or your code just crashed (<em>mine</em> never has bugs, but
  1354.    they tell me it happens). Or your software is written
  1355.    in Java and unfortunately went into a 45-second stop-the-world garbage-collection stall.</p>
  1356.    <p>When any of these happen, the eventing software will get the idea that you didn’t successfully receive the data, and since
  1357.    its primary purpose in life is to deliver reliably, it will try again. Which means you end up
  1358.    processing it twice.</p>
  1359.    <p>On the way in…</p>
  1360.    <img src="https://www.tbray.org/ongoing/When/202x/2020/03/08/events-in.png" alt="events in" />
  1361.    <p>…on the way out.</p>
  1362.    <img src="https://www.tbray.org/ongoing/When/202x/2020/03/08/events-out.png" alt="events out, with dupes" />
  1363.    <div class='caption'><p>Production note: For pictures of events in action, I’ve adopted this technique, created by
  1364.    <a href="https://www.notesfromthesound.com/">Colm MacCárthaigh</a>, of using emojois. It’d be a good all-purpose solution except
  1365.    for I’ve found the emjoi repertoire and representation highly nonportable between Mac and Windows.</p></div>
  1366.    <p>Sometimes duplicates aren’t your fault. Suppose one data center gets cut off from all the others in an AWS
  1367.    region. Yes, we try really hard to arrange for multiple redundant connections, but shit happens. A little birdie told that one
  1368.    day a few years ago, a badly-built bridge in Beijing fell down and it had been carrying three different network providers’ fibres.</p>
  1369.    <p>We call this situation a network partition. When it happens, both sides of the partition will try really hard not to
  1370.    lose any data. The details can get complicated, but duplicate messages are a common result.</p>
  1371.    <p>Obviously this could be a problem for your app.</p>
  1372.    <h2 id='p-3'>What to do?</h2>
  1373.    <p>There are situations where you can ignore the problem. Since duplicates are rare, in an analytics app or anything else
  1374.    that’s doing stats, they’re probably not a big deal. But there are lots of situations where they are.</p>
  1375.    <p>If your app has a database that does transactions, you’re probably OK, because you can safely remember when you’ve
  1376.    seen each event’s unique ID, and just discard duplicates.</p>
  1377.    <p>Another useful technique is
  1378.    <a href="https://en.wikipedia.org/wiki/Idempotence">idempotency</a>.  That is to say, structure your application such that API calls
  1379.    can be repeated without changing the result.  An example is anything that can be expressed as a pure HTTP PUT request. You can set
  1380.    a field to a given value as many times as you like without doing any damage.  Designing an app to work this way is
  1381.    tricky. But it’s an option that’s worth investigating, because having idempotent operations tends to produce apps that are
  1382.    robust in the face of all sorts of common failure scenarios.</p>
  1383.    <p>Here’s one thing to note: Duplicates are rare, but when they do happen, they tend to come in clusters (think about that
  1384.    bridge in Beijing).  I don’t know if that fact is useful in the context of your app, but just in case.</p>
  1385.    <p>But there are some apps that just can’t live with dupes.</p>
  1386.    <h2 id='p-4'>“Exactly once”</h2>
  1387.    <p>That‘s the terminology used for software that comes with built-in de-duping. One example would be
  1388.    <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html">SQS FIFO</a>.  Upon
  1389.    encountering this capability, you might ask yourself “why don’t I just use this for everything?”  It turns out, just as with
  1390.    <a href="/ongoing/When/202x/2020/03/07/FIFO-Facet">FIFO</a>, de-duping isn’t free and in fact isn’t particularly cheap.</p>
  1391.    <p>It can also get kind of complicated and there are more details than you might think.  Consider this blog:
  1392.    <a href="https://aws.amazon.com/blogs/developer/how-the-amazon-sqs-fifo-api-works/">How the Amazon SQS FIFO API Works</a>. It
  1393.    dives deep on all the details, which ends up taking over two thousand words.</p>
  1394.    <p>My advice would be to teach your software to live with duplicates, if at all possible. “At least once” systems are just part
  1395.    of the cloud landscape.</p>
  1396. </div></content></entry>
  1397.  
  1398. <entry>
  1399. <title>On Soaking the Rich</title>
  1400. <link href='https://www.tbray.org/ongoing/When/202x/2020/02/19/Soak-the-rich' />
  1401. <link rel='replies'        thr:count='5'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/02/19/Soak-the-rich#comments' />
  1402. <id>https://www.tbray.org/ongoing/When/202x/2020/02/19/Soak-the-rich</id>
  1403. <published>2020-02-19T12:00:00-08:00</published>
  1404. <updated>2020-02-19T23:24:21-08:00</updated>
  1405. <category scheme='https://www.tbray.org/ongoing/What/' term='The World/Politics' />
  1406. <category scheme='https://www.tbray.org/ongoing/What/' term='The World' />
  1407. <category scheme='https://www.tbray.org/ongoing/What/' term='Politics' />
  1408. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>The government of   <a href='https://en.wikipedia.org/wiki/British_Columbia'>BC</a>, the Canadian province where I live, just released a new   budget which, among other things, raises tax on high incomes.   <a href='https://www.cbc.ca/news/canada/british-columbia/bc-budget-2020-analysis-1.5467815?cmp=rss'>Here</a> is an overview.   The top marginal tax on incomes over C$220,000 goes from 16.8%   to 20.5%. This is just provincial tax; what with the Feds, the total top marginal rate is now 53%.  Not everyone is delighted.   For example Garth Turner, finance/real-estate blogger, who <a href='https://www.greaterfool.ca/2020/02/19/dr-moneybags/'>emits a   howl of right-wing grief</a>.   I’m comfortable speaking about this since I’m personally affected</div></summary>
  1409. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  1410.  <p>The government of
  1411.  <a href="https://en.wikipedia.org/wiki/British_Columbia">BC</a>, the Canadian province where I live, just released a new
  1412.  budget which, among other things, raises tax on high incomes.
  1413.  <a href="https://www.cbc.ca/news/canada/british-columbia/bc-budget-2020-analysis-1.5467815?cmp=rss">Here</a> is an overview.
  1414.  The top marginal tax on incomes over C$220,000 goes from 16.8%
  1415.  to 20.5%. This is just provincial tax; what with the Feds, the total top marginal rate is now 53%.  Not everyone is delighted.
  1416.  For example Garth Turner, finance/real-estate blogger, who <a href="https://www.greaterfool.ca/2020/02/19/dr-moneybags/">emits a
  1417.  howl of right-wing grief</a>.
  1418.  I’m comfortable speaking about this since I’m personally affected.</p>
  1419.  <p><i>[These days, a Canuck buck is worth about $0.75 American.]</i></p>
  1420.  <p>It’s worth noting that our province is led by the
  1421.  <a href="https://www.bcndp.ca/">New Democratic Party</a> (everyone says  “NDP”) who are Social Democrats; by and large
  1422.  pretty moderate by world standards. On the American spectrum they’re red-toothed commies; not too far off
  1423.  what Bernie Sanders would like to see.</p>
  1424.  <h2 id='p-1'>Numbers</h2>
  1425.  <p>Let’s do some math, OK?  Garth Turner points out how awful this will be for people living in super-expensive Vancouver and
  1426.  making a paltry $250K. Hmm, the marginal increase is 3.7% on income over $220K. So those poor $250K people are going
  1427.  to be paying an extra $1110/year.  Garth also excerpts a letter from an aggrieved doctor’s husband who says they’ll be paying an
  1428.  extra $1K a month; by my arithmetic the doctor’s making over $540K taxable, thus taking home over $250K, ignoring the tax dodges
  1429.  available to the self-employed, then there’s the husband’s income.  Maybe I’m missing something, but neither of these calculations yield
  1430.  what feel to me like lifestyle-changing numbers.</p>
  1431.  <p>Side-note: I don’t want to diss Garth too much because he’s a really excellent writer on personal finance in general and
  1432.  real-estate in particular; if you’re in Canada and have money to invest you should totally read him.  Also he’s funny and
  1433.  runs cute dog pictures.  He becomes less interesting as he veers into overly-predictable right-wing tax-grouch tropes.  And,
  1434.  disclosure: I’m
  1435.  <a href="http://www.turnerinvestments.ca/">a customer</a>.</p>
  1436.  <h2 id='p-2'>Saving and spending</h2>
  1437.  <p>Here’s the thing: A high proportion of people subject to the tax hike are taking home more than they need to live on, so the
  1438.  effect of this move is that their savings (and wealth) will grow more slowly.  Most such people
  1439.  retire with enough to live on, and are in their forties or older.  So the tax hike won’t result in any short-term spending
  1440.  decreases to speak of, but starting say ten
  1441.  years from now, a demographic of well-off retirees will have a little less to pump into that future economy.</p>
  1442.  <p>And those slightly-smaller savings, what about them?  They join a world-wide glut of capital, surging around looking for a
  1443.  decent return. Plenty will look for it outside Canada, or in ventures where most of the profits go to money people.</p>
  1444.  <p>On the other hand, the extra $200M or so a year this tax maneuver pulls in will <em>all</em> get spent by the
  1445.  government, the vast majority in paychecks to middle- and working-class civil servants and contractors, who will in
  1446.  turn spend almost all of it in the next twelve months right here in BC.  Maybe I’m missing something, but this policy feels like
  1447.  an economic win/win. The only losers are one-percenters like me; modestly, a decade or two from now.  Maybe I’ll only be able to
  1448.  vacation once a year.</p>
  1449.  <p>Or is that kind of thinking dangerously socialist?  Here’s some data from <cite>The Economist</cite>:
  1450.  <a href="https://www.economist.com/finance-and-economics/2020/02/13/wage-gains-for-low-earners-have-helped-sustain-americas-economic-expansion">Wage gains for low earners have helped sustain America’s economic expansion</a>.
  1451.  It’s data-rich and convincing. Inequality not only sucks, it’s bloody inefficient and generally bad for the economy.</p>
  1452.  <p>In fact, redistributionist policies might improve the economy enough that the return on my savings will make up for their
  1453.  slightly smaller size.</p>
  1454.  <h2 id='p-3'>Conclusion</h2>
  1455.  <p>Raise the  minimum wage and the marginal tax rates on people like me.  Both as a left-winger and an investor, I
  1456.  approve.</p>
  1457. </div></content></entry>
  1458.  
  1459. <entry>
  1460. <title>Sally Leekie</title>
  1461. <link href='https://www.tbray.org/ongoing/When/202x/2020/02/14/Valentines-Feast' />
  1462. <link rel='replies'        thr:count='2'        type='application/xhtml+xml'        href='/ongoing/When/202x/2020/02/14/Valentines-Feast#comments' />
  1463. <id>https://www.tbray.org/ongoing/When/202x/2020/02/14/Valentines-Feast</id>
  1464. <published>2020-02-14T12:00:00-08:00</published>
  1465. <updated>2020-02-15T14:11:52-08:00</updated>
  1466. <category scheme='https://www.tbray.org/ongoing/What/' term='The World/Food and Drink' />
  1467. <category scheme='https://www.tbray.org/ongoing/What/' term='The World' />
  1468. <category scheme='https://www.tbray.org/ongoing/What/' term='Food and Drink' />
  1469. <summary type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>Happy Valentines! A day that celebrates love comes, in my mind, second only to the one that celebrates giving thanks. I didn’t do roses or chocolate, but I made dinner for a couple of people I love; one of the dishes was improvised and came out well, so this recipe is my valentine to the world</div></summary>
  1470. <content type='xhtml'><div xmlns='http://www.w3.org/1999/xhtml'>
  1471. <p>Happy Valentines!
  1472. A day that celebrates love comes, in my mind, second only to the one that celebrates giving thanks. I
  1473. didn’t do roses or chocolate, but I made dinner for a couple of people I love; one of the dishes was improvised and came out well,
  1474. so this recipe is my valentine to the world.</p>
  1475. <p>The main ingredients are salmon and leeks; thus “Sally Leekie”.</p>
  1476. <h2 id='p-1'>Ingredients</h2>
  1477. <ol>
  1478. <li><p><i>Salmon:</i> We use wild frozen-at-sea Pacific Sockeye, usually obtainable in Vancouver.  750g fed three generously.</p></li>
  1479. <li><p><i>Leeks:</i> A couple of big ones.</p></li>
  1480. <li><p><i>Garlic:</i> Confession: For this dish, I used minced garlic out of a jar that I bought in a supermarket; a couple of
  1481. heaping teaspoons-full.</p></li>
  1482. <li><p><i>Seasoning:</i> Oregano, Fennel, and black pepper, all served liberally.</p></li>
  1483. <li><p><i>Oil:</i> Now this is interesting. I started with olive oil because it was at the front of the cupboard, then thought
  1484. “Leeks? Everyone knows you cook them in butter!” so I added some of that too.  It came out nicely.</p></li>
  1485. </ol>
  1486. <img src="https://www.tbray.org/ongoing/When/202x/2020/02/14/IMG_20200214_175257.png" alt="Sauteeing leeks" />
  1487. <div class='caption'><p>Sauteeing leeks.</p></div>
  1488. <h2 id='p-2'>Process</h2>
  1489. <ol>
  1490. <li><p>I covered the bottom of a sautee pan with oil, tossed in the garlic and seasoning, and heated it to not-quite-bubbling-or-smoking
  1491. for ten or fifteen minutes.</p></li>
  1492. <li><p>While this was happening, I chopped the leeks and salmon. If you haven’t done leeks before, you have to cut them
  1493. <em>lengthwise</em> then take them over to the sink and wash out the mud and gunk that tends to occur; leeks are just not hygenic
  1494. vegetables. Salmon into bite-size chunks.</p></li>
  1495. <li><p>I turned up the heat to the point that a few bubbles were occurring, and put the chopped leeks through in three
  1496. batches. Watch out; they cook down tremendously, so you want to start with more than you need.  Maybe five minutes a batch;
  1497. until they’ve lost their curl but not their color.
  1498. I used a bowl in the warming drawer under the oven to accumulate the leeks and keep them warm.</p></li>
  1499. <li><p>Once the leeks were done I dropped the salmon into the garlicky spiced leek-flavored oil and sauteed that for maybe ten
  1500. minutes, till all the sides were sealed and it was getting ready to eat.</p></li>
  1501. <li><p>Finally, I took the leeks out of the warming drawer, tossed them in with the salmon, turned the heat up so there was a bit of
  1502. smoke, and tossed the mixture together.</p></li>
  1503. </ol>
  1504. <p>It looked neat, the salmon pink contrasting with the leek green. There were no leftovers and sincere compliments.</p>
  1505. </div></content></entry>
  1506.  
  1507. </feed>
  1508.  

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

  1. Download the "valid Atom 1.0" banner.

  2. Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)

  3. Add this HTML to your page (change the image src attribute if necessary):

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=http%3A//www.tbray.org/ongoing/ongoing.atom

Copyright © 2002-9 Sam Ruby, Mark Pilgrim, Joseph Walton, and Phil Ringnalda