[Valid Atom 1.0] This is a valid Atom 1.0 feed.


This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.


  1. <?xml version='1.0' encoding='UTF-8'?><?xml-stylesheet href="" type="text/css"?><feed xmlns='' xmlns:openSearch='' xmlns:blogger='' xmlns:georss='' xmlns:gd="" xmlns:thr=''><id>,1999:blog-7528831701633643336</id><updated>2017-03-28T10:35:50.695-07:00</updated><category term="-"/><category term="--"/><category term="afe GLDv3"/><category term="atom"/><category term="blogger"/><category term="feedburner"/><category term="illumos arcmsr"/><category term="illumos gsoc"/><category term="joyent illumos zfs"/><category term="mxfe GLDv3"/><category term="oracle opensolaris"/><category term="rss"/><category term="sdcard"/><category term="zfs nexenta fma"/><title type='text'>/dev/dump</title><subtitle type='html'>illumos and storage.  Rockets and whitewater.  Go figure.</subtitle><link rel='' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href=''/><link rel='hub' href=''/><link rel='next' type='application/atom+xml' href=';start-index=26&amp;max-results=25'/><author><name>Garrett D&#39;Amore</name><email></email><gd:image rel='' width='32' height='32' src='//'/></author><generator version='7.00' uri=''>Blogger</generator><openSearch:totalResults>339</openSearch:totalResults><openSearch:startIndex>1</openSearch:startIndex><openSearch:itemsPerPage>25</openSearch:itemsPerPage><entry><id>,</id><published>2016-10-24T21:53:00.000-07:00</published><updated>2016-10-24T21:53:03.943-07:00</updated><title type='text'>MacOS X Mystery (Challenge)</title><content type='html'>(Maybe my MacOS X expert friends will know the answer.)&lt;br /&gt;&lt;br /&gt;This is a mystery that I cannot seem to figure out. &amp;nbsp;I think its a bug in the operating system, but I cannot seem to figure out the solution, or even explain the behavior to my satisfaction.&lt;br /&gt;&lt;br /&gt;Occasionally, a shell window (iTerm2) will appear to &quot;forget&quot; my identity. &lt;br /&gt;&lt;br /&gt;For example:&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;% whoami&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;501&lt;/span&gt;&lt;br /&gt;&lt;br /&gt;That&#39;s half right... The same command in other window is more correct:&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;% whoami&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;garrett&lt;/span&gt;&lt;br /&gt;&lt;br /&gt;Further, id -a reports differently:&lt;br /&gt;&lt;br /&gt;The broken window:&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;% id -a&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;uid=501 gid=20(staff) groups=20(staff),501,12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),33(_appstore),100(_lpoperator),204(_developer),395(,398(,399(;/span&gt;&lt;br /&gt;&lt;br /&gt;The working one:&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;% id -a&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;uid=501(garrett) gid=20(staff) groups=20(staff),501(access_bpf),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),33(_appstore),100(_lpoperator),204(_developer),395(,398(,399(;/span&gt;&lt;br /&gt;&lt;br /&gt;It appears that the shell (and this broken behavior seems to be inherited by child shells, by the way), somehow loses the ability to map numeric Unix ids to login names.&lt;br /&gt;&lt;br /&gt;So I tried another command:&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;% dscl . -read /Users/garrett&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;Operation failed with error: eServerError&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;&lt;br /&gt;&lt;/span&gt;The same works properly in my other window (I&#39;m not posting the entire output, since its really long).&lt;br /&gt;&lt;br /&gt;I am wondering what could possibly be different. &amp;nbsp;The behavior doesn&#39;t seem to depend on environment variables (I&#39;ve tried stripping those out).&lt;br /&gt;&lt;br /&gt;I&#39;m thinking that there is something in the process table (in the MacOS X equivalent of the uarea?) that gives me access to directory services -- and that this is somehow clobbered. &amp;nbsp;As indicated, whatever the thing is, it appears to be inherited across fork(2).&lt;br /&gt;&lt;br /&gt;I thought maybe I could figure this out with DTrace or dtruss... but Apple have crippled DTrace on the platform and this is one of those binaries that I am unable to introspect. &amp;nbsp;Arrgh!&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;sudo dtruss dscl . -read /Users/garrett&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;Password:&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;dtrace: system integrity protection is on, some features will not be available&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;&lt;br /&gt;&lt;/span&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;dtrace: failed to execute dscl: dtrace cannot control executables signed with restricted entitlements&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;&lt;br /&gt;&lt;/span&gt;Btw, I&#39;m running the latest MacOS X:&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;% uname -a&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;Darwin Triton.local 16.0.0 Darwin Kernel Version 16.0.0: Mon Aug 29 17:56:20 PDT 2016; root:xnu-3789.1.32~3/RELEASE_X86_64 x86_64&lt;/span&gt;&lt;br /&gt;&lt;br /&gt;So, for my MacOS X expert friends -- anyone know how directory services &lt;i&gt;really&lt;/i&gt;&amp;nbsp;works? &amp;nbsp;(As in how it works under the hood?) &amp;nbsp;I don&#39;t think we&#39;re in UNIX land anymore, Toto!&lt;br /&gt;&lt;br /&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=4932888243302548895' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='MacOS X Mystery (Challenge)'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>0</thr:total></entry><entry><id>,</id><published>2016-10-24T11:44:00.002-07:00</published><updated>2016-10-24T11:44:58.610-07:00</updated><title type='text'>Security Advice to IoT Firmware Engineers</title><content type='html'>Last Friday (October 16, 2016), a &lt;a href=&quot;;&gt;major DDoS&lt;/a&gt; attack brought down a number of sites across the Internet. &amp;nbsp;My own &lt;a href=&quot;;&gt;employer&lt;/a&gt; was amongst those affected by the wide spread DNS outage.&lt;br /&gt;&lt;br /&gt;It turns out that the sheer scale (millions of unique botnet members) was made possible by the IoT, and rather shoddy engineering practices.&lt;br /&gt;&lt;br /&gt;Its time for device manufacturers and firmware engineers to &quot;grow up&quot;, and learn how to properly engineer these things for the hostile Internet, so that they don&#39;t have to subsequently issue recalls when their customers&#39; devices are weaponized by attackers without their owners knowledge.&lt;br /&gt;&lt;br /&gt;This blog is meant to offer some advice to firmware engineers and manufacturers in the hope that it may help them prevent their devices from being used in these kinds of attacks in the future.&lt;br /&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;Passwords&lt;/h4&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;Passwords are the root of most of the problems, and so much of the advice here is about improving the way these are handled.&lt;/div&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;No Default Passwords&lt;/h4&gt;&lt;br /&gt;The idea of using a simple default password and user name, like &quot;admin/admin&quot;, is a practice from the 90&#39;s, and is intended to facilitate service personnel, and eliminate management considerations from dealing with many different passwords. &amp;nbsp;Unfortunately, this is probably the single biggest problem -- bad usernames and passwords. &amp;nbsp;Its far worse in an IoT world, where there are many thousands, or even millions, of devices that have the same user name and password.&lt;br /&gt;&lt;br /&gt;The proper solution is to allocate a unique password to each and every device. &amp;nbsp;Much like we already do manage unique MAC addresses, we need every device to have a unique password. &amp;nbsp;(Critically, the password must &lt;i&gt;not&lt;/i&gt;&amp;nbsp;be derived from the MAC address though.)&lt;br /&gt;&lt;br /&gt;My advice is to simply have a small amount of ROM that is factory burned with either a unique password, or a numeric key that can be used to create one. &amp;nbsp;(If you have enough memory to store a dictionary in generic firmware -- say 32k words, you can get very nice human manageable default passwords by storing just four 16-bit numbers, each representing an index into the dictionary (so only 15 bits of unique data, but thats 60 bits of total entropy, which is plenty to ensure that every device has its own password -- and only requires storing a 64-bit random number in ROM.)&lt;br /&gt;&lt;br /&gt;Then you have nice human parseable passwords like &quot;bigger-stampede-plasma-pandering&quot;. &amp;nbsp;These can be printed on the same sticker that MAC passwords are typically given. &amp;nbsp;(You could also accept a hexadecimal representation of the underlying 64-bit value, or just use that instead of human readable passwords if you are unable to accommodate an English dictionary. &amp;nbsp;Devices localized for use in other countries could use locale-appropriate dictionaries as well.)&lt;br /&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;Mandatory Authorization Delay&lt;/h4&gt;&lt;br /&gt;Second, IoT devices should inject a minimum delay after password authentication attempts (regardless of whether successful or otherwise). &amp;nbsp;Just a few seconds is enough to substantially slow down dictionary attacks against poorly chosen end-user passwords. &amp;nbsp;(2 seconds means that only 1800 unique attempts can be performed per hour under automation - 5 seconds reduces that to 720. &amp;nbsp;It will be difficult to iterate a million passwords against a device that does this.)&lt;br /&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;Strong Password Enforcement&lt;/h4&gt;&lt;br /&gt;User chosen passwords should not be a single dictionary word; indeed, the default should be to use a randomly generated password using the same dictionary approach above (generate a 64-bit random number, break into chunks, and index into a stock dictionary). &amp;nbsp;It may be necessary to provide an end-user override, but it should be somewhat difficult to get at by default, and when activate should display large warnings about the compromise to security that user-chosen passwords typically represent.&lt;br /&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;Networks&lt;/h4&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;Dealing with the network, and securing the use of the network, is the other part of the problem that IoT vendors need to get right.&lt;/div&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;Local Network Authentication Only&lt;/h4&gt;&lt;br /&gt;IoT devices generally know the network they are on; if the device has a separate management port or LAN-only port (like a WiFi Router), it should only by default allow administrator access from that port.&lt;br /&gt;&lt;br /&gt;Devices with only a single port, or that exist on a WiFi network, should prevent administrator access from &quot;routed&quot; networks, by default. &amp;nbsp; That is, devices should not allow login attempts from a remote IP address that is not on a local subnet, by default. &amp;nbsp;While this won&#39;t stop many attacks (especially those on public WiFis), it makes attacking them from a global botnet, or managing them as part of a global botnet, that much harder. &amp;nbsp; (Again, there has to be a provision to disable this limitation, but it should present a warning.)&lt;br /&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;Encrypted Access Only&lt;/h4&gt;&lt;br /&gt;Use of unsecured channels (HTTP or telnet) is unacceptable in this day and age. &amp;nbsp;TLS and/or SSH are the preferred ways to do this, and will let your customers deploy these devices somewhat more securely.&lt;br /&gt;&lt;br /&gt;&lt;h4&gt;Secure All Other Ports&lt;/h4&gt;&lt;br /&gt;Devices should disable any network services that are not specifically part of the service they offer, or intrinsic to their management. &amp;nbsp; System administrators have known to do this on systems for decades now, but it seems some firmwares still have stock services enabled that can be used as attack vectors.&lt;br /&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;Don&#39;t Advertise Yourself&lt;/h4&gt;&lt;br /&gt;This one is probably the hardest. &amp;nbsp;&lt;a href=&quot;;&gt;mDNS&lt;/a&gt; and device discovery over &quot;standard&quot; networks is one of the ways that attackers find devices to target. &amp;nbsp;Its far far better to have this disabled by default -- if discovery is needed during device configuration, then it can be enabled briefly, when the device is being configured. &amp;nbsp;Having a &quot;pairing&quot; button to give end-users the ability to enable this briefly is useful -- but mDNS should be used only with caution.&lt;br /&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;Secure Your Channel Home&lt;/h4&gt;&lt;br /&gt;Devices often want to call-home for reporting, or web-centric command &amp;amp; control. &amp;nbsp;(E.g. remote management of your thermostat.) &amp;nbsp;This is one of the major attack vectors. &amp;nbsp; (If you can avoid calling home altogether, this is even better!)&lt;br /&gt;&lt;br /&gt;Users must be able to disable this function (it should be disabled by default in fact). &amp;nbsp;Furthermore, the channels must be properly secured entirely through your network, with provision for dealing with a compromise (e.g. leaked private keys at the server side). &amp;nbsp;Get a security expert to review your protocols, and your internal security practices.&lt;br /&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;Mesh Securely&lt;/h4&gt;&lt;br /&gt;Building local mesh networks of devices, e.g. to create a local cloud, means having strong pairing technology. &amp;nbsp;The strongest forms of this require administrator action to approve -- just like pairing a bluetooth keyboard or other peripheral.&lt;br /&gt;&lt;br /&gt;If you want to automate secure mesh provisioning, you have to have secure networking in place -- technologies like VPN or &lt;a href=&quot;;&gt;ZeroTier&lt;/a&gt; can help build networking layers that are secure by default.&lt;br /&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;Don&#39;t Invent Your Own Protocols&lt;/h4&gt;&lt;br /&gt;The roadside is littered with the corpses of protocols and products that attempted to invent their own protocols or use cryptography in non-standard ways. &amp;nbsp;The best example of this is &lt;a href=&quot;;&gt;WEP&lt;/a&gt;, which took a relatively secure crypto layer (RC4 was not broken at the time), but deployed it naively and brokenly. &amp;nbsp;RC4 got a very bad rap for this, but it was actually WEP that was broken. &amp;nbsp;(Since then, RC4 itself has been shown to have some weaknesses, but this is relatively new compared to the brokenness that was WEP.) &lt;br /&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;General Wisdoms&lt;/h4&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;Next we have some advice that most people should already be aware of, but yet bears repeating.&lt;/div&gt;&lt;h3&gt;&lt;br /&gt;&lt;/h3&gt;&lt;h4&gt;Don&#39;t Rely on Obscurity&lt;/h4&gt;&lt;br /&gt;Its an old adage that &quot;security by obscurity is no security at all&quot;. &amp;nbsp;Yet we often see naive engineers trying to harden systems by making them more obscure. &amp;nbsp;This really doesn&#39;t help anything long term, and can actually hinder security efforts by giving a false sense of security or creating barriers to security analysis.&lt;br /&gt;&lt;h4&gt;&lt;br /&gt;&lt;/h4&gt;&lt;h4&gt;Audit&lt;/h4&gt;&lt;br /&gt;Get an independent security expert to audit your work. &amp;nbsp;Special focus should be paid to the items pointed out above. &amp;nbsp;This should include a review of the product, as well as your internal practices around engineering, including secure coding, use of mitigation technologies, and business practices for dealing with keying material, code signing, and other sensitive data.&lt;br /&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=4956875839204442298' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='Security Advice to IoT Firmware Engineers'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>0</thr:total></entry><entry><id>,</id><published>2016-05-14T09:24:00.000-07:00</published><updated>2016-05-14T09:24:35.297-07:00</updated><title type='text'>Microsoft Hates My Name (Not Me, Just My Name)</title><content type='html'>In order to debug &lt;a href=&quot;;&gt;nanomsg&lt;/a&gt; problems on Windows, I recently installed a copy of Windows 8.1 in a VMWare guest VM, along with Visual Studio 14 and CMake 3.5.2. &amp;nbsp;(Yes, I&#39;ve entered a special plane of Hell, reserved for just for people who try to maintain cross-platform open source software. &amp;nbsp;I think this one might be the tenth plane, that Dante skipped because it was just too damned horrible.)&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;Every time I tried to build, I got bizarre errors from the CMake / build process ... like this:&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;       &lt;blockquote class=&quot;tr_bq&quot;&gt;&lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;Cannot evaluate the item metadata &quot;%(FullPath)&lt;/span&gt;&lt;/blockquote&gt;&lt;div class=&quot;p1&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;Turns out that when I created my account, using the &quot;easy&quot; installation in VMWare, it created my Windows account using my full name. &amp;nbsp;&quot;&lt;b&gt;Garrett D&#39;Amore&lt;/b&gt;&quot;. &amp;nbsp;Turns out that the software is buggy, and can&#39;t cope with the apostrophe in my full name, when it appears in a filesystem path.&amp;nbsp;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;Moving the project directory to C:\Projects\nanomsg solved the problem.&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;Really Microsoft? &amp;nbsp;This is 2016. &amp;nbsp;I expected programs to struggle and for me to find bugs in programs (often root exploits &amp;nbsp;-- all hackers should try using punctuation in their login and personal names) with the apostrophe in my name back in the 1990s. &amp;nbsp;Not in this decade.&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;Not only that, but the error message was so incredibly cryptic that it took a Google search to figure out that it was a problem with the path. &amp;nbsp;(Other people encountered this problem with paths &amp;gt; 260 characters. &amp;nbsp;I knew that wasn&#39;t my problem, but I hypothesized, and proved, that it was my name.) &amp;nbsp;I have no idea how to file a bug on Visual Studio to Microsoft. &amp;nbsp;I&#39;m not a paying user of it, so maybe I shouldn&#39;t complain, and I really have no recourse. &amp;nbsp;Still, they need to fix this.&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;Normally, I&#39;d never intentionally create a path with an apostrophe in it, but in this case I was being lazy and just accepted some defaults. &amp;nbsp;I staunchly refuse to change my&amp;nbsp;&lt;i&gt;name&lt;/i&gt;&amp;nbsp;because some software is too stupid to cope with it -- this is a pet peeve for me.&amp;nbsp;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;We&#39;re in the new millennium, and have been for a decade and half. &amp;nbsp;Large numbers of folks with heritage from countries like Italy, France, and Ireland have this character in their surname. &amp;nbsp;(And more recently -- since like the 1960s! -- the African-American community has been using this character in their &lt;b&gt;first&lt;/b&gt;&amp;nbsp;names too!) &amp;nbsp;If your software can&#39;t accommodate this common character in names, then it&#39;s broken, and you need to fix it. &amp;nbsp;There are literally millions of us that are angered by this sort of brokenness every day; do us all a favor and make your software just a little less rage inducing by letting us use the names we were born with please.&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=4945004197183723437' title='2 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='Microsoft Hates My Name (Not Me, Just My Name)'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>2</thr:total></entry><entry><id>,</id><published>2016-02-23T12:03:00.000-08:00</published><updated>2016-02-23T12:03:13.379-08:00</updated><title type='text'>Leaving github</title><content type='html'>&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;(Brief reminder that this represents my own personal opinion, not necessarily that of any employer or larger open source project.)&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;&lt;br /&gt;&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;I am planning to move my personal git repositories (including &lt;a href=&quot;;&gt;mangos&lt;/a&gt;, &lt;a href=&quot;;&gt;tcell&lt;/a&gt;, &lt;a href=&quot;;&gt;govisor&lt;/a&gt;, &lt;a href=&quot;;&gt;less-fork&lt;/a&gt;, etc.) &amp;nbsp;from &lt;a href=&quot;;&gt;GitHub&lt;/a&gt; to &lt;a href=&quot;;&gt;;/a&gt;.&amp;nbsp;&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&lt;span class=&quot;s1&quot;&gt;&lt;/span&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;The reasons for this are fairly simple. &amp;nbsp;They have nothing to do whatsoever with technology. &amp;nbsp;I love the GitHub platform, and have been a happy user of it for years now. I would dearly love it if I could proceed with GitHub. &amp;nbsp;Fortunately GitLab seems to have &lt;a href=&quot;;&gt;feature parity&lt;/a&gt; with GitHub (and a growing user and project base), so I&#39;m not trapped.&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;The reason for leaving GitHub is because of the&amp;nbsp;&lt;a href=&quot;;&gt;hostility&lt;/a&gt;&amp;nbsp;of it&#39;s&amp;nbsp;leadership towards certain classes of people makes me feel that I cannot in good conscience continue to support them. In particular, their HR department is &lt;a href=&quot;;&gt;engaging&lt;/a&gt; in what is nothing less than race&amp;nbsp;&lt;a href=&quot;;&gt;warfare&lt;/a&gt; against white people. &amp;nbsp;(Especially men, but even white women are being discriminated against.) By the way, I&#39;d take the same position if the hostility were instead towards any other racial or gender group other than my own.&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;I&#39;m not alone in asking GitHub to fix this; yet they&#39;ve remained silent on the matter, leading me to believe that the problematic policies have support within the highest levels of the company. &amp;nbsp;(Github itself is &lt;a href=&quot;;&gt;in trouble&lt;/a&gt;, and I have doubts about its &lt;a href=&quot;;&gt;future&lt;/a&gt;, as both &lt;a href=&quot;;&gt;developers&lt;/a&gt; and employees are leaving in droves.)&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;&lt;br /&gt;&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;Post &lt;a href=&quot;;&gt;Tom Preston-Werner&lt;/a&gt;, GitHub&#39;s leadership apparently sees the company as a platform for prosecuting the Social Justice War, and it even has a &lt;a href=&quot;;&gt;Social Impact Team&lt;/a&gt; just to that effect.&amp;nbsp;&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;In GitHub&#39;s own words:&lt;/span&gt;&lt;/div&gt;&lt;blockquote class=&quot;tr_bq&quot;&gt;&lt;span class=&quot;s1&quot;&gt;&quot;T&lt;/span&gt;he Social Impact team will be focused on these three areas: - Diversity &amp;amp; Inclusion - both internally and within the Open Source Community - Community Engagement - we have a net positive impact in local and online communities via partnerships - Leveraging GitHub for Positive Impact - supporting people from varied communities to use in innovative ways&quot;&lt;/blockquote&gt;&lt;div class=&quot;p2&quot;&gt;&lt;span class=&quot;s1&quot;&gt;&lt;/span&gt;It&#39;s no accident that they list &quot;Diversity &amp;amp; Inclusion&quot; as the first item here either. &amp;nbsp;Apparently this has been&amp;nbsp;&lt;a href=&quot;;&gt;more of a priority for GitHub than improving their platform&lt;/a&gt; or addressing long standing customer issues.&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;Those of you who have followed me know that I’m strongly in favor of &lt;i&gt;&lt;b&gt;inclusion&lt;/b&gt;&lt;/i&gt;, and making an environment friendly for &lt;i&gt;&lt;b&gt;all&lt;/b&gt;&lt;/i&gt; people, regardless of race or gender or religion (provided your religion respects my basic rights -- religious fundamentalist nut-jobs need not apply).&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;Lack of diversity cannot be fixed through exclusion. &amp;nbsp;Attempts to do so are inherently misguided. &amp;nbsp;Furthermore, as a company engages in &lt;i&gt;any exclusive hiring practices&lt;/i&gt;&amp;nbsp;they are inherently limiting their own access to talent. &amp;nbsp;Racist or sexist (or ageist) approaches are self-destructive, and companies that engage in such behavior deserve to fail.&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;The way to fix an un-level playing field is to level the playing field -- not to swing it back in the other direction. &amp;nbsp;You can&#39;t fix social injustice with more injustice; we should guarantee equal opportunity not equal results.&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;There are plenty of people of diverse ethnic backgrounds who have overcome significant social and economic barriers to achieve success. &amp;nbsp;And many who have not. &amp;nbsp;News flash -- you will find white men and women in both lists, as well as blacks, latinos, women, gays, and people of &quot;other gender identification&quot;. &amp;nbsp;Any hiring approach or policy (written or otherwise) that only looks at the color of a person&#39;s skin or gender is unfair, and probably illegal outside of a very limited few and specific instances (e.g. casting for movie roles).&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&amp;nbsp;Note that this does not mean that I do not support efforts to reach out to encourage people from other groups to engage more in technology (or any other field). &amp;nbsp;As I said, I encourage efforts to include &lt;i&gt;everyone&lt;/i&gt;&amp;nbsp;-- the larger talent pool that we can engage with, the more successful we are likely to be. &amp;nbsp;And we should do everything we can as a society and as an industry to make sure that the talent pool is as big as we can make it.&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;We should neither exclude any future&amp;nbsp;&lt;a href=&quot;;&gt;Marie Curie&lt;/a&gt; or &lt;a href=&quot;;&gt;Daniel Hale Williams&lt;/a&gt; from achieving the highest levels of success, nor should we exclude a future&amp;nbsp;&lt;a href=&quot;;&gt;Isaac Newton&lt;/a&gt;&amp;nbsp;just because of his race or gender.&amp;nbsp; The best way to avoid that, is to be inclusive of everyone, and make sure that everyone has the best opportunities to achieve success possible.&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;Sadly I will probably be labeled racist or sexist, or some other -ist, because I&#39;m not supportive of the divisive agendas supported by people like&amp;nbsp;&lt;a href=&quot;;&gt;Nicole Sanchez&lt;/a&gt;&amp;nbsp;and Danilo Libre, and because I am a heterosexual white middle class male (hence automatically an entitled enemy in their eyes.) &amp;nbsp;It seems that they would rather have me as an enemy rather than a friendly supporter -- at least that is what their actions demonstrate. &amp;nbsp;It&#39;s certainly easier to apply an -ist label than to engage in rationale dialogue.&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;&lt;br /&gt;&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;I am however deeply supportive of efforts to reach out to underrepresented groups in early stages. &amp;nbsp;Show more girls, blacks, and latinos filling the role of technophiles in popular culture (movies and shows) that market towards children. &amp;nbsp;Spend money (wisely!) to improve education in poorer school districts. &amp;nbsp;Teach kids that they truly can be successful regardless of color or gender, and make sure that they have the tools (including access to technology) to &lt;i&gt;achieve success&amp;nbsp;based on merit, not because of their grouping.&lt;/i&gt;&amp;nbsp; These efforts have to be made at the primary and secondary school levels, where inspiration can have the biggest effects. &amp;nbsp;(By the way, these lessons apply equally well to white boys; teaching children to respect one another as individuals rather than as labels is a good thing, in all directions.)&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;&lt;br /&gt;&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;span class=&quot;s1&quot;&gt;By the time someone in is choosing a college or sitting in front of a recruiter, it&#39;s far too late (and far too expensive). &amp;nbsp;The only tools that can be applied at later stages are only punitive in nature, and therefore the only reasonable thing to do at this late stage is to punish unjust behaviors (i.e. zero tolerance towards bigotry, harassment, and so forth.)&lt;/span&gt;&lt;/div&gt;&lt;div class=&quot;p1&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;I&#39;ll have more detail as to the moves of the specific repos over the coming days.&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;PS: &amp;nbsp;GitLab does&amp;nbsp;&lt;a href=&quot;;&gt;support&lt;/a&gt;&amp;nbsp;diversity as well, which is a good thing, but they do it without engaging in the social justice war, or exclusive policies.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;p2&quot;&gt;&lt;br /&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=8365365563163650994' title='2 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='Leaving github'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>2</thr:total></entry><entry><id>,</id><published>2016-01-06T10:43:00.000-08:00</published><updated>2016-01-06T10:43:45.918-08:00</updated><title type='text'>Stepping Down</title><content type='html'>(Quick reminder that this blog represents my own opinion, and not necessarily that of any open source project or employer.)&lt;br /&gt;&lt;br /&gt;For nearly a year, I&#39;ve been primary maintainer of &lt;a href=&quot;;&gt;nanomsg&lt;/a&gt;, a library of common lightweight messaging patterns written in C.&lt;br /&gt;&lt;br /&gt;I was &lt;a href=&quot;,1&quot;&gt;given&lt;/a&gt; this mantle when I &lt;a href=&quot;;&gt;asked&lt;/a&gt; for the nanomsg community to take some action to get forward progress on some changes I had to fix some core bugs, one of which was a protocol bug. &amp;nbsp;(I am also the creator of &lt;a href=&quot;;&gt;mangos&lt;/a&gt;, a wire-compatible library supporting the same patterns written in Go, which is why I came to care about fixing nanomsg.)&lt;br /&gt;&lt;br /&gt;Today, I am stepping down as maintainer.&lt;br /&gt;&lt;br /&gt;There are several reasons for this, but the most relevant right now is my frustration with this community, and its response to what I believed to be a benign &lt;a href=&quot;;&gt;proposal&lt;/a&gt;, that to adopt a &lt;a href=&quot;;&gt;Code of Conduct&lt;/a&gt;, in an attempt to make the project more inviting to a broader audience.&lt;br /&gt;&lt;br /&gt;I was unprepared for the backlash.&lt;br /&gt;&lt;br /&gt;And frankly, I haven&#39;t got enough love of the project to want to continue to lead it, when its clearly unwilling to codify what are frankly some sound and reasonable communication practices.&lt;br /&gt;&lt;br /&gt;As maintainer, I could have just enforced my will upon the project, but since the project existed before I came to it, that doesn&#39;t feel right. &amp;nbsp;So instead, I&#39;m just stepping down.&lt;br /&gt;&lt;br /&gt;I&#39;m not sure who will succeed me. &amp;nbsp;I can nominate a party, but at this point there are several other parties with git commit privileges to the project; I think they should nominate one. &amp;nbsp;Martin (the founder) still has administrative privileges as well.&lt;br /&gt;&lt;br /&gt;&lt;i&gt;To be clear, I think both sides of the Code of Conduct are &lt;b&gt;wrong&lt;/b&gt; -- a bunch of whinny kids really.&amp;nbsp;&lt;/i&gt;&lt;br /&gt;&lt;i&gt;&lt;br /&gt;&lt;/i&gt;On the one side, we have people who seem to feel that the existence of a document means something.&lt;br /&gt;&lt;br /&gt;I think that&#39;s a stupid view; it may have meaning when you have larger democratic projects and you need therefore written rules to justify actions -- and in that case a Code of Conduct is really a way to justify punishing someone, rather than prevention or education. &amp;nbsp;To those of you who think you need such a document in order to participate in a project -- I think you&#39;re acting like a bunch of spineless wimps.&lt;br /&gt;&lt;br /&gt;This isn&#39;t to say you should have to put up with abuse or toxic conduct. &amp;nbsp;But if you think a document creates a &quot;safe space&quot;, you&#39;re smoking something funny. &amp;nbsp;Instead, look at the actual conduct of the project, and the actions of leadership. &amp;nbsp;A paper Code of Conduct isn&#39;t going to fix brokenness, and I have my doubts that it can prevent brokenness from occurring in the first place.&lt;br /&gt;&lt;br /&gt;If the leadership needs a CoC to correct toxic behavior, then the leadership of the project is busted. &amp;nbsp;Strong leadership leads by example, and takes the appropriate action to ensure that the communities that they lead are pleasant places to be. &amp;nbsp; (That&#39;s not necessarily the same as being conflict-free; much technical goodness comes about as a consequence of heartfelt debate, and developers can be just as passionate about the things they care about as anyone else. &amp;nbsp;Keeping the tone of such debate on topic and non-personal and professional is one of the signs of good leadership.)&lt;br /&gt;&lt;br /&gt;On the other side, are those who rail against such a document. &amp;nbsp;Are you so afraid of your own speech that you don&#39;t think you can agree to a document that basically says we are to treat each other respectfully? &amp;nbsp;The word I use for such people is &quot;chickenshit&quot;. &amp;nbsp; If you can&#39;t or won&#39;t agree to be respectful towards others in the open source projects I lead, then I don&#39;t want your involvement.&lt;br /&gt;&lt;br /&gt;There&#39;s no doubt that there exists real abuse and intolerance in open source communities, and those who would cast aspersions on someone because of race, religion, physical attribute, or gender (or preference), are themselves slime, who really only underscore for everyone else their own ignorance and stupidity. &amp;nbsp;I have no tolerance for such bigotry, and I don&#39;t think anyone else should either.&lt;br /&gt;&lt;br /&gt;Don&#39;t misunderstand me; I&#39;m not advocating for CoCs. &amp;nbsp;I think they are nearly worthless, and I resent the movement that demands that every project adopt one. &amp;nbsp;But I equally resent the strenuous opposition to their existence. &amp;nbsp;If a CoC does no good, it seems to me that it does no harm either. &amp;nbsp;So even if it is just a placebo effect, if it can avoid conflict and make a project more widely acceptable, then its worth having one, precisely because the cost of doing so is so low.&lt;br /&gt;&lt;br /&gt;Yes, this is &quot;slacktivism&quot;.&lt;br /&gt;&lt;br /&gt;I&#39;ve been taught that actions speak louder than words though.&lt;br /&gt;&lt;br /&gt;So today I&#39;m stepping down.&lt;br /&gt;&lt;br /&gt;I&#39;m retaining my BDFL of mangos, of course, so I&#39;ll still be around the nanomsg community, but I will be giving it far less of my energy.&lt;br /&gt;&lt;br /&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=3668235664676989390' title='8 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='Stepping Down'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>8</thr:total></entry><entry><id>,</id><published>2015-12-11T10:46:00.002-08:00</published><updated>2015-12-11T10:52:21.420-08:00</updated><title type='text'>What Microsoft Can Do to Make Me Hate Windows a Little Less</title><content type='html'>Those who know me know that I have little love for Microsoft Windows. &amp;nbsp;The platform is a special snowflake, and coming from a Unix background (real &lt;a href=&quot;;&gt;UNIX&lt;/a&gt;, not Linux, btw), every time I&#39;m faced with Windows I feel like I&#39;m in some alternate dimension where everything is a little strange and painful.&lt;br /&gt;&lt;br /&gt;I have to deal with Windows because of applications. &amp;nbsp;My wife runs Quickbooks (which is one of the more chaotic and poorly designed bits of software I&#39;ve run across), the kids have video games they like. &amp;nbsp;I&#39;ve had to run it myself historically because some expense report site back at former employer AMD was only compatible with IE. &amp;nbsp;I also have a flight simulator for RC aircraft that only works in Windows (better to practice on the sim, no glue needed when you crash, just hit the reset button.)&lt;br /&gt;&lt;br /&gt;All of those are merely annoyances, and I keep Windows around on one of my computers for this reason. &amp;nbsp;It&#39;s not one I use primarily, nor one I carry with me when I travel.&lt;br /&gt;&lt;br /&gt;But I also have created and support software that runs on Windows, or that people want to use on Windows. &amp;nbsp;Software like &lt;a href=&quot;;&gt;nanomsg&lt;/a&gt;, &lt;a href=&quot;;&gt;mangos&lt;/a&gt;, &lt;a href=&quot;;&gt;tcell&lt;/a&gt;, etc. &amp;nbsp;This is stuff that supports other developers. &amp;nbsp;Its free and open software, and I make no money from any of it.&lt;br /&gt;&lt;br /&gt;Supporting that software is a pain on Windows, largely due to the fact that I don&#39;t have a Windows license to run Windows in a VM. &amp;nbsp;The only reason I&#39;d buy such a license for my development laptop would be to support my free software development efforts. &amp;nbsp;Which would actually help and benefit the Windows ecosystem.&lt;br /&gt;&lt;br /&gt;I rely on &lt;a href=&quot;;&gt;AppVeyor&lt;/a&gt; (which is an excellent service btw) to help me overcome my lack of a Windows instance on my development system. &amp;nbsp;This has allowed me to support some things pretty well, but the lack of an interactive command line means that some experiments are nigh impossible for me to try; others make me wait for the CI to build and test this, which takes a while. &amp;nbsp;Leading to lost time during the development cycle, all of which make me loathe working on the platform even more.&lt;br /&gt;&lt;br /&gt;Microsoft can fix this. &amp;nbsp;In their latest &quot;incarnation&quot;, they are claiming to be open source friendly, and they&#39;ve even made big strides here in supporting open source developers. &amp;nbsp;&lt;a href=&quot;;&gt;Visual Studio is free&lt;/a&gt; (as in beer). &amp;nbsp;Their &lt;a href=&quot;;&gt;latest code editor&lt;/a&gt; is even open source. &amp;nbsp;The .Net framework itself is &lt;a href=&quot;;&gt;open source&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;But the biggest barrier is the license for the platform itself. &amp;nbsp;I&#39;m simply not going to run Windows on the bare metal -- I&#39;m a Mac/UNIX guy and that is not going to change. &amp;nbsp;But I can and would be happier to occasionally run Windows to better support that platform in a VM, just like I do for &lt;a href=&quot;;&gt;illumos&lt;/a&gt; or Linux or FreeBSD.&lt;br /&gt;&lt;br /&gt;So, Microsoft, here&#39;s your chance to make me hate your platform a little less. &amp;nbsp;Give open source developers access to free Windows licenses; to avoid cannibalizing your business you could have license terms that only allow these free licenses to be used when Windows is run in a virtual machine for non-commercial purposes. &amp;nbsp;This is a small thing you could do, to extend your reach to a set of developers who&#39;ve mostly abandoned you.&lt;br /&gt;&lt;br /&gt;(And Apple, there&#39;s a similar lesson there for you. &amp;nbsp;I&#39;m a devoted MacOS X fan, but imagine how much wider your developer audience could be if you let people run MacOS X in a VM for non-commercial use?)&lt;br /&gt;&lt;br /&gt;In the meantime, if you use software I develop, please don&#39;t be surprised if you find that I treat Windows as a distinctly second class citizen. &amp;nbsp;After all, its no worse than how Microsoft has treated me as an open source developer.</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=2789136103532913749' title='4 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='What Microsoft Can Do to Make Me Hate Windows a Little Less'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>4</thr:total></entry><entry><id>,</id><published>2015-12-08T16:41:00.000-08:00</published><updated>2015-12-08T16:42:33.526-08:00</updated><title type='text'>On Misunderstandings</title><content type='html'>Yesterday there was a flurry of activity on Twitter, and in retrospect, it seems that some have come away with interpretations of what I said that are other than what I intended. &amp;nbsp;Some of that misunderstanding is pretty unfortunate, so I&#39;d like to set the record straight on a couple of items now.&lt;br /&gt;&lt;br /&gt;First off, let me begin by saying that this blog, and my Twitter account, are mine alone, and are used by me to express my opinions. &amp;nbsp;They represent neither &lt;a href=&quot;;&gt;illumos&lt;/a&gt; nor &lt;a href=&quot;;&gt;Lucera&lt;/a&gt;, nor anyone or anything else.&lt;br /&gt;&lt;br /&gt;Second, I have to apologize for it seems that I&#39;ve come across as somehow advocating either against diversity (whether in the community or in the workplace) or in favor of toxicity.&lt;br /&gt;&lt;br /&gt;Nothing could be further from the truth. &amp;nbsp;I believe strongly in diversity and an inclusive environment, both for illumos, and in the work place. &amp;nbsp;I &lt;a href=&quot;;amp;;amp;list=PLH8r-Scm3-2VmZhZ76tFPAhPOG0pvmjdA#t=13m30s&quot;&gt;talked about this at illumos day last year&lt;/a&gt;&amp;nbsp;(see about 13:30 into the video, slides &lt;a href=&quot;;&gt;here&lt;/a&gt;), and I&#39;ve also put my &lt;a href=&quot;;&gt;money where my mouth&lt;/a&gt; is. &amp;nbsp;Clearly, it hasn&#39;t been enough, and I think we all can and should do better. &amp;nbsp;I&#39;m interested in finding ways to increase the diversity in illumos in particular, and the industry in general. &amp;nbsp;Feel free to post your suggestions in the comments following this blog.&lt;br /&gt;&lt;br /&gt;Additionally, no, I don&#39;t believe that anyone should have to put up with &quot;high performing toxic people&quot;. &amp;nbsp;The illumos community has appropriately censured people for toxic behavior in the past, and I was supportive of that action back then, and still am now. &amp;nbsp;Maintaining a comfortable work place and a comfortable community leads to increased personal satisfaction, and that leads to increased productivity. &amp;nbsp;Toxicity drives people away, and that flies in the face of the aforementioned desire for diversity (as well as the unstated ones for a growing and a healthy community.)&lt;br /&gt;&lt;br /&gt;Finally, I didn&#39;t mean to offend anyone. &amp;nbsp;If I&#39;ve done so in my recent tweets, please be assured that this was not intentional, and I hope you&#39;ll accept my heartfelt apology.&lt;br /&gt;&lt;br /&gt;Thanks.</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=1493228101611806802' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='On Misunderstandings'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>0</thr:total></entry><entry><id>,</id><published>2015-10-19T04:29:00.001-07:00</published><updated>2015-10-19T04:29:36.762-07:00</updated><title type='text'>A Space Shooter in Curses</title><content type='html'>Some of you who follow me may know that I have recently built a pretty nifty framework for working with terminals. &amp;nbsp;ANSI, ASCII, VT100, Windows Console, etc. &amp;nbsp;Its called &lt;a href=&quot;;&gt;Tcell&lt;/a&gt;, and located on github. &amp;nbsp;(Its a Go framework though.) &amp;nbsp;It offers many of the same features as curses, though it is most definitely not a clone of curses.&lt;br /&gt;&lt;br /&gt;Anyway, I decided it should be possible to write a game in this framework, so I wrote one.&lt;br /&gt;&lt;br /&gt;I give you &lt;a href=&quot;;&gt;Escape From Proxima 5&lt;/a&gt;, a 2D multi-axis scrolling space shooter written entirely in Go, designed to operate in your text terminal&lt;br /&gt;&lt;br /&gt;The game is fairly primordial, but there is a playable level complete with enemies and hazards. &amp;nbsp; It&#39;s actually reasonably difficult to get past just this first level.&lt;br /&gt;&lt;br /&gt;Mostly the idea here is that you can get a sense of what the game engine is capable of, and see Tcell in action.&lt;br /&gt;&lt;br /&gt;As part of this, I wrote a pretty complete 2D game engine. &amp;nbsp;Its got rich sprite management with collision detection, palettes, an events subsystem, scrolling maps, and support for keyboards and mice. &amp;nbsp;Its also got pretty nice extensibility as assets are defined in YAML files that are converted and compiled into the program. &amp;nbsp;(I guess an asset editor needs to be written. :-)&lt;br /&gt;&lt;br /&gt;The code is &lt;a href=&quot;;&gt;Apache 2 licensed&lt;/a&gt;, so feel free to borrow bits for your own projects. &amp;nbsp;I&#39;d love to hear about it.&lt;br /&gt;&lt;br /&gt;Anyway, I thought I&#39;d post this here. &amp;nbsp;I made two videos. &amp;nbsp;The longer one, at about 3:30, shows most of the features of the game, animated sprites, some nice explosions, gravity effects, beam field effects, etc.&lt;br /&gt;&lt;br /&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;iframe width=&quot;320&quot; height=&quot;266&quot; class=&quot;YOUTUBE-iframe-video&quot; data-thumbnail-src=&quot;; src=&quot;; frameborder=&quot;0&quot; allowfullscreen&gt;&lt;/iframe&gt;&lt;/div&gt;&lt;br /&gt;&lt;br /&gt;The second video shows what this looks like on less rich terminals -- say a VT100 with only 7-bit ASCII characters available. &amp;nbsp;The richer your locale, the nicer it will look. &amp;nbsp;But it falls down as gracefully as one can expect.&lt;br /&gt;&lt;br /&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;iframe width=&quot;320&quot; height=&quot;266&quot; class=&quot;YOUTUBE-iframe-video&quot; data-thumbnail-src=&quot;; src=&quot;; frameborder=&quot;0&quot; allowfullscreen&gt;&lt;/iframe&gt;&lt;/div&gt;&lt;br /&gt;Btw, this framework is now basically design complete, so it should be super easy to product a lot of simples kinds of games -- for example a clone of Missile Command or Space Invaders should be doable in an afternoon. &amp;nbsp; What makes this game a little bigger is the number if different kinds of objects and object interactions we can have.</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=6042188200741696921' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='A Space Shooter in Curses'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><media:thumbnail xmlns:media="" url="" height="72" width="72"/><thr:total>0</thr:total></entry><entry><id>,</id><published>2015-10-05T00:02:00.001-07:00</published><updated>2015-10-05T00:02:07.569-07:00</updated><title type='text'>Fun with terminals, character sets, Unicode, and Go</title><content type='html'>As part of my recent work on &lt;a href=&quot;;&gt;Tcell&lt;/a&gt;, I&#39;ve recently added some pretty cool functionality for folks who want to have applications that can work reasonably in many different locales.&lt;br /&gt;&lt;br /&gt;For example, if your terminal is running in UTF-8, you can access a &lt;i&gt;huge&lt;/i&gt; repertoire of glyphs / characters.&lt;br /&gt;&lt;br /&gt;But if you&#39;re running in a non-UTF-8 terminal, such as an older ISO 8859-1 (Latin1) or KOI8-R (Russian) locale, you might have problems. &amp;nbsp;Your terminal won&#39;t be able to display UTF-8, and your key strokes will probably be reported to the application using some 8-bit variant that is incompatible with UTF-8. &amp;nbsp;(For ASCII characters, everything will work, but if you want to enter a different character, like Я (Russian for &quot;ya&quot;), you&#39;re going to have difficulties.&lt;br /&gt;&lt;br /&gt;If you work on the console of your operating system, you probably have somewhere around 220 characters to play with. &amp;nbsp;You&#39;re going to miss some of those glyphs.&lt;br /&gt;&lt;br /&gt;&lt;a href=&quot;;&gt;Go&lt;/a&gt; of course works with UTF-8 natively. &amp;nbsp;Which is just awesome.&lt;br /&gt;&lt;br /&gt;Until you have to work in one of these legacy environments. &amp;nbsp; And some of the environments are not precisely &quot;legacy&quot;. &amp;nbsp;(&lt;a href=&quot;;&gt;GB18030&lt;/a&gt; has the same repertoire as UTF-8, but uses a different encoding scheme and is legally mandatory within China.)&lt;br /&gt;&lt;br /&gt;If you use Tcell for your application&#39;s user interface, this is now &quot;fixed&quot;.&lt;br /&gt;&lt;br /&gt;Tcell will attempt to convert to characters that the user&#39;s terminal understands on output, provided the user&#39;s environment variables are set properly ($LANG, $LC_ALL, $LC_CTYPE, per POSIX). &amp;nbsp;It will also convert the user&#39;s key strokes from your native locale to UTF-8. &amp;nbsp;This means that YOU, the application developer, can just worry about UTF-8, and skip the rest. &amp;nbsp;(Unless you want to add new Encodings, which is entirely possible.)&lt;br /&gt;&lt;br /&gt;Tcell even goes further.&lt;br /&gt;&lt;br /&gt;It will use the alternate character set (ACS) to convert Unicode drawing characters to the characters supported by the terminal, if they exist -- or to reasonable ASCII fallbacks if they don&#39;t. &amp;nbsp;(Just like &lt;a href=&quot;;&gt;ncurses&lt;/a&gt;!)&lt;br /&gt;&lt;br /&gt;It will also cope with both East Asian full-width (or ambiguous width) characters, and even properly handles combining characters. &amp;nbsp;(If your terminal supports it, these are rendered properly on the terminal. &amp;nbsp;If it doesn&#39;t, Tcell makes a concerted effort to make a best attempt at rendering -- preserving layout and presenting the primary character even if the combining character cannot be rendered.)&lt;br /&gt;&lt;br /&gt;The Unicode (and non-Unicode translation) handling capabilities in Tcell far exceed any other terminal handling package I&#39;m aware of.&lt;br /&gt;&lt;br /&gt;Here are some interesting screen caps, taken on a Mac using the provided unicode.go test program.&lt;br /&gt;&lt;br /&gt;First the UTF-8. &amp;nbsp;Note the Arabic, the correct spacing of the Chinese glyphs, and the correct rendering of Combining characters. &amp;nbsp;(Note also that emoji are reported as width one, instead of two, and so take up more space than they should. &amp;nbsp;This is a font bug on my system -- Unicode says these are Narrow characters.)&lt;br /&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;a href=&quot;; imageanchor=&quot;1&quot; style=&quot;margin-left: 1em; margin-right: 1em;&quot;&gt;&lt;img border=&quot;0&quot; height=&quot;208&quot; src=&quot;; width=&quot;320&quot; /&gt;&lt;/a&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;Then we run in ISO8859-1 (Latin 1). &amp;nbsp;Here you can see the accented character available in the Icelandic word, and some terminal specific replacements have been made for the drawing glyphs. &amp;nbsp;ISO 8859-1 lacks most of the unusual or Asian glyphs, and so those are rendered as &quot;?&quot;. &amp;nbsp;This is done by Tcell -- the terminal &lt;i&gt;never sees the raw Unicode/UTF-8&lt;/i&gt;. &amp;nbsp;That&#39;s important, since sending the raw UTF-8 could cause my terminal to do bad things.&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;Note also that the widths are properly handled, so that even though we cannot display the combining characters, nor the full-width Chinese characters, the widths are correct -- 1 cell is taken for the combining character combinations, and 2 cells are taken by the full width Chinese characters.&lt;/div&gt;&lt;br /&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;a href=&quot;; imageanchor=&quot;1&quot; style=&quot;margin-left: 1em; margin-right: 1em;&quot;&gt;&lt;img border=&quot;0&quot; height=&quot;208&quot; src=&quot;; width=&quot;320&quot; /&gt;&lt;/a&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;Then we show off legacy Russian (KOI8-R): &amp;nbsp; Here you can see Cyrillic is rendered properly, as well as the basic ASCII and the alternate (ACS) drawing characters (mostly), while the rest are filled with place holder ?&#39;s.&lt;/div&gt;&lt;br /&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;a href=&quot;; imageanchor=&quot;1&quot; style=&quot;margin-left: 1em; margin-right: 1em;&quot;&gt;&lt;img border=&quot;0&quot; height=&quot;208&quot; src=&quot;; width=&quot;320&quot; /&gt;&lt;/a&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;And, for those of you in mainland China, here&#39;s GB18030: &amp;nbsp;Its somewhat amusing that the system font seems to not be able to cope with the combining enclosure here. &amp;nbsp;Again, this is a font deficiency in the system.&lt;/div&gt;&lt;br /&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;a href=&quot;; imageanchor=&quot;1&quot; style=&quot;margin-left: 1em; margin-right: 1em;&quot;&gt;&lt;img border=&quot;0&quot; height=&quot;208&quot; src=&quot;; width=&quot;320&quot; /&gt;&lt;/a&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;As you can see, we have a lot of rendering options. &amp;nbsp;Input is filtered and converted too. &amp;nbsp;Unfortunately, the mouse test program I use to verify this doesn&#39;t really show this (since you can&#39;t see what I typed), but the Right Thing happens on input too.&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;Btw, those of you looking for mouse usage in your terminal should be very very happy with Tcell. &amp;nbsp;As far as I can tell, Tcell offers improved mouse handling on stock XTerm over every other terminal package. &amp;nbsp;This includes live mouse reporting, click-drag reporting, etc. &amp;nbsp; Here&#39;s what the test program looks like on my system, after I&#39;ve click-dragged to create a few boxes:&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;a href=&quot;; imageanchor=&quot;1&quot; style=&quot;margin-left: 1em; margin-right: 1em;&quot;&gt;&lt;img border=&quot;0&quot; height=&quot;306&quot; src=&quot;; width=&quot;320&quot; /&gt;&lt;/a&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;I&#39;m super tempted to put all this together to write an old DOS-style game. &amp;nbsp;I think Tcell has everything necessary here to be used as the basis for some really cool terminal hacks.&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;Give it a whirl if you like, and let me know what you think.&lt;/div&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: left;&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;br /&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=2568373107269807122' title='1 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='Fun with terminals, character sets, Unicode, and Go'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><media:thumbnail xmlns:media="" url="" height="72" width="72"/><thr:total>1</thr:total></entry><entry><id>,</id><published>2015-10-02T22:25:00.000-07:00</published><updated>2015-10-02T22:25:06.667-07:00</updated><title type='text'>Tcell  - Terminal functionality for Pure Go apps</title><content type='html'>Introducing &lt;a href=&quot;;&gt;Tcell&lt;/a&gt; &amp;nbsp;- Terminal functionality for Pure Go apps&lt;br /&gt;&lt;br /&gt;As part of the work I&#39;ve done on &lt;a href=&quot;;&gt;govisor&lt;/a&gt;, I had a desire for rich terminal functionality so that I could build a portable curses-style management application.&lt;br /&gt;&lt;br /&gt;This turned out to be both easier and harder than I thought.&lt;br /&gt;&lt;br /&gt;Easier, because there was an implementation to start from --&amp;nbsp;&lt;a href=&quot;;&gt;termbox-go&lt;/a&gt;, but harder because that version wasn&#39;t portable to the &lt;a href=&quot;;&gt;OS I cared most about&lt;/a&gt;, and fell short in some ways that I felt were important. &amp;nbsp;(Specifically, reduced functionality for mice, colors, and Unicode.)&lt;br /&gt;&lt;br /&gt;This led me to create my own library; I&#39;ve made some very very different design choices. &amp;nbsp;These design choices have let me both support more platforms (pretty much all POSIX platforms and Windows are included), increase support for a much richer set of terminals and terminal capabilities, etc.&lt;br /&gt;&lt;br /&gt;The work is called &quot;&lt;a href=&quot;;&gt;Tcell&lt;/a&gt;&quot; (for terminal cells, which is the unit we operate on -- if you don&#39;t like the name ... well I&#39;m open to suggestions. &amp;nbsp;I freely admit that I suck at naming -- and its widely &lt;a href=&quot;;&gt;acknowledged&lt;/a&gt; that naming is one of the &quot;hard&quot; problems in computer science.)&lt;br /&gt;&lt;br /&gt;As part of this work, I&#39;ve implemented a full Terminfo string parser/expander in Go. &amp;nbsp;This isn&#39;t quite as trivial as you might think, since the parameterized strings actually have a kind of stack based &quot;&lt;a href=&quot;;&gt;minilanguage&lt;/a&gt;&quot;, including conditionals, in them.&lt;br /&gt;&lt;br /&gt;Furthermore, I&#39;ve wound up exploring lots more about mice, and terminals, and the most common emulators. &amp;nbsp;As a result, I think I have created a framework that can support very rich mouse reporting, key events, colors, and Unicode. &amp;nbsp;Of course, the functionality will vary somewhat by platform and terminal (your &lt;a href=&quot;;&gt;vt52&lt;/a&gt; isn&#39;t going to support rich colors, Unicode, or even a mouse, after all. &amp;nbsp;But Xterm can, as can most modern emulators.)&lt;br /&gt;&lt;br /&gt;This work would be a nice basis for portable readline implementations, getpassphrase, etc. &amp;nbsp;Or a full up curses implementation. &amp;nbsp;It does support a compatibility mode for termbox, and most termbox applications work with only changing the import line. &amp;nbsp; I still need to change govisor, and the topsl panels library, to use tcell, but part of that work will be including much more capable mouse reporting, which tcell now facilitates.&lt;br /&gt;&lt;br /&gt;Admittedly, if you cannot tolerate cgo in any of your work -- this isn&#39;t for you -- yet. &amp;nbsp;I will probably start adding non-cgo implementations for particular ports such as Linux, Darwin, and FreeBSD. &amp;nbsp;But for other platforms there will always be cgo as a fallback. &amp;nbsp;Sadly, unless the standard Go library evolves to add sufficient richness that all of the termios functionality are available natively on all platforms (which is a real possibility), cgo will remain necessary. &amp;nbsp;(Note that Windows platform support does &lt;i&gt;not&lt;/i&gt; use CGO.)</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=9177932589493626125' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='Tcell  - Terminal functionality for Pure Go apps'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>0</thr:total></entry><entry><id>,</id><published>2015-09-22T12:05:00.001-07:00</published><updated>2015-09-22T12:05:49.293-07:00</updated><title type='text'>On Go, Portability, and System Interfaces</title><content type='html'>I&#39;ve been noticing more and more lately that we have a plethora of libraries and programs written for Go, which don&#39;t work on one platform or another. &amp;nbsp;The root cause of these is often the use of direct system call coding to system calls such as ioctl(). &amp;nbsp;On some platforms (illumos/solaris!) there is no such system call.&lt;br /&gt;&lt;br /&gt;&lt;h3&gt;The Problems&lt;/h3&gt;&lt;br /&gt;But this underscores a far far worse problem, that has become common (mal)-practice in the &lt;a href=&quot;;&gt;Go&lt;/a&gt; community. &amp;nbsp;That is, the coding of system calls directly into high level libraries and even application programs. &amp;nbsp;For example, it isn&#39;t uncommon to see something like this (taken from &lt;a href=&quot;;&gt;termbox-go&lt;/a&gt;):&lt;br /&gt;&lt;blockquote class=&quot;tr_bq&quot;&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&lt;span style=&quot;color: #cc0000;&quot;&gt;func&lt;/span&gt; tcsetattr(fd uintptr, termios *syscall_Termios) &lt;span style=&quot;color: #cc0000;&quot;&gt;error&lt;/span&gt; {&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; r, _, e := syscall.Syscall(syscall.SYS_IOCTL,&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; fd, uintptr(syscall_TCSETS), uintptr(unsafe.Pointer(termios)))&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; if r != 0 {&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &lt;span style=&quot;color: #cc0000;&quot;&gt;return&lt;/span&gt; os.NewSyscallError(&quot;&lt;span style=&quot;color: blue;&quot;&gt;SYS_IOCTL&lt;/span&gt;&quot;, e)&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; }&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &lt;span style=&quot;color: #cc0000;&quot;&gt;return nil&lt;/span&gt;&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;}&lt;/span&gt;&lt;/blockquote&gt;&lt;br /&gt;This has quite a few problems with it.&lt;br /&gt;&lt;ol&gt;&lt;li&gt;It&#39;s not platform portable. &amp;nbsp;This function depends on a specific implementation of &lt;a href=&quot;;&gt;tcsetattr&lt;/a&gt;() that is done in terms of specific ioctl()s. &amp;nbsp;For example, TCSETS may be used on one platform, but on others TIOCSETA might be used.&lt;/li&gt;&lt;li&gt;It&#39;s not Go portable, since SYS_IOCTL isn&#39;t implemented on platforms like &lt;a href=&quot;;&gt;illumos&lt;/a&gt;, even though as a &lt;a href=&quot;;&gt;POSIX&lt;/a&gt; system we do have a working tcsetattr().&amp;nbsp;&lt;/li&gt;&lt;li&gt;The code is actually pretty unreadable, and somewhat challenging to write the first time correctly.&lt;/li&gt;&lt;li&gt;The code uses unsafe.Pointer(), which is clearly something we ought to avoid.&lt;/li&gt;&lt;li&gt;On some platforms, the details of the ioctls are subject to change, so that the coding above is actually fragile. &amp;nbsp;(In illumos &amp;amp; Solaris system call interfaces are &quot;undocumented&quot;, and one must use the C library to access system services. &amp;nbsp;This is our &quot;stable API boundary&quot;. &amp;nbsp;This is somewhat different from Linux practice; the reasons for this difference is both historical and related to the fact that Linux delivers only a kernel, while illumos delivers a system that includes both the kernel and core libraries.)&lt;/li&gt;&lt;/ol&gt;&lt;b&gt;&lt;i&gt;How did we wind up in this ugly situation?&lt;/i&gt;&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;The problem I believe stems from some misconceptions, and some historical precedents in the Go community. &amp;nbsp; First the Go community has long touted static linking as one of its significant advantages. &amp;nbsp;However, I believe this has been taken too far.&lt;br /&gt;&lt;br /&gt;Why is static linking beneficial? &amp;nbsp;The obvious (to me at any rate) reason is to avoid the dependency nightmares and breakage that occurs with other systems where many dynamic libraries are brought together. &amp;nbsp;For example, if A depends directly on both B and C, and B depends on C, but some future version of B depends on a newer version of C that is incompatible with the version of C that A was using, then we cannot update A to use the new B. &amp;nbsp;And when the system components are shared across the entire system, the web of dependencies gets to be so challenging that managing these dependencies in real environments can become a full time job, consuming an entire engineering salary.&lt;br /&gt;&lt;br /&gt;You can get into surprising results where upgrading one library can cause unexpected failures in some other application. &amp;nbsp;So the desire to avoid this kind of breakage is to encode the entire binary together, in a single stand-alone executable, so that we need never have a fear as to whether our application will work in the future or not. &amp;nbsp;As I will show, we&#39;ve not really achieved this with 100% statically linked executables in Go, though I&#39;ll grant that we have greatly reduced the risk.&lt;br /&gt;&lt;br /&gt;This is truly necessary because much of the open source ecosystem has no idea about interface stability nor versioning interfaces. &amp;nbsp;This is gradually changing, such that we now have ideas like semver coming around as if they are somehow new and great ideas. &amp;nbsp;The reality is that commercial operating system vendors have understood the importance of stable API boundaries for a very very long time. &amp;nbsp;Some, like Sun, even made legally binding promises around the stability of their interfaces. &amp;nbsp;However, in each of these cases, the boundary has to a greater or lesser extent been at the discretion of the vendor.&lt;br /&gt;&lt;br /&gt;Until we consider standards such as POSIX 1003.1. &amp;nbsp;Some mistakenly believe that POSIX defines system calls. &amp;nbsp;It does not. &amp;nbsp;It defines a C function call interface. &amp;nbsp;The expectation is that many of these interfaces have 1:1 mappings with system calls, but the details of those system calls are completely unspecified by POSIX.&lt;br /&gt;&lt;br /&gt;Basically, the Go folks want to minimize external dependencies and the web of failure that can lead to. &amp;nbsp;Fixing that is a goal I heartily agree with. &amp;nbsp;However, we cannot eliminate our dependency on the platform. &amp;nbsp;And using system calls directly is actually worse, because it moves our dependency from something that is stable and defined by standards bodies, to an interface that is undocumented, not portable, and may change at any time.&lt;br /&gt;&lt;br /&gt;If you&#39;re not willing to have a dynamic link dependency on the C library, why would you be willing to have a dependency on the operating system kernel? &amp;nbsp;In fact, the former is far safer than the latter! (And on Solaris, you don&#39;t have a choice -- the Go compiler always links against the system libraries.)&lt;br /&gt;&lt;br /&gt;&lt;b&gt;&lt;i&gt;Harmful results that occur with static linking&lt;/i&gt;&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;If the application depends on a library that has a critical security update, it becomes necessary to recompile the application. &amp;nbsp;If you have a low level library such as a TLS or HTTP client, and a security fix for a TLS bug is necessary (&lt;i&gt;and we&#39;ve never ever ever had any bugs in TLS or SSL implementation, right?&lt;/i&gt;), this could mean recompiling a very large body of software to be sure you&#39;ve closed the gaps.&lt;br /&gt;With statically linked programs, even knowing which applications need to be updated can be difficult or impossible. &amp;nbsp;They defy the most easy kinds of inspection, using tools like ldd or otool to see what they are built on top of.&lt;br /&gt;&lt;br /&gt;What is also tragic, is that static executables wind up encoding the details of the kernel system call interface into the binary. &amp;nbsp;On some systems this isn&#39;t a big deal because they have a stable system call interface. &amp;nbsp;(Linux mostly has this -- although glibc still has to cope with quite a few differences here by handling ENOSYS, and don&#39;t even get me started on systemd related changes.) &amp;nbsp;But on systems like Solaris and illumos, we&#39;ve historically considered those details a private implementation detail between libc and kernel. &amp;nbsp;And to prevent applications from abusing this, we don&#39;t even deliver a static libc. &amp;nbsp;This gives us the freedom to change the kernel/userland interface fairly freely, without affecting applications.&lt;br /&gt;&lt;br /&gt;When you consider standards specifications like POSIX or X/OPEN, this approach makes a lot of sense. &amp;nbsp;They standardize the C function call interface, and leave the kernel implementation up to the implementor.&lt;br /&gt;&lt;br /&gt;But statically linked Go programs break this, badly. &amp;nbsp;If that kernel interface changes, we can wind up breaking all of the Go programs that use it, although &quot;correct&quot; programs that only use libc will continue to work fine.&lt;br /&gt;&lt;br /&gt;&lt;b&gt;&lt;i&gt;The elephant in the room (licensing)&lt;/i&gt;&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;The other problem with static linking is that it can create a license condition that is very undesirable. &amp;nbsp;For example, glibc is LGPL. &amp;nbsp;That means that per the terms of the LGPL it must be possible to relink against a different glibc, if you link statically.&lt;br /&gt;&lt;br /&gt;Go programs avoid this by not including any of the C library statically. Even when cgo is used, the system libraries are linked dynamically. &amp;nbsp;(This is usually the C library, but can include things like a pthreads library or other base system libraries.)&lt;br /&gt;&lt;br /&gt;In terms of the system, the primary practice for Go programmers has been to use licenses like MIT, BSD, or Apache, that are permissive enough that static linking of 3rd party Go libraries is usually not a problem. &amp;nbsp;I suppose that this is a great benefit in that it will serve to help prevent GPL and LGPL code from infecting the bulk of the corpus of Go software.&lt;br /&gt;&lt;br /&gt;&lt;h3&gt;The Solutions&lt;/h3&gt;&lt;br /&gt;The solution here is rather straightforward.&lt;br /&gt;&lt;br /&gt;First, we should not eschew use of the C library, or other libraries that are part of the standard system image. &amp;nbsp;I&#39;m talking about things like libm, libc, and for those that have them, libpthread, libnsl, libsocket. &amp;nbsp;Basically the standard libraries that every non-trivial program has to include. &amp;nbsp;On most platforms this is just libc. &amp;nbsp;If recoded to use the system&#39;s tcsetattr (which is defined to exist by POSIX), the above function looks like this:&lt;br /&gt;&lt;blockquote class=&quot;tr_bq&quot;&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&lt;span style=&quot;color: #6aa84f;&quot;&gt;// include &amp;lt;termios.h&amp;gt;&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;color: #cc0000;&quot;&gt;import&lt;/span&gt; &quot;&lt;span style=&quot;color: blue;&quot;&gt;C&lt;/span&gt;&quot;&lt;br /&gt;&lt;span style=&quot;color: #cc0000;&quot;&gt;import&lt;/span&gt; &quot;&lt;span style=&quot;color: blue;&quot;&gt;os&lt;/span&gt;&quot;&lt;/span&gt;&amp;nbsp;&lt;/blockquote&gt;&lt;blockquote class=&quot;tr_bq&quot;&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&lt;span style=&quot;color: #cc0000;&quot;&gt;func&lt;/span&gt; tcsetattr(f *os.File, termios *C.struct_termios) &lt;span style=&quot;color: #cc0000;&quot;&gt;error&lt;/span&gt; {&lt;br /&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; _, e := C.tcsetattr(, C.TCSANOW, termios)&lt;br /&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &lt;span style=&quot;color: #cc0000;&quot;&gt;return&lt;/span&gt; e&lt;br /&gt;}&lt;/span&gt;&lt;/blockquote&gt;The above implementation will cause your library or program to dynamically link against and use the standard C library on the platform. &amp;nbsp;And it works on all POSIX systems everywhere and because it uses a stable documented standard API, it is pretty much immune to breakage from changes elsewhere in the system. &amp;nbsp;(At least any change that broke this implementation would also break so many other things that the platform would be unusable. &amp;nbsp;Generally we can usually trust people who make the operating system kernel and C library to not screw things up too badly.)&lt;br /&gt;&lt;br /&gt;What would be even better, and cleaner, would be to abstract that interface above behind some Go code, converting between a Go struct and the C struct as needed, just as is done in much of the rest of the Go runtime. &amp;nbsp;The logical place to do this would be in the standard Go system libraries. &amp;nbsp;I&#39;d argue rather strongly that core services like termio handling ought to be made available to Go developers in the standard system libraries that are part of Go, or perhaps more appropriately, with the &lt;a href=&quot;;&gt;;/a&gt; repository.&lt;br /&gt;&lt;br /&gt;In any event, if you&#39;re a Go programmer, please consider NOT directly calling syscall interfaces, but instead using higher level interfaces, and when those aren&#39;t already provided in Go, don&#39;t be afraid to use cgo to access standard functions in the C library. &amp;nbsp;Its far far better for everyone that you do this, than that you code to low level system calls.&lt;br /&gt;&lt;br /&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=4853466830148344524' title='5 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='On Go, Portability, and System Interfaces'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>5</thr:total></entry><entry><id>,</id><published>2015-09-20T09:12:00.000-07:00</published><updated>2015-09-20T09:12:30.638-07:00</updated><title type='text'>Announcing govisor 1.0</title><content type='html'>I&#39;m happy to announce that I feel I&#39;ve wrapped up &lt;a href=&quot;;&gt;Govisor&lt;/a&gt; to a point where its ready for public consumption.&lt;br /&gt;&lt;br /&gt;Govisor is a service similar to &lt;a href=&quot;;&gt;supervisord&lt;/a&gt;, in that it can be used to manage a bunch of processes. &amp;nbsp;However, it is much richer in that it understands process dependencies, conflicts, and also offers capabilities for self-healing, and consolidated log management.&lt;br /&gt;&lt;br /&gt;It runs as an ordinary user process, and while it has some things in common with programs like init, upstart, and Solaris SMF, it is not a replacement for any of those things. &amp;nbsp;Instead think of this is a portable way to manage a group of processes without requiring root. &amp;nbsp;In my case I wanted something that could manage a tree of microservices that was deployable by normal users. &amp;nbsp;Govisor is my answer to that problem.&lt;br /&gt;&lt;br /&gt;Govisor is also written entirely in Go, and is embeddable in other projects. &amp;nbsp;The REST server interface uses a stock http.ServeHTTP interface, so it can be used with various middleware or frameworks like the Gorilla toolkit.&lt;br /&gt;&lt;br /&gt;Services can be implemented as processes, or in native Go.&lt;br /&gt;&lt;br /&gt;Govisor also comes with a nice terminal oriented user interface (I&#39;d welcome a JavaScript based UI, but I don&#39;t write JS myself). &amp;nbsp;Obligatory screen shots below.&lt;br /&gt;&lt;br /&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;a href=&quot;; imageanchor=&quot;1&quot; style=&quot;margin-left: 1em; margin-right: 1em;&quot;&gt;&lt;img border=&quot;0&quot; height=&quot;238&quot; src=&quot;; width=&quot;320&quot; /&gt;&lt;/a&gt;&lt;/div&gt;&lt;br /&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;a href=&quot;; imageanchor=&quot;1&quot; style=&quot;margin-left: 1em; margin-right: 1em;&quot;&gt;&lt;img border=&quot;0&quot; height=&quot;238&quot; src=&quot;; width=&quot;320&quot; /&gt;&lt;/a&gt;&lt;/div&gt;&lt;br /&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;a href=&quot;; imageanchor=&quot;1&quot; style=&quot;margin-left: 1em; margin-right: 1em;&quot;&gt;&lt;img border=&quot;0&quot; height=&quot;238&quot; src=&quot;; width=&quot;320&quot; /&gt;&lt;/a&gt;&lt;/div&gt;&lt;br /&gt;Which actually brings up the topic of &quot;&lt;a href=&quot;;&gt;tops&#39;l&lt;/a&gt;&quot; &amp;nbsp;(which is a contraction of top-sail, often used in sailing). &amp;nbsp;Govisor depends the package tops -- &quot;terminal oriented panels support library&quot;), which provides a mid-level API for interacting with terminals. &amp;nbsp; I created topsl specifically to help me with creating the govisor client application. &amp;nbsp;I do consider topsl ready for use by Govisor, but I advise caution if you want to use it your own programs -- its really young and there are probably breaking changes looming in its future.&lt;br /&gt;&lt;br /&gt;The documentation is a bit thin on the ground, but if you want to help or have questions, just let me know! &amp;nbsp;In the meantime, enjoy!</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=1719934006778171557' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='Announcing govisor 1.0'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><media:thumbnail xmlns:media="" url="" height="72" width="72"/><thr:total>0</thr:total></entry><entry><id>,</id><published>2015-05-02T17:35:00.001-07:00</published><updated>2015-05-02T17:35:09.966-07:00</updated><title type='text'>MacOS X 10.10.3 Update is *TOXIC*</title><content type='html'>As a PSA (public service announcement), I&#39;m reporting here that updating your Yosemite system to 10.10.3 is incredibly toxic if you use WiFi.&lt;br /&gt;&lt;br /&gt;I&#39;ve seen other reports of this, and I&#39;ve experienced it myself. &amp;nbsp;What happened is that the update for 10.10.3 seems to have done something tragically bad to the WiFi drivers, such that it completely hammers the network to the point of making it unusable for everyone else on the network.&lt;br /&gt;&lt;br /&gt;I have late 2013 iMac 27&quot;, and after I updated, I found that other systems started badly badly misbehaving. &amp;nbsp;I blamed my ISP, and the router, because I was seeing ping times of tens of &lt;i&gt;seconds&lt;/i&gt;!&lt;br /&gt;(No, not milliseconds, &lt;i&gt;seconds!!!&lt;/i&gt;&amp;nbsp; In one case I saw responses over 64 seconds.) &amp;nbsp;This was on other systems that were not upgraded. &amp;nbsp;Needless to say, that basically left the network unusable.&lt;br /&gt;&lt;br /&gt;(The behavior was cyclical -- I&#39;d get a few tens of seconds where pings to would be in the 20 msec range, and then it would start to jump up very quickly until maxing up around a minute or so. &amp;nbsp;It would stay there for a minute or two, then rest or drop back to sane times. &amp;nbsp; But only very briefly.)&lt;br /&gt;&lt;br /&gt;This was most severe when using a 5GHz network. &amp;nbsp;Switching down to 2.4GHz reduced some of the symptoms -- although still over 10 seconds to get traffic through and thoroughly unusable for a wide variety of applications.&lt;br /&gt;&lt;br /&gt;There are reports that disabling Bluetooth may alleviate this, and also some people reported some success with clearing certain settings files. &amp;nbsp;I&#39;ve not tried either of these yet. &amp;nbsp;Google around for the answer if you want to. &amp;nbsp;For now, my iMac 27&quot; is powered off, until I can take the chance to disrupt the network again to try these &quot;fixes&quot;.&lt;br /&gt;&lt;br /&gt;Apple, I&#39;m seriously seriously disappointed here. &amp;nbsp;I&#39;m not sure at all how this testing got past you, but you need to fix this. &amp;nbsp;Its absolutely criminal that applying a recommended update with security critical fixes in it should turn my computer into a DoS device for my local network. &amp;nbsp;I&#39;m shocked that several days later I&#39;ve not seen a release update from Apple to fix this critical problem.&lt;br /&gt;&lt;br /&gt;Anyway, my advice is, if possible, hold off for the update to 10.10.3. &amp;nbsp;Its tragically, horribly toxic not just to the upgraded device, but probably to the entire network it sits on. &amp;nbsp;I&#39;m a little astounded that a bug in the code could hose an entire WiFi network as badly as this does -- I would have previously thought this impossible (and this was part of the reason why it took a while to diagnose this down to the computer -- I thought the ridiculous ping responses had to be a problem with my upstream provider!)&lt;br /&gt;&lt;br /&gt;I&#39;ll post an update here if one becomes available.</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=5822226322037061865' title='31 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='MacOS X 10.10.3 Update is *TOXIC*'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>31</thr:total></entry><entry><id>,</id><published>2015-04-14T11:13:00.002-07:00</published><updated>2015-04-14T11:13:18.947-07:00</updated><title type='text'>vtprint - blast from the past</title><content type='html'>I recently decided to have a look back at some &lt;i&gt;old&lt;/i&gt;&amp;nbsp;stuff I wrote. &amp;nbsp;Blast from the past stuff. &amp;nbsp;One of the things I decided to look at was the very first open source program I wrote -- something called &lt;b&gt;&lt;i&gt;&lt;a href=&quot;;&gt;vtprint&lt;/a&gt;&lt;/i&gt;&lt;/b&gt;.&lt;br /&gt;&lt;br /&gt;&lt;i&gt;&lt;b&gt;vtprint&lt;/b&gt;&lt;/i&gt; is a tool that borrowed ideas stolen from the pine mail program. &amp;nbsp;This comes from the days of serial dialups, and legacy (&lt;a href=&quot;;&gt;Kermit&lt;/a&gt;&amp;nbsp;or &lt;a href=&quot;;&gt;ProComm&lt;/a&gt; for those who remember such things) serial connectivity over 14.4K modems. &amp;nbsp;Printing a file on a remote server was hard back then; you could transfer the file using &lt;a href=&quot;;&gt;xmodem&lt;/a&gt; using rx(1), or its &quot;newer&quot; variants, rb(1) or rz(1), but this was awkward. &amp;nbsp;It turns out that most physical terminals had support for an escape sequence that would start hardcopy to a locally attached printer, and another that would stop it. &amp;nbsp;And indeed, many terminal emulators have the same support. &amp;nbsp;(I added support for this myself to the &lt;a href=&quot;;&gt;rxvt&lt;/a&gt;(1) terminal emulator back in the mid 90&#39;s, so any emulators derived from rxvt inherit this support as well.)&lt;br /&gt;&lt;br /&gt;So vtprint, which in retrospect could have been written in a few lines of shell code, is a C program. &amp;nbsp;It supports configuration via a custom file called &quot;vtprintcap&quot;, that can provide information about different $TERM values and the escape sequences they use. &amp;nbsp;So for example, TVI925 terminals use a different escape sequence than VT100 or VT220 terminals.&lt;br /&gt;&lt;br /&gt;The first version of it, 1.0, is lost in time, and was written back in 1991 or 1992 while I was working as a &quot;Computer Consultant&quot; (fancy name for &lt;a href=&quot;;&gt;BOFH&lt;/a&gt;, though I also did a lot of programming for the school&#39;s Windows 3.1 network and Novell NetWare 3.12 server) at the &lt;a href=&quot;;&gt;Haas School of Business at UC Berkeley&lt;/a&gt;. &amp;nbsp;(I was a Chemical Engineering student there, and I flunked out in thermodynamics -- I still blame &lt;a href=&quot;;&gt;Gibbs Free Energy&lt;/a&gt;&amp;nbsp;ΔG as a concept I never truly could get my mind around. &amp;nbsp;It took a couple of years before I finally gave up fighting &quot;hard engineering&quot; and accepted my true calling as a software engineer.)&lt;br /&gt;&lt;br /&gt;Anyway, I transferred to &lt;a href=&quot;;&gt;SDSU&lt;/a&gt; into the Computer Science department. &amp;nbsp;While there, back in 1993, I updated vtprint significantly, and that is the release that lives on at SourceForge.&lt;br /&gt;&lt;br /&gt;Today I imported that project to &lt;a href=&quot;;&gt;github&lt;/a&gt;. &amp;nbsp;(And let me tell you, it took some hoops to do this. &amp;nbsp;Apparently all the old CVS projects that were going to have converted to CVS are supposed to already have done so, because at this point even the conversion tools are mostly crufty and broken. &amp;nbsp;Maybe I&#39;ll post what I did there.) &amp;nbsp;Most of the files indicate an age of 11 years ago. &amp;nbsp;That&#39;s because 11 years ago I imported the source into CVS for the benefit of &lt;a href=&quot;;&gt;SourceForge&lt;/a&gt;. &amp;nbsp;The actual files date back to 1994, and you can even see my old email address -- which hasn&#39;t worked for a couple of decades, in the files.&lt;br /&gt;&lt;br /&gt;So, I gave vtprint a whirl today for the first time in years:&lt;br /&gt;&lt;br /&gt;&lt;blockquote class=&quot;tr_bq&quot;&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;garrett@hipster{4}&amp;gt; ./vtprint README&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&lt;br /&gt;&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;*** vtprint (v2.0.2) ***&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&lt;br /&gt;&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;Copyright 1993-1994, Garrett D&#39;Amore&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;Last revised October 25, 1994.&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&lt;br /&gt;&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;NO WARRANTY! &amp;nbsp;Use &quot;vtprint -w&quot; for info.&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;Freely redistributable. &amp;nbsp;Use &quot;vtprint -l&quot; for info.&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;&lt;br /&gt;&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;vtprint: Can&#39;t open /usr/local/lib/vtprint/vtprintcap, using builtin codes.&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;vtprint: Using &amp;lt;stdout&amp;gt; for output.&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;vtprint: Output flags: formfeed&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;vtprint: Printed README.&lt;/span&gt;&lt;span style=&quot;font-family: Verdana, sans-serif; font-size: x-small;&quot;&gt;vtprint: Successfully printed 1 file (1 specified).&lt;/span&gt;&lt;/blockquote&gt;&lt;br /&gt;Sadly, MacOS X does not appear to emulate the escape sequences properly. &amp;nbsp;But &lt;a href=&quot;;&gt;iTerm2&lt;/a&gt; does so very nicely. &amp;nbsp;It even generates a nice print preview using the standard MacOS X print dialog. &amp;nbsp;Sadly, it does not pass through PostScript unmolested, but for printing ASCII files it works beautifully. &lt;br /&gt;&lt;div class=&quot;separator&quot; style=&quot;clear: both; text-align: center;&quot;&gt;&lt;a href=&quot;; imageanchor=&quot;1&quot; style=&quot;margin-left: 1em; margin-right: 1em;&quot;&gt;&lt;img border=&quot;0&quot; src=&quot;; height=&quot;320&quot; width=&quot;278&quot; /&gt;&lt;/a&gt;&lt;/div&gt;&lt;br /&gt;I think that I&#39;m going to start using vtprint again, when I need to print a file located on a virtual machine, or over a connection that only offers SSH (such as a secured access machine). &amp;nbsp;With firewalled networks, I suspect that this program has new usefulness. &amp;nbsp;I&#39;m also switching from (as many people have suggested) to iTerm2.&lt;br /&gt;&lt;br /&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=8778211508689408995' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='vtprint - blast from the past'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><media:thumbnail xmlns:media="" url="" height="72" width="72"/><thr:total>0</thr:total></entry><entry><id>,</id><published>2015-02-22T11:48:00.000-08:00</published><updated>2015-02-22T13:07:37.116-08:00</updated><title type='text'>IPv6 and IPv4 name resolution with Go</title><content type='html'>As part of a work-related project, I&#39;m writing code that needs to resolve DNS names using &lt;a href=&quot;;&gt;Go&lt;/a&gt;, on &lt;a href=&quot;;&gt;illumos&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;While doing this work, I noticed a very surprising thing. &amp;nbsp;When a host has both IPv6 and IPv4 addresses associated with a name (such as &lt;i&gt;localhost&lt;/i&gt;), Go prefers to resolve to the IPv4 version of the name, unless one has asked specifically for v6 names.&lt;br /&gt;&lt;br /&gt;This flies in the fact of existing practice on illumos &amp;amp; Solaris systems, where resolving a name tends to give an IPv6 result, assuming that any IPv6 address is plumbed on the system. &amp;nbsp;(And on modern installations, that is the default -- at least the loopback interface of ::1 is always plumbed by default. &amp;nbsp;And not only that, but services listening on that address will automatically serve up both v6 and v4 clients that connect on either ::1 or;br /&gt;&lt;br /&gt;The rationale for this logic is buried in the Go &lt;a href=&quot;;&gt;&lt;span style=&quot;font-family: inherit;&quot;&gt;net/ipsock.go&lt;/span&gt;&lt;/a&gt; file, in comments for the function &lt;span style=&quot;font-family: inherit;&quot;&gt;&lt;a href=&quot;;&gt;firstFavoriteAddr&lt;/a&gt;&amp;nbsp;()&lt;/span&gt;:&lt;br /&gt;&lt;pre style=&quot;background-color: #e9e9e9; border-bottom-left-radius: 5px; border-bottom-right-radius: 5px; border-top-left-radius: 5px; border-top-right-radius: 5px; color: #222222; line-height: 18px; margin: 20px; padding: 10px;&quot;&gt;&lt;span style=&quot;font-family: Verdana, sans-serif;&quot;&gt;&lt;span class=&quot;ln&quot; id=&quot;L76&quot; style=&quot;color: #999999;&quot;&gt;&lt;span style=&quot;font-size: 14px;&quot;&gt; &lt;/span&gt;&lt;span style=&quot;font-size: x-small;&quot;&gt;   76&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;font-size: x-small;&quot;&gt; &lt;span class=&quot;comment&quot; style=&quot;color: #006600;&quot;&gt;// We&#39;ll take any IP address, but since the dialing&lt;/span&gt;&lt;br /&gt;&lt;span class=&quot;ln&quot; id=&quot;L77&quot; style=&quot;color: #999999;&quot;&gt;    77&lt;/span&gt; &lt;span class=&quot;comment&quot; style=&quot;color: #006600;&quot;&gt;// code does not yet try multiple addresses&lt;/span&gt;&lt;br /&gt;&lt;span class=&quot;ln&quot; id=&quot;L78&quot; style=&quot;color: #999999;&quot;&gt;    78&lt;/span&gt; &lt;span class=&quot;comment&quot; style=&quot;color: #006600;&quot;&gt;// effectively, prefer to use an IPv4 address if&lt;/span&gt;&lt;br /&gt;&lt;span class=&quot;ln&quot; id=&quot;L79&quot; style=&quot;color: #999999;&quot;&gt;    79&lt;/span&gt; &lt;span class=&quot;comment&quot; style=&quot;color: #006600;&quot;&gt;// possible. This is especially relevant if localhost&lt;/span&gt;&lt;br /&gt;&lt;span class=&quot;ln&quot; id=&quot;L80&quot; style=&quot;color: #999999;&quot;&gt;    80&lt;/span&gt; &lt;span class=&quot;comment&quot; style=&quot;color: #006600;&quot;&gt;// resolves to [ipv6-localhost, ipv4-localhost]. Too&lt;/span&gt;&lt;br /&gt;&lt;span class=&quot;ln&quot; id=&quot;L81&quot; style=&quot;color: #999999;&quot;&gt;    81&lt;/span&gt; &lt;span class=&quot;comment&quot; style=&quot;color: #006600;&quot;&gt;// much code assumes localhost == ipv4-localhost.&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/pre&gt;This is a really surprising result. &amp;nbsp;If you want to get IPv6 names by default, with Go, you could use the &lt;span style=&quot;font-family: inherit;&quot;&gt;&lt;a href=&quot;;&gt;net.ResolveIPAddr&lt;/a&gt;() (or &lt;a href=&quot;;&gt;ResolveTCPAddr&lt;/a&gt;() or &lt;a href=&quot;;&gt;ResolveUDPAddr&lt;/a&gt;())&lt;/span&gt; functions with the network type of &quot;ip6&quot;, &quot;tcp6&quot;, or &quot;udp6&quot; first. &amp;nbsp;Then if that resolution fails, you can try the standard versions, or the v4 specific versions (doing the latter is probably slightly more efficient.) &amp;nbsp;Here&#39;s what that code looks like:&lt;br /&gt;&lt;pre style=&quot;background-color: #e9e9e9; border-bottom-left-radius: 5px; border-bottom-right-radius: 5px; border-top-left-radius: 5px; border-top-right-radius: 5px; line-height: 18px; margin: 20px; padding: 10px;&quot;&gt;&lt;span style=&quot;font-family: Verdana, sans-serif;&quot;&gt;&lt;span style=&quot;font-size: x-small;&quot;&gt;&lt;span style=&quot;color: #222222;&quot;&gt;        name := &quot;&lt;/span&gt;&lt;span style=&quot;color: #6fa8dc;&quot;&gt;localhost&lt;/span&gt;&lt;span style=&quot;color: #222222;&quot;&gt;&quot;&lt;br /&gt;&lt;br /&gt;        &lt;/span&gt;&lt;span style=&quot;color: #38761d;&quot;&gt;&lt;span class=&quot;comment&quot;&gt;// First we try IPv6. &amp;nbsp;Note that we *hope* this fails if the host&lt;/span&gt;&lt;br /&gt;        &lt;span class=&quot;comment&quot;&gt;// stack does not actually support IPv6.&lt;/span&gt;&lt;br /&gt;&lt;/span&gt;&lt;span style=&quot;color: #222222;&quot;&gt;        err, ip := net.ResolveIP(&quot;&lt;/span&gt;&lt;span style=&quot;color: #6fa8dc;&quot;&gt;ip6&lt;/span&gt;&lt;span style=&quot;color: #222222;&quot;&gt;&quot;, name)&lt;br /&gt;        if err != nil {&lt;br /&gt;                &lt;/span&gt;&lt;span class=&quot;comment&quot; style=&quot;color: #38761d;&quot;&gt;// IPv6 not found, maybe IPv4?&lt;/span&gt;&lt;span style=&quot;color: #222222;&quot;&gt;&lt;br /&gt;                err, ip = net.ResolveIP(&quot;&lt;/span&gt;&lt;span style=&quot;color: #6fa8dc;&quot;&gt;ip4&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;color: #222222;&quot;&gt;&lt;span style=&quot;font-size: x-small;&quot;&gt;&quot;, name)&lt;br /&gt;        }&lt;/span&gt;&lt;span style=&quot;font-size: xx-small;&quot;&gt;&lt;br /&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/pre&gt;However, this choice turns out to also be &lt;i&gt;evil&lt;/i&gt;, because while ::1 often works locally as an IPv6 address and is functional, other addresses, for example, will resolve to IPv6 addresses which will &lt;i&gt;not&lt;/i&gt;&amp;nbsp;work unless you have a clean IPv6 path all the way through. &amp;nbsp;For example, the above gives me this for;nbsp;2607:f8b0:4007:804::1010, but if I try to telnet to it, it won&#39;t work -- no route to host (of course, because I don&#39;t have an IPv6 path to the Internet, both my home gateway and my ISP are IPv4 only.)&lt;br /&gt;&lt;br /&gt;&lt;br /&gt;Its kind of a sad that the Go people felt that they had to make this choice -- at some level it robs the choice from the administrator, and encourages the existing broken code to remain so. &amp;nbsp;I&#39;m not sure what the other systems use, but at least on illumos, we have a stack that understands the situation, and resolves &lt;i&gt;optimally&lt;/i&gt;&amp;nbsp;for the given the situation of the user. &amp;nbsp;Sadly, Go shoves that intelligence aside, and uses its own overrides.&lt;br /&gt;&lt;br /&gt;One moral of the story here is -- always use either explicit v4 or v6 addresses if you care, or ensure that your names resolve properly.</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=3253875900774655656' title='1 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='IPv6 and IPv4 name resolution with Go'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>1</thr:total></entry><entry><id>,</id><published>2015-02-11T21:35:00.001-08:00</published><updated>2015-02-11T21:35:58.661-08:00</updated><title type='text'>Rise of mangos</title><content type='html'>&lt;h2&gt;&lt;i&gt;What is mangos?&lt;/i&gt;&lt;/h2&gt;&lt;br /&gt;Those of you who follow me may have heard about this project I&#39;ve created called &lt;a href=&quot;;&gt;mangos&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;Mangos is a lightweight messaging system designed to be wire compatible with &lt;a href=&quot;;&gt;nanomsg&lt;/a&gt;, but is implemented entirely in &lt;a href=&quot;;&gt;Go&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;There is a very nice &lt;a href=&quot;;&gt;write up&lt;/a&gt; of mangos by Tyler Treat, which might help explain some things.&lt;br /&gt;&lt;br /&gt;&lt;h3&gt;&lt;i&gt;Recent Activity&lt;/i&gt;&lt;/h3&gt;&lt;div&gt;&lt;i&gt;&lt;br /&gt;&lt;/i&gt;&lt;/div&gt;As a consequence of a few things, the last two weeks has seen a substantial rise of use of mangos. &lt;br /&gt;&lt;br /&gt;First off, there was Tyler&#39;s excellent article. &amp;nbsp;(By the way, he&#39;s done a series comparing and contrasting other messaging systems -- highly recommended reading.)&lt;br /&gt;&lt;br /&gt;Second, mangos got mentioned on &lt;a href=&quot;;&gt;Hacker News&lt;/a&gt;. &amp;nbsp;That drew a large number visitors to my github repo.&lt;br /&gt;&lt;br /&gt;Then another open source project, &lt;a href=&quot;;&gt;Goq&lt;/a&gt;, switched from using libnanomsg to mangos, using the compatibility shim I provided for such use. &amp;nbsp;As a consequence of that work, several bugs were identified, and subsequently squashed.&lt;br /&gt;&lt;br /&gt;The upshot of all that is that I saw the number unique visitors sky rocket. &amp;nbsp;On Saturday Feb 7, there were over &lt;a href=&quot;;&gt;2500 unique visitors&lt;/a&gt; to the github page, and 29 unique people took clones. &amp;nbsp;Sunday it tapered sharply to just over 1k visitors, and today there were only 7. &amp;nbsp;Peaks rarely get sharper than that.&lt;br /&gt;&lt;br /&gt;&lt;h3&gt;&lt;i&gt;Improvements&lt;/i&gt;&lt;/h3&gt;&lt;br /&gt;Over the past week or so I&#39;ve made a large number of changes and improvements. &amp;nbsp;Recently, mangos has grown support for &lt;a href=&quot;;&gt;RFC 6455&lt;/a&gt; (websocket), including websocket over &lt;a href=&quot;;&gt;TLS&lt;/a&gt;, and has had numerous internal improvements.&lt;br /&gt;&lt;br /&gt;Some of these changes have broken API. &amp;nbsp;If you use mangos, I&#39;m sorry about the breakage -- please let me know if you&#39;re hurt by this. &amp;nbsp;(I have created tagged releases for v1.0.0 and v1.1.0 in an attempt to mitigate the risk, but tip still has some interesting changes in it.)&lt;br /&gt;&lt;br /&gt;Unlike libnanomsg, mangos (tip only) can notify you when a connection is added or removed, and you can access interesting information about the connection. &amp;nbsp;This is in the &lt;a href=&quot;;&gt;Port API&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;&lt;h3&gt;&lt;i&gt;Futures&lt;/i&gt;&lt;/h3&gt;&lt;br /&gt;We are using mangos internally at Lucera, and I know now of several cases of production use. &amp;nbsp;This is kind of scary at one level, since I wrote this originally as a hobby project about a year ago (to learn Go.) &amp;nbsp;But it has become useful -- frankly extending mangos is far far more pleasurable than working in the C libnanomsg implementation -- a lot of this is thanks to Go which is utterly pleasurable to work in (no matter how bad the guts may be reputed to be). &amp;nbsp;Being able to write a new TLS transport, or even websocket, in the course of an afternoon or two (actually for TLS it was more like an hour), is really nice.&lt;br /&gt;&lt;br /&gt;I&#39;m hoping that more people will find it useful, and that folks who want to experiment with the underlying messaging patterns may find it easier to work with than the C code. &amp;nbsp;Ideally, there will be more collaborators here, as we start exploring new directions for this stuff.&lt;br /&gt;&lt;br /&gt;In the meantime, I&#39;m going to continue to work to improve and extend mangos, because its become one of the tools at my day job. &amp;nbsp;Its nice when work and pleasure come together!</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=1317329752004791569' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='Rise of mangos'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>0</thr:total></entry><entry><id>,</id><published>2014-11-13T09:33:00.000-08:00</published><updated>2014-11-13T09:33:21.082-08:00</updated><title type='text'>A better illumos...</title><content type='html'>&lt;div dir=&quot;ltr&quot; style=&quot;text-align: left;&quot; trbidi=&quot;on&quot;&gt;If you follow &lt;a href=&quot;;&gt;illumos&lt;/a&gt; very closely, you may already know some of this.&lt;br /&gt;&lt;br /&gt;&lt;h3 style=&quot;text-align: left;&quot;&gt;A New Fork&lt;/h3&gt;&lt;br /&gt;Several months ago, I forked &lt;a href=&quot;;&gt;illumos-gate&lt;/a&gt; (the primary source code repository for the kernel and system components of illumos) into &lt;a href=&quot;;&gt;illumos-core&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;I had started upstreaming my work from illumos-core into illumos-gate. &amp;nbsp;I&#39;ve since ceased that effort, largely because I simply have no time for the various arguments that my work often generates. &amp;nbsp;I think this is largely because my &lt;a href=&quot;;&gt;vision&lt;/a&gt; for illumos is somewhat different from that of other folks, and sadly illumos proper lacks anything resembling a guiding vision now, which means that only entirely non-contentious changes can get integrated into illumos. &lt;br /&gt;&lt;br /&gt;However, I still want to proceed apace with illumos-core, because I believe that work has real value, and I firmly believe that my vision for illumos is the one that will lead to greater adoption by users, and by distributors as well, since much of what I&#39;m trying to achieve in illumos-gate is aimed at reducing barriers to adoption and to developers both of illumos itself and of systems that want to build on top of or integrate illumos. &amp;nbsp;(An example of reducing barriers to adoption -- I recently &lt;a href=&quot;;&gt;implemented a BSD compatible flock() within libc&lt;/a&gt;, which is sometimes used by applications developed for BSD or Linux.)&lt;br /&gt;&lt;br /&gt;&lt;h3 style=&quot;text-align: left;&quot;&gt;Relationship to Upstream&lt;/h3&gt;&lt;br /&gt;I do also invite other parties to cherry-pick from illumos-core into illumos-gate. &amp;nbsp;I suspect that a large number of the enhancements I&#39;ve made, such as the support for the &lt;a href=&quot;;&gt;fexecve&lt;/a&gt;() function &lt;a href=&quot;;&gt;specified&lt;/a&gt; by POSIX 2008, are likely to be more widely useful. &amp;nbsp;Within illumos-core, I want to retain a high standard of quality, and facilitate the effort of upstreaming for those who want to make the effort to do so.&lt;br /&gt;&lt;br /&gt;I do want to reiterate that unlike other projects that have forked from illumos, it is not my intent to divorce myself from the community -- rather I see this illumos-core as an experimental branch aimed at exploring new directions that I ultimately hope will be embraced by the wider illumos community some day; by doing this in a separate repository/branch/fork, illumos-core can drive towards these goals without getting mired in questions that would prevent progress on these goals within illumos-gate proper.&lt;br /&gt;&lt;br /&gt;The focus here is on delivery, rather than on discussion. &amp;nbsp;(In fact, one of my taglines on social media has for many years been &quot;&lt;a href=&quot;;&gt;Code first, questions later&lt;/a&gt;.&quot; &amp;nbsp;The illumos-core effort represents a return to that core value.)&lt;br /&gt;&lt;br /&gt;&lt;h3 style=&quot;text-align: left;&quot;&gt;Call for Participation&lt;/h3&gt;&lt;br /&gt;I&#39;m also interested in having co-collaborators on this project. &amp;nbsp;The goals are large, and while I hope to achieve them someday even if I have to do it all myself, I&#39;m certain that the project will move quite a lot faster with help. &amp;nbsp;Also, because of our lack of bureaucracy, I hope that illumos-core can be an easier path to integration than illumos-gate. &amp;nbsp;I just use a simple &lt;a href=&quot;;&gt;github&lt;/a&gt; pull-request for integration at present.&lt;br /&gt;&lt;br /&gt;There is an opportunity for folks at all different technical levels to participate. &amp;nbsp;We need work that involves systems programming, but also there is work around documentation, research, shell scripting, test development and release engineering to be performed. &amp;nbsp;I&#39;m happy to mentor folks who want to help out, based on their skill level.&lt;br /&gt;&lt;br /&gt;And, of course, for folks who want to focus primarily on improving illumos-gate upstream, there is effort that could be spent to figure out what to cherry-pick and to do the various illumos-gate process wrangling steps to get those bits integrated.&lt;br /&gt;&lt;br /&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=845552299681278252' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='A better illumos...'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>0</thr:total></entry><entry><id>,</id><published>2014-10-17T23:20:00.002-07:00</published><updated>2014-10-17T23:20:57.527-07:00</updated><title type='text'>Your language sucks...</title><content type='html'>&lt;div dir=&quot;ltr&quot; style=&quot;text-align: left;&quot; trbidi=&quot;on&quot;&gt;As a result of work I&#39;ve been doing for &lt;a href=&quot;;&gt;illumos&lt;/a&gt;, I&#39;ve recently gotten re-engaged with internationalization, and the support for this in libc and localedef (I am the original author for our localedef.)&lt;br /&gt;&lt;br /&gt;I&#39;ve decided that human languages suck. &amp;nbsp;Some suck worse than others though, so I thought I&#39;d write up a guide. &amp;nbsp;You can take this as &quot;your language sucks if...&quot;, or perhaps a better view might be &quot;your program sucks if you make assumptions this breaks...&quot;&lt;br /&gt;&lt;br /&gt;(Full disclosure, I&#39;m spoiled. &amp;nbsp;I am a native speaker of English. &amp;nbsp;English is pretty awesome for data-processing, at least at the written level. &amp;nbsp;I&#39;m not going to concern myself with questions about deeper issues like grammar, natural language recognition, speech synthesis, or recognition, automatic translation, etc. &amp;nbsp;Instead this is focused strictly on the most basic display and simple operations like collation (sorting), case conversion, and character classification.)&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;1. Too many code points.&amp;nbsp;&lt;/h2&gt;Some languages (from Eastern Asia) have way way too many code points. &amp;nbsp;There are so many that these languages can&#39;t actually fit into 16-bits all by themselves. &amp;nbsp;Yes, I&#39;m saying that there are languages with over 65,000 characters in them! &amp;nbsp;This explosion means that generating data for languages results in intermediate lookup tables that are megabytes in size. &amp;nbsp;For Unicode, this impacts all languages. &amp;nbsp;The intermediate sources for the Unicode supported in illumos blow up to over 2GB when support for the additional code planes is included.&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;2. Your language requires me to write custom code for symbol names.&amp;nbsp;&lt;/h2&gt;Hangul Jamo, I&#39;m looking at you. &amp;nbsp;Of all the languages in Unicode, only this one is so bizarre that it requires multiple lookup tables to determine the names of the characters, because the characters are made up of smaller bits of phonetic portions (vowels and consonants.) &amp;nbsp;It even has its own section in the basic conformance document for Unicode (section 3.12). &amp;nbsp;I don&#39;t speak Korean, but I had to learn about Jamo.&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;3. Your language&#39;s character set is continuing to evolve.&amp;nbsp;&lt;/h2&gt;Yes, that&#39;s Asia again (mostly China I think). &amp;nbsp; The rate at which new Asian characters are added rivals that of updates to the timezone database. &amp;nbsp;The approach your language uses is wrong!&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;4. Characters in your language are of multiple different cell widths.&amp;nbsp;&lt;/h2&gt;Again, this is mostly, but not exclusively, Asian languages. &amp;nbsp;Asian languages require 2 cells to display many of their characters. &amp;nbsp;But, to make matters far far worse, some times the number f code points used to represent a character is more than one, which means that the width of a character when displayed may be 0, 1, or 2 cells. &amp;nbsp; Worse, some languages have both half- and full-width forms for many common symbols. &amp;nbsp;Argh.&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;5. The width of the character depends on the context.&amp;nbsp;&lt;/h2&gt;Some widths depend on the encoding because of historical practice (Asia again!), but then you have composite characters as well. &amp;nbsp;For example, a Jamo vowel sound could in theory be displayed on its own. &amp;nbsp;But if it follows a leading consonant, then it changes the consonant character and they become a new character (at least to the human viewer).&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;6. Your language has unstable case conversions.&lt;/h2&gt;There are some evil ones here, and thankfully they are rare. &amp;nbsp;But some languages have case conversions which are not reversible! &amp;nbsp;Case itself is kind of silly, but this is just insane! &amp;nbsp;Armenian has a letter with this property, I believe.&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;7. Your language&#39;s collation order is context-dependent.&amp;nbsp;&lt;/h2&gt;(French, I&#39;m looking at you!) &amp;nbsp;Some languages have sorting orders that depend not just on the character itself, but on the characters that precede or follow it. &amp;nbsp;Some of the rules are really hard. &amp;nbsp;The collation code required to deal with this generally is really really scary looking.&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;8. Your language has equivalent alternates (ligatures).&amp;nbsp;&lt;/h2&gt;German, your ß character, which stands in for &quot;ss&quot;, is a poster child here. &amp;nbsp;This is a single code point, but for sorting it is equivalent to &quot;ss&quot;. &amp;nbsp;This is just historical decoration, because it&#39;s &quot;fancy&quot;. &amp;nbsp;Stop making my programming life hard.&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;9. Your language can&#39;t decide on a script.&amp;nbsp;&lt;/h2&gt;Some languages can be written in more than one script. &amp;nbsp;For example, Mongolian can be written using Mongolian script or Cyrillic. &amp;nbsp;But the winner (loser?) here is Serbian, which in some places uses both Latin and Cyrillic characters &lt;i&gt;interchangeably&lt;/i&gt;! Pick a script already! I think the people who live like this are just schizophrenic. &amp;nbsp;(Given all the political nonsense surrounding language in these places, that&#39;s no real surprise.)&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;10. Your language has Titlecase.&amp;nbsp;&lt;/h2&gt;POSIX doesn&#39;t do Titlecase. &amp;nbsp;This happens because your language also uses ligatures instead of just allocating a separate cell and code point for each character. &amp;nbsp;Most people talk about titlecase used in a phrase or string of words. &amp;nbsp;But yes, titlecase can apply to a &lt;i&gt;SINGLE CHARACTER&lt;/i&gt;. &amp;nbsp;For example, Dž is just such a character.&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;11. Your language doesn&#39;t use the same display / ordering we expect.&lt;/h2&gt;So some languages use right to left, which is backwards, but whatever. &amp;nbsp; Others, crazy ones (but maybe crazy smart, if you think about it) use back and forth bidirectional. &amp;nbsp;And still others use vertical ordering. &amp;nbsp;But the worst of them are those languages (Asia again, dammit!) where the orientation of text can change. &amp;nbsp;Worse, some cases even rotate individual characters, depending upon context (e.g. titles are rotated 90 degrees and placed on the right edge). &amp;nbsp;How did you ever figure out how to use a computer with this crazy stuff?&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;12. Your encoding collides control codes.&lt;/h2&gt;We use the first 32 or so character codes to mean special things for terminal control, etc. &amp;nbsp;If we can&#39;t use these, your language is going to suck over certain kinds of communication lines.&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;13. Your encoding uses conflicting values at ASCII code points.&lt;/h2&gt;ASCII is universal. &amp;nbsp;Why did you fight it? &amp;nbsp;But that&#39;s probably just me being mostly Anglo-centric / bigoted.&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;14. Your language encoding uses shift characters.&amp;nbsp;&lt;/h2&gt;(Code page, etc.) &amp;nbsp;Some East Asian languages used this hack in the old days. &amp;nbsp;Stateful encodings are JUST HORRIBLY BROKEN. &amp;nbsp; A given sequence of characters should not depend on some state value that was sent a long time earlier.&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;15. Your language encoding uses zero values in the middle of valid characters.&amp;nbsp;&lt;/h2&gt;Thankfully this doesn&#39;t happen with modern encodings in common use anymore. &amp;nbsp;(Or maybe I just have decided that I won&#39;t support any encoding system this busted. &amp;nbsp;Such an encoding is so broken that I just flat out refuse to work with it.)&lt;br /&gt;&lt;br /&gt;&lt;h2 style=&quot;text-align: left;&quot;&gt;Non-Broken Languages&lt;/h2&gt;&lt;br /&gt;So, there are some good examples of languages that are famously not broken.&lt;br /&gt;&lt;br /&gt;a. &lt;b&gt;English&lt;/b&gt;. &amp;nbsp;Written English has simple sorting rules, and a very simple character set. &amp;nbsp;Dipthongs are never ligatures. &amp;nbsp;This is so useful for data processing that I think it has had a great deal to do with why English is the common language for computer scientists around the world. &amp;nbsp;US-ASCII -- and English character set, is the &quot;base&quot; character set for Unicode, and pretty much all other encodings use ASCII encodings in the lower 7 bits.&lt;br /&gt;&lt;br /&gt;b. &lt;b&gt;Russian&lt;/b&gt;. &amp;nbsp;(And likely others that use Cyrillic, but not all of them!) &amp;nbsp;Russian has a very simple alphabet, strictly phonetic. &amp;nbsp;The number of characters is small, there are no composite characters, and no special sorting rules. &amp;nbsp;Hmm... I seem to recall that Russia (Soviet era) had a pretty robust computing industry. &amp;nbsp;And these days Russians mostly own the Internet, right? &amp;nbsp;Coincidence? &amp;nbsp;Or maybe they just don&#39;t have to waste a lot of time fighting with the language just to get stuff done?&lt;br /&gt;&lt;br /&gt;I think there are probably others. &amp;nbsp;(At a glance, Geoergian looks pretty straight-forward. &amp;nbsp; I suspect that there are languages using both Cyrillic and Latin character sets that are sane. &amp;nbsp;Ethiopic actually looks pretty simple and sane too. &amp;nbsp;(Again, just from a text processing standpoint.)&lt;br /&gt;&lt;br /&gt;But sadly, the vast majority of natural languages have written forms &amp;amp; rules that completely and utterly suck for text processing.&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=8778482429302443995' title='20 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='Your language sucks...'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>20</thr:total></entry><entry><id>,</id><published>2014-10-12T14:00:00.001-07:00</published><updated>2014-10-13T16:03:12.827-07:00</updated><title type='text'>My Problem with Feminism</title><content type='html'>&lt;div dir=&quot;ltr&quot; style=&quot;text-align: left;&quot; trbidi=&quot;on&quot;&gt;I&#39;m going to say some things here that may be controversial. &amp;nbsp;Certainly that headline is. &amp;nbsp;But please, bear with me, and read this before you judge too harshly.&lt;br /&gt;&lt;br /&gt;As another writer said, 2014 has been a terrible year for women in tech. &amp;nbsp;(Whether in the industry, or in gaming.) &amp;nbsp;Arguably, this is not a &lt;i&gt;new&lt;/i&gt;&amp;nbsp;thing, but rather events are reaching a head. &amp;nbsp;Women (some at any rate) are being more vocal, and awareness of women&#39;s issues is up. &amp;nbsp;On the face of it, this should be a good thing.&lt;br /&gt;&lt;br /&gt;And yet, we have incredible conflict between women and men. &amp;nbsp;And this is at the heart of my problem with &quot;Feminism&quot;.&lt;br /&gt;&lt;div style=&quot;text-align: left;&quot;&gt;&lt;br /&gt;&lt;/div&gt;&lt;h3 style=&quot;text-align: left;&quot;&gt;The F-Word&lt;/h3&gt;&lt;br /&gt;Don&#39;t get me wrong. &amp;nbsp;I strongly believe that women should be treated fairly and with respect; in the professional place they should receive the same level of professional respect -- and compensation! -- as their male counterparts can expect. &amp;nbsp;I believe this passionately -- as a nerd, I prefer to judge people on the merits of their work, rather than on their race, creed, gender, or sexual preference. &amp;nbsp;A similar principle applies to gaming -- after all, how do you really know the gender of the player on the other side of the MMO? &amp;nbsp;Does it even matter? &amp;nbsp;When did gaming become a venue for channeling hate instead of fun?&lt;br /&gt;&lt;br /&gt;The problem with &quot;feminism&quot; is that instead of repairing inequality and trying to bring men and women closer together, so much of it seems to be divisive. &amp;nbsp;The very word itself basically suggests a gender based conflict, and I think this, as well as much of the recent approach, is counterproductive.&lt;br /&gt;&lt;br /&gt;Instead of calling attention to inequalities and improper behaviors (lets face it, nobody wants to deal with sexual harassment, discrimination, or some of the very much worse behavior that a few terribly bad actors are guilty of), we&#39;ve become focused on gender bias and &quot;fixing&quot; gender bias as a goal in and of itself, rather than instead focusing on fair and equal treatment for all.&lt;br /&gt;&lt;br /&gt;Every day I&#39;m inundated with tweets and Facebook postings extolling the terrible plight of women at the expense of men. &amp;nbsp;Many of these posts seem intended to make me either angry at men, or ashamed of being one. &amp;nbsp;This basically drives a wedge between people, even unconsciously, to the point that it has become impossible to avoid being a soldier on one side or the other of this war. &amp;nbsp;And don&#39;t get me wrong, it has indeed degenerated to a total war.&lt;br /&gt;&lt;br /&gt;I don&#39;t think this is what most feminists or their advocates really want. &amp;nbsp;(Though, I think it is what some of them want. &amp;nbsp;The side of feminism has its bad actors who thrive on conflict just as much as the other side has. &amp;nbsp;Extremism is gender and color and religion blind, as we&#39;ve ample evidence of.)&lt;br /&gt;&lt;br /&gt;I think one thing that advocates for women in tech can do, is to pick a different term, and a different way of stating their goals, and perhaps a different approach. &amp;nbsp;I think we&#39;ve reached the critical mass necessary for awareness, so the constant tweets about how terrible it is to be a woman are no longer helpful.&lt;br /&gt;&lt;br /&gt;I&#39;m not sure what &quot;term&quot; should replace feminism -- in the workplace I&#39;d suggest &quot;professionalism&quot;. &amp;nbsp;After all everyone wants to be treated professionally, not just women. &amp;nbsp;(Btw, I&#39;d say that in the gaming community, the value should be &quot;sportsmanship&quot;. &amp;nbsp;Sadly some will see that word is gender biased, but I don&#39;t ascribe to the notion that we have to completely change our language in order to be more politically correct. &amp;nbsp;You know what I mean.)&lt;br /&gt;&lt;br /&gt;Likewise, instead of dog piling on the one person (as I&#39;m sure will happen in response to this post) on someone who doesn&#39;t immediately appear to support the feminist agenda, perhaps a little more tolerance, and education should be used in the approach. &amp;nbsp;Focus should, IMO, be on public praise for the parties who are working to make conditions better.&lt;br /&gt;&lt;br /&gt;Educate instead of punish. &amp;nbsp;Make allies instead of enemies.&lt;br /&gt;&lt;br /&gt;&lt;h3 style=&quot;text-align: left;&quot;&gt;Salary Gap&lt;/h3&gt;&lt;br /&gt;&lt;div style=&quot;text-align: left;&quot;&gt;The salary gap issue that was raised recently by Microsoft is another case in point.&lt;br /&gt;&lt;br /&gt;I don&#39;t agree with &lt;a href=&quot;;&gt;Satya Nadella&#39;s comments&lt;/a&gt; saying that women should not ask for raises, but I think many women are nearly as likely to get a raise upon requesting one as a man of similar accomplishments. &amp;nbsp;(Yes, it would be better if this statement could have been said without &quot;nearly&quot;.) &amp;nbsp; Far too few women feel comfortable asking for a merit based raise in the first place -- that is something that should change.  But using race or gender as a bias to demand pay increases is a recipe for further division. &amp;nbsp;Indeed, men may begin to wonder if women are being compensated unfairly because they are women, but in the reverse direction.&amp;nbsp;&lt;/div&gt;&lt;div style=&quot;text-align: left;&quot;&gt;&lt;br /&gt;&lt;/div&gt;Likewise, bringing up discrimination in a salary discussion puts the other party on the defensive. &amp;nbsp;It presumes to imply prior wrong-doing. &amp;nbsp;This may be the case, but it may well not be. &amp;nbsp;After all, I&#39;ve known many men that were under compensated simply because they sold themselves short, or were not comfortable asking for more money. &amp;nbsp; Why look for a fight when there isn&#39;t one? &amp;nbsp;(I suspect this is what Satya was really trying to get at.)&lt;br /&gt;&lt;br /&gt;None of this helps the cause of &quot;professionalism&quot;, and probably not the cause of &quot;feminism&quot;.&lt;br /&gt;&lt;br /&gt;Average tech salary figures are easily obtainable. &amp;nbsp;If a worker, man or woman, feels under compensated -- for any reason -- then they should take it to his employer and ask for a correction. &amp;nbsp;But to presume that the reason is gender, starts the conversation from a point of conflict.&lt;br /&gt;&lt;br /&gt;Far far better is to demand far pay based on work performance and merit, relative to industry norms as appropriate. &amp;nbsp; If an employer won&#39;t compensate fairly, just leave. &amp;nbsp;There is no shortage of tech jobs in the industry. &amp;nbsp;If you&#39;re a woman, maybe look for jobs at companies that employ (and successfully retain) women. &amp;nbsp;Ask the people who work at a prospective employer about conditions, etc. &amp;nbsp;That&#39;s true for minorities too! &amp;nbsp;Ultimately, an employer who discriminates will find itself at a severe competitive advantage, as both the discriminated-against parties, and their allies refuse to do business with them.&lt;br /&gt;&lt;br /&gt;An employer is not obligated to pay you &quot;more&quot; because of your gender. &amp;nbsp;But they must also not pay you less because of gender. &amp;nbsp;And yet every company will generally try to pay as little as they think they can get away with. &amp;nbsp;So don&#39;t let them -- but keep discrimination out of the conversation unless there is really compelling proof of wrong doing. &amp;nbsp;(And if there is such evidence, I&#39;d recommend looking elsewhere, and possibly explore stronger legal measures.)&lt;br /&gt;&lt;br /&gt;And yes, I strongly strongly believe that most men feel as I do. &amp;nbsp;They support the notion that everyone should be treated equally and professionally, and would like to stamp out sexism in the workplace, but many of us are starting to show symptoms of battle fatigue, and even more of us just don&#39;t want to be involved in a conflict at all. &amp;nbsp; Frankly, I think a lot of us are annoyed at feminist attempts to draw us into the conflict, even though we do support many of the stated goals of equal pay, fair treatment, etc. etc.&lt;br /&gt;&lt;h3 style=&quot;text-align: left;&quot;&gt;Closing Thoughts&lt;/h3&gt;As for me, I support the plight of women who find themselves discriminated against based on their gender, and I would like to see more women in my industry. &amp;nbsp;And I&#39;ve put my money where my mouth is.&amp;nbsp;&lt;br /&gt;&lt;br /&gt;But at the same time, you won&#39;t find me supporting &quot;feminism&quot;. &amp;nbsp;I want to heal the rift, and work with awesome people -- and I happen to believe at least half of the awesome people in the world are of a different gender than I am. &amp;nbsp;Why would I want to alienate them?&lt;br /&gt;&lt;/br /&gt;I happen to believe that many well meaning people of many causes damage their cause by basically forcing people to deal with their &quot;diversity&quot; first, instead of of being able to deal with people as people on their own merit. &amp;nbsp;Its so much harder to appreciate a person on her own merits, when at least half of what she is saying is that she&#39;s unfairly treated because of gender, race, sexual preference, etc. &amp;nbsp;This true for everyone. &amp;nbsp;Show me how you&#39;re excellent, and I promise to appreciate you for your awesomeness, and to treat you fairly and with the same respect I would for anyone of my own gender/race/sexual preference.&lt;br /&gt;&lt;br /&gt;You are awesome because of your accomplishments/innovations/contributions, not because of your gender or race or sexual preference.&lt;br /&gt;&lt;br /&gt;But, if you won&#39;t let me look past your race/gender/etc. identity, then please don&#39;t be offended if I don&#39;t see anything else. &amp;nbsp;If you want to be treated like a &quot;person&quot;, then let me see the person instead of just some classification in an equal opportunity survey.&lt;br /&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=6417258591013467960' title='3 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='My Problem with Feminism'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>3</thr:total></entry><entry><id>,</id><published>2014-10-02T14:24:00.003-07:00</published><updated>2014-10-02T14:24:45.479-07:00</updated><title type='text'>Supporting Women in Open Source</title><content type='html'>Please have a look at &lt;a href=&quot;;&gt;Sage Weil&#39;s blog post&lt;/a&gt; on supporting the Ada Initiative, which supports women in open source development.&lt;br /&gt;&lt;br /&gt;Sage is sponsoring an $8192 matching grant, to support women in open source development of open storage technology.&lt;br /&gt;&lt;br /&gt;You may have heard my talk recently, where I expressed that there have been no female contributions to illumos (that includes ZFS by the way!) &amp;nbsp;This is kind of a tragedy; intelligence and creativity of at least half the population are simply not represented here, and we are worser for it.&lt;br /&gt;&lt;br /&gt;If you want to try to do something about it, heres a small thing. &amp;nbsp;There&#39;s a week remaining to do so, so I encourage folks to step up. &amp;nbsp;($3392 has already been granted.)&lt;br /&gt;&lt;br /&gt;I&#39;m making a donation myself, if you think supporting more women in open source is a worthwhile cause, please join me!&lt;br /&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=2247759060905481071' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='Supporting Women in Open Source'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>0</thr:total></entry><entry><id>,</id><published>2014-09-07T09:10:00.000-07:00</published><updated>2014-09-07T18:31:49.286-07:00</updated><title type='text'>Modernizing &quot;less&quot;</title><content type='html'>&lt;div dir=&quot;ltr&quot; style=&quot;text-align: left;&quot; trbidi=&quot;on&quot;&gt;I have just spent an all-nighter doing something I didn&#39;t expect to do. &lt;br /&gt;&lt;br /&gt;I&#39;ve &quot;&lt;a href=&quot;;&gt;modernized&lt;/a&gt;&quot; &lt;b&gt;less&lt;/b&gt;(1). &amp;nbsp;(That link is to the changeset.)&lt;br /&gt;&lt;br /&gt;First off, let me explain the motivation. &amp;nbsp;We need a pager for illumos that can meet the requirements for POSIX IEEE 2003.1-2008 &lt;a href=&quot;;&gt;&lt;b&gt;more&lt;/b&gt;&lt;/a&gt;(1). &amp;nbsp;We have a suitable pager (barely), in closed source form only, delivered into &lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;/usr/xpg4/bin/more&lt;/span&gt;. &amp;nbsp;We have an open source &lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;&lt;a href=&quot;;&gt;/usr/bin/more&lt;/a&gt;&lt;/span&gt;, but it is incredibly limited, hearkening back to the days of printed hard copy I think. &amp;nbsp;(It even has Microsoft copyrights in it!)&lt;br /&gt;&lt;br /&gt;So closed source is kind of a no go for me. &lt;br /&gt;&lt;br /&gt;&lt;a href=&quot;;&gt;&lt;b&gt;less&lt;/b&gt;&lt;/a&gt;(1) looks attractive. &amp;nbsp;It&#39;s widely used, and has been used to fill in for more(1) to achieve POSIX compliance on other systems (such as MacOS X.)&lt;br /&gt;&lt;br /&gt;So I started by unpacking it into our tree, and trying to get it to work with an illumos build system.&lt;br /&gt;&lt;br /&gt;That&#39;s when I discovered the crazy contortions autoconf was doing that basically wound up leaving it with just legacy BSD termcap. &amp;nbsp; Ewww. &amp;nbsp; I wanted it to use &lt;a href=&quot;;&gt;X/Open Curses&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;When I started trying to do that, I found that there were severe crasher bugs in less, involving the way it uses scratch buffer space. &amp;nbsp;I started trying to debug just that problem, but pretty soon the effort mushroomed.&lt;br /&gt;&lt;br /&gt;Legacy less supports all kinds of crufty and ancient systems. &amp;nbsp; Systems like MS-DOS (actually many different versions with different compiler options!) and Ultrix and OS/2 and OS9, and OSK, etc. &amp;nbsp;In fact, it apparently had to support systems where the C preprocessor didn&#39;t understand &lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;#elif&lt;/span&gt;, so the &lt;span style=&quot;font-family: Courier New, Courier, monospace;&quot;&gt;#ifdef&lt;/span&gt; maze was truly nightmarish. &amp;nbsp;The code is K&amp;amp;R style C even.&lt;br /&gt;&lt;br /&gt;I decided it was high time to modernize this application for POSIX systems. &amp;nbsp;So I went ahead and did a sweeping update. &amp;nbsp;In the process I ditched thousands of lines of code (the screen handling bits in screen.c are less than half as big as they were).&lt;br /&gt;&lt;br /&gt;So, now it:&lt;br /&gt;&lt;br /&gt;&lt;br /&gt;&lt;ul style=&quot;text-align: left;&quot;&gt;&lt;li&gt;Speaks terminfo (X/Open Curses) instead of ancient BSD termcap&lt;/li&gt;&lt;li&gt;Uses &lt;b&gt;glob&lt;/b&gt;(3C) instead of a hack involving the shell and a helper program (&lt;b&gt;lessecho&lt;/b&gt;, which I&#39;ve removed from my tree.)&lt;/li&gt;&lt;li&gt;Functions properly as /usr/bin/more, both with and without -e (even on broken xterms)&lt;/li&gt;&lt;li&gt;Is fully ANSI C (or ISO C, if you prefer)&lt;/li&gt;&lt;li&gt;Passes illumos&#39; cstyle code style checks&lt;/li&gt;&lt;li&gt;Is &lt;b&gt;lint&lt;/b&gt;(1) clean&lt;/li&gt;&lt;/ul&gt;&lt;br /&gt;&lt;br /&gt;There is more work to do in the future if someone wants to. &amp;nbsp;Here are the ideas for the future:&lt;br /&gt;&lt;br /&gt;&lt;br /&gt;&lt;ul style=&quot;text-align: left;&quot;&gt;&lt;li&gt;Internationalization. &amp;nbsp;This is a pretty easy task involving &lt;b&gt;gettext&lt;/b&gt;().&lt;/li&gt;&lt;li&gt;Make less use getopt() instead of its byzantine option parser (it needed that for PC operating systems. &amp;nbsp;We don&#39;t need or want this complexity on POSIX.)&lt;/li&gt;&lt;li&gt;Fix its character set handling so it can use the &lt;b&gt;mbstring&lt;/b&gt; and &lt;b&gt;wcstring&lt;/b&gt; routines in the platform instead of relying on it&#39;s own implementation of UTF-8. &amp;nbsp;(This would make it support other multibyte locales.)&lt;/li&gt;&lt;li&gt;Make it support port events instead of sleeping when acting in &quot;tail -f&quot; mode.&lt;/li&gt;&lt;/ul&gt;&lt;br /&gt;&lt;br /&gt;If someone wants to pick up any of this work, let me know. &amp;nbsp;I&#39;m happy to advise. &amp;nbsp;Oh, and this isn&#39;t in illumos proper yet. &amp;nbsp;It&#39;s unclear when, if ever, it will get into illumos -- I expect a lot of grief from people who think I shouldn&#39;t have forked this project, and I&#39;m not interested in having &amp;nbsp;a battle with them. &amp;nbsp;The upstream has to be a crazy maze because of the platforms it has to support. &amp;nbsp;We can do better, and I think this was a worthwhile case. &amp;nbsp;(In any event, I now know quite a lot more about less internals than I did before. &amp;nbsp;Not that this is a good thing.)&lt;br /&gt;&lt;br /&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=4888050947994817475' title='12 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='Modernizing &quot;less&quot;'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>12</thr:total></entry><entry><id>,</id><published>2014-08-20T22:15:00.001-07:00</published><updated>2014-08-20T22:15:12.688-07:00</updated><title type='text'>It&#39;s time already</title><content type='html'>&lt;div&gt;(Sorry for the political/religious slant this post takes... I&#39;ve been trying to stay focused on technology, but sometimes events are simply too large to ignore...)&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;The &lt;a href=&quot;;&gt;execution of John Foley&lt;/a&gt; is just the latest. &amp;nbsp;But for me, its the straw that broke the camel&#39;s back.&amp;nbsp;&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;Over the past weeks, I&#39;ve become extremely frustrated and angry. &amp;nbsp;The &quot;radical Islamists&quot; have become the single biggest threat to world peace since Hitler&#39;s Nazi&#39;s. &amp;nbsp;And they are worse than the Nazi&#39;s. &amp;nbsp;Which takes some doing. &amp;nbsp;(Nazi&#39;s &quot;merely&quot; exterminated Jews. &amp;nbsp;The Islamists want to exterminate everyone who does&#39;t believe exactly their own particular version of extreme religion.)&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;I&#39;m not a Muslim. &amp;nbsp;I&#39;m probably not even a Christian when you get down to it. &amp;nbsp;I do believe in God, I suppose. &amp;nbsp;And I do believe that God certainly didn&#39;t intend for one group of believes to exterminate another simply because they have different beliefs.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;Parts of the Muslim world &lt;i&gt;claim&lt;/i&gt;&amp;nbsp;that ISIS and those of its ilk are a scourge, primarily, I think, because they are turning the rest of the world against Islam. &amp;nbsp;If that&#39;s true, then the entire Muslim world who rejects ISIS and radical fundamentalist Islam (and it&#39;s not clear to me that rejecting one is the same as the other) needs to come together and eliminate ISIS, and those who follow its beliefs or even sympathize with it.&amp;nbsp;&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;That hasn&#39;t happened. &amp;nbsp;I don&#39;t see a huge military invasion of ISIS territory by forces from Arabia, Indonesia, and other Muslim nations. &amp;nbsp;Why not?&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;I don&#39;t believe it is possible to be a peace loving person (Muslim or otherwise), and stand idly by (or advocate standing by) why the terrorist forces who want nothing more than to destroy the very fabric of human society work to achieve their evil ends.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;Just as Nazi Germany and Imperial Japan were an Axis of Evil during our grandparents&#39; generation, so now we have a new Axis of Evil that has taken root in the middle east.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;It&#39;s time now to recognize that there is absolutely &lt;i&gt;no&lt;/i&gt;&amp;nbsp;chance for a peaceful coexistence with these people. &amp;nbsp;They are, frankly, subhuman, and their very &lt;i&gt;existence&lt;/i&gt;&amp;nbsp;is at odds with that of everyone everywhere else in the world.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;It&#39;s time for those of us in civilized nations to stop with our petty nonsense bickering. &amp;nbsp;The actions taking place in Ukraine, unless you live there (an in many case even if you do live there), are a diversion. &amp;nbsp;Putin and Obama need to stop their petty bickering, and cooperate to eliminate the real threat to civilization, which is radical Islam.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;To be clear, I believe that the time has now come for the rest of the world to pick up and take action, where the Muslim world has failed. &amp;nbsp;We need to clean house. &amp;nbsp;We can no longer cite &quot;freedom of expression&quot; and &quot;freedom of religion&quot; as reasons to let imam&#39;s recruit young men into death cults. &amp;nbsp;We must recognize that these acts of incitement to terrorism are indeed what they are, and the perpetrators have no more right to life and liberty than Charles Manson.&amp;nbsp;&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;These are forces that seek to overthrow from within, by recruitment, by terrorism, or by any means they can. &amp;nbsp;These are forces that place no value on human life. &amp;nbsp;These are forces with which are inimical to the very concept of civilization.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;There can be no tolerance for them. &amp;nbsp;None, whatsoever.&amp;nbsp;&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;To be clear, I&#39;m advocating that when a member of one of these organizations willing self identifies as such, we should simply kill them. &amp;nbsp;Wherever they are. &amp;nbsp;These are the enemy, and there is no separate battlefield, and they do not recognize &quot;civilians&quot; or &quot;innocents&quot;; therefore, like a cancer, radical Islam must be purged from the very earth, by any means necessary.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;The militaries of the world should unit, and work together, to eradicate entrenched forces of radical Islam wherever it exists in the world. &amp;nbsp;This includes all those forms that practice Sharia law, where a man and woman can be stoned to death simply for marrying without parental consent, as well as those groups that seek to eliminate the state of Israel, that seek to kill those who don&#39;t believe exactly as they do, that would issue a &lt;i&gt;fatwa&lt;/i&gt;&amp;nbsp;demanding the death of a cartoonist simply for depicting their prophet, &amp;nbsp;and those who seek to reduce women to the status of mere cattle.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;To be clear, we have to do the hard work, all nations of the world, to eliminate this scourge, and eliminate it at its source. &amp;nbsp;Mosques where radicalism are preached must no longer be sanctuaries. &amp;nbsp;Schools where &quot;teachers&quot; train their students in the killing of Christians and Jews, and that their God demands the death of &quot;unbelievers&quot; and rewards suicide bombers with paradise, need to be recognized as the training camps they are. &amp;nbsp;Even if the students are women and children.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;Your right to free speech and to religion does not trump my right to live. &amp;nbsp;Nor, by the way, does it trump my own rights to free speech and religion.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;I suppose this means that we have to be willing to accept some losses of combat, in the fight against radicalism. &amp;nbsp;We also have to accept that &quot;collateral damage&quot; is inevitable. &amp;nbsp;As with rooting out a cancer, some healthy cells are going to be destroyed. &amp;nbsp;But these losses have to be endured if the entire organism that is civilization is to survive.&amp;nbsp;&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;If this sounds like I&#39;m a hawk, perhaps that&#39;s true. &amp;nbsp;I think, rather, I&#39;m merely someone who wants to survive, and wants the world to be a place where my own children and grandchildren can live without having to endure a constant fear of nut jobs who want to kill them simply because they exist and think differently.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;Btw, if Islam as a religion is to survive in the long run, it &lt;i&gt;must&lt;/i&gt;&amp;nbsp;see these forces purged. &amp;nbsp;Because otherwise the only end result becomes an all out war of survival between Muslims and the rest of the world. &amp;nbsp;And guess which side has the biggest armies and weapons? And who will be the biggest losers in a conflict between Muslims and everyone else?&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;So, it&#39;s time to choose a side. &amp;nbsp;There is no middle ground. &amp;nbsp;Radical Islam tolerates no neutrality. &amp;nbsp;So, what&#39;s it going to be?&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;As for me, I choose civilization and survival. &amp;nbsp;That means a world without radical Islam. &amp;nbsp;Period.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=3767819768476018717' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='It&#39;s time already'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>0</thr:total></entry><entry><id>,</id><published>2014-07-11T20:54:00.000-07:00</published><updated>2014-07-11T20:54:25.769-07:00</updated><title type='text'>POSIX 2008 locale support integrated (illumos)</title><content type='html'>A year in the making... and finally the code is pushed. &amp;nbsp;Hooray!&lt;br /&gt;&lt;br /&gt;I&#39;ve just pushed &lt;a href=&quot;;&gt;2964&lt;/a&gt;&amp;nbsp;into &lt;a href=&quot;;&gt;illumos&lt;/a&gt;, which adds support for a bunch of new libc calls for thread safe and thread-specific locales, as well as explicit locale_t objects. &amp;nbsp; Some of the interfaces added fall under the BSD/MacOS X &quot;&lt;a href=&quot;;&gt;xlocale&lt;/a&gt;&quot; class of functions.&lt;br /&gt;&lt;br /&gt;Note that not all of the xlocale functions supplied by BSD/MacOS are present. &amp;nbsp;However, all of the routines that were added by &lt;a href=&quot;;&gt;POSIX 2008&lt;/a&gt; for this class are present, and should conform to the POSIX 2008 / XPG Issue 7 standards. &amp;nbsp;(Note that we are not yet compliant with POSIX 2008, this is just a first step -- albeit a rather major one.)&lt;br /&gt;&lt;br /&gt;The &lt;a href=&quot;;&gt;webrev&lt;/a&gt; is also available for folks who want to look at the code.&lt;br /&gt;&lt;br /&gt;The new APIs are documented in newlocale(3c), uselocale(3c), etc. &amp;nbsp; (Sadly, man pages are not indexed yet so I can&#39;t put links here.)&lt;br /&gt;&lt;br /&gt;Also, documentation for some APIs that was missing (e.g. strfmon(3c)) are now added.&lt;br /&gt;&lt;br /&gt;This project has taken over a year to integrate, but I&#39;m glad it is now done.&lt;br /&gt;&lt;br /&gt;I want to say a big -- huge -- thank you to Robert Mustacchi who not only code reviewed a huge amount of change (and provided numerous useful and constructive feedback), but also contributed a rather large swath of man page content in support of this effort, working on is own spare time. &amp;nbsp;Thanks Robert!&lt;br /&gt;&lt;br /&gt;Also, thanks to both Gordon Ross and Dan McDonald who also contributed useful review feedback and facilitated the integration of this project. &amp;nbsp;Thanks guys!&lt;br /&gt;&lt;br /&gt;Any errors in this effort are mine, of course. &amp;nbsp;I would be extremely interested in hearing constructive feedback. &amp;nbsp;I expect there will be some minor level of performance impact (unavoidable due to the way the standards were written to require a thread-specific check on all locale sensitive routines), but I hope it will be minor.&lt;br /&gt;&lt;br /&gt;I&#39;m also extremely interested in feedback from folks who are making use of these new routines. &amp;nbsp;I&#39;m told the Apache Standard C++ library depends on these interfaces -- I hope someone will try it out and let me know how it goes. &amp;nbsp; Also, if someone wants/needs xlocale interfaces that I didn&#39;t include in this effort, please drop me a note and I&#39;ll try to get to it.&lt;br /&gt;&lt;br /&gt;As this is a big change, it is not entirely without risk. &amp;nbsp;I&#39;ve done what I could to minimize that risk, and test as much as I could. &amp;nbsp;If I missed something, please let me know, and I&#39;ll attempt to fix in a timely fashion.&lt;br /&gt;&lt;br /&gt;Thanks!&lt;br /&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=7476308653358552214' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='POSIX 2008 locale support integrated (illumos)'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>0</thr:total></entry><entry><id>,</id><published>2014-05-27T10:30:00.000-07:00</published><updated>2014-05-27T10:30:07.661-07:00</updated><title type='text'>illumos, identification, and versions</title><content type='html'>Recently, there has been a bit of a &lt;a href=&quot;;&gt;debate&lt;/a&gt; on the &lt;a href=&quot;;&gt;illumos&lt;/a&gt; mailing lists, beginning I suppose with an initial &lt;a href=&quot;;&gt;proposal&lt;/a&gt; I made concerning the output from the &lt;b&gt;&lt;a href=&quot;;&gt;uname&lt;/a&gt;&lt;/b&gt;(1) command on illumos, which today, when presented with &quot;-s -r&quot; as options, emits &quot;SunOS 5.11&quot;.&lt;br /&gt;&lt;br /&gt;A lot of the debate centers on various ways that software can or cannot identify the platform it is running on, and use that information to make programmatic decisions about how to operate.  Most often these are decisions made at compile time, by tools such as &lt;b&gt;autoconf&lt;/b&gt; and &lt;b&gt;automake&lt;/b&gt;.&lt;br /&gt;&lt;br /&gt;Clearly, it would be better for software not to rely on uname for programmatic uses, and detractors of my original proposal are correct that even in the Linux community, the value from uname cannot be relied upon for such use. &amp;nbsp;There are indeed better mechanisms to use, such as &lt;b&gt;&lt;a href=&quot;;&gt;sysconf&lt;/a&gt;&lt;/b&gt;(3c), or the ZFS feature flags, to determine individual capabilities. &amp;nbsp;Indeed, the GNU autotools contains many individual tests for such things, and arguably discourages the use of uname except as a last resort.&lt;br /&gt;&lt;br /&gt;Yet there can be no question that there are a number of packages that do make such use. &amp;nbsp;And changes to the output from uname become risky to such packages.&lt;br /&gt;&lt;br /&gt;But perversely, &lt;i&gt;not changing&lt;/i&gt;&amp;nbsp;the output from uname also creates risk for such packages, as the various incarnations of SunOS 5.11 become ever less like one another. &amp;nbsp;Indeed, illumos != SunOS, and uname has become something of a lie over the past 4 years or so.&lt;br /&gt;&lt;br /&gt;Clearly, the focus for programmatic platform determination -- particularly for enabling features or behaviors, should be to move away from uname. &amp;nbsp;(Actually, changing uname may actually help package maintainers in identifying this questionable behavior as questionable, although there is no doubt that such a change would be disruptive to them.)&lt;br /&gt;&lt;br /&gt;But all this debate completely misses the other major purpose of uname&#39;s output, which is to identify the &lt;i&gt;platform&lt;/i&gt;&amp;nbsp;to humans. &amp;nbsp;Be they administrators, or developers, or distributors. &amp;nbsp;There is no question in mind that illumos&#39; failure to self identify, and to have a regular &quot;release&quot; schedule (for whatever a release really means in this regard) is harmful.&lt;br /&gt;&lt;br /&gt;The distributors, such as &lt;a href=&quot;;&gt;Joyent&lt;/a&gt; (&lt;a href=&quot;;&gt;SmartOS&lt;/a&gt;), would prefer that people only identify their particular distribution, and I believe that much of the current argument from them stems from two primary factors. &amp;nbsp;First, they see additional effort in any change, with no direct benefit to them. &amp;nbsp;Second, in some ways these distributors are disinclined to emphasize illumos itself. &amp;nbsp; Some of the messages sent (either over IRC or in the email thread) clearly portray some resentment towards the rest of the illumos ecosystem, especially some of the niche players), as well as a very heavy handed approach by one commercial concern towards the rest of the ecosystem.&lt;br /&gt;&lt;br /&gt;Clearly, this flies in the spirit of community cooperation in which illumos was founded. &amp;nbsp;illumos should not be a slave to any single commercial concern, and nobody should be able to exercise a unilateral veto over illumos, or shut down conversations with a single &quot;&lt;a href=&quot;;&gt;thermonuclear&lt;/a&gt;&quot; post.&lt;br /&gt;&lt;br /&gt;So, without regard to the merits, or lack thereof, of changing uname&#39;s output, I&#39;m quite certain, with sufficient clear evidence of my own gathering, that illumos absolutely needs to have a regularly scheduled &quot;releases&quot; of illumos, with humanly understandable release identifiers. &amp;nbsp;The lack of both the releases, and the identifiers to go with them, hinders meaningful reviews, hurts distributors (particularly smaller ones), and makes support for the platform harder, particularly when the questions of support are considered across distributions. &amp;nbsp;All of these hinder &lt;i&gt;adoption&lt;/i&gt;&amp;nbsp;of the illumos platform; clearly an undesirable outcome.&lt;br /&gt;&lt;br /&gt;Some would argue that we could use &lt;b&gt;&lt;a href=&quot;;&gt;git&lt;/a&gt;&lt;/b&gt; tags for the identifiers. &amp;nbsp;From a programmatic standpoint, these would be easy to collect. &amp;nbsp;Although they have problems as well (particularly for distributions which neither use git, or use a private fork that doesn&#39;t preserve our git versions), there are worse problems.&lt;br /&gt;&lt;br /&gt;Specifically, a humans aren&#39;t meant to derive meaning from something like &quot;e3de96f25bd2ea4282eea2d1a86c1bebac8950cb&quot;. &amp;nbsp; While &lt;a href=&quot;;&gt;Neo&lt;/a&gt; could understand this, most of use merely mortal individuals simply can&#39;t understand such tags. &amp;nbsp;Worse, there is no implied sequencing here. &amp;nbsp;Is &quot;e3de96f25bd2ea4282eea2d1a86c1bebac8950cb&quot; newer than &quot;d1007364f5b14efdd7d6ba27aa458669a6365d48&quot; ? &amp;nbsp;You can&#39;t do a meaningful comparison without examining the actual git history.&lt;br /&gt;&lt;br /&gt;This makes it hard to guess whether a given running release has a bug integrated or not. &amp;nbsp;It makes it hard to have conversations about the platform. &amp;nbsp;It even makes it hard for independent reviewers of the platform &amp;nbsp;to identify anything meaningful about the platform in the context of reviewing a distribution.&lt;br /&gt;&lt;br /&gt;Furthermore, when we talk about the platform, its useful for version numbers to convey more than just serial meaning. &amp;nbsp;In particular, version numbers help set expectations for developers. &amp;nbsp;Historically Solaris (or rather SunOS, from which illumos is derived), set those expectations in the form of stability guarantees. &amp;nbsp; A given library interface might be declared as Stable, or Evolving, (or Committed or Uncommitted), or Obsolete. &amp;nbsp;This was a way to convey to developers the relative risks of an interface (in terms of interface change), and it set some ground rules for rate of change. &amp;nbsp;Indeed, Solaris (and SunOS) relied upon a form of &lt;a href=&quot;;&gt;semantic versioning&lt;/a&gt;, and many of the debates in Sun&#39;s architectural leadership for Solaris (PSARC) revolved around these commitments.&lt;br /&gt;&lt;br /&gt;Yet today, the illumos distributors seem over willing to throw that bit of our heritage into the waste bin. &amp;nbsp;A trend, I fear, which ultimately leads to chaos, and an increase in the difficulty of adoption by ISVs and developers.&lt;br /&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;illumos is one component -- albeit probably the most important by far -- of a distribution. &amp;nbsp;Like all components, it is useful to be able to determine and talk about it. &amp;nbsp;This is not noticeably different than Linux, Xorg, gnome, or pretty much any of the other systems which you are likely to find as part of a complete Ubuntu or RedHat distribution. &amp;nbsp;In fact, our situation is entirely analogous other than we combine our libc and some key utilities and commands with the kernel.&lt;br /&gt;&lt;br /&gt;Technically, in Solaris parlance, illumos is a &lt;i&gt;consolidation&lt;/i&gt;. &amp;nbsp;In fact, this distinction has alway been clear. &amp;nbsp;And historically the way the consolidation is identified is with the contents of the utsname structure, which is what is emitted by uname. &amp;nbsp;&lt;br /&gt;&lt;br /&gt;Furthermore, when we talk about the platform, its useful for version numbers to convey more than just serial meaning&lt;br /&gt;&lt;br /&gt;In summary, regardless of whether we feel uname should return illumos or not, there is a critical need, although one not necessarily agreed upon by certain commercial concerns, for the illumos platform to have a release number at a minimum, and this release number must be useful to convey meaningful information to end-users, developers, and distributors alike. &amp;nbsp;It would be useful if this release number were obtainable in the traditional fashion (uname), but its more important that the numbers convey meaning in the same way across distributions (which means packaging metadata cannot be used, at least not exclusively).&lt;br /&gt;&lt;br /&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=8248767765081061272' title='2 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='illumos, identification, and versions'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>2</thr:total></entry><entry><id>,</id><published>2014-04-06T20:31:00.000-07:00</published><updated>2014-04-06T21:20:13.061-07:00</updated><title type='text'>SP protocols improved again!</title><content type='html'>&lt;h2&gt;Introduction&lt;/h2&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;As a result of some investigations performed in response to my &lt;a href=&quot;;&gt;first performance tests&lt;/a&gt; for my &lt;a href=&quot;;&gt;SP implementation&lt;/a&gt;, I&#39;ve made a bunch of changes to my code.&lt;br /&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;First off, I discovered that my code was rather racy. &amp;nbsp;When I started bumping up GOMAXPROCS, and and used the -race flag to go test, I found lots of issues.&amp;nbsp;&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;Second, there were failure scenarios where the performance fell off a cliff, as the code dropped messages, needed to retry, etc.&amp;nbsp;&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;I&#39;ve made a lot of changes to fix the errors. &amp;nbsp;But, I&#39;ve also made a major set of changes which enable a vastly better level of performance, particularly for throughput sensitive workloads. Note that to get these numbers, the application should &quot;recycle&quot; the Messages it uses (using a new &lt;a href=&quot;;&gt;Free()&lt;/a&gt; API... there is also a &lt;a href=&quot;;&gt;NewMessage() &lt;/a&gt;API to allocate from the cache), which will cache and recycle used buffers, greatly reducing the garbage collector workload.&lt;br /&gt;&lt;h2&gt;&lt;/h2&gt;&lt;h2&gt;Throughput&lt;/h2&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;So, here are the new numbers for throughput, compared against my previous runs on the same hardware, including tests against the &lt;a href=&quot;;&gt;nanomsg&lt;/a&gt; reference itself.&lt;br /&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;&lt;table border=&quot;2&quot; style=&quot;color: #333333; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 15px; line-height: 20.790000915527344px;&quot;&gt;&lt;caption&gt;Throughput Comparision&lt;br /&gt;(Mb/s)&lt;/caption&gt;&lt;tbody&gt;&lt;tr&gt;&lt;th&gt;transport&lt;/th&gt;&lt;th&gt;nanomsg 0.3beta&lt;/th&gt;&lt;th&gt;old gdamore/sp&lt;/th&gt;&lt;th&gt;new&lt;br /&gt;(1 thread)&lt;/th&gt;&lt;th&gt;new&lt;br /&gt;(2 threads)&lt;/th&gt;&lt;th&gt;new&lt;br /&gt;(4 threads)&lt;/th&gt;&lt;th&gt;new&lt;br /&gt;(8 threads)&lt;/th&gt;&lt;/tr&gt;&lt;tr&gt;&lt;th&gt;inproc 4k&lt;/th&gt;&lt;td align=&quot;right&quot;&gt;4322&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;5551&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;6629&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;7751&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;8654&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;8841&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;th&gt;ipc 4k&lt;/th&gt;&lt;td align=&quot;right&quot;&gt;9470&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;2379&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;6176&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;6615&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;5025&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;5040&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;th&gt;tcp 4k&lt;/th&gt;&lt;td align=&quot;right&quot;&gt;9744&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;2515&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;3785&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;4279&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;4411&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;4420&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;th&gt;inproc 64k&lt;/th&gt;&lt;td align=&quot;right&quot;&gt;83904&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;21615&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;45618&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;35044&lt;sup&gt;b&lt;/sup&gt;&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;44312&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;47077&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;th&gt;ipc 64k&lt;/th&gt;&lt;td align=&quot;right&quot;&gt;38929&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;7831&lt;sup&gt;a&lt;/sup&gt;&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;48400&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;65190&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;64471&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;63506&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;th&gt;tcp 64k&lt;/th&gt;&lt;td align=&quot;right&quot;&gt;30979&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;12598&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;34994&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;49608&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;53064&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;53432&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;br /&gt;&lt;div&gt;&lt;sup&gt;a&lt;/sup&gt; I think this poor result is from retries or resubmits inside the old implementation.&lt;br /&gt;&lt;sup&gt;b&lt;/sup&gt; I cannot explain this dip; I think maybe unrelated activity or GC activity may be to blame&lt;br /&gt;&lt;br /&gt;The biggest gains are with large frames (64K), although there are gains for the 4K size as well. &amp;nbsp;nanomsg still out performs for the 4K size, but with 64K my message caching changes pay dividends and my code actually beats nanomsg rather handily for the TCP and IPC cases.&lt;br /&gt;&lt;br /&gt;I think for 4K, we&#39;re hurting due to inefficiencies in the Go TCP handling below my code. &amp;nbsp;My guess is that there is a higher per packet cost here, and that is what is killing us. &amp;nbsp;This may be true for the IPC case as well. &amp;nbsp;Still, these are very respectable numbers, and for some very real and useful workloads my implementation compares and even beats the reference.&lt;br /&gt;&lt;br /&gt;The new code really shows some nice gains for concurrency, and makes good use of multiple CPU cores.&lt;br /&gt;&lt;br /&gt;There are a few mysteries though. &amp;nbsp;Notes &quot;a&quot; and &quot;b&quot; point to two of them. &amp;nbsp;The third is that the IPC performance takes a dip when moving from 2 threads to 4. &amp;nbsp;It still significantly outperforms the TCP side though, and is still performing more than twice as fast as my first implementation, so I guess I shouldn&#39;t complain too much.&lt;br /&gt;&lt;h2&gt;&lt;/h2&gt;&lt;h2&gt;Latency&lt;/h2&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;The latency has shown some marked improvements as well. &amp;nbsp;Here are new latency numbers.&lt;/div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;&lt;table border=&quot;2&quot; style=&quot;background-color: white; color: #333333; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 15px; line-height: 20.790000915527344px;&quot;&gt;&lt;caption&gt;Latency Comparision&lt;br /&gt;(usec/op)&lt;/caption&gt;&lt;tbody&gt;&lt;tr&gt;&lt;th&gt;transport&lt;/th&gt;&lt;th&gt;nanomsg 0.3beta&lt;/th&gt;&lt;th&gt;old gdamore/sp&lt;/th&gt;&lt;th&gt;new&lt;br /&gt;(1 thread)&lt;/th&gt;&lt;th&gt;new&lt;br /&gt;(2 threads)&lt;/th&gt;&lt;th&gt;new&lt;br /&gt;(4 threads)&lt;/th&gt;&lt;th&gt;new&lt;br /&gt;(8 threads)&lt;/th&gt;&lt;/tr&gt;&lt;tr&gt;&lt;th&gt;inproc&lt;/th&gt;&lt;td align=&quot;right&quot;&gt;6.23&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;8.47&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;6.56&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;9.93&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;11.0&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;11.2&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;th&gt;ipc&lt;/th&gt;&lt;td align=&quot;right&quot;&gt;15.7&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;22.6&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;27.7&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;29.1&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;31.3&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;31.0&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;th&gt;tcp&lt;/th&gt;&lt;td align=&quot;right&quot;&gt;24.8&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;50.5&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;41.0&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;42.7&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;42.9&lt;/td&gt;&lt;td align=&quot;right&quot;&gt;42.9&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;div&gt;&lt;br /&gt;All in all, the round trip times are reasonably respectable. I am especially proud of how close I&#39;ve come within the best inproc time -- a mere 330 nsec separates the Go implementation from the nanomsg native C version. &amp;nbsp;When you factor in the heavy use of go routines, this is truly impressive. &amp;nbsp; To be honest, I suspect that most of those 330 nsec are actually lost in the extra data copy that my inproc implementation has to perform to simulate the &quot;streaming&quot; nature of real transports (i.e. data and headers are not separate on message ingress.)&lt;br /&gt;&lt;br /&gt;There&#39;s a sad side to story as well. &amp;nbsp;TCP handling seems to be less than ideal in Go. &amp;nbsp;I&#39;m guessing that some effort is done to use larger TCP windows, and Nagle may be at play here as well (I&#39;ve not checked.) Even so, I&#39;ve made a 20% improvement in latencies for TCP from my first pass.&lt;br /&gt;&lt;br /&gt;The other really nice thing is near linear scalability when threads (via bumping GOMAXPROCS) are added. &amp;nbsp;There is very, very little contention in my implementation. &amp;nbsp;(I presume some underlying contention for the channels exists, but this seems to be on the order of only a usec or so.) &amp;nbsp;Programs that utilize multiple goroutines are likely to benefit well from this.&lt;br /&gt;&lt;h2&gt;&lt;/h2&gt;&lt;h2&gt;Conclusion&lt;/h2&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;Simplifying the code to avoid certain indirection (extra passes through additional channels and goroutines), and adding a message pooling layer, have yielded enormous performance gains. &amp;nbsp;Go performs quite respectably in this messaging application, comparing favorably with a native C implementation. &amp;nbsp;It also benefits from additional concurrency.&lt;br /&gt;&lt;br /&gt;One thing I really found was that it took some extra time to get my layering model correct. &amp;nbsp;I traded complexity in the core for some extra complexity in the Protocol implementations. &amp;nbsp;But this avoided a whole other round of context switches, and enormous complexity. &amp;nbsp;My use of linked lists, and the ugliest bits of mutex and channel synchronization around list-based queues, were removed. &amp;nbsp;While this means more work for protocol implementors, the reduction in overall complexity leads to marked performance and reliability gains.&lt;br /&gt;&lt;br /&gt;I&#39;m now looking forward to putting this code into production use.&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='' title='Post Comments'/><link rel='replies' type='text/html' href=';postID=4903361205291680757' title='0 Comments'/><link rel='edit' type='application/atom+xml' href=''/><link rel='self' type='application/atom+xml' href=''/><link rel='alternate' type='text/html' href='' title='SP protocols improved again!'/><author><name>Garrett D&#39;Amore</name><uri></uri><email></email><gd:image rel='' width='32' height='32' src='//'/></author><thr:total>0</thr:total></entry></feed>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

  1. Download the "valid Atom 1.0" banner.

  2. Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)

  3. Add this HTML to your page (change the image src attribute if necessary):

If you would like to create a text link instead, here is the URL you can use:

Copyright © 2002-9 Sam Ruby, Mark Pilgrim, Joseph Walton, and Phil Ringnalda