MadMode http://www.madmode.com/ Dan Connolly's tinkering notebook en-us Mon, 19 Feb 2024 17:27:31 +0000 Mon, 19 Feb 2024 17:27:31 +0000 Ubuntu 5.10, archive.org, and .torrent files http://www.madmode.com//2023/archive-ubuntu-torrent/ <p>I'm ready to say goodbye to my copy of <a href="https://archive.org/details/ubuntu-5.10-pc">Ubuntu 5.10 for i386</a> on CD, after nearly 2 decades of keeping it as a combination keepsake and software supply chain anchor. I donated it to archive.org:</p> <p><a href="https://archive.org/details/ubuntu-5.10-pc"><img alt="Ubuntu 5.10" src="https://ia804507.us.archive.org/14/items/ubuntu-5.10-pc/__ia_thumb.jpg" /></a></p> <p>While brainstorming about Merkle trees for file access, I noticed that not only does archive.org OCR the PDF I gave them of the cover and support browsing of the contents of <a href="https://ia904507.us.archive.org/view_archive.php?archive=/14/items/ubuntu-5.10-pc/Ubuntu%205.10%20i386.iso">Ubuntu 5.10 i386.iso</a>, but they provide <a href="https://archive.org/download/ubuntu-5.10-pc/ubuntu-5.10-pc_archive.torrent">ubuntu-5.10-pc_archive.torrent</a>, which means I can have high-performance access to the the full contents of the CDs for just 29k of storage. And brave supports <code>.torrent</code> files natively with WebTorrent (<a href="https://webtorrent.io/intro">WebTorrent Tutorial</a> looks pretty straightforward.)</p> <p>But what's in that <code>.torrent</code> file? Aha! <a href="https://en.wikipedia.org/wiki/Bencode">bencode</a> from <a href="https://www.bittorrent.org/beps/bep_0003.html#bencoding">BEP 3</a>! I've heard of it in <a href="https://github.com/ocapn/ocapn/issues/1#issuecomment-819652299">OCapN discussion</a> but didn't realize it comes from bittorrent. <a href="https://www.nayuki.io/page/bittorrent-bencode-format-tools">BitTorrent bencode format tools</a> is really handy, including stopping in a debugger to see the details:</p> <p><img alt="image" src="https://github.com/dckc/madmode-blog/assets/150986/39fc8f12-4679-4fe3-91df-30814143c001" /></p> <p><a href="https://github.com/johngunderman/haskell-torrent/blob/0385d0d4af9778053317d2b7725ef027ae870daf/src/Protocol/BCode.hs#L60-L65">BCode.hs from haskell-torrent</a> has a crisp specification:</p> <pre><code class="language-hs">-- | BCode represents the structure of a bencoded file data BCode = BInt Integer -- ^ An integer | BString B.ByteString -- ^ A string of bytes | BArray [BCode] -- ^ An array | BDict (M.Map B.ByteString BCode) -- ^ A key, value map deriving (Show, Eq) </code></pre> <p>...</p> <pre><code class="language-hs">-- | Return the hash of the info-dict in a torrent file hashInfoDict :: BCode -&gt; IO Digest hashInfoDict bc = do ih &lt;- case info bc of Nothing -&gt; fail &quot;Could not find infoHash&quot; Just x -&gt; return x let encoded = encode ih digest $ L.fromChunks $ [encoded] </code></pre> <p>Playing with <a href="https://www.npmjs.com/package/parse-torrent">parse-torrent</a> in a <a href="https://stackblitz.com/edit/stackblitz-starters-gcvlq7">parse-torrent-ubuntu-5.10 project on StackBlitz</a> is handy in that it shows the <code>infoHash</code>, <code>b890d2e1174a809d1cd0437de30400c542e0a939</code>, but its JSON output misled me about the real structure: there is no <code>infoHash</code> key in the file; there's an <code>info</code> dictionary that gets hashed.</p> <p>Say... Ubuntu offers bittorrent as a download option; maybe they keep a 5.10 <code>.torrent</code> file around? I didn't find one from them, but I did find:</p> <ul> <li><a href="https://archive.org/details/Ubuntu-5.10">Ubuntu 5.10 (Breezy Badger) : Canonical Ltd., Ubuntu community : Internet Archive</a><br /> Source <a href="https://archive.org/search.php?query=source%3A%22torrent%3Aurn%3Asha1%3A329a357ebd51db73417e1ad767b041291f598ae8%22">torrent:urn:sha1:329a357ebd51db73417e1ad767b041291f598ae8</a> Addeddate: 2017-06-20 14:16:31 Identifier: Ubuntu-5.10</li> </ul> <p>Note the <strong>Source</strong>; yes, <a href="https://help.archive.org/help/archive-bittorrents/">Internet Archive ingests BitTorrents</a>.</p> <p>Somehow my <code>Ubuntu 5.10 i386.iso</code> is 632,262 kb, which is 300 kb larger than theirs (631,962 kb). Maybe some <a href="https://gitlab.gnome.org/GNOME/gnome-disk-utility/-/issues/321">unused space captured by gnome-disk-utility</a> when I ripped the CD?</p> Thu, 21 Dec 2023 00:00:00 +0000 http://www.madmode.com//2023/archive-ubuntu-torrent/ walk-n-talk? http://www.madmode.com//2023/walk-n-talk/ <p>walk-n-talk?</p> <p>That's short for: would you like to talk with me as I take my dog on a walk?</p> <p>When the pandemic hit, I traded in my commute to <a href="/label/cube%20life/">KU Med Center</a> for a walk with my dog Mojo.</p> <figure> <img alt="my dog Mojo on a walk in the fall in KC" src="https://lh3.googleusercontent.com/pw/ADCreHchkU0fj04I0dsci4RXy-w4nbcVRW7yG7ntZ3EHb-6wI-_Ewp0gLPVGwcQfLg89aTncmoGfUhL1ipecUI8DJKpa-WkMdPWptdE4LXZUANLT4ZxvCAje7GldCUreHtXrXEfVTdvyDLOZ6Oi8UetRq-GV=w616-h765-s-no-gm" width="400" /> </figure> <p>While we're out walking, I like to talk with a friend, family member, or some colleagues. Mojo really likes it because I lose track of time and she gets to smell everything more.</p> <p>Usually it's a plain 1-1 phone call (or <a href="https://support.signal.org/hc/en-us/articles/360007060492-Voice-or-Video-Calling">signal call</a>). But sometiems it's a group thing. It's great to talk-real time with colleagues around the globe, but sometimes the tech reminds you that there's nothing like being there:</p> <ul> <li>Zoom: the dominant player. Pretty reliable</li> <li><a href="https://meet.jit.si/">Jitsi Meet</a>: based on open tech. Also reliable in my experience. The main drawback is that recording isn't as well integrated.</li> <li>Discord void: disappointing, especially since I gather the reason it was created was to provide a voice side-channel for gaming.</li> </ul> <p><a href="/contact/">Contact me</a> if you're interested to chat some time.</p> Sat, 18 Nov 2023 00:00:00 +0000 http://www.madmode.com//2023/walk-n-talk/ Office Hours: Patterns of Cooperation without Vulnerability http://www.madmode.com//2023/office-hours/ <p>One of my favorite patterns of cooperation is <dfn>Office Hours</dfn>. I open my office, virtually, and welcome questions and other open discussion, mostly about the way <a href="http://erights.org/">object capabilities</a> can be used to form patterns of cooperation without vulnerabilities, but also wider discussion of Web Architecture, open source, project management... and occasionally, to fill gaps, guitar :)</p> <p><a href="https://github.com/Agoric/agoric-sdk/wiki/Office-Hours">Agoric Dev Office Hours</a> on Wednesdays is the main series these days.</p> <figure> <a href="https://github.com/Agoric/agoric-sdk/discussions/8489"> <img alt="Dan's talking head next to an Agoric web page" src="https://user-images.githubusercontent.com/150986/284014913-a1623da3-d9ba-4da4-abb5-e1a96fbabdbc.png" /></a> </figure> <p><a href="https://github.com/Agoric/agoric-sdk/discussions/categories/office-hours?discussions_q=is%3Aopen+category%3A%22Office+Hours%22+label%3Avideo-recording">Notes with video recordings</a> are the norm. <a href="https://github.com/Agoric/agoric-sdk/discussions/categories/office-hours?discussions_q=is%3Aopen+category%3A%22Office+Hours%22+">All episodes</a> have at least a tab dump of the links discussed in the session (<em>thanks to the <a href="https://chrome.google.com/webstore/detail/tabcopy/micdllihgoppmejpecmkilggmaagfdmb">TabCopy</a> extension</em>).</p> <p>There's a lot going on at Agoric!</p> <ul> <li><a href="https://community.agoric.com/t/big-news-at-agoric-appjam-at-cosmoverse-liquid-staked-atom-for-vaults/507">Big News at Agoric: AppJam at Cosmoverse, Liquid Staked ATOM for Vaults - Agoric Community Forum</a> Oct 15</li> </ul> <p>I started doing Saturday <a href="https://github.com/rchain/bounties/issues/403">RChain community building office hours: Story Telling and Test Cases</a> in 2018. but as the <a href="/2019/rchain-blues">RChain Blues</a> set in, I re-scoped my Saturday time as <a href="https://github.com/dckc/awesome-ocap/issues/22">awesome-ocap office hours</a>.</p> <p>I haven't had critical mass on a Saturday morning for a while, so I join <a href="https://fossandcrafts.org/hack-and-craft/">Hack &amp; Craft</a> now and then. Darn... this is an off week; they only run 1st and 3rd Saturdays of the month.</p> Sat, 18 Nov 2023 00:00:00 +0000 http://www.madmode.com//2023/office-hours/ Toward capabilities all the way down with Genode on a Thinkpad http://www.madmode.com//2023/genode-thinkpad-dual-boot/ <p>In this year's holiday lull, I got closer to a "capabilities all the way down" workstation using <a href="https://genode.org/">Genode</a>, an OS framework with capability-based security.</p> <p>Sculpt is a Genode-based general-purpose OS.</p> <p><img alt="Sculpt system overview" src="https://genode.org/documentation/articles/sculpt_overview.png" /></p> <p>Genode has a choice of low level microkernels at the bottom. Sculpt uses the <a href="http://hypervisor.org/">NOVA microhypervisor</a>:</p> <blockquote> <p>... the NOVA microhypervisor uses a capability-based authorization model and provides only basic mechanisms for virtualization, spatial and temporal separation, scheduling, communication, and management of platform resources.</p> </blockquote> <p>NOVA isn't formally verified like seL4, but it's tiny (9,000 LOC):</p> <blockquote> <p><img alt="figure 1" src="https://user-images.githubusercontent.com/150986/210296418-9ce6f1e6-ce31-4328-96e9-8009a815d7cc.png" /><br /> Figure 1: Comparison of the TCB size of virtual environments. -- <a href="http://hypervisor.org/eurosys2010.pdf">Steinberg 2010</a></p> </blockquote> <p>... not to mention mature: that peer-reviewed release was in 2010 and the <a href="https://github.com/alex-ab/NOVA/blob/a34076e/doc/specification.pdf">spec</a> last changed in 2014. It's clearly trustworthy enough for my hobby usage.</p> <p>For reference, <a href="https://github.com/genodelabs/genode/blob/sculpt-22.10/repos/base-nova/ports/nova.port">genode sculpt-22.10 <code>ports/nova.port</code></a> points to <a href="https://github.com/alex-ab/NOVA/tree/a34076e">NOVA a34076e</a>, modified Sep 29. So they do continue to make the occasional tweak here and there to the C++ code... prompted by Sculpt usage, I suspect.</p> <p>The Sculpt release notes include a tutorial on how to boot it from a USB drive and play around with it; I managed to get that far back in 2018. This time, I got a more clear understanding of how <a href="https://genode.org/documentation/articles/sculpt-22-10#Making_customizations_permanent">persistent configuration</a> on a partition of the USB drive works. A big clue is that the built-in "inspect" view is plenty for command-line file manipulation; there's no need to install and configure all the components of a shell.</p> <p>I wanted to reproduce Schlatow's results from <a href="https://genodians.org/jschlatow/2021-04-23-start-existing-linux-from-sculpt">Starting an existing Linux installation from Sculpt</a>. I managed to</p> <ol> <li><a href="https://github.com/dckc/madmode-blog/issues/49#issuecomment-1356447232">partition the hard drive</a><ul> <li>considered <a href="https://github.com/nix-community/disko">nix declarative disk partitioning with disko</a> a la <a href="https://lobste.rs/s/aamjm7/setting_up_my_new_laptop_nix_style">McGee's notes</a><ul> <li>learned a bit more about <a href="https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix3-flake.html">nix flakes</a></li> <li>tried to make my own flake; re-discovered <a href="https://lobste.rs/s/ff54p1/how_nix_nixos_get_so_close_perfect#c_po5s5h">my problem with nix</a>: unlike <code>apt</code> where if you get an option wrong, some C code tells you that you got an option wrong, nix just passes the wrong option down into interpreted code where you eventually get “string found where integer expected” or some such. As Phil Karlton would say, "yet another interpreted language without a debugger."</li> </ul> </li> </ul> </li> <li>install linux on the internal SDD</li> <li>install genode on another partition<ul> <li>though I couldn't get the grub config to work automatically</li> </ul> </li> </ol> <p>and I downloaded the components to run a VM, but when I tried to start it up, it couldn't find <code>/machine.vbox6</code>:</p> <pre><code>[runtime] child &quot;vbox6&quot; [runtime] RAM quota: 4402952K [runtime] cap quota: 7966 ... [runtime -&gt; vbox6 -&gt; vbox] main Executable: /virtualbox6 [runtime -&gt; vbox6 -&gt; vbox] Error: failed to init machine from settings [runtime -&gt; vbox6 -&gt; vbox] Runtime error opening '/machine.vbox6' for reading: -102 (File not found.). [runtime -&gt; vbox6 -&gt; vbox] /data/depot/genodelabs/src/vbox6/2022-10-11/src/virtualbox6/src/VBox/Main/src-server/MachineImpl.cpp[499] (nsresult Machine::initFromSettings(VirtualBox*, const com::Utf8Str&amp;, const com::Guid*)) [core] attempted exec at non-executable memory (EXEC pf_addr=0x271dd78 pf_ip=0x271dd78 from pager_object: pd='init -&gt; runtime -&gt; vbox6 -&gt; vbox' thread='ep') [core] page fault, pd='init -&gt; runtime -&gt; vbox6 -&gt; vbox' thread='ep' cpu=0 ip=0x271dd78 address=0x271dd78 stack pointer=0x403fe6b8 qualifiers=0x15 IrUwP reason=3 [core] no RM attachment (READ pf_addr=0x0 pf_ip=0x16bb302 from pager_object: pd='init -&gt; runtime -&gt; vbox6 -&gt; vbox' thread='Watcher') [core] page fault, pd='init -&gt; runtime -&gt; vbox6 -&gt; vbox' thread='Watcher' cpu=0 ip=0x16bb302 address=0x0 stack pointer=0x409feb00 qualifiers=0x4 irUwp reason=1 </code></pre> <p>After a <a href="https://www.diigo.com/user/dckc-madmode">a bit of diigoing</a>, I came to the conclusion that there's a significant gap in my education around using VirtualBox in general, never mind in Genode.</p> Tue, 03 Jan 2023 00:00:00 +0000 http://www.madmode.com//2023/genode-thinkpad-dual-boot/ Divesting from Flickr in the Annual File Purge http://www.madmode.com//2023/file-purge-flickr-divest/ <p>I spent much of this year's annual file purge recovering from <a href="https://blog.flickr.net/en/2022/04/19/update-free-account-limit-changes-and-enforcement-start-may-1-2022/">Flickr going back on their 1TB storage offer</a>.</p> <p>While <a href="https://github.com/dckc/madmode-blog/issues/49">tinkering with Genode</a> and catching up on Metamath (RIP, Norm Megill), I made a lot of use of github issues as my lab notebook; I can search copies of my comments in my email since <a href="https://www.madmode.com/2021/closet-librarian-approach-cloud-services">I'm a closet librarian and I don't trust cloud services completely</a>. One of these searches led me to the pile of monthly <strong>"account in violation free account limits"</strong> nasty-grams building up since May:</p> <ul> <li><a href="https://blog.flickr.net/en/2022/04/19/update-free-account-limit-changes-and-enforcement-start-may-1-2022/">Update: Free account limit changes and enforcement start May 1, 2022. \| Flickr Blog</a></li> </ul> <p>Back in 2015, I mostly knew better, but I did take them up on their terabyte storage offer:</p> <blockquote> <p>My photostream on flickr goes back to <a href="https://www.flickr.com/photos/dckc/archives/date-posted/2004/12/calendar/">Dec 2004</a> when it was big in the open web community. I could never bring myself to go premium, but in May 2013 when they announced the terabyte storage offer, I dusted it off. - <a href="../2015/photo-flickr-explore.html">MadMode: Syncing a 5 Year iPhoto Library with flickr</a></p> </blockquote> <p>I have about 50GB of files from a Flickr data request Feb 17, 2019 on an external SSD. I didn't take the time to keep the private data separate from the code and other detailed notes, but briefly, I</p> <ol> <li>verified access to 47GB of data from a March 2019 Flickr data request (<code>72157706876334384_ff8f2206a90f_part1.zip</code> etc.) by copying it from an external SSD to an internal NVMe.<ul> <li>Why did that take 20 minutes? Oops... I used a USB2 cable and so lost USB3 "SuperSpeed"</li> </ul> </li> <li>verified that I can recover a favorite album from iPhoto<ul> <li>dealt with the fact that the <code>photo_NNN.json</code> files don't actually contain the name of the <code>abc_NNN_xyz.jpg</code> media files</li> <li>joined the flickr ids with iPhoto ids using records from the 2015 upload process<ul> <li>used nix to bring up a juypter notebook environment with the relevant goodies: <code>nix-shell -p "python3.withPackages(ps: with ps; [ipython jupyter numpy pandas pillow flickrapi progress crc32c])"</code></li> </ul> </li> <li>made a simple HTML list of links to photos</li> <li>reverse-engineered the way Web-iPhoto would make an album of those photos from iPhoto files:<ul> <li>wrote out <code>albums</code> and <code>photos</code> JSON<ul> <li>discovered the README docs were incomplete and the code needs <code>tree</code> too.</li> </ul> </li> </ul> </li> </ul> </li> <li>verified that I can recover a favorite keyword from Apple photos<ul> <li>reified the keyword as a directory with symlinks to the relevant photo files</li> <li>toured <a href="https://simonwillison.net/2020/May/21/dogsheep-photos/">simonwillison's dogsheep-photos work</a> and <a href="https://github.com/RhetTbull/osxphotos">osxphotos</a> while decoding the Apple photos database.</li> <li>evaluated <a href="https://photoprism.app/">photoprism</a>, an "AI-Powered Photos App for the Decentralized Web" in hopes that open source AI would help me curate interesting photos the way Apple's applied <a href="https://machinelearning.apple.com/research/recognizing-people-photos">computer-vision research</a> did for Simon<ul> <li>wow! nicely packaged!</li> <li>bulk import with move option for canonical naming: <code>2015/10/20150510_015146_88F59DFB.jpg</code><ul> <li>that hash is a Castagnoli <a href="https://pypi.org/project/crc32c/">crc32c</a>, the one with hardware support, not the one from the python stdlib.</li> </ul> </li> </ul> </li> </ul> </li> <li>deleted all 10,000+ non-private photos using the <a href="https://stuvel.eu/software/flickrapi/">flickr API</a> so I'll stop getting those monthly nasty-grams.<ul> <li>learned to use the <a href="https://pypi.org/project/progress/">progress</a> package to see that it would take about an hour</li> </ul> </li> </ol> <p>I made lots of <a href="https://www.diigo.com/user/dckc-madmode">diigo bookmarks and annotations</a> too.</p> <h3>Footnote on Apple photos dates</h3> <p><a href="https://discussions.apple.com/message/27873467#27873467">Apple support discussion Apr 2015</a> gives us some clues about the database schema, which seems to be an <a href="https://en.wikipedia.org/wiki/Photos_(application)">Aperture</a> database (<code>apdb</code>).</p> <blockquote> <p>Aperture uses Core Data, which is a database-independent abstraction layer, and thus does not use the native SQLite encoding for dates (juliandate), but rather the NSDate format, which should be a double-precision number of seconds since the reference date (2001-01-01 00:00:00 GMT). -- <a href="https://majid.info/blog/aperture-internals/#comment-4860">majid 2011-05-03</a></p> </blockquote> Tue, 03 Jan 2023 00:00:00 +0000 http://www.madmode.com//2023/file-purge-flickr-divest/ Unbelievably good noise cancellation http://www.madmode.com//2022/09-amazing-headset/ <p>It was loud. I mean <strong>really</strong> loud. Like guns going off over my head. The noise from the nailguns and hammers from the hardwood floor installation going on right above my office.</p> <p>I apologized to the other folks in the meeting about the background noise, but since my business in the meeting was urgent and the noise wasn't going to get any better, I pressed on.</p> <p>They dind't hear it. At all. They reported my voice was coming through just fine.</p> <p>I still hardly believe it.</p> <p>Hats off to Corsair!</p> <p>And thanks, B!</p> <pre><code>$ lsusb | grep -i headset Bus 001 Device 018: ID 1b1c:0a17 Corsair Corsair VOID PRO USB Gaming Headset </code></pre> Fri, 09 Sep 2022 00:00:00 +0000 http://www.madmode.com//2022/09-amazing-headset/ Hello Spritely Institute! And Haunt. And Guix, again http://www.madmode.com//2022/spritely-haunt-guix/ <p>Hello, <a href="https://spritely.institute/">Spritely Institute</a>!</p> <blockquote> <p>... ActivityPub, social networks, smart contracts, object capabilities, and secure distributed virtual worlds. ... freely licensed open source ...</p> </blockquote> <p>Yum, yum, gimme some! Can't wait for mind-boggling stuff like this <a href="https://blog.pimaker.at/texts/rvc1/">VRChat hack</a>, but powered by ocaps and open source!</p> <iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/G2u7NOpzcBQ?controls=0&amp;start=5110" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> <blockquote> <p>Privacy &amp; moderation issues have triggered the permanent shutdown of millions of networked communities and the destruction of their relationships and artifacts.</p> </blockquote> <p>I was just talking with <a href="https://twitter.com/simonw">@simonw</a> about <a href="https://twitter.com/dckc/status/1479108723494699021">portable posts using ERTP</a>. More on that later...</p> <p>Nice <a href="https://spritely.institute/">site</a> btw... shiny!</p> <blockquote> <p>Powered by <a href="https://dthompson.us/projects/haunt.html">Haunt</a>.</p> </blockquote> <p>Keep talking... looks clean...</p> <blockquote> <p>gives authors the ability to treat websites as Scheme programs.</p> </blockquote> <p>Or maybe Scheme programs that are also <a href="https://agoric.com/documentation/guides/js-programming/hardened-js.html">Hardened JavaScript</a> programs, using James / Jessie on <a href="https://github.com/cwebber/rockit">rockit</a>?</p> <p>Let's give it a spin as-is before stretching it...</p> <blockquote> <p>To install Haunt ..., run: <code>guix install haunt</code></p> </blockquote> <p>Ah yes... <a href="https://github.com/dckc/madmode-blog/issues/144">guix for ocaps at the systemd level</a> is on my wish-list too. I have <code>guix</code> on my Ubuntu workstation, don't I? Let's see... <code>guix install haunt</code>... <strong title="haunt 0.2.5 /gnu/store/5pk6cwrh11bgchm68phj556srvcvdlvb-profile.drv">bingo!</strong></p> <h2>Catching up with Guix after 55 days</h2> <p>meanwhile...</p> <pre><code>guix install: warning: Your Guix installation is 55 days old. guix install: warning: Consider running 'guix pull' followed by 'guix package -u' to get up-to-date packages and security updates. </code></pre> <p>so...</p> <pre><code>15:38 jambox$ guix pull Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'... Authenticating channel 'guix', commits 9edb3f6 to 2dfbd03 (5,405 new commits)... ... </code></pre> <p>Ugh... taking a while... I wonder what it's doing, so I hop over to <a href="https://matrix.to/#/#guix:libera.chat">#guix</a> and learn that the 5K commits include a big merge recently. I'm just about to give up on it when it comes back to life...</p> <pre><code>185.4 MB will be downloaded building /gnu/store/7y8wijc8zmbf8il1yzrv4ivmggi5zx7i-compute-guix-derivation.drv... Computing Guix derivation for 'x86_64-linux'... News for channel 'guix' Icedove 91: profile folder moved to `~/.thunderbird' `gdm-service-type' now supports Wayland 16:10 jambox$ guix package -u The following derivation will be built: /gnu/store/gpf86rpdl5k21v65xdshv7qdypwc3w89-profile.drv 33.2 MB will be downloaded 16:15 jambox$ guix install haunt The following derivation will be built: /gnu/store/kciishw2a3xb52ql3m55m3g16dry7p0h-profile.drv </code></pre> <p>Thansk for the "News..." bit. The editorial discretion shows user-centered design. <code>guix search</code> is another breath of fresh air in contrast to the way <code>apt</code> and <code>nix</code> seem to grudgingly provide a search feature. Likewise:</p> <pre><code>hint: Consider setting the necessary environment variables by running: GUIX_PROFILE=&quot;/home/connolly/.guix-profile&quot; . &quot;$GUIX_PROFILE/etc/profile </code></pre> <p><code>guix install emacs</code> gives 27.2 (2021) where Ubuntu gives 26.3 (2019). And <code>guix install gnucash</code> has the new sqlite3 and postgres support. Yes, this could be my tribe. I can't go all in yet because I use Brave and I think there's a reason guix doesn't support it.</p> <p>Now let's pop back...</p> <h2>Blogging with Haunt eval, cont.</h2> <p>Stubbed my toe on <code>ERROR: In procedure setlocale: Invalid argument</code>. But <code>guix search locale</code> guided me nicely to <code>guix install glibc-locales</code>. Then <code>asset processing failed with errno: "images" 2</code> because the homepage has <code>(static-directory "images")</code> in the example config but neglects <code>mkdir images</code>. (The tutorial doesn't have this bug.) Now we're rolling:</p> <pre><code>09:48 connolly@jambox$ haunt serve serving site on port 8080 </code></pre> <p>So close! If it were <code>serving site on http://localhost:8080</code> I could just follow the link.</p> <blockquote> <p>First post! by Eva Luator</p> </blockquote> <p>There's my tribe again :)</p> <p>But I doubt netlify supports guix as well as they support python and node.js. Here's hoping they do it well enough...</p> Sat, 08 Jan 2022 00:00:00 +0000 http://www.madmode.com//2022/spritely-haunt-guix/ Closet Librarian Approach to Cloud Services http://www.madmode.com//2021/closet-librarian-approach-cloud-services/ <p>A colleague suggested we shouldn't delete calendar items of old meetings. I said I don't rely on my calendar for records of the past. Then I admitted that actually, I keep version-controlled backups of my cloud-hosted calendars.</p> <p>I'm content to use the <a href="https://support.google.com/calendar/answer/37111?hl=en">Google calendar export feature</a> manually. I tend to do it in preparation for travel or perhaps when reviewing a trip:</p> <pre><code>gdata$ hg log10 790:9efa3d723bb1 2021-10-16 before SFO trip 789:03f577c7a24e 2021-08-08 RedRocks Road Trip KML data from Google Timeline </code></pre> <p>In general, I don't trust any cloud services. I keep backups of linkedin contacts, tweets, etc.</p> <p>I'm much more comfortable using github issues to capture my thoughts since I discovered "Include your own updates" in the <a href="https://docs.github.com/en/account-and-profile/managing-subscriptions-and-notifications-on-github/setting-up-notifications/configuring-notifications#customizing-your-email-notifications">email notification settings</a>.</p> <p>I generally follow the <a href="https://indieweb.org/PESOS">IndieWeb PESOS</a> pattern: Publish Elsewhere, Syndicate (to your) Own Site rather than <a href="https://indieweb.org/POSSE">POSSE</a>: Publish (on your) Own Site, Syndicate Elsewhere. The cost of keeping backups in an open format is manageable (I avoid services that don't support it) but economies of scale naturally result in rich knowledge capture and sharing user interfaces for large communities--why should I limit myself to the writing tools I can maintain on my own site?</p> <p>The ownership risk seems manageable: in large communities including many of my peers, I rely on them to alert me to poor terms of service, and when I venture out on new ground, I read the TOS myself fairly carefully.</p> <p>PESOS also gives me editorial discretion over which syndicated copies go on the front page of my blog and which go on backups in a closet.</p> Sat, 18 Dec 2021 00:00:00 +0000 http://www.madmode.com//2021/closet-librarian-approach-cloud-services/ What's Next: Agoric Computing http://www.madmode.com//2021/next-gig-agoric/ <p>After 15 years at W3C and 10 years at KU Med Center, my next gig is at <a href="https://agoric.com/">Agoric</a>. Here I answer some questions, some recently asked and some anticipated.</p> <p>Q: I see a <a href="https://www.nytimes.com/2021/01/10/technology/tim-berners-lee-privacy-internet.html">NY Times feature</a> about Tim Berners Lee and his new company, Inrupt. Did you work with Tim Berners-Lee?<br /> A: yes, from the early 1990s to 2010 we worked together building the Web at W3C.</p> <p>Q: Are you working with Tim at Inrupt?<br /> A: No, but I am starting a new job at Agoric.</p> <p>Q: What is Agoric? What do they do?<br /> A: Agoric provides a safer, simpler way to program smart contracts. We believe smart contracts enable the future of global economic cooperation.</p> <p>Q: Why is it called "Agoric"?<br /> A: <em>Agoric</em> stems from <em>agora</em>, the Greek term for a meeting and market place. An agoric system is a software system using market mechanisms. (<a href="https://agoric.com/papers/markets-and-computation-agoric-open-systems/abstract/">Miller and Drexler, 1988</a>)</p> <p>Q: Smart contracts? What's that?<br /> A: If you've used amazon, ebay, or a vending machine, you've used a smart contract: a contract-like arrangement expressed in software, where the behavior of the software enforces the terms of the contract.</p> <p>At W3C, a big part of my job was developing the W3C standards process and figuring out how much of it could and should automate; looking back, I think of it as smart contract design.</p> <p>Q: Are these smart contracts for blockchains and cryptocurrencies like Bitcoin and Ethereum?<br /> A: Yes, Agoric plans to build on the high-integrity shared compute infrastructure provided by blockchains, though the architecture scales down to private clusters and single machines as well. (<a href="https://www.youtube.com/watch?v=iyuo0ymTt4g&amp;list=PLhuBigpl7lqth_Ow_eQWZs7NFxmeDw9W8">Miller, 2019</a>)</p> <p>Agoric is not building on Bitcoin or Ethereum directly, but we are building on mature blockchain technology, the Cosmos <abbr title="Software Development Kit">SDK</abbr>. In order to bridge to Ethereum and Bitcoin, Agoric is a leading contributor to <a href="https://cosmos.network/ibc">IBC</a>, the Inter-Blockchain Communication protocol.</p> <p>Q: How are smart contracts safer using Agoric's technology?<br /> A: At least three ways:</p> <ol> <li>Agoric provides <strong>offer safety</strong>: when you place an offer, either you'll get what you wanted or you get a refund, regardless of the (mis)behavior of the underlying smart contract. This API (Zoe) is built using a couple of more fundamental mechanisms:</li> <li>Agoric supports patterns of cooperation without vulnerability using <strong>object capabilities</strong> (OCaps).</li> <li>Agoric avoids reentrancy hazards (such as the $50M DAO bug) using asynchronous <strong>eventual-sends</strong>.</li> </ol> <p>Q: Do developers have to learn a new programming language to get all this?<br /> A: No; Agoric smart contracts are written in a <a href="https://github.com/Agoric/Jessie#subsetting-ecmascript">secure subset of JavaScript</a> that mostly involves sticking to established best practices.</p> <p>Q: Object capabilities? What's that?<br /> A: Capability-based security is like the way we control access to our cars: I use a key to drive my car. I can delegate driving the car to you by handing you the key. But keys are impractical to forge, so effectively, unauthorized people can't drive it.</p> <p>In contrast, the traditional way to control access to computing resources is with access control lists (ACLs): each file or database table has a list of who can read and write it. If cars worked this way, the car would let me drive it only if I present the right driver's license; perhaps my wife would be on the list too. But if I wanted to delegate to you, I'd have to update the list of drivers in the car. And what if my son got hurt and I wasn't around to update the list of drivers so his friend could get him to the doctor? (<a href="http://erights.org/talks/efun/SecurityPictureBook.pdf">Stiegler, 2004</a>)</p> <p>Valet parking would be pretty tedious. And the trunk would have to have a separate access control list to do what valet keys do.</p> <p>Worse: what we normally do when we run programs on our computers is like giving my driver's license to you or the valet to let you drive the car. I have to run the risk that you'll do other things with the driver's license like open a bank account in my name. Maybe I trust you not to do that, but every program on my computer can do anything I can do, including mess up all my files and demand a ransom to get them back. And with our computers connected to millions of other computers via the Internet, we're vulnerable to more than just our friends. With capabilities, making things like valet keys is easy so that each program, and each part of a program, gets access to only what it needs in order to do its job; capabilities support the <strong>principle of least authority</strong> much better than access control lists. (<a href="https://www.hpl.hp.com/techreports/2009/HPL-2009-20.html">Close, 2009</a>)</p> <p>I have been excited about capability-based security since 2001 when I discovered OCaps and composable smart contracts(<a href="http://erights.org/elib/capability/ode/index.html">Miller, Morningstar, Frantz, 2000</a>). Since 2010 I have been responsible for the security of a million or so health records in a clinical data research warehouse at KU Med Center. Being constrained to use ACL-based filesystems, databases, and web applictions drives me crazy! The chance to work on smart contracts with OCaps as a day job is a dream come true.</p> <p>Q: And Berners-Lee's Inrupt? What do they do?<br /> A: Inrupt "aims to reset the balance of power on the web" by giving users control of their data in "pods," personal online data stores. "Each person could control his or her own data — websites visited, credit card purchases, workout routines, music streamed — in an individual data safe, typically a sliver of server space" (<a href="https://www.nytimes.com/2021/01/10/technology/tim-berners-lee-privacy-internet.html">NY Times Jan 10</a>). It uses SOLID, a set of open technologies.</p> <p>Q: What do you think of Inrupt and SOLID?<br /> A: I certainly agree when Tim says that "too much power and too much personal data reside with the tech giants like Google and Facebook". But</p> <ol> <li>SOLID Web Access Control (<a href="https://github.com/solid/web-access-control-spec/">WAC</a>) uses ACLs, not capabilities.</li> <li>I don't see an integrated economic model in SOLID.</li> </ol> <p>To free users from Google, we have to provide the same sub-second search for all of their email, and I don't see how to do that without bringing the application code to where the data is. We're going to want mash-ups of multiple applications. We need the kind of cooperation without vulnerability that only capability-based security brings. I wonder what would happen if we mixed the Agoric platform's ability to scale down to clusters and single machines with the notion of a SOLID pod.</p> <p>In 2005, I learned that Internet pioneer Dave Clark took a year off to study economics. Since then it has been pretty clear to me that whatever comes next in the architecture of the Internet and the Web, economics needs to be an integral part of the protocols. Bitcoin came along in 2008 and Ethereum in 2014. Mixing in economics increases the motivation for fraud, which has made me hesitant to commit to the cryptocurrency industry as a career. But the Agoric platform provides an increasing level of safety and most of the team has been working on smart contracts with capability security as long as I've been working on the Web, so I just can't pass up the opportunity to join them as they scale it up to global economic cooperation.</p> Mon, 11 Jan 2021 00:00:00 +0000 http://www.madmode.com//2021/next-gig-agoric/ Fun with @ski at Agoric "Hack the Orb" http://www.madmode.com//2020/agoric-hackathon-prize/ <blockquote class="twitter-tweet"> <p lang="en" dir="ltr">This was great fun... especially working with <a href="https://twitter.com/suhailski?ref_src=twsrc%5Etfw">@suhailski</a>. <a href="https://t.co/11uJri3Wvv">https://t.co/11uJri3Wvv</a> </p>&mdash; Dan Connolly (@dckc) <a href="https://twitter.com/dckc/status/1336061189013581826?ref_src=twsrc%5Etfw">December 7, 2020</a> </blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> <noscript> <blockquote> @agoric · Dec 7 <p> 2/ 📽️ Umqhele: @dckc, @suhailski, and @tgrecojs built a #dapp that allows you to auction and trade #NFTs that grant access to live video streams. This dapp demonstrates Agoric contract composition, offer safety, and object capabilities. https://github.com/ski/umqhele </p> 3:33 PM · Dec 7, 2020·Twitter Web App </blockquote> </noscript> <blockquote> Suhail Manzoor @suhailski · Dec 7 Replying to @dckc <p> The pleasure was all mine Dan :) </p> </blockquote> <p><a href="https://github.com/dckc/umqhele">dckc/umqhele</a> 20cdf68 on Nov 23 forked form <a href="https://github.com/ski/umqhele">ski/umqhele</a></p> <div class="h-event vevent"> <a href="https://www.hacktheorb.com/"> Hack the Orb: the Agoric and Chainlink Hackathon <time class="dt-start dtstart">November 6 - 21, 2020</time> </div> <div class="h-event vevent"> <a href="https://www.youtube.com/watch?v=wVrX06owyeY"> <b class="p-name summary">Agoric Monthly Community Call #3</b> </a> <time class="dt-start dtstart">2020-12-02</time> </div> <h3>Relevant people</h3> <ul> <li class="h-card hCard"><b class="fn">Dan Connolly</b> @dckc <em class="p-note">writing software to support health research; using capability security whenever I can</em></li> <li class="h-card"><b class="fn">Suhail Manzoor</b> @suhailski <em class="p-note">A cultural refugee in Edinburgh who is still trying to search and sort his way through life. Retweets are often endorsement but sometimes not.</em></li> <li class="h-card"><b>Agoric</b> @agoric <em class="p-note">We believe smart contracts enable the future of global economic cooperation. Agoric provides a safer, simpler way to program smart contracts.</em></li> </ul> Mon, 07 Dec 2020 00:00:00 +0000 http://www.madmode.com//2020/agoric-hackathon-prize/ Javascript on genode, genode on HP Envy 15 http://www.madmode.com//2020/genode-js-hp-envy/ <p>Genode is a microkernel operating system with capability-based security; in the 20.05 release they added <a href="https://genode.org/documentation/release-notes/20.05#Capability-based_security_using_seccomp_on_Linux">capability-based security using seccomp on Linux</a>. That got me excited enough to make some real progress on JavaScript on Genode using the Moddable XS engine and on Genode on an HP Envy laptop.</p> <p>In <a href="https://github.com/dckc/genode-js-xs">genode-js-xs</a> I got this JavaScript code running on genode:</p> <pre><code>export default function main() { let message = &quot;Hello, world - sample&quot;; trace(message + &quot;\n&quot;); } </code></pre> <p>It works like this, once it's built (see below):</p> <pre><code>goa-hello$ goa run [goa-hello:make] recursive make: make Genode 20.02-1-gac1b2ec24e 17592186044415 MiB RAM and 8997 caps assigned to init [init -&gt; goa-hello] Hello before libc [init -&gt; goa-hello] hello via stdio [init -&gt; goa-hello] Hello in libc [init -&gt; goa-hello] lin_xs_cli: loading top-level main.js [init -&gt; goa-hello] lin_xs_cli: loaded [init -&gt; goa-hello] lin_xs_cli: invoking main(argv) [init -&gt; goa-hello] Hello, world - sample [init -&gt; goa-hello] main() returned immediate value (not a promise). exiting [init -&gt; goa-hello] Warning: rtc not configured, returning 0 Warning: blocking canceled in entrypoint constructor [init] child &quot;goa-hello&quot; exited with exit value 0 </code></pre> <p>I started with the hello package from then <a href="https://genodians.org/nfeske/2019-11-25-goa">Nov 2019 article introducing goa</a>.</p> <p>Then I grabbed the <a href="https://github.com/Moddable-OpenSource/moddable/tree/public/examples/helloworld">helloworld</a> example from the Moddable XS SDK, generated C sources, and got it to build.</p> <p>Then I worked out getting <code>goa run</code> to work. My <code>artifacts</code> is a bit of a kludge: it reaches out from <code>var/build</code> back to <code>src/bin</code>.</p> <h3>before <code>goa build</code>: <code>gensrc</code>, <code>genode_platform</code></h3> <p>There's a bit of an impedence mismatch between <code>goa build</code> and the <a href="https://github.com/Moddable-OpenSource/moddable/">Moddable SDK</a>, so use <code>make -C src gensrc</code> to generate C etc. before running <code>goa build</code>.</p> <p>We also extend the Moddable SDK to add a <code>genode</code> platform using <code>make -C src genode_platform</code>.</p> <h3>Next Steps</h3> <ul> <li> <p><strong>event loop</strong>: Figure out how to integrate the <a href="https://github.com/genodelabs/genode-world/blob/master/ports/glib.port">port of glib to genode</a> to turn the event loop, which is currently commented out, back on.</p> </li> <li> <p><strong>nix and dhall</strong>: Compare goa to <a href="https://git.sr.ht/~ehmry/genodepkgs">genodepkgs</a></p> </li> </ul> <h2>NixOS 20.03 and Genode Sculpt 20.02 on the HP Envy 15</h2> <p>When my son upgraded his gaming laptop last year, I got his hand-me-down <a href="https://support.hp.com/us-en/product/hp-envy-15-j100-notebook-pc-series/5401187/model/6521450?sku=E1R44AV">HP ENVY 15z-j100</a> and tried to boot it from free software. I got Windows so messed up that it could neither boot nor repair itself, so I set it aside in frustration.</p> <p>Prompted by my success with js on genode, I tried a Ubuntu 20.04 USB stick (that I had made for my other son, who uses linux for his gaming PC) and lo, it worked the first time. Ubuntu has gone to the trouble to get their installation media working with secure boot.</p> <p><a href="https://github.com/NixOS/nixpkgs/pull/53901">secure boot for NixOS</a> looks really bleeding edge, but with a better understanding based on <a href="https://www.rodsbooks.com/efi-bootloaders/secureboot.html#hp705">EFI Boot Loaders for Linux: Dealing with Secure Boot</a> by Rod Smith, I managed to turn secure boot off and get NixOS installed.</p> <p>With all this momentum, I did not give up when booting <a href="https://genode.org/download/sculpt">Sculpt OS 20.02</a> quit with a just <a href="https://github.com/alex-ab/g2fg/issues/1">screen full of (?)s</a>. They only document support for Intel CPUs and this has an AMD A10-5750M, so it looked like an AHCI driver problem or some such. But the screen wasn't locked altogether. It responded to enter and to <code>clear</code>... aha! It's a grub prompt with a missing font. <code>terminal_output console</code> dealt with the font issue and the solution turned out to be changing <code>hd0</code> to <code>hd1</code> in a grub config file. Genode has no ath9k driver, so wifi isn't working, but I have plenty of ethernet ports and cables. :)</p> Sun, 07 Jun 2020 00:00:00 +0000 http://www.madmode.com//2020/genode-js-hp-envy/ Rosetta Stone: Taking propositions-as-types to the next level http://www.madmode.com//2020/grok-idris-ct/ <blockquote class="twitter-tweet"><p lang="en" dir="ltr">I read CS papers by turning notation into (<a href="https://twitter.com/idrislang?ref_src=twsrc%5Etfw">@idrislang</a>) code. Slow going, but the only way I really grok. e.g. <a href="https://t.co/8Y34KMgiXd">https://t.co/8Y34KMgiXd</a> more: <a href="https://t.co/AecWRmyFJG">https://t.co/AecWRmyFJG</a></p>&mdash; Dan Connolly (@dckc) <a href="https://twitter.com/dckc/status/1230157668888780800?ref_src=twsrc%5Etfw">February 19, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> <p>The <a href="https://arxiv.org/abs/0903.0340">Rosetta Stone</a> paper by Baez and Stay has been on my to-read list since a conversation with Stay on a bus a few years ago. The scope is a bit intimidating:</p> <table> <tr> <th>Category Theory</th> <th>Physics</th> <th>Topology</th> <th>Logic</th> <th>Computation</th> </tr> <tr> <td>object</td> <td>system</td> <td>manifold</td> <td>proposition</td> <td>data type</td> <tr> <td>morphism</td> <td>process</td> <td>cobordism</td> <td>proof</td> <td>program</td> </tr> <caption>Table 1: The Rosetta Stone (pocket version)</caption> </table> <p>But in <a href="https://blog.statebox.org/fun-with-categories-70c64649b8e0">Fun with Categories</a>, the Statebox folks laid the groundwork for grokking category theory with idris in their <a href="https://github.com/statebox/idris-ct">idris-ct</a> library.</p> <p>And the RChain community includes a <a href="https://discordapp.com/channels/257555930173014017/548583847235682304">#computational-calculi</a> channel where people get together to study such things... I get to fill in gaps from skipping grad school without all the tuition and tests :). Jake and several other people there immediately agreed when I suggested we read this paper together. Jake dug up <a href="http://math.ucr.edu/~mike/rosettaslides.pdf">Mike Stay's slides</a> and <a href="http://math.ucr.edu/~mike/rosetta%20slides.txt">notes</a> and it took just a little work to get the details into <a href="https://calendar.google.com/calendar/embed?src=2cj152c9nidh6glpr1d5g4eq28%40group.calendar.google.com">the calendar</a>. Jake connected us to the statebox folks via a <a href="https://categorytheory.zulipchat.com/">Category Theory</a> chat group started by Christian W.</p> <h2>Quantum Physics Gap</h2> <p>Of the five domains in the Rosetta Stone, Quantum Physics is definitely the most indimidating, to me. <a href="https://www.lesswrong.com/posts/hc9Eg6erp6hk9bWhn/the-quantum-physics-sequence">The Quantum Physics Sequence</a> by Yudkowsky in 2008 is the closest thing I have found so far to something my speed, but I think I made it only 1/3rd of the way thru.</p> <p>I had to look up <a href="https://en.wikipedia.org/wiki/Hilbert_space">Hilbert space</a>; fortunately my linear algebra and topology is fresh enough that "a real or complex inner product space that is also a complete metric space with respect to the distance function induced by the inner product" only took a few minutes to grok.</p> <p>I just watched <a href="https://youtu.be/cuJY14BYac4">a lecture by Edward Witten about knots and QM</a>, a recommended Tomaslov passed along from Greg. Quantifier noted <em>A Quantum Mechanics Primer</em> by D.T.Gillespie as highly regarded. Jake sai Baez has a <a href="http://math.ucr.edu/home/baez/books.html">good list of recommend resources</a>.</p> <p>Fab from Statebox recommended <a href="https://www.amazon.com/Picturing-Quantum-Processes-Diagrammatic-Reasoning-ebook/dp/B06XB1K71P">Picturing Quantum Processes</a>. Here's hoping...</p> <h2>Intro to Idris</h2> <p>I gave an intro to idris this week:</p> <blockquote> <p><a href="https://www.idris-lang.org/">Idris</a> is a programming language designed to encourage <em>Type-Driven Development</em>.</p> <p>In type-driven development, types are tools for constructing programs. We treat the type as the plan for a program, and use the compiler and type checker as our assistant, guiding us to a complete program that satisfies the type. The more expressive the type is that we give up front, the more confidence we can have that the resulting program will be correct.</p> <p>In Idris, types are first-class constructs in the langauge. This means types can be passed as arguments to functions, and returned from functions just like any other value ...</p> </blockquote> <p>We have a <a href="https://youtu.be/IXqTq839pIo">recording</a>, though it was a pretty informal session.</p> <h3>Installing Idris</h3> <p>Idris docs recommend <code>cabal install idris</code> but that's a pain; <a href="https://nixos.org/nixpkgs/manual/#idris">nix support for idris</a> reduces installation (and use!) to just[^1]:</p> <pre><code class="language-bash">$ nix-shell -p 'idrisPackages.with-packages (with idrisPackages; [ contrib pruviloj ])' </code></pre> <p>I like to combine it with <a href="https://direnv.net/">direnv</a> support for nix so that when I <code>cd</code> into an idris project directory, it all Just Works.</p> <p>[^1]: if you get <code>idris-modules/build-builtin-package.nix:23 is marked as broken, refusing to evaluate</code>, your nix installation may be out of date; <code>nix update-nix</code> worked for me.</p> <p>There seems to be good <a href="https://github.com/zjhmale/vscode-idris">vs-code support for idris</a>, but I don't know how to get it to look at the <code>contrib</code> package, so for today, I tried emacs, though even that showed some rough edges.</p> <h3>Formalizing Knowledge with Idris</h3> <p>LaTeX is fine for <em>typesetting</em> knowledge, but my goal is to <em>formalize</em> knowledge such that the machine can help exploit it.</p> <p>The example I gave in my <a href="https://twitter.com/dckc/status/1230157668888780800?ref_src=twsrc%5Etfw">Feb 29 tweet</a> was a <a href="https://www-sop.inria.fr/lemme/Tamara.Rezk/publication/csf16Capabilities.pdf">2016 paper on capabilities and confused deputy attacks</a>.</p> <p>The first definitions I captured were:</p> <blockquote> <p>We assume two countable sets, $Loc$ of mutable references and $Prin$ of principals. ... Values consist of integers ($n$), booleans ($tt$, $ff$) and pointers or mutable references $R_r$ and $W_r$.</p> </blockquote> <p>In idris, this takes the somewhat familiar of a haskell / ML data type:</p> <pre><code class="language-idris">data Loc = MkLoc Nat data Prin = MkPrin Nat data Value = IntVal ZZ | True | False | Read Loc | Write Loc </code></pre> <p>This allows the idris compiler to, for example, let me know when I neglected to handle all kinds of <code>Value</code> later in my transcription to code.</p> <h2>Extending idris-ct for the Rosetta Stone</h2> <p>On page 5 of the Rosetta Stone paper we find this nice diagram relating definitions in category theory:</p> <p><img src="/static/img/rosetta-fig-p5.svg" /></p> <p><a href="https://github.com/statebox/idris-ct">idris-ct</a> already formalizes monoidal categories and symmetric monoidal categories but not closed categories and cartesian categories. Jake started laying the ground work for <a href="https://gitlab.com/jake.gillberg/programmingcats/-/blob/master/src/ProgrammingCats/MonoidalCategory/CartesianCategory.idr">CartesianCategory.idr</a> and such.</p> <p>In discussion of cup and cap, Fabrizio answered a question by way of a figure from some of his <a href="https://arxiv.org/abs/1805.05988">work on petri nets</a>. He also mentioned Samson Abramsky's work; <a href="https://www.sciencedirect.com/science/article/pii/030439759390181R">Computational interpretations of linear logic</a> has been on my list since Greg said it "changed the direction of his career." Here's hoping for time to study that one closely too.</p> Sat, 02 May 2020 00:00:00 +0000 http://www.madmode.com//2020/grok-idris-ct/ Trying znc IRC bouncer to stay in touch with the ocap community http://www.madmode.com//2020/what-i-use-irc/ <p>I'm typically connected to IRC channels for a few ocap projects etc.:</p> <p><a href="http://www.erights.org/">#erights</a> <a href="https://genode.org/community/index">#genode</a> <a href="https://www.idris-lang.org/">#idris</a> <a href="https://indieweb.org/IRC">#indieweb</a> <a href="http://www.monte-language.org/#IRC">#monte</a> <a href="https://ocaml.org/community/mailing_lists.html">#ocaml</a> <a href="https://sandstorm.io/community">#sandstorm</a> <a href="https://www.w3.org/2001/sw/interest/#swig_chan">#swig</a></p> <p>I use <a href="https://packages.ubuntu.com/bionic/hexchat">hexchat</a> to connect to these channels on <a href="https://freenode.net/">freenode</a> but netsplits and other failure modes inherent in <a href="https://en.wikipedia.org/wiki/Internet_Relay_Chat">IRC</a> make for annoying little gaps in the connection from time to time.</p> <p>An ice storm here in Kansas City gives me a chance to install the <a href="https://packages.ubuntu.com/bionic/net/znc">znc</a> IRC bouncer in my home office, after a few months of positive experience at work.</p> <p>Both <code>znc --makeconf</code> and <a href="https://wiki.znc.in/HexChat">HexChat znc config</a> want me to specify passwords, servers, and channels. I don't have a clear mental model of how it's supposed to work, but I managed to muddle through, I think. <em>Where to back up the config? Passwords don't fit in a dotfiles repo... ok, kbfs += <code>znc-irc-bouncer-config.tgz</code>, <code>hexchat-irc-client-config.tgz</code>.</em></p> Fri, 17 Jan 2020 00:00:00 +0000 http://www.madmode.com//2020/what-i-use-irc/ Toward secure distributed computing with Agoric at SFBW '19 http://www.madmode.com//2019/agoric-xs-sfbw/ <p>After months of indecision, I decided to go to <a href="https://sfblockchainweek.io/">San Francisco Blockchain Week</a> in hopes of syncing up with the <a href="https://agoric.com/">Agoric</a> team. I'm glad I did!</p> <p><a href="https://twitter.com/agoric/status/1188952857002696704"> <img src="https://pbs.twimg.com/media/EIACUZfU4AAeYxA?format=jpg&name=small" alt="#Agoric Team at #CESC Day 1." /> </a></p> <p>I have been working remotely for a few months on <a href="https://github.com/Agoric/SwingSet/issues/126">SwingSet on xs</a>:</p> <blockquote> <p>A key to Agoric's smart contract platform is compatibility between widespread understanding of best practices in JavaScript development with object capabilities and fail-stop deterministic execution. Initial development of cosmic-swingset is based on the popular and feature-rich node.js platform, but node.js represents an uncomfortably large amount of code to include in a trusted computing base as well as a substantial risk to deterministic execution. The xs engine in the Moddable SDK is designed for constrained microcontroller environments. It has some limitations, but the supported feature set seems to be an excellent match for the Agoric platform.</p> </blockquote> <p>SwingSet has about 500 tests across 15 files. Once I got about 200 in 3 files, I started looking for someone to reproduce my work. Switching from my linux desktop to a macbook for this trip meant I could be that someone. In the Agoric office on Wednesday, I found out that some of the kludges I had put in place weren't portable between linux and macOS, so only about 170 tests are working cross-platform. It was great to be able to chat in person with Brian and Michael about this stuff.</p> <h2>Zoe and Offer Safety</h2> <p>Agoric leveled up their platform this week with <a href="https://agoric.com/zoe-vs-status-quo/">Zoe</a>, which provides <strong>offer safety</strong>: when you place an offer, either you'll get what you wanted or you get a refund, regardless of the (mis)behavior of the underlying smart contract.</p> <p>For example, <a href="https://github.com/Agoric/autoswap-frontend">autoswap</a> is a cool service, but conventional interfaces provide little assurance as to what is going to happen when you send it some tokens in a transaction. In contrast, a <a href="https://github.com/Agoric/wallet-frontend">wallet working with Zoe</a> presents offers in an intelligible form and guarantees that the offer is either executed as stated or you get a refund.</p> <p>There are some rough edges in getting these demos running, but I had done the <a href="https://agoric.com/testnet-pixel-demo/">testnet demo</a> the previous week so it only took me a little chatting with JF and co to get them running.</p> <p>And the Agoric folks are not alone in smoothing out the UI: Dan Finlay built a <a href="https://github.com/danfinlay/metamask-agoric-plugin">MetaMask Agoric Plugin</a> at the DeFi hackathon.</p> <p>I think it would be fun to formally verify the offer safety property. Mark Miller and I chatted about this with Eric McCarthy, who works on the <a href="https://www.kestrel.edu/home/projects/ethereum/index.html">ACL2 Ethereum</a> project. I have some history with ACL2, and in some sense the untyped style of ACL2 goes well with JavaScript, but I wonder if that's a false economy. I'm more facile with ML style propositions-as-types systems (idris, ocaml) lately.</p> <h2>CESC: Ouroboros, Kara from Oasis Labs, ...</h2> <p>At CESC (crypto economics security conference) Monday and Tuesday, some talks were too full of greek letters for me to follow and others were more like TV commercials than academic contributions. Two that stood were:</p> <ul> <li> <p>Vassilis Zikas on <em>The Evolution of Ouroboros: From Proof of Work to Proof of Stake</em>. He started with an introduction to bitcoin that seemed a little remedial for this venue but he built on the structure of that intro to make a clear presentation of the rest.</p> </li> <li> <p><a href="https://kara.cloud/#/">Kara</a> from <a href="https://www.oasislabs.com">Oasis Labs</a>: privacy-preserving health data analytics. "AI for healthcare is limited by the fact, that the most valuable data is locked in data silos. Kara breaks up these silos and allow researchers to train their models on data that is otherwise hidden from science - extracting value from it, while never exposing the data itself." The talk was a little on the TV commercial side, but the <a href="https://arxiv.org/abs/1804.05141">Ekiden</a> on how they mix private compute nodes with the blockchain is pretty convincing. The evaluation includes some machine learning stuff for medical diagnostics.</p> </li> </ul> <h2>Epicenter: CasperLabs, Chainlink</h2> <p>I went to the <a href="https://twitter.com/meetCasperLabs/status/1190015712208666624">CasperLabs workshop by Michael</a> with Joshy, a <a href="https://joshyorndorff.com/blog/keeping-the-rchain-community-together">fellow rchain expat</a>.</p> <p>The Chainlink marketplace for connecting smart contracts to real world data sources (oracles) and payment systems looked interesting. I'm a little bummed I missed the <a href="https://docs.google.com/document/d/e/2PACX-1vTfuGrmqJO_il0PzZY5iU5_HtmiEExu5t7XHzrj6rRVKZnOdvy3fUJzlIlTgd-FMrUl2A-9T9ndP7Nj/pub">chainlink workshop</a>, but I think I made the right call hanging out with the Agoric team instead.</p> Sun, 03 Nov 2019 00:00:00 +0000 http://www.madmode.com//2019/agoric-xs-sfbw/ spendr: toward an rchain gRPC client in rust using tokio and async / await http://www.madmode.com//2019/spendr-rust-async/ <p>Inspired by mention of a gRPC client and server library in <a href="https://tokio.rs/blog/2019-08-alphas/">Tokio alpha release with async &amp; await</a>, I tried building an rchain wallet client in rust.</p> <p>A couple hours later, I had the hello-world client from <a href="https://github.com/tower-rs/tower-grpc/issues/198">tower-grpc</a> adapted to talk to the shiny new <a href="https://github.com/rchain/rchain/releases/tag/v0.9.12">0.9.12 release of rnode</a>:</p> <pre><code>~/projects/spendr$ cargo run Compiling spendr v0.1.0 (/home/connolly/projects/spendr) Finished dev [unoptimized + debuginfo] target(s) in 3.00s Running `target/debug/spendr` RESPONSE = Response { metadata: MetadataMap { headers: {&quot;content-type&quot;: &quot;application/grpc&quot;, &quot;grpc-encoding&quot;: &quot;identity&quot;, &quot;grpc-accept-encoding&quot;: &quot;gzip&quot;} }, message: Streaming } </code></pre> <p>But that client code doesn't use the yummy new async / await stuff. It doesn't look like async / await has made it up to the gRPC level yet. I think I mis-read the blog post; I think it was just saying that tokio has a gRPC client and server library.</p> Tue, 10 Sep 2019 00:00:00 +0000 http://www.madmode.com//2019/spendr-rust-async/ Practical production python scripts http://www.madmode.com//2019/python-eng/ <p>In <a href="https://vincent.bernat.ch/en/blog/2019-sustainable-python-script">Writing sustainable Python scripts</a> (<a href="https://lobste.rs/s/zoo6tm/sustainable_python_scripts">lobsters discussion</a>), the initial prototype looks like:</p> <pre><code class="language-python">import sys for n in range(int(sys.argv[1]), int(sys.argv[2])): if n % 3 == 0 and n % 5 == 0: print(&quot;fizzbuzz&quot;) elif n % 3 == 0: print(&quot;fizz&quot;) elif n % 5 == 0: print(&quot;buzz&quot;) else: print(n) </code></pre> <p>I have a process for turning this sort of thing into production code that I quite enjoy. I just did it for this code in a <a href="https://gist.github.com/dckc/d58a04b154d44d46de9abcda84b2a2f0">fizzbuzz.py gist</a>:</p> <ol> <li>1ac9082</li> <li>621ddad only run when invoked as script</li> <li>4cfa693 ocap discipline: explicit access to argv, stdout</li> <li>35e74c6 quick-n-dirty arg parsing</li> <li>7cfeeea factor out fizzbuzz() from main()</li> <li>8d85c04 module doctest of usage</li> </ol> <p>The bottom line in the <a href="https://informatics.kumc.edu/work/wiki/CodeReviewNotes">code review checklist</a> in the shop I run is: <strong>is this code I would be happy to maintain?</strong></p> <p>While I appreciate writing documentation first, it usually takes me a few iterations of prototyping before I know what it is I want to document.</p> <p>I haven't made strict habit of test-driven development, but I do find factoring out tricky bits of logic and adding unit tests for them frees my mind up during development. And I do make it a rule to write tests before fixing any bugs.</p> <h2>Only run when invoked as script</h2> <p>If this code is good, we want to be able to reuse it by importing it from other modules. And even if we never reuse this code, we want the ability to unit test it. So we use the <a href="https://docs.python.org/3/library/__main__.html">top level convention</a> (<code>if __name__ == "__main__":</code>) to test whether this code is invoked as a script, but we minimize the code there. As a rule, module level code shouldn't do anything but define things:</p> <pre><code class="language-python"> def main(): for n in range(int(sys.argv[1]), int(sys.argv[2])): if n % 3 == 0 and n % 5 == 0: print(&quot;fizzbuzz&quot;) elif n % 3 == 0: print(&quot;fizz&quot;) elif n % 5 == 0: print(&quot;buzz&quot;) else: print(n) if __name__ == '__main__': main() </code></pre> <h2>Ocap discipline: explicit access to IO</h2> <p>To facilitate patterns of cooperation without vulnerability, object capability (<a href="https://en.wikipedia.org/wiki/Object-capability_model">ocap</a>) discipline says the only thing globally accessible should be immutable data. This also supports <a href="http://misko.hevery.com/code-reviewers-guide/">testability</a>. <code>argv</code> and <code>stdout</code> are IO, or at least sources of non-determinism; object capability discipline calls for passing these around explicitly.</p> <p>Of course we have to get them from somewhere. In an object capability language, they would be passed (directly or indirectly) to <code>main()</code>; we simulate this using <code>_script_io()</code>, a function that we only define and invoke if <code>__name__ == '__main__</code>:</p> <pre><code class="language-python"> def main(argv, stdout): for n in range(int(argv[1]), int(argv[2])): if n % 3 == 0 and n % 5 == 0: print(&quot;fizzbuzz&quot;, file=stdout) elif n % 3 == 0: print(&quot;fizz&quot;, file=stdout) elif n % 5 == 0: print(&quot;buzz&quot;, file=stdout) else: print(n, file=stdout) if __name__ == '__main__': def _script_io(): from sys import argv, stdout main(argv, stdout) _script_io() </code></pre> <h2>Quick-n-dirty arg parsing</h2> <p>The code is now testable, so we start looking inside the functions. Mixing in arg parsing with other logic doesn't smell right:</p> <pre><code>def main(argv, stdout): for n in range(int(argv[1]), int(argv[2])) </code></pre> <p>For a little script like this, one or two carefully crafted lines is all I'd spend on arg parsing:</p> <pre><code class="language-python"> def main(argv, stdout): [lo_, hi_] = argv[1:3] lo, hi = int(lo_), int(hi_) for n in range(lo, hi): ... </code></pre> <p>If we just invoke this script with no args, the resulting traceback is often enough of a usage clue, given that the userbase of these scripts is a handful of devs in the shop (plus jenkins):</p> <pre><code>$ python3 fizzbuzz.py Traceback ... File &quot;fizzbuzz.py&quot;, line 4, in main [lo_, hi_] = argv[1:3] ... </code></pre> <p>For more involved arg parsing, or perhaps if the userbase were larger, I prefer <a href="http://docopt.org/">docopt</a> to writing (or worse: reading!) elaborate argparse calls.</p> <h3>Aside: who asked for fizz and buzz to be parameterized?</h3> <p>Parameterizing fizz and buzz adds testing and maintenance burden. Unless we've got a customer requirement, let's skip all that.</p> <h2>Factor out fizzbuzz() from main()</h2> <p>I like to keep our <code>main()</code> functions small and separate logic from IO. Using <code>yield</code> in place of <code>print</code> is particularly convenient. Let's add one quick doctest while we're at it:</p> <pre><code class="language-python">def main(argv, stdout): [lo_, hi_] = argv[1:3] lo, hi = int(lo_), int(hi_) for word in fizzbuzz(lo, hi): print(word, file=stdout) def fizzbuzz(lo, hi): &quot;&quot;&quot; &gt;&gt;&gt; list(fizzbuzz(1, 6)) [1, 2, 'fizz', 4, 'buzz'] &quot;&quot;&quot; for n in range(lo, hi): if n % 3 == 0 and n % 5 == 0: yield &quot;fizzbuzz&quot; elif n % 3 == 0: yield &quot;fizz&quot; elif n % 5 == 0: yield &quot;buzz&quot; else: yield n </code></pre> <h2>Module doctest of usage</h2> <p>Now we can document and test usage together in the module docstring.</p> <p>I actually started the module doctest earlier in the process, but I didn't commit it until the end in order to make each commit demonstrate one part of the process.</p> <pre><code>&quot;&quot;&quot;Simple fizzbuzz generator. The game proceeds with players announcing numbers increasing sequentially, except for multiples of 3 and 5: &gt;&gt;&gt; usage = 'fizzbuzz 1 20' &gt;&gt;&gt; from io import StringIO; stdout = StringIO() &gt;&gt;&gt; main(usage.split(), stdout) &gt;&gt;&gt; print(stdout.getvalue()) ... #doctest: +ELLIPSIS 1 2 fizz 4 buzz fizz 7 ... 14 fizzbuzz 16 ... &quot;&quot;&quot; def main(argv, stdout): [lo_, hi_] = argv[1:3] lo, hi = int(lo_), int(hi_) for word in fizzbuzz(lo, hi): print(word, file=stdout) def fizzbuzz(lo, hi): &quot;&quot;&quot; &gt;&gt;&gt; list(fizzbuzz(1, 6)) [1, 2, 'fizz', 4, 'buzz'] &quot;&quot;&quot; for n in range(lo, hi): if n % 3 == 0 and n % 5 == 0: yield &quot;fizzbuzz&quot; elif n % 3 == 0: yield &quot;fizz&quot; elif n % 5 == 0: yield &quot;buzz&quot; else: yield n if __name__ == '__main__': def _script_io(): from sys import argv, stdout main(argv, stdout) _script_io() </code></pre> <h2>Reflection</h2> <p>In this example, our 10 line (working!) prototype has turned into 60 lines of code. If a developer asked me to review the original 10 lines after using it succefully for a few days, and if the cost of a failure was just a few minutes, I might not bother to touch it.</p> <p>But the code did mix IO with logic in a way that makes me uneasy.</p> <p>More often, the cost of a failure is re-running a Jenkins job that takes hours. And even in early prototype form, scripts are usually longer than 10 lines and applying these techniques turns up a bug or two or at least clarifies the security properties of the code.</p> <p>And in this case, of the resulting 60 lines:</p> <ul> <li>25 are documentation<ul> <li>10 show simulated script output, which is very simple</li> </ul> </li> <li>The <code>main()</code> function is 6 lines that clearly do nothing but parse args, call a function, and output the results</li> <li>The <code>_script_io()</code> function is formulaic</li> </ul> <p>The <code>fizbuzz()</code> function retains structure of the original 10 line prototype, with just 4 lines added for documentation / testing.</p> Mon, 22 Jul 2019 00:00:00 +0000 http://www.madmode.com//2019/python-eng/ RChain Blues http://www.madmode.com//2019/rchain-blues/ <p>Working on RChain was fun for a while, but I am no longer be comfortable with the RChain Cooperative's business practices.</p> <p>I am inclined to keep my distance until I see</p> <ul> <li>quarterly financials on time ("no later than 20 days after the quarter ends" per IOB 1 from the <a href="https://blog.rchain.coop/blog/2018/10/26/rchain-2018-election-results/">2018 annual meeting</a>)</li> <li>the <a href="https://github.com/rchain-community/sm19a/issues/3">board carry out IOB 4 member participation in governance</a></li> <li><a href="https://github.com/rchain-community/sm19a/issues/11">timely board agendas and minutes</a></li> <li>some <a href="https://github.com/rchain-community/sm19a/issues/14#issuecomment-495039918">details re Barcelona</a></li> <li>this open-source cooperative <a href="https://github.com/rchain-community/sm19a/issues/9#issuecomment-495452506">divest any significant interest in a proprietary audio codec license</a></li> </ul> <p>The story of how <a href="https://coinmarketcap.com/currencies/rchain/">RChain's CoinMarketCap rank</a> fell from 30 to over 200 in 2018 would take more energy to tell than I have just now, but to emphasize why something as supposedly mundane as quarterly financial reports has me so uncomfortable...</p> <p>I assumed since I got involved in Nov 2017 that any organization that had raised $30M from investors would have someone keeping books, so that producing a balance sheets or cashflow statements was a one-click task... I assumed that people were busy and they just hadn't bothered to click that button and share the results.</p> <p>I should have known this was yet another case where the web reflects reality.</p> <p>In the board meeting of July 23, 2018, when they decided to spend tens of millions of dollars on a proprietary audio codec, they <em>did not know how much money the coop actually had</em>, according to Meredith at the <a href="https://www.youtube.com/watch?v=xdi6ZCsX4K8">Oct 2018 annual meeting</a>.</p> <p>Some new board members were elected and tried to improve corporate governance, but one by one they reached a point of diminishing returns and moved on to other things.</p> <p>After the April <a href="https://blog.rchain.coop/blog/2019/04/26/rchain-update-04-26-19/">Barcelona incident</a>, one member put it this way:</p> <blockquote> <p>... it became near impossible to responsibly put one's reputation on the line for this project.</p> </blockquote> <p>I attempted to steer member engagement in a useful direction in a <a href="https://github.com/rchain-community/sm19a">Special Meeting 19A</a> process; it was a long-shot and not much resulted from it, but I did manage to get over 100 members on record thanking those who had served on the board since the October annual meeting, trying to right the ship:</p> <blockquote> <p>Whereas Barry Cynamon, Mark Pui, and Kevin Goldstein served with distinction on the RChain Cooperative board, let us thank them for their service.</p> </blockquote> Wed, 05 Jun 2019 00:00:00 +0000 http://www.madmode.com//2019/rchain-blues/ Migrated from github pages to netlify for continuous deployment, SSL http://www.madmode.com//2019/netlify-deploy-https/ <p>A lot of the friction around my blog has been running the build step: re-constituting python dependencies, running the build, manually tweaking the results, and committing the build results to github pages[^1].</p> <p><a href="https://github.com/pmoorman">pmoorman</a> turned me on to <a href="https://www.netlify.com/">Netlify</a> last fall, so I gave it a try: <strong>Holy shnikeys! it worked 1st try with no tweaks whatsoever!</strong> I told it my build command was <code>./site build</code>[^2] and it picked up my <code>requirements.txt</code>, installed the dependencies, built the site and deployed it.</p> <p>Fiddling with DNS took quite a bit longer, but I got free SSL for my trouble. Netflify pointed out a zillion mixed content issues... enough that I wrote a little <a href="https://github.com/dckc/madmode-blog/blob/netlify-https/fix_insecure.py">fix_insecure.py</a> script to parse their log and make the edits.</p> <p>While I'm addressing hygene issues, I fixed the dependency vulerabilities that github pointed out.[^3] <a href="https://github.com/dckc/madmode-blog/blob/netlify-https/CONTRIBUTING.md">CONTRIBUTING.md</a> shows how I bootstrap with direnv and pipenv.</p> <p>[^1]: I migrated from nearlyfreespeech.net to github pages in 2015.</p> <p>[^2]: When did I switch to frozen flask? Wow... it was 2012. Time flies.</p> <p>[^3]: Darn. This footnote markup doesn't work because flask-markdown predates <a href="https://python-markdown.github.io/change_log/release-3.0/#positional-arguments-deprecated">deprecating positional arguments</a>.</p> Thu, 30 May 2019 00:00:00 +0000 http://www.madmode.com//2019/netlify-deploy-https/ Smart Contracts for Health Research Data Sharing http://www.madmode.com//2018/boulder-bos/ <p>I'm under contract with RChain, two days a week from June to December, to work on <a href="https://github.com/rchain/bounties/issues/788">smart contracts for health research data sharing</a>. I'm still at KUMC for the other three days a week.</p> <h2>RChain Devcon Boulder</h2> <p>In April, I was invited to <a href="https://developer.rchain.coop/conference">RChain Devcon</a> in Boulder. I enjoyed Greg's <a href="https://www.youtube.com/watch?v=3R3IL1bGm0s&amp;t=617s">RChain VM talk</a>. (<em>More on that later, I hope.</em>) Vlad's CBC Casper talk was mostly familiar; I think I got most of it from <a href="https://www.youtube.com/watch?v=z3sY8zZRPtw">his devcon three talk</a>, which was just my speed.</p> <p>I gave a short talk about <a href="https://docs.google.com/presentation/d/1B2Vu8o3ACwruY6HY1ayXRQ4qkNKsMy4hdbOdxrCHI2o/edit">Getting Involved in the RChain Bounty Program</a> (<a href="https://www.youtube.com/watch?v=HsQTDNEIbjs&amp;t=1s">video</a>).</p> <p>TimBL had written <a href="https://webfoundation.org/2018/03/web-birthday-29/">The web is under threat</a>, which I used as a springboard for <a href="https://docs.google.com/presentation/d/1GOboFZj311rfGExrtCFp3aaQ2vYhxHqX3qnix21nqo4/edit">Can RChain Answer?</a> (<a href="https://www.youtube.com/watch?v=ZnBbi6ifzdo&amp;t=849s">video</a>). My summary slide is:</p> <blockquote> <p>It has a lot of the right pieces:</p> <ul> <li>Object Capability Discipline</li> <li>Support for Formal Verification</li> <li>Integrated Economics with Network Architecture</li> <li>Scalability</li> <li>Down to transistors</li> <li>Up to the globe</li> <li>Visionary Leadership</li> <li>Cooperative Governance</li> </ul> </blockquote> <p>I did a quick run up of capabilities from simple closure-based objects to sealing and unsealing to capability-based money as in <a href="http://erights.org/elib/capability/ode/index.html">Miller, Morningstar, and Franz</a> from FC 2000, including:</p> <blockquote> <p>Before presenting the following simple example of capability-based money, we must attempt to head off a confusion this example repeatedly causes. <strong>We are not proposing to actually do money this way!</strong> A desirable money system must also provide for: ... blinding, non-repudiation, accounting controls, and backing (redeemability).</p> </blockquote> <p>Then—tada!—I showed that <strong>RChain <em>does</em> propose to do money this way</strong> by translating the E code to <a href="https://github.com/rchain/rchain/blob/master/casper/src/main/rholang/MakeMint.rho">MakeMint.rho</a>:</p> <pre><code> contract MakeMint(return) = { new pairCh, thisMint, internalMakePurse in { MakeBrandPair!(*pairCh) | for(@p &lt;- pairCh) { ... </code></pre> <h2>HERON Clinical Data Repository Access Policy</h2> <p>The main smart contract I want to work on is an evolution of the <a href="https://informatics.kumc.edu/work/wiki/HeronAdminDev">HERON policy enforcement layer</a>, a bunch of python / PHP / SQL code that enforces the policy around access to KUMC's clinical data repository on ~500K patients: you have to be qualified faculty or sponsored by one, and your human subjects training has to be current and you have to sign a couple forms:</p> <p><img alt="flow of authority" src="https://informatics.kumc.edu/work/graphviz/a02d7fe066856aadf1894e50f41f4b2aa27ca3b4.dot.png" /></p> <p>Once you get through all that, you can use the i2b2 ad-hoc query interface to do "cohort discovery"; for example, to find out if enough of the right kind of patients come through KU Med for the study you want to do.</p> <h2>i2b2 Harvard Symposium</h2> <p>The <a href="http://transmartfoundation.org/harvard-symposium-2018/">i2b2 tranSMART Foundation Harvard Symposium</a> was this year's annual meeting of the i2b2 user community. Automation of regulatory processes is a regular topic and this year <a href="http://geekdoctor.blogspot.com/">John Halamka</a> spoke on <em>Emerging Models of Data Flow - APIs, Blockchain and Cloud</em>. He cut through a lot of the mystery around blockchain technology with a great illustration of one-way hashing. :)</p> <p><img src="https://lh3.googleusercontent.com/6w32pE2Tpc6ZHzHSWKR-TmNvIvL1Nl_21z3UAYFdCukNr-MJcDlPpVTy1HRnTOcrVD2jTt59TQeaD4WEmcftD8PtxAQWeA-OIlRJ8mageizVjdaTUCCL2ENSjmHulogCqPtwcYeQ-Q=w640-h480" width="640" height="480" /></p> <h2>Decentralized identity, verifiable claims</h2> <p>While killing time in the airport, I managed to reach Manu Sporny. His work around <a href="https://veres.io/use-cases/verifiable-prescriptions/">verifyable prescriptions</a> has certain parallels with research data sharing. It wasn't all good news, but he did put me in touch with a couple of his colleagues in the Boston area.</p> <p>I had hoped to meet with <a href="https://dustycloud.org/">Chris Webber</a> and sync up on js libraries for <a href="https://w3c-ccg.github.io/ocap-ld/">linked data capabilities</a>. Capability security seems necessary and sufficient, to me, for decentralized access control. As Stiegler put it in the <a href="http://erights.org/talks/efun/SecurityPictureBook.pdf">picture book</a>:</p> <blockquote> <p>The patterns described in this picturebook are simple because they discard the modern fascination with the identities of the participants. Individual Authentication is so pervasive, it is now a part of the problem.</p> <p>Suppose that your car, instead of accepting a delegatable key, demanded that your driver’s license match the car’s title registry, which happens to be in your spouse’s name. Entrepreneurs would leap forward to develop ever more powerful "identity management" for automobiles. We would subcontract to security experts so our teenage daughters could borrow the car to buy milk. Heaven forfend that the daughter, breaking her leg, had to delegate to her best friend to get to the hospital.</p> </blockquote> <p>Unfortunately, while Chris is closer to Boston these days, he's still a couple hours away.</p> <p>But I did get to meet <a href="http://computingjoy.com/">Dmitri Zagidulin</a> over breakfast. He has done javascript work both with TimBL and company on <a href="https://github.com/solid/solid">solid</a> and with Manu on decentralized identifiers (e.g. <a href="https://github.com/digitalbazaar/did-io">did-io</a>). He isn't yet working on linked data capabilities yet, so I twisted his arm in that direction, and <a href="http://www.madmode.com/2011/07/secure-mashups-csrf-resistent.html">away from WebID</a>.</p> <p>Things started to click for him when I talked about verifyable claims in terms of insurance and markets:</p> <blockquote> <p>Proof of identity, in so much as it involves revelation of the profile, or enables its revelation through the use of unique identifiers, is trade in an asset when the information revealed is more than the minimum required with current technology. -- <a href="https://www.w3.org/2000/12/drm-ws/pp/hp-poorvi2.html">Vora et. al from HP at W3C 2001 DRM Workshop</a></p> </blockquote> <p>Take proof of age, for example. A smart contract for a client might ask "who will bet me $10 at 50-to-1 that this request comes from someone over 21 years of age?" A business where people are routinely physically present is in a position to take such bets with high confidence.</p> <h2>Personal Health Records</h2> <p>Manu also put me in touch with <a href="http://healthurl.com/www/Blogs_+.html">Adrian Gropper</a>, CTO of Patient Privacy Rights. Meeting in the airport as he was arriving and I was departing was a bit hectic, so I couldn't be sure whether I had read the paper he and Mark Miller worked:</p> <ul> <li><a href="https://github.com/WebOfTrustInfo/rebooting-the-web-of-trust-fall2017/blob/master/final-documents/identity-hubs-capabilities-perspective.md">Identity Hubs Capabilities Perspective</a> Rebooting the Web of Trust V, October 13, 2017 by Adrian Gropper, Drummond Reed, Mark S. Miller</li> </ul> <p>His description of <a href="http://hieofone.org/">HIEofone</a> increased the priority of UMA in my things-to-study-in-more-depth list:</p> <blockquote> <p>HIE of One (Health Information Exchange of One) is a volunteer-driven open source project to combine emerging standards for access authorization (OpenID HEART) and emerging standards for blockchain-based self-sovereign identity (DID) into a patient-centered health record infrastructure.</p> </blockquote> <p>He was one of several people on this trip that talked about FHIR deployment in ways that make me want to look into it again. In particular, he mentioned a FHIR sandbox by CMS, the medicare folks.</p> <h2>OCap-safe JavaScript</h2> <p>On the way home, I got stuck in the Washington D.C. airport overnight due to weather. I used the time to start <a href="https://github.com/dckc/tinyses2rho/">tinyses2rho</a>, some exploratory scala code to integrate TinySES with RChain.</p> <p>TinySES is a small subset of JavaScript designed so that non-experts can use to write non-trivial non-exploitable smart contracts. It comes from Mark Miller and company, who have been working on object capabilities and smart contracts at least as far back as the early '90s, and recently <a href="https://www.coindesk.com/new-startup-zooko-naval-betting-better-crypto-contracts/">launched Agoric</a>.</p> <p><a href='https://photos.google.com/share/AF1QipPEmCV0T84sLHj1L1zNUQ-eldo2SVxeNYYf49RhnQF-5kp6kHZua4BAbfgCDOrICw?key=cmVENTBqc2YwY2g2QzZxQkJjSUVvVXFYYXg3QTVB&source=ctrlq.org'><img alt='DCA in the wee hours' src='https://lh3.googleusercontent.com/rv9s-h-_E58jZftv4H1XcBlgGtx1hszNMMpXrQyMVDGuath90K8OtXn7_ItZxR0G6n-_1dEVujUf0ED_nKtPq8qZElbDRsAY7PkWvKyOGejZgAZVU6HLKmQ3cKOdF0Rf-gCrh0zM8w=w2400' /></a></p> Thu, 12 Jul 2018 00:00:00 +0000 http://www.madmode.com//2018/boulder-bos/ dreaming big at the RChain Governance Forum http://www.madmode.com//2018/rchain-gov-forum-SEA/ <p>I'm back from the <a href="https://www.rchain.coop/">RChain</a> Governance Forum in Seattle, which was full of big dreams, just like the first Web conference in Geneva.</p> <blockquote> <p>Why RChain? Governance and blockchain can't be separated; I felt compelled to connect them. -- <a href="https://platform.coop/2017/schedule/breakout-panels-next-money-new-tech-for-new-finance">Greg Meredith Sep 2017</a></p> </blockquote> <p>I went in with almost no idea what that means. But at the closing brunch, I found myself telling the next person at the table that the economics of email—the original decentralized Internet communication platform—was what led me to trade my privacy to Google for spam protection.</p> <p>Many of us who built the Web had in mind a decentralized system where access to information and freedom of expression would enrich our lives. At some level, we knew sharp tools cut both ways, but ...</p> <blockquote> <p>When REST was being framed, it seemed inconceivable that two billion people would all agree to use one website (Facebook); -- <a href="https://research.google.com/pubs/pub46310.html">Fielding et. al. 2017</a></p> </blockquote> <p>The result is a vulnerability that Kate Charlet, who worked on Cyber Policy in the U.S. Department of Defense for the last decade, <a href="http://www.irckc.org/m/event_details.asp?id=1041848">says</a> nation states have successfully used to destabilize our democracy.</p> <p>Perhaps there's something to all this blockchain stuff; perhaps we can repair the imbalance by integrating economics into the next generation network architecture.</p> <p>The imbalance in the system has a glib description:</p> <blockquote> <p>If you are not paying for it, you're not the customer; you're the product being sold. -- <a href="https://www.metafilter.com/95152/Userdriven-discontent#32560467">blue_beetle Aug 2010</a></p> </blockquote> <p>But another way to put it is:</p> <blockquote> <p>There's a strangle-hold on social media today; people don't get a chance to participate in the economic proposition in an equitable way. -- <a href="https://youtu.be/p0a0zu5APd4">Greg Meredith, Jan 2017</a></p> </blockquote> <p>He emphasizes that we need new ways of coordinating, inspired by resilient living systems and reflective intelligence. Tai Chi on Saturday morning was a whole body experience of it.</p> <p>And music!</p> <p>I had great conversations with Peter and Jonathan and Derek and Rudy, including Ted Nelson's ideas on the <a href="http://www.madmode.com/2008/01/technology-that-inspires.html">evolution of movie making</a>.</p> <p>Greg performed in a group of mind-bogglingly improvisational guitar players. Mostly I felt stressed and out of place when listening to it. But I was right at home when the twelve bar blues chord progression emerged in the middle of it.</p> <p>I was also pretty much at home when I saw lightning talks on the agenda, so I offered to lead the session. That was <a href="https://youtu.be/Mmkae9E93tk?t=1h46m13s">great fun</a>. I gave two talks myself and enjoyed a lot of the follow-up discussions.</p> <ul> <li><a href="https://docs.google.com/presentation/d/1XxdZxk9mWbB4cAely1_Ly5DZo8sTIeE4jq-s5gqiaqw/edit?usp=sharing">Toward a Trust Metric for Authority in the RChain Bounty System</a></li> <li><a href="https://docs.google.com/presentation/d/1dW1nsNDJjXBd6YTBOxQ21GBGqHMt5Q8f3B1RIG3bNao/edit#slide=id.p">RChain Life, a learning game</a></li> </ul> <p>The technical architecture of RChain also sunk in a bit more. In addition to <a href="http://www.madmode.com/2017/rchain-dev-retreat.html">capability security and namespaces</a>, I'm beginning to see reflection (the R in RChain) as a possible solution to the <a href="https://lists.w3.org/Archives/Public/public-cwm-talk/2016JanMar/0000.html">quoting and diagonalization issues</a> TimBL and I struggled with in the Semantic Web.</p> <h2>References</h2> <ul> <li>Fielding, Taylor, Erenkrantz, Gorlick, Whitehead, Khare, Oreizy, <a href="https://research.google.com/pubs/pub46310.html">Reflections on the REST Architectural Style and “Principled Design of the Modern Web Architecture”</a> ESEC/FSE 2017<ul> <li><a href="https://www.youtube.com/watch?v=6oFAmQUM8ws">video</a></li> </ul> </li> <li>Arndt, Verborgh, De Roo, Sun, Mannens, Van De Walle, <a href="http://link.springer.com/chapter/10.1007/978-3-319-21542-6_9">Semantics of Notation3 Logic: A Solution for Implicit Quantification</a> RuleML 2015</li> <li>Berners-Lee, Connolly, Kagal, Hendler, and Schraf, <a href="https://arxiv.org/abs/0711.1533">N3Logic: A Logical Framework for the World Wide Web</a> Journal of Theory and Practice of Logic Programming (TPLP), Special Issue on Logic Programming and the Web, 2008</li> </ul> Wed, 21 Feb 2018 00:00:00 +0000 http://www.madmode.com//2018/rchain-gov-forum-SEA/ Smart Contracts and Capabilities Up Close http://www.madmode.com//2017/rchain-dev-retreat/ <p>This year I was invited to a <a href="https://foresight.org/event/next-frontier-blockchain-meets-object-capabilities/">Blockchain meets Object Capabilities panel</a> in San Francisco July 3 and the <a href="https://github.com/rchain/Members/issues/191">RChain developer retreat</a> in Park City Nov 12-17. Leaving Park City was... <em>interesting</em>.</p> <p>My flight to San Francisco was funded from the <a href="http://www.madmode.com/2016/eth-dao.html">$30 that I put into Ethereum</a> in mid 2016--as an educational expense, I thought; I hardly expected to square my investment.</p> <p>Jorge Lopez of <a href="https://economicspace.agency/gravity">Gravity</a> hosted long-time capabilities leaders Mark Miller, Zooko Wilcox, and Brian Warner as well as Arthur Breitman of <a href="https://www.tezos.com/">Tezos</a>. Zooko also represented <a href="https://z.cash/">Zcash</a>.</p> <figure> <a href="https://photos.app.goo.gl/XnUs6dddBlRTl94F2"> <img src="https://lh3.googleusercontent.com/-47cAa-jvInI/WknX6v9rDuI/AAAAAAAAEsk/0WCmJqTB3YAJLdm1gux1PRZ86BF32kU3QCJoC/w530-h224-n-rw/IMG_20170703_182953499.jpg" alt="" /> <figcaption>Mark Miller with Lopez and Warner to his right and Breitman and Wilcox to his left</figcaption> </a> </figure> <p>Brian gave a great summary of the bitcoin market forces but I was too engrossed in the event to take notes; the one quote I managed to write down was "Satoshi is probably in the room."</p> <p><em>Aside: <a href="https://oasis.sandstorm.io/">Sandstorm Oasis</a> let me export the grain I took my notes in even though my subscription had lapsed. That's pretty cool.</em></p> <p>The evening before the panel, @zooko gave a <a href="https://twitter.com/zooko/status/881656410777411584">shout</a> out to my <a href="https://github.com/dckc/awesome-ocap">awesome-ocap</a> list, and in a <a href="https://twitter.com/robertobrien/status/905362724292509696">reply from @robertobrien</a>, I discovered <a href="http://rchain-architecture.readthedocs.io/en/latest/contracts/namespaces.html">RChain's Namespace Logic</a>.</p> <p>I saw Mike Stay's name, which was familiar from <a href="https://groups.google.com/forum/#!forum/cap-talk">cap-talk</a> discussion, on some of the citations in the RChain Architecture document, and asked him some questions in email. Before long I was contributing <a href="https://github.com/rchain/rchain/pull/40">little tweaks</a> to the code and wrangled an invitation to the developer retreat.</p> <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">RChain developer&#39;s retreat at Flanagan&#39;s <a href="https://t.co/5by4odQ85J">pic.twitter.com/5by4odQ85J</a></p>&mdash; Dan Connolly (@dckc) <a href="https://twitter.com/dckc/status/930621928116592640?ref_src=twsrc%5Etfw">November 15, 2017</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> <p>Again, I was too engrossed to take good notes, but they made <a href="https://www.youtube.com/playlist?list=PLf2bbiic5ZjCPzin3gCSMGiBtbT8UO5o2">recordings of the main sessions</a>.</p> <p>The RChain goals are ambitious, to say the least: "content delivery at the scale of Facebook and transactions at the speed of Visa." Not to mention a an "e pluribus unum 2.0, a new center on a scale-invariant axis reconciling the wisdom of collective and the wisdom of the individual" for collaborating on problems such as global warming. An <a href="https://www.youtube.com/watch?v=p0a0zu5APd4">interview with Greg Meredith</a> from the <a href="https://bitcointalk.org/index.php?topic=1747033.0">January RChain announcement</a> filled in a lot of the background for me.</p> <p>Nash and the gang did an amazing job hosting. We had dinner out most nights and breakfast in the houses in the morning, and it's a tough call which was better.</p> <p>On the last day, after a mind-blowing week, we were all lounging around, zoned out on our phones, when we noticed the snow. For some, it was their first snowstorm... nice and pretty. Then Nash snapped us out of it: "<strong>Get off the mountain now</strong> before they close the pass and you miss your flights!" As we were getting in the big rental SUV, I asked, "Have you driven in snow before?" and since she hadn't, the keys passed to me.</p> <p>The drive out was a little hairy, but <a href="https://twitter.com/VladZamfir">Vlad</a> shared some tunes and we all got to know each other a little better.</p> <figure> <a href="https://photos.app.goo.gl/eTkY2Pb8L8Errglf2"> <img src="https://lh3.googleusercontent.com/-aUKMLj97inQ/Wk3Dn5U6f5I/AAAAAAAAEuY/-CMagGS7HuQ6RVwCGm42zhNAYfbZkvP2gCJoC/w530-h323-n-rw/IMG_0572.jpg" alt="" /> <figcaption>Escaping Park City</figcaption> </a> </figure> Sun, 31 Dec 2017 00:00:00 +0000 http://www.madmode.com//2017/rchain-dev-retreat/ College Expense Tracking in BASIC09 http://www.madmode.com//2017/ut-austin-expenses/ <p>It's no wonder my kids struggle so much more to pay for college:</p> <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Hours of minimum wage work needed to pay for four years of public college<br><br>Boomer: 306<br>Millennial: 4,449<br><br>Source: <a href="https://t.co/3ZZDpC9Fgw">https://t.co/3ZZDpC9Fgw</a></p>&mdash; Ryan Carson (@ryancarson) <a href="https://twitter.com/ryancarson/status/943468921834717185?ref_src=twsrc%5Etfw">December 20, 2017</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> <p>In my freshman year at U.T. Austin, I wrote a <a href="https://en.wikipedia.org/wiki/BASIC09">BASIC09</a> program to track my expenses:</p> <pre><code class="language-basic">PRINT CHR$(12); &quot;Expenses -- by Dan Connolly&quot; PRINT &quot;&lt;A&gt; - Edit Accounts&quot; PRINT &quot;&lt;E&gt; - Journal Entry&quot; PRINT &quot;&lt;R&gt; - Generate Report&quot; PRINT &quot;&lt;C&gt; - Clean up file&quot; PRINT &quot;&lt;Q&gt; - Quit&quot; RUN Choose(&quot;Choice: &quot;,&quot;AERCQ&quot;,Choice) </code></pre> <p>I found a <code>Rpt02.22</code> report that shows tuition of about $500, mostly covered by a scholarship:</p> <pre><code class="language-python">def _cocodisks(): from pathlib import Path return Path('1986-cocodisks') EXP = _cocodisks() / 'archive' / 'PRG-x' / 'EXP' tx_lines = list((EXP / 'Rpt02.22').open()) tx_lines = tx_lines[1:] # skip blank line tx_lines[:2] + tx_lines[8:11] </code></pre> <pre><code>[u'Date Description Amount Source Name Src Bal Dest Name Dest Bal\n', u'----------- ------------- ------- ------------- ------- ------------- --------\n', u' 9-01-86:11 Books 117.85 Cash 182.15 Books/Supplie 117.85\n', u' 9-15-86:11 Scholarship 296.46 National Meri -296.46 U T 496.28\n', u' 9-15-86:15 Scholarship 78.54 National Meri -375.00 Cash 260.69\n'] </code></pre> <p>The last page of the report shows account balances:</p> <pre><code class="language-python">acct_hd_ix = next(ix for ix, line in enumerate(tx_lines) if line.strip().startswith('Num')) acct_lines = tx_lines[acct_hd_ix:] acct_lines[:5] </code></pre> <pre><code>[u' Num Account Name Balance\n', u'---- ------------- -------\n', u' 1: Cash 63.51\n', u' 2: Checks 28.00\n', u' 3: Bank Account 888.52\n'] </code></pre> <p>But the <code>Jrnl</code> data file goes thru March 12...</p> <pre><code>1986-cocodisks/archive/PRG-x/EXP$ ls -l Jrnl -r--r--r-- 1 connolly connolly 5443 Mar 12 1987 Jrnl 986-cocodisks/archive/PRG-x/EXP$ sha1sum Jrnl 3f75dbc8bcdac51874259c44ef1fcad55fe068e0 Jrnl </code></pre> <p>... where that report only goes thru Feb 22:</p> <pre><code class="language-python">tx_lines[acct_hd_ix - 7: acct_hd_ix - 5] </code></pre> <pre><code>[u' 2-22-87:11 Bus .50 Cash 63.51 Living Expens 138.94\n', u' 2-22-87:15 NOW ----- .00 .00 .00\n'] </code></pre> <h2>Porting BASIC09 File Reading Code to Python</h2> <p>I spent some time poring over the <a href="https://bitbucket.org/DanC/madmode-blog/src/08c15cccd1ef?at=default">EXP</a> source code (<code>08c15cc</code>) to get the data out:</p> <pre><code>1986-cocodisks/archive/PRG-x/EXP$ wc *.b 0 38 300 Acct.b 3 506 4490 Entry.b 7 564 5141 Exp.b 1 125 1021 Rec.b 2 142 1269 Report.b 13 1375 12221 total </code></pre> <p>The <code>file</code> command even recognizes the compiled format:</p> <pre><code>1986-cocodisks/archive/PRG-x/EXP$ file Expenses Expenses: OS9/6809 module: BASIC I-code subroutine </code></pre> <pre><code class="language-python">import pandas as pd import numpy as np dict(pandas=pd.__version__, numpy=np.__version__) </code></pre> <pre><code>{'numpy': '1.10.1', 'pandas': u'0.17.1'} </code></pre> <p>The file is just 5K. These days it's trivial to read that into memory, but my coco only had 16K of RAM, upgraded from 4K.</p> <pre><code class="language-python">Jrnl = (EXP / 'Jrnl').open('rb').read() len(Jrnl) </code></pre> <pre><code>5443 </code></pre> <p>The transaction format is mostly straightforward, though I'm glad I had the source code to decode the <code>key</code> field:</p> <pre><code class="language-python">import datetime from collections import namedtuple, OrderedDict import struct class Trans(namedtuple('Trans', 'key, desc, amt, db, cr')): &quot;&quot;&quot; TYPE Trans=Key:INTEGER; Desc:STRING[13]; Amt:REAL; DB,CR:BYTE &quot;&quot;&quot; struct = struct.Struct('&gt;h13s5sBB') @classmethod def unpack(cls, data): it = cls(*cls.struct.unpack(data[:cls.struct.size])) it = it._replace(desc=Basic09.string(it.desc), amt=Basic09.real(it.amt)) return it @property def indx(self): return self.key % 32 + 1 @property def date(self): r&quot;&quot;&quot; port of PROCEDURE DateStr Indx=MOD(Key,32)+1 \Copy=Key/32 Day=MOD(Copy,31)+1 \Copy=Copy/31 Month=MOD(Copy,12)+1 \Copy=Copy/12 Year=86+Copy &quot;&quot;&quot; copy = self.key / 32 day = copy % 31 + 1 copy = copy / 31 month = copy % 12 + 1 copy = copy / 12 year = 1986 + copy try: return datetime.date(year, month, day) except ValueError: # Nov 31??? return datetime.date(year, month, day - 1) def as_dict(self): return dict(date=self.date, indx=self.indx, desc=self.desc, amt=round(self.amt, 2), db=self.db, cr=self.cr) </code></pre> <p>I couldn't figure out how to decode the floating point account balances until I realized I was comparing them against the Feb 22 report rather than their March 12 values.</p> <blockquote> <p>Type REAL</p> <p>REAL numbers are stored in 5 consecutive memory bytes. The first byte is the (8-bit) exponent in binary two's-complement representation. The next four bytes are the binary sign-and-magnitude representation of the mantissa; the mantissa in the first 31 bits, and the sign of the mantissa in the last (least significant) bit of the last byte of the real quantity. -- <a href="http://www.roug.org/soren/6809/basic09.pdf">BASIC09: Programming Language Reference Manual</a> Copyright (c) 1983 Microware Systems Corporation</p> </blockquote> <pre><code class="language-python">class Basic09(object): @classmethod def string(cls, data): return data[:data.find('\xff')] if '\xff' in data else data @classmethod def real(cls, b5): exp, mag = struct.unpack('&gt;bI', b5) sgn = -1 if (mag % 2) else 1 mag = mag &gt;&gt; 1 mag = mag * 1.0 / (2 ** 31) return mag * (2 ** exp) * sgn </code></pre> <p>The overall file format is a linked list:</p> <pre><code class="language-python">class Global(namedtuple('Global', 'trx, head, tail, rec, avail, name, bal, file')): &quot;&quot;&quot; TYPE Global=Trx:Trans; Head,Tail,Rec,Avail:INTEGER; Name(32):STRING &quot;&quot;&quot; struct = struct.Struct('&gt;hhhh%ds%dsB' % (32 * 13, 32 * 5)) @classmethod def unpack(cls, data): trx = Trans.unpack(data) data = data[Trans.struct.size:] it = cls(*((trx,) + cls.struct.unpack(data[:cls.struct.size]))) ea = 13 name = [Basic09.string(it.name[ea * ix:ea * (ix + 1)]) for ix in range(32)] ea = 5 bal = [Basic09.real(it.bal[ea * ix:ea * (ix + 1)]) for ix in range(32)] return it._replace(name=name, bal=bal) def accounts(self): a = pd.DataFrame(dict(name=self.name, bal=self.bal), columns=['name', 'bal']) a.index = a.index + 1 return a def iter_trans(self, jrnl): here = self.rec while True: after, before = struct.unpack('&gt;HH', jrnl[here + Trans.struct.size:][:4]) here = after if here == 0: break yield Trans.unpack(jrnl[here:]) G = Global.unpack(Jrnl) print G.trx print dict(head=G.head, tail=G.tail, rec=G.rec, avail=G.avail) ut_accounts = G.accounts() ut_accounts.head(3) </code></pre> <pre><code>Trans(key=32767, desc='Delphi Bill', amt=46.80000001192093, db=10, cr=19) {'avail': 5443, 'rec': 0, 'tail': 5313, 'head': 607} </code></pre> <div> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>name</th> <th>bal</th> </tr> </thead> <tbody> <tr> <th>1</th> <td>Cash</td> <td>76.220001</td> </tr> <tr> <th>2</th> <td>Checks</td> <td>552.950000</td> </tr> <tr> <th>3</th> <td>Bank Account</td> <td>890.420005</td> </tr> </tbody> </table> </div> <pre><code class="language-python">journal = pd.DataFrame.from_records( (tx.as_dict() for tx in G.iter_trans(Jrnl)), columns=['date', 'indx', 'desc', 'amt', 'db', 'cr']).set_index(['date', 'indx']) journal = journal.merge(ut_accounts[['name']], left_on='db', right_index=True) journal = journal.rename(columns=dict(name='Source Name')) journal = journal.merge(ut_accounts[['name']], left_on='cr', right_index=True) journal = journal.rename(columns=dict(name='Dest Name')) journal = journal.sort_index() journal.iloc[6:9] </code></pre> <div> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th></th> <th>desc</th> <th>amt</th> <th>db</th> <th>cr</th> <th>Source Name</th> <th>Dest Name</th> </tr> <tr> <th>date</th> <th>indx</th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> </tr> </thead> <tbody> <tr> <th>1986-09-01</th> <th>11</th> <td>Books</td> <td>117.85</td> <td>1</td> <td>23</td> <td>Cash</td> <td>Books/Supplie</td> </tr> <tr> <th rowspan="2" valign="top">1986-09-15</th> <th>11</th> <td>Scholarship</td> <td>296.46</td> <td>7</td> <td>20</td> <td>National Meri</td> <td>U T</td> </tr> <tr> <th>15</th> <td>Scholarship</td> <td>78.54</td> <td>7</td> <td>1</td> <td>National Meri</td> <td>Cash</td> </tr> </tbody> </table> </div> <p>Computing running balances with pandas with <code>cumsum</code> was fun.</p> <pre><code class="language-python">def running_balance(journal): cr = journal[['cr', 'amt']].rename(columns=dict(cr='acct')) cr['col'] = 'cr' db = journal[['db', 'amt']].rename(columns=dict(db='acct')) db['col'] = 'db' db.amt = -db.amt ea = cr.append(db).sort_index() ea['bal'] = ea.groupby('acct').amt.cumsum() cum = ea.reset_index().pivot_table(index=['date', 'indx'], columns='col', values=['bal']) journal = journal.copy() journal.insert(len(journal.columns) - 1, 'Src Bal', cum.bal.db) journal['Dest Bal'] = cum.bal.cr return journal running_balance(journal).to_csv('ut-austin-journal.csv') running_balance(journal).iloc[6:9] </code></pre> <div> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th></th> <th>desc</th> <th>amt</th> <th>db</th> <th>cr</th> <th>Source Name</th> <th>Src Bal</th> <th>Dest Name</th> <th>Dest Bal</th> </tr> <tr> <th>date</th> <th>indx</th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> </tr> </thead> <tbody> <tr> <th>1986-09-01</th> <th>11</th> <td>Books</td> <td>117.85</td> <td>1</td> <td>23</td> <td>Cash</td> <td>182.15</td> <td>Books/Supplie</td> <td>117.85</td> </tr> <tr> <th rowspan="2" valign="top">1986-09-15</th> <th>11</th> <td>Scholarship</td> <td>296.46</td> <td>7</td> <td>20</td> <td>National Meri</td> <td>-296.46</td> <td>U T</td> <td>496.28</td> </tr> <tr> <th>15</th> <td>Scholarship</td> <td>78.54</td> <td>7</td> <td>1</td> <td>National Meri</td> <td>-375.00</td> <td>Cash</td> <td>260.69</td> </tr> </tbody> </table> </div> <p>And with that, we have recovered the journal data from the report:</p> <pre><code class="language-python">tx_lines[:2] + tx_lines[8:11] </code></pre> <pre><code>[u'Date Description Amount Source Name Src Bal Dest Name Dest Bal\n', u'----------- ------------- ------- ------------- ------- ------------- --------\n', u' 9-01-86:11 Books 117.85 Cash 182.15 Books/Supplie 117.85\n', u' 9-15-86:11 Scholarship 296.46 National Meri -296.46 U T 496.28\n', u' 9-15-86:15 Scholarship 78.54 National Meri -375.00 Cash 260.69\n'] </code></pre> <p>And we can compute account balances:</p> <pre><code class="language-python">src_bal = journal.groupby('db')[['amt']].sum() dst_bal = journal.groupby('cr')[['amt']].sum() bal = src_bal.merge(dst_bal, left_index=True, right_index=True, how='outer', suffixes=['_src', '_dst']).fillna(0) bal['balance'] = bal.amt_dst - bal.amt_src bal = bal.drop(['amt_src', 'amt_dst'], axis=1) bal = bal.merge(ut_accounts[['name']], left_index=True, right_index=True)[['name', 'balance']] bal.to_csv('ut-austin-accounts.csv', index_label='acct_num') bal[:3] </code></pre> <div> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>name</th> <th>balance</th> </tr> </thead> <tbody> <tr> <th>1</th> <td>Cash</td> <td>76.22</td> </tr> <tr> <th>2</th> <td>Checks</td> <td>552.95</td> </tr> <tr> <th>3</th> <td>Bank Account</td> <td>890.42</td> </tr> </tbody> </table> </div> Sat, 30 Dec 2017 00:00:00 +0000 http://www.madmode.com//2017/ut-austin-expenses/ Migrating Breadcrumbs http://www.madmode.com//2017/breadcrumbs-migrate/ <p><em>Breadcrumbs</em> was the blog of <a href="http://dig.csail.mit.edu/">DIG</a>, the Decentralized Information Group at MIT CSAIL.</p> <p>In a <a href="http://logs.glob.uno/?c=freenode%23microformats&amp;s=20+Jun+2015&amp;e=20+Jun+2015#c81549">2015 #microformats chat</a>, I discovered that it was down:</p> <blockquote> <p>DanC&gt; grr... the blog is down. http://dig.csail.mit.edu/breadcrumbs/node/228<br /> "Unable to connect to database server"<br /> <em>DanC verifies that he has an export of his work there...</em><br /> DanC&gt; interesting... my backup is evidently python pickles of XMLRPC responses from the API of that CMS (drupal?)<br /> &gt;&gt;&gt; x['dateCreated']<br /> &lt;DateTime '20080306T17:00:05' at 7f20e8aef5f0&gt;<br /> &gt;&gt;&gt; x['dateCreated'].<strong>class</strong><br /> &lt;class xmlrpclib.DateTime at 0x7f20e444eef0&gt; </p> </blockquote> <p>The files are numbered:</p> <pre><code class="language-python">def _numbered_files(pattern='[0-9]*', breadcrumbs='/home/connolly/sites/breadcrumbs'): from pathlib import Path return Path(breadcrumbs).glob(pattern) breadcrumbs_bak = list(_numbered_files()) sorted(int(f.parts[-1]) for f in breadcrumbs_bak)[:10] </code></pre> <pre><code>[4, 5, 6, 7, 8, 9, 10, 11, 12, 13] </code></pre> <p>Each is a pickled XMLRPC response:</p> <pre><code class="language-python">import pickle breadcrumbs_xmlrpc = dict((int(f.parts[-1]), pickle.load(f.open('rb'))) for f in breadcrumbs_bak) x = breadcrumbs_xmlrpc[228] x['title'], x['dateCreated'], x['dateCreated'].__class__ </code></pre> <pre><code>('hAudio for microformats mixtapes, in progress', &lt;DateTime '20080306T17:00:05' at 7fa8242a5320&gt;, &lt;class xmlrpclib.DateTime at 0x7fa82427cf58&gt;) </code></pre> <h2>MadMode blog pages</h2> <pre><code class="language-python">from collections import OrderedDict from __future__ import print_function from sys import stderr class BlogWriter(object): def __init__(self, pages): self._pages = pages def addPage(self, body, title, date, tags, published, slug): datestr = date.isoformat() headings = OrderedDict(title=repr(title), date=datestr[:10], tags=&quot;[%s]&quot; % (', '.join(&quot;'%s'&quot; % tag for tag in tags)), published=published) header = '\n'.join([&quot;%s: %s&quot; % (k, v) for k, v in headings.iteritems()]) yyyy = datestr[:4] page = (self._pages / yyyy / slug).with_suffix('.md') print(&quot;addPage: &quot;, page, tags, file=stderr) with page.open('wb') as out: out.write(header) out.write('\n\n') out.write(body.encode('utf-8')) def _madmode(): from pathlib import Path return BlogWriter(Path('/home/connolly/sites') / 'madmode-blog' / 'pages') mmwr = _madmode() </code></pre> <pre><code class="language-python">from time import mktime from datetime import datetime import re def drupal2md(body): body = body.split('&lt;/title&gt;', 1)[1] # remove redundant title body = body.replace('\r', '') # unix newlines return body def findTags(body): tags = [] for txt in body.split('&lt;'): if txt.startswith('a '): txt = txt[len('a '):] attrs = {} while '=' in txt and not txt.startswith('&gt;'): name, txt = txt.split('=', 1) name = name.strip() txt = txt.strip() _, value, txt = txt.split('&quot;', 2) attrs[name] = value txt = txt.strip() href = attrs.get('href', '') if 'tag' in attrs.get('rel', '') or 'del.icio.us' in href: if href.endswith('/'): href = href[:-1] tags.append(href.split('/')[-1]) return tags for postid, item in sorted(breadcrumbs_xmlrpc.items()): print(postid, item['title'], file=stderr) dt = datetime.fromtimestamp(mktime(item['dateCreated'].timetuple())) tags = ['breadcrumbs'] + findTags(item['content']) mmwr.addPage(drupal2md(item['content']), title=item['title'], date=dt, tags=tags, published=True, slug='breadcrumbs_%04d' % postid) </code></pre> <pre><code>4 On OpenID and comment policies addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0004.md ['breadcrumbs'] 5 little burst of PAW demo hacking addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0005.md ['breadcrumbs'] 6 DIG blog wish list addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0006.md ['breadcrumbs', 'connolly'] 7 Fire at Southampton... hope everything's alright soon addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0007.md ['breadcrumbs'] 8 Sourceforge is the place... to sell soap? addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0008.md ['breadcrumbs'] 9 Reflecting blog structure into the Semantic Web with SIOC? addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0009.md ['breadcrumbs'] 10 I'd rather be... addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0010.md ['breadcrumbs'] 11 PHP angst addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0011.md ['breadcrumbs'] 12 Shopping for a client-side blogging editor addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0012.md ['breadcrumbs', 'authoring'] 13 presented Issues in Semantic Web Logic to 6.898 addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0013.md ['breadcrumbs'] 14 xchat RFE: "mail a log of this chat to mbox@domain" macro addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0014.md ['breadcrumbs'] 15 U.S. papertrail: the federal register addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0015.md ['breadcrumbs'] 16 XHTML for computer science research papers and bibliographies addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0016.md ['breadcrumbs'] 17 ISWC buzz addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0017.md ['breadcrumbs'] 18 Why isn't bill payee set-up integrated with address book or yellow pages lookup? addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0018.md ['breadcrumbs'] 23 RDF Calendar, GRDDL, Microformats, and all that at XML2005 in Atlanta addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0023.md ['breadcrumbs', 'quality'] 24 SKOS, SIOC, and drupal taxonomy addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0024.md ['breadcrumbs'] 25 sorry about overriding your font size addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0025.md ['breadcrumbs'] 26 Ray Ozzie's take on diff/sync addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0026.md ['breadcrumbs'] 27 a fly-by of XACML addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0027.md ['breadcrumbs'] 28 MathML as a rule interchange format addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0028.md ['breadcrumbs'] 29 GRDDL transform wanted: National Information Exchange Model (NIEM) addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0029.md ['breadcrumbs'] 30 Go-Karting rush tainted by lack of OpenID for bug reporting about hypertext editing addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0030.md ['breadcrumbs'] 45 Toward richtext syndicated feed addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0045.md ['breadcrumbs'] 46 Toward better documentation of some schemas for the W3C digital library addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0046.md ['breadcrumbs'] 47 Brought my hockey skates with me this time addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0047.md ['breadcrumbs'] 52 Connecting DIG Student Projects to the MIT UROP listing addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0052.md ['breadcrumbs'] 55 Drupal, OpenID, and the Mac OS X Keychain addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0055.md ['breadcrumbs'] 56 Wikicompany? addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0056.md ['breadcrumbs'] 57 upgrade to CivicSpace? addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0057.md ['breadcrumbs'] 61 frbr:embodiment is enough without frbr:embodimentOf, no? addPage: /home/connolly/sites/madmode-blog/pages/2005/breadcrumbs_0061.md ['breadcrumbs'] 63 On Google, Jabber, and Jingle and good and evil in IM and IP networks addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0063.md ['breadcrumbs'] 66 Arpeggio in D, a little three chord ditty addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0066.md ['breadcrumbs'] 69 Fun with Policy Aware Web at UMD, AFS/SVN at CSAIL addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0069.md ['breadcrumbs'] 70 Using truth maintenance techniques in RDF stores? addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0070.md ['breadcrumbs'] 77 MadScientistMode addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0077.md ['breadcrumbs'] 78 RSS is dead; long live RSS addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0078.md ['breadcrumbs'] 82 python, javascript, and PHP, oh my! addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0082.md ['breadcrumbs', 'installation', 'javascript', 'python', 'quality', 'testing', 'programming'] 84 tabulator use cases: when can we meet? and PathCross addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0084.md ['breadcrumbs'] 85 bnf2turtle -- write a turtle version of an EBNF grammar addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0085.md ['breadcrumbs'] 86 formally closing the feedback loop addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0086.md ['breadcrumbs'] 87 Using RDF and OWL to model language evolution addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0087.md ['breadcrumbs'] 88 Toward integration of cwm's proof structures with InferenceWeb's PML addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0088.md ['breadcrumbs'] 89 Investigating logical reflection, constructive proof, and explicit provability addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0089.md ['breadcrumbs'] 90 Fun with Embedded RDF and DOAP for the GRDDL profile addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0090.md ['breadcrumbs'] 91 Toward Semantic Web data from Wikipedia addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0091.md ['breadcrumbs', u'connolly'] 92 Reflections on the W3C Technical Plenary week addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0092.md ['breadcrumbs', 'NCE'] 93 Getting (dis)organized for SxSWi in Austin addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0093.md ['breadcrumbs', 'Austin'] 94 Dates in drupal vs planetrdf addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0094.md ['breadcrumbs'] 96 Getting my Personal Finance data back with hCalendar and hCard addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0096.md ['breadcrumbs'] 97 A look at emerging Web security architectures from a Semantic Web perspective addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0097.md ['breadcrumbs'] 98 a quick take on Kiko, a nifty looking calendar service addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0098.md ['breadcrumbs'] 99 using JSON and templates to produce microformat data addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0099.md ['breadcrumbs'] 100 geocoding and hCards for airports from wikipedia addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0100.md ['breadcrumbs', 'geo'] 101 time, context, quoting, and reification addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0101.md ['breadcrumbs'] 102 no more life in a textarea: MozEx and emacs to the rescue! addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0102.md ['breadcrumbs'] 107 hacking soccer schedules into hCalendar and into my sidekick addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0107.md ['breadcrumbs'] 123 A step forward with python and sshagent, and a walk around gnome security tools addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0123.md ['breadcrumbs', 'web', 'policy', 'security', 'python', 'programming'] 124 Consensus and community review in open source and open standards addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0124.md ['breadcrumbs'] 127 busy day in #microformats addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0127.md ['breadcrumbs'] 129 Access control and version control: an over-constrained problem? addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0129.md ['breadcrumbs'] 130 citing W3C specs from WWW conference papers addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0130.md ['breadcrumbs'] 131 On GData, SPARQL update, and RDF Diff/Sync addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0131.md ['breadcrumbs', 'diff', 'sync', 'sparql', 'calendar', 'web+architecture'] 133 RDF, Microformats, and Javascript hacking in person at the 'tute addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0133.md ['breadcrumbs', 'mobile', 'javascript', 'microformats', 'travel', 'calendar', 'BOS', 'bos'] 135 webizing TaskJuggler addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0135.md ['breadcrumbs', 'calendar'] 139 WWW2006 in Edinburgh: Identity, Reference, and Meaning addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0139.md ['breadcrumbs', 'www2006', 'EDI', 'travel', 'web+architecture', 'URI'] 140 Exporting databases in the Semantic Web with SPARQL, D2R, dbview, ARC, and such addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0140.md ['breadcrumbs', 'www2006', 'EDI', 'travel', 'sparql'] 141 Equality and inconsistency in the rules layer addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0141.md ['breadcrumbs'] 142 fun with flock addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0142.md ['breadcrumbs', u'flock', u'writing', u'editing', u'drupal'] 146 converting vcard .vcf syntax to hcard and catching up on CALSIFY addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0146.md ['breadcrumbs'] 148 a walk thru the tabulator calendar view addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0148.md ['breadcrumbs', 'calendar', 'SeedApplications'] 151 Choosing flight itineraries using tabulator and data from Wikipedia addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0151.md ['breadcrumbs'] 154 OpenID, verisign, and my life: mediawiki, bugzilla, mailman, roundup, ... addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0154.md ['breadcrumbs'] 155 tabulator maps in Argentina addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0155.md ['breadcrumbs'] 156 how much do I want to know about drupal? addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0156.md ['breadcrumbs'] 157 on Wikimania 2006, from a few hundred miles away addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0157.md ['breadcrumbs'] 158 Stitching the Semantic Web together with OWL at AAAI-06 addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0158.md ['breadcrumbs', 'RdfAndSql', 'AAAI', 'public-sparql-dev', 'citation'] 159 On the Future of Research Libraries at U.T. Austin addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0159.md ['breadcrumbs', 'Austin', 'URI', 'web+architecture'] 160 ACL 2 seminar at U.T. Austin: Toward proof exchange in the Semantic Web addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0160.md ['breadcrumbs', 'Austin', 'semantic', 'web', 'logic', 'research'] 161 Talking with U.T. Austin students about the Microformats, Drug Discovery, the Tabulator, and the Semantic Web addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0161.md ['breadcrumbs', 'Austin', 'semantic', 'web'] 162 Wishing for XOXO microformat support in OmniOutliner addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0162.md ['breadcrumbs'] 163 Trip reporting with flock addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0163.md ['breadcrumbs'] 164 Adding Shoenfield, Brachman books to my bookshelf? addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0164.md ['breadcrumbs'] 165 Now is a good time to try the tabulator addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0165.md ['breadcrumbs'] 171 Celebrating OWL interoperability and spec quality addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0171.md ['breadcrumbs'] 172 A new Basketball season brings a new episode in the personal information disaster addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0172.md ['breadcrumbs'] 178 Modelling HTTP cache configuration in the Semantic Web addPage: /home/connolly/sites/madmode-blog/pages/2006/breadcrumbs_0178.md ['breadcrumbs'] 179 She's a witch and I have the proof (in N3) addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0179.md ['breadcrumbs'] 180 A design for web content labels built from GRDDL and rules addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0180.md ['breadcrumbs'] 187 The Mercurial SCM: great for lots of stuff, but not the holy grail addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0187.md ['breadcrumbs', 'python+scm'] 192 Collaboration and crime at a distance at HASTAC, WWW2007 addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0192.md ['breadcrumbs', 'openid', 'hastac', 'Duke', 'RDU', 'digital+media'] 193 IKL by Hayes et al. provides a semantics for N3? addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0193.md ['breadcrumbs'] 194 Linked Data at WWW2007: GRDDL, SPARQL, and Wikipedia, oh my! addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0194.md ['breadcrumbs', u'banff', u'grddl', u'www2007', u'travel'] 198 Units of measure and property chaining addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0198.md ['breadcrumbs'] 201 Soccer schedules, flight itineraries, timezones, and python web frameworks addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0201.md ['breadcrumbs'] 206 FOAF and OpenID: two great tastes that taste great together addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0206.md ['breadcrumbs'] 207 brainstorming, issue tracking, and problem reporting... with tabulator? addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0207.md ['breadcrumbs'] 214 Free Culture: Why buy the Amazon Kindle when you can give and get an OLPC XO-1 for the same price? addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0214.md ['breadcrumbs'] 221 I can only imagine... addPage: /home/connolly/sites/madmode-blog/pages/2007/breadcrumbs_0221.md ['breadcrumbs'] 228 hAudio for microformats mixtapes, in progress addPage: /home/connolly/sites/madmode-blog/pages/2008/breadcrumbs_0228.md ['breadcrumbs'] 229 sidekick calendar subscription for SXSW addPage: /home/connolly/sites/madmode-blog/pages/2008/breadcrumbs_0229.md ['breadcrumbs'] 240 The details of data in documents; GRDDL, profiles, and HTML5 addPage: /home/connolly/sites/madmode-blog/pages/2008/breadcrumbs_0240.md ['breadcrumbs'] 246 OpenID "Hello World" on apache still deep magic addPage: /home/connolly/sites/madmode-blog/pages/2009/breadcrumbs_0246.md ['breadcrumbs'] 250 DIG losing the battle with spammers again addPage: /home/connolly/sites/madmode-blog/pages/2009/breadcrumbs_0250.md ['breadcrumbs'] 251 migrating from danger/sidekick to android/G1 addPage: /home/connolly/sites/madmode-blog/pages/2009/breadcrumbs_0251.md ['breadcrumbs'] 252 Existentials in ACL2 and Milawa make sense; how about level breakers? addPage: /home/connolly/sites/madmode-blog/pages/2010/breadcrumbs_0252.md ['breadcrumbs'] 253 Map and Territory in RDF APIs addPage: /home/connolly/sites/madmode-blog/pages/2010/breadcrumbs_0253.md ['breadcrumbs'] </code></pre> <h2>PyData Tools</h2> <pre><code class="language-python">import pandas as pd dict(pandas=pd.__version__) </code></pre> <pre><code>{'pandas': u'0.17.1'} </code></pre> <pre><code class="language-python">items = pd.DataFrame.from_records(breadcrumbs_xmlrpc.values()) items.postid = items.postid.astype(int) items = items.set_index('postid') print(items.dtypes) items[['title', 'dateCreated']].sort_values('dateCreated').head() </code></pre> <pre><code>content object dateCreated object description object link object mt_allow_comments int64 mt_convert_breaks object permaLink object title object userid object dtype: object </code></pre> <div> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>title</th> <th>dateCreated</th> </tr> <tr> <th>postid</th> <th></th> <th></th> </tr> </thead> <tbody> <tr> <th>4</th> <td>On OpenID and comment policies</td> <td>20051024T23:28:49</td> </tr> <tr> <th>5</th> <td>little burst of PAW demo hacking</td> <td>20051026T20:12:18</td> </tr> <tr> <th>6</th> <td>DIG blog wish list</td> <td>20051026T20:14:27</td> </tr> <tr> <th>7</th> <td>Fire at Southampton... hope everything's alrig...</td> <td>20051031T11:59:08</td> </tr> <tr> <th>9</th> <td>Reflecting blog structure into the Semantic We...</td> <td>20051031T13:18:51</td> </tr> </tbody> </table> </div> <pre><code class="language-python">items.loc[[228], ['title', 'dateCreated']] </code></pre> <div> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>title</th> <th>dateCreated</th> </tr> <tr> <th>postid</th> <th></th> <th></th> </tr> </thead> <tbody> <tr> <th>228</th> <td>hAudio for microformats mixtapes, in progress</td> <td>20080306T17:00:05</td> </tr> </tbody> </table> </div> Fri, 29 Dec 2017 00:00:00 +0000 http://www.madmode.com//2017/breadcrumbs-migrate/ Making Secure IoT: seL4 on my Raspberry Pi 3B http://www.madmode.com//2017/sel4-rpi/ <p>I got seL4 running on my Raspberry Pi 3B tonight.</p> <p>Even though I worked with Dale Dougherty in the early '90s, I've been on the sidelines of the whole maker thing until September when Micro Center bundled a <a href="http://www.microcenter.com/site/content/google_aiy_preorder.aspx">Google AIY VOICE KIT with Raspberry Pi 3B for $35</a>.</p> <p><a data-flickr-embed="true" data-footer="true" href="https://www.flickr.com/photos/dckc/26502865629/in/album-72157690394355946/" title="AIY Kit"><img src="https://farm5.staticflickr.com/4517/26502865629_a8f62d67b5.jpg" width="500" height="305" alt="AIY Kit"></a> <br /> <script async src="//embedr.flickr.com/assets/client-code.js" charset="utf-8"></script> </p> <p>After I verified that it works as a voice device, I remembered the tantalizing <a href="https://research.csiro.au/tsblog/sel4-raspberry-pi-3/">seL4 on the Raspberry Pi 3</a> item from early this year. The build dependencies were greatly simplified by the <a href="https://github.com/SEL4PROJ/seL4-CAmkES-L4v-dockerfiles">seL4 dockerfiles</a>. Building the seL4-Test project went without a hitch:</p> <pre><code>mkdir sel4test &amp;&amp; cd sel4test repo init --config-name -u git@github.com:seL4/sel4test-manifest.git repo sync make rpi3_debug_xml_defconfig &amp;&amp; make ... [GEN_IMAGE] sel4test-driver-image-arm-bcm2837 </code></pre> <p>The only way to see the seL4 test project do its thing is via the serial console. Before I overwrote the working voice kit SD card, I wanted to test connectivity. I have plenty of experience with RS-232 serial cables (I even had a job in high school where I helped a tech by putting together serial terminal cables) but <a href="https://www.sparkfun.com/tutorials/215">RS-232 vs. TTL serial</a> is not just a matter of wires and connectors; the voltage levels are different. USB to TTL cables usually go for around $15, which is more than half of what I paid for the Pi!</p> <p>Meanwhile, this summer Micro Center had a beaglebone green wireless, which usually goes for around $50, on clearance for $20, and I couldn't pass it up. The beaglebone uses the same TTL levels and works fine as an ssh server, so I put together a cable</p> <p><a data-flickr-embed="true" data-footer="true" href="https://www.flickr.com/photos/dckc/38223910826/in/album-72157690394355946/" title="IMG_20171106_212900488"><img src="https://farm5.staticflickr.com/4569/38223910826_a8fe8f7bdf.jpg" width="500" height="281" alt="IMG_20171106_212900488"></a><script async src="//embedr.flickr.com/assets/client-code.js" charset="utf-8"></script></p> <p>After some <code>config-pin</code> tinkering, I confirmed that I could get UART1 and UART2 on the beaglebone to talk to each other (UART0 is debug outputonly), but I couldn't get any console output from the Pi to show up.</p> <p>After discovering a <a href="https://www.amazon.com/gp/product/B00QT7LQ88/">JBtek Raspberry Pi USB to TTL Serial Cable</a> on Amazon for $7, I ordered it and set the project aside.</p> <p>That didn't work either until I connected an HDMI monitor and keyboard and used <code>raspi-config</code> to enable the serial console. <em>I wonder if the beaglebone would have worked given that fix.</em></p> <p>With serial console hardware issues in hand, I loaded the SD card as instructed. The first few boot stages worked, but I struggled to <code>Hit any key to stop autoboot</code>. Minicom (remember minicom?) showed "Offline" so I turned off hardware flow control. Bingo:</p> <pre><code>U-Boot&gt; fatls mmc 0 50248 bootcode.bin 2818372 start.elf 35 config.txt 393408 u-boot.bin 4064956 sel4test-driver-image-arm-bcm2837 U-Boot&gt; fatload mmc 0 0x100000 sel4test-driver-image-arm-bcm2837 reading sel4test-driver-image-arm-bcm2837 4064956 bytes read in 328 ms (11.8 MiB/s) U-Boot&gt; bootelf 0x100000 ... Test suite passed. 119 tests passed. 42 tests disabled. All is well in the universe </code></pre> Thu, 09 Nov 2017 00:00:00 +0000 http://www.madmode.com//2017/sel4-rpi/ My Capability Security 2017 Wish-List http://www.madmode.com//2017/ocap-wish-list/ <p>Computers are getting faster, smaller, more connected, and more capable, but when it comes to security, <a href="https://medium.com/message/everything-is-broken-81e5f33a24e1">everything is broken</a>. Along with correct-by-construction software (e.g. <a href="http://adam.chlipala.net/cpdt/">Certified Programming with Dependent Types</a>), the best weapon I see is <strong>object capability discipline</strong>.</p> <p>Before I get into my wish list of projects and issues, I'd like to point out <a href="https://github.com/dckc/awesome-ocap">dckc/awesome-ocap</a>, my list of capability security technology that is ready to use today, including everything from <a href="https://sel4.systems/">seL4</a>, an open source operating-system kernel with an end-to-end proof of implementation correctness and security enforcement to <a href="https://sandstorm.io/">Sandstorm</a>, a self-hostable SAAS platform.</p> <h2>Secure module loading for node.js</h2> <p>The good parts of JavaScript line up well with object capability discipline, but support in node.js for some of the best parts is lacking and hence there's no guarantee that calling and enhanced <code>sqrt()</code> from some npm module will not send an HTTP request to launch missiles.</p> <p>Mark Miller demonstrated the feasibility of secure loading as far back as 2011 with <a href="https://github.com/google/caja/blob/master/src/com/google/caja/ses/makeSimpleAMDLoader.js">makeSimpleAMDLoader.js</a>. I'm trying to fully understand node's incomplete support for <code>Object.defineProperty</code> in <a href="https://github.com/drses/ses/issues/6">ses/issues/6</a>.</p> <p>Meanwhile, I'm having fun with Capper; see <a href="https://bitbucket.org/DanC/finquick">finquick</a>.</p> <h2>Sandstorm dev tools on Ubuntu 16.04</h2> <p>Ubuntu's 16.04 kernels handle pid namespaces in a way that interferes with the sandstorm dev tools. Tracking issue: <a href="https://github.com/sandstorm-io/sandstorm/issues/2526">2526</a>.</p> <p>I figured out <a href="https://gist.github.com/dckc/2e6b5c8029246ab38c16e254fc3d3f4d">how to build sandstorm apps with nix</a>, but without the dev tools, the edit/test/debug cycle time is too long.</p> <h2>Capability security for mainstream linux with CloudABI, capsicum</h2> <p>While considering alternative kernels, I ran into <a href="https://github.com/dckc/madmode-blog/issues/20">a linking issue</a> when trying to build a linux kernel that supports capsicum and CloudABI. (There was a PPA for capsicum for a while...)</p> <p>CloudABI or capsicum at work would be <em>so great</em>. But it's a long way off... we're struggling to migrate to SuSE 12 so we can try out Docker.</p> <p>For example, a research workflow app I maintain needs to be able to send mail, but - only from one address - only using templated bodies - only to users who have in some way asked for it</p> <p>Design sketch: at at investigator request time, user grants "capability to send app-template mail to addresses X, Y, Z".</p> <p>As a demonstration, I'm <a href="https://github.com/dckc/ZeroVault/tree/cloudabi_wsgi">porting ZeroVault to CloudABI</a> using a <a href="https://atlas.hashicorp.com/freebsd/boxes/FreeBSD-11.0-STABLE">FreeBSD vagrant box VM</a>. It's pretty fun since Ed fixes <a href="https://github.com/NuxiNL/cloudabi-ports/issues?utf8=%E2%9C%93&amp;q=%20is%3Aissue%20author%3Adckc%20">my issues</a> within a few hours of when I report them.</p> <h3>bus1, Capability-based IPC for Linux</h3> <p>I'm heartened by <a href="https://lwn.net/Articles/697191/">momentum around bus1</a>.</p> <p>On top of the lack of composability in the chmod/chgroup, there's a mounting kludge tower of stuff like SELinux and (to a lesser extent?) AppArmour. I was doing a storage audit and learning how <code>lsblk</code> gets the serial number of my drives. I had heard of udev and systemd, but I had no idea it uses <a href="https://en.wikipedia.org/wiki/Netlink">netlink</a> ("a more flexible alternative to ioctl") to communicate with the kernel.</p> <h2>Object capability discipline for Docker</h2> <p>Is this even possible? I can't get my head around the Docker security story.</p> <h2>Uniform, composable FFI and stdlib for pony</h2> <p>Pony aims to be a high-performance capability-secure language. I would love to see it make some inroads on golang: while go addresses (many of) the memory-safety issues of C/C++, its standard library is full of ambient authority and its type system dooms developers to lots of boilerplate maintenance.</p> <p>I'm struggling (mostly for time) to convince the pony community that a reasonably simple <a href="https://github.com/dckc/rfcs/blob/ffi-taming/text/0000-ffi-taming.md">policy</a> can eradicate ambient authority from the standard library.</p> <p>In discussion of my <a href="https://github.com/ponylang/ponyc/pull/301">network API PR</a>, I learned that the pony designers don't (yet) see interposition as a key component of robust composition.</p> <h2>Safe systems programming on seL4 and genode</h2> <p>The <a href="https://genode.org/documentation/release-notes/16.05">genode May 2016 release</a> included initial support for rust. I haven't managed to try it out. Support for pony on genode has only gotten as far as a <a href="https://twitter.com/ponylang/status/671971997753212928">Dec 2015 twitter exchange</a> as far as I know.</p> <p><a href="https://robigalia.org/">Robigalia</a> aims to be a persistent capability OS built on seL4 and rust.</p> <p>Rust on seL4 is pretty bleeding-edge: <a href="https://github.com/SEL4PROJ/rust-camkes-samples/issues/1">SEL4PROJ/rust-camkes-samples/issues/1</a> documents my trials and tribulations. https://github.com/seL4/refos looks interesting.</p> <h2>Object capability discipline support in rust</h2> <p>We supporters of the <a href="https://github.com/rust-lang/rust/issues/3094#issuecomment-9589749">2012 proposals to isolate ambient authority in the rust stdlib</a> didn't make our case well enough for the 1.0 cut-off, but there is renewed interst in <a href="https://internals.rust-lang.org/t/refactoring-std-for-ultimate-portability/4301">refactoring std for ultimate portability</a>. One result of this could be a std alternative with no ambient authority.</p> Mon, 02 Jan 2017 00:00:00 +0000 http://www.madmode.com//2017/ocap-wish-list/