Thursday, September 18, 2014

TL;DW for Clojure Data Science

Edmund Jackson talked at the 2012 Clojure/Conj, and you can see his talk here.

I took these notes as I watched it:
  1. What is "data science"?
    1. "That realm of endeavor that requires, simultaneously, advanced computational and statistical methods."
    2. Some people aren't sure whether "data science" is a thing, or just data analysis dressed up with a fancy name. That question amuses me.
  2. What's new, such that everybody suddenly cares about data science?
    1. widely available computing resources, open source tools such as R, and large amounts of data available in private companies and in public
    2. Compares to early days of Linux, when there was a bunch of new stuff that everybody could hack on
  3. Interactive tools aren't enough; you're not taking some data, analyzing it, and coming back with the answer. You need platform features like native language speed, data structures, language constructs, connectivity, and QC in order to embed your analysis in business processes.
  4. The tools with better analysis features (e.g., R, Mathematica) lack the platform features, and the tools with better platform features (he focuses primarily on C++ as his example here) lack the analysis features.
  5. Python is in the sweet spot, with platform features and (via numpy, scipy, and pandas) analysis features. But:
    1. It's full of mutable data!
    2. The mode of expression in imperative languages poorly matches the content of expression when you're dealing with maths.
  6. F#, Scala, and Clojure are all functional, and therefore (immutable data, more natural expression of maths) better alternatives than Python.
  7. Clojure yay! points:
    1. Native: Incanter, Storm, Cascalog, Datomic
    2. JVM: Mahout (ML on Hadoop), jBLAS, Weka (Java lib with many ML algorithms)
    3. Interop: Rincanter (call out to R), JNI
  8. From here he goes into calculating the entropy of a distribution, and the relative entropy of different distributions.
  9. Demonstrates using relative entropy fns in Datomic queries

Wednesday, September 3, 2014



  1. 6:27: Characteristics of a good API:
    1. Easy to learn
    2. Easy to use, even without documentation
    3. Hard to misuse
    4. Easy to read and maintain code that uses it
    5. Sufficiently powerful to satisfy requirements
    6. Easy to evolve
    7. Appropriate to audience
    1. Agility trumps completeness at this point.
    3. Flesh the spec out as you gain confidence.
    1. Write at least three plugins before your release.
  6. 19:35: Maintain realistic expectations.
    1. You won't please everyone.
    2. Aim to displease everyone equally.
    1. Functionality should be easy to explain.
  9. 26:27: Implementation should not impact API.
    2. Don't leak implementation details such as SQL exceptions!
  10. 29:36: Minimize accessibility of everything.
    1. Make names self-explanatory.
    2. Be consistent.
  12. 32:32: Documentation matters.
    3. Good design usually coincides with good performance.
  14. 40:00: Minimize mutability
    1. Rich Hickey makes a similar point here.
    1. Here's a bad example:
  20. 57:15: Avoid long parameter lists.

Friday, February 14, 2014

hostnames as commands

Several years ago, I adopted a practice I've realized I should write down. I have two shell scripts that live in ~/bin/:
james.mojo.home ~ $ cat bin/ssh-host

remote_host=`basename $0`
if ! ssh $remote_host $*; then
    echo from $start to
james.mojo.home ~ $ cat bin/mosh-host

remote_host=`basename $0`
if ! mosh $remote_host -- $*; then
    echo from $start to
And I have many symlinks in ~/bin/ that point to those scripts. For example:
lrwxr-xr-x 1 moquist staff 8 Jul 12 2013 aristotle -> ssh-host
lrwxr-xr-x 1 moquist staff 8 Jul 12 2013 -> ssh-host 
lrwxr-xr-x 1 moquist staff 8 Jul 12 2013 -> mosh-host 

Of course I also have ~/.ssh/config set up, and my SSH keys are all in the appropriate ~/.ssh/authorized_keys files on remote systems.

But once all that's done, if I want to log in to a system, I can just type the name of the system (with tab completion). If I want to pipe something into or out of a command on a remote system (via ssh-host only), the system name just becomes another command:
james.mojo.home ~ $ aristotle "w | grep eviluser || echo eviluser is absent"
eviluser is absent
james.mojo.home ~ $ aristotle cat somefile | grep bits-i-want
### elided ###
james.mojo.home ~ $ for h in aristotle plato plantinga kant; do echo ====$h====; $h ls | grep lostfile; done
Obviously these are contrived examples, and there are plenty of other ways to do the same things. I've just found it convenient to think of hosts as commands, and this approach has let me do that.