Last week I was in Raleigh, attending the second Clojure/Conj. The last time that I attended a Lisp conference was the 1986 ACM Conference on Lisp and Functional Programming. I am a Lisp guy. I took the famed “Structure and Interpretation of Computer Programs” course from Sussman and Abelson. I spent some time doing undergraduate research on Symbolics Lisp Machines. When Apple invested some energy into Dylan, I hoped that I’d be able to use a Lisp on a personal computer. Java pretty much ruined that. Over the years, I pretty much gave up on the idea of being able to use Lisp for my day to day work. So much so, that when I first heard Rich Hickey talk about Clojure, my reaction going in was unenthusiastic. By the end of Rich’s talk, he had my attention. Clojure has been doing some growing up since then, and I really wanted to attend last year’s Clojure/Conj, but wasn’t able to.
Almost all of my conversations at the conference involved the questions, “Why are you at Clojure/Conj” and “How did you get interested in Clojure”. I’ve answered the second question in the previous paragraph. The question of “why” boils down to three themes: Clojure itself, Data, and Clojurescript. I’m going to use these threes theme to report on the conference talks.
Clojure itself
Clojure is a Lisp dialect that runs on the JVM and has great interoperability with existing Java code. It has great support for functional programming, as well as several innovative features for dealing with concurrency.
Stuart Sierra started off with a talk that pointed out areas where people could learn beyond the books and online exercises that are available. In each of those areas, he also proposed projects that people could work on. One of the things that stood out for me was his use of the Clojure reader to deal with Java Resources. I always found Resources to be annoying, and the use of the Reader is a clever way to make them more palatable and useful.
Clojail is a system for executing Clojure code in a sandbox. The system is quite flexible and the applications aren’t just limited to security. I can imagine using clojail to implement something like the Sponsors described in the original Actor model. Anthony Grimes, one of the committers for clojail gave the presentation. He is 17 years old.
One thing that made me happy was to see the bridge building between the Scala and Clojure communities. Phil Bagwell, who pioneered many of the persistent data structures in Clojure is now at Typesafe, the Scala company. He came and gave a nice talk about Scala’s parallel collection classes. Perhaps these classes will one day find their way into Clojure Daniel Spiewak gave a very solid presentation on the computer science behind the persistent data structures in Clojure.
At many conferences a talk like Clojure on Android would be at the higher end. The technical level of the talks at the Conj was high enough to make the task of getting Clojure on Android seem mundane. This is to take nothing away from the very impressive work that has been done. There are some issues remaining like footprint and startup time, but it looks like some effort is going to happen at the Clojure core team level to make some of this possible. The thought of talking to a REPL running on a phone, or tablet is a tasty one.
Rich Hickey’s keynote reminded me very much of a Guido keynote at PyCon: a discussion of language issues that he was looking at, and a solicitation for discussion. Rich was very careful to say that the stuff he was discussing was not a roadmap, so I’ll repeat that disclaimer. Here are some of the items that stood out to me. Plans to allow multiple builds of Clojure – a regular version, a leaner deployment version, a really lean Android version, a super deluxe development/debugging version and so on. There is discussion about allowing the reader to be extensible, in order to allow new data types to be round tripped. I didn’t follow the history of ClojureScript, so it was useful to see that Rich is pretty committed to this idea, and that bits of technology might even be flowing “backward” from the ClojureScript compiler into Clojure on the JVM. I was also very interested on Rich’s view that the use of a logic system like that in core.logic would be a far better tool than a traditional type system. More on the logic system below.
The last talk of the conference was Sam Aaron’s talk on Overtone, which is a computer music system written in Clojure. The major point was that he used Clojure to define a language for describing computer music, much in the sam way that sheet music describes regular music. There was lots of cool music along the way, including a pretty good simulation of the sound portion of the THX commercial that often plays before movies. The description of that commercial fit in a single projected screen of code.
Data
One thing that I’ve been looking at recently is exploratory environments for working with “federated” data. I’ve grown to dislike the term Big Data, because it’s come to mean almost nothing, however, the ship has already sailed on that one. Most people would be familiar with the idea of sitting down in front of their relational database SQL command prompts, and issuing ad-hoc queries. As the use of varied kinds of storage systems grows, we are losing that kind of interactive relationship with data. Some of the people in the Clojure community have built some interesting data systems, and Clojure is itself amenable to exploratory work with data, between it’s orientation around functional programming, and a development style oriented around a REPL.
David McNeil talked about Revelytix’s federated (among RDBMS and RDF triple store) SPARQL query engine. Their system uses s-expressions to represents the nodes in a graph of stream processing nodes. These expressions are then compiled down to a form that can be executed in parallel using the Java Fork/Join framework. The operators in the s-expessions are mirrors of built in Clojure sequence functions, and can use and be used in Clojure expressions. It’s not hard to imagine extending the set of federatable storage systems.
Heroku’s Mark McGranaghan talked about viewing logs data. What he really meant was viewing log data as akin to a native data type on Clojure and being able to use Clojure’s built in functions on log data in a natural way. Heroku has built a system call Pulse which takes this view. I particularly liked the small functions that he defined for expressing the intervals for recomputing statistics. It’s the cleanest formulation of that kind of thing that I’ve seen, and it’s enabled by his thesis view and Clojure.
Nathan Marz has been doing some great work at BackType and now Twitter. At StrangeLoop he open sourced Storm, a set of general primitives for doing realtime computation. At the Conj, he was talking about Cascalog, which is a Clojure DSL for Hadoop. Both Cascalog and Storm are in use at Twitter. Cascalog is inspired by Datalog and targets the same space as Pig. Cascalog has the full power of Clojure available to it, as well as the power of Datalog. It’s a little unclear to me exactly how much of Datalog is supported, but this is a powerful idea. Imagine combining the best of Cascalog and the Revelytix system. The source code to Marz’s examples is on Github.
Clojure has a logic programming library, core.logic which is based on the miniKanren system developed at Indiana University by Daniel Friedman, William Byrd, and Oleg Kiselyov. Ambrose Bonnaire-Sergeant presented an excellent tutorial on logic programming in general, and miniKanren in particular. David Nolen talked about predicate dispatching, a much more general way of doing method dispatch, and talked about his plans to tie that together with core.logic. The surprise highlight in this area was that Dan Friedman and William Byrd came to the conference and did a BOF on miniKanren and their constraint extensions to miniKanren. The BOF was surprisingly well attended (over 60 people), due in part to Ambrose’s excellent talk earlier that day. A key philosophical point about miniKanren is that there is a straight forward mechanical conversion from a functional program to logical/relational (miniKanren) program. This looks very promising, and it has me thinking about mashups of miniKanren (core.logic) and Datalog (cascalog). Professor Friedman and his students have done some very important work in the Scheme area over the years, and it was a great experience to meet him and spend some time over dinner. After dinner, we were sitting in the hotel lobby, and David Nolen was walking Friedman and Byrd through the implementation of core.logic, which was ported from the Scheme version of miniKanren, and then optimized for Clojure. There was a free flow of ideas back and forth, and it was a great example of a collaboration between academia and practice (it’s hard to say industry because Nolen and company are doing this in their free time). This is one of the things that I’ve always hoped for around open source, and it was nice to see such a concrete example. MiniKanren is described in Byrd’s PhD dissertation, and in the book “The Reasoned Schemer“.
ClojureScript
ClojureScript is a Clojure compiler which emits Javascript, which is then run though Google’s Closure compiler. I’ve been doing some prototyping work using Node.js and HTML/Javascript, so ClojureScript looks kind of interesting, particularly because it is good at some the data intensive stuff that Javascript is so laborious at. There were three ClojureScript sessions. Chris Houser took us on a deep dive of the compiler, Kevin Lynagh show us some basic applications of ClojureScript in the browser, and David Nolen did a BOF where he showed off the browser connected REPL for Javascript. ClojureScript is still in its infancy, but it’s interesting nonetheless. Once David gets the constraint version of core.logic working in ClojureScript, it should get a lot more interesting.
Community
The thing that stood out to me about the Clojure community was the presence of the “young Jedi”, Anthony Grimes, and Ambrose Bonnaire-Sergeant. Both of them were able to attend their first Clojure/Conj (Anthony’s was last year) due to fundraising campaign initiated by Chas Emerick. Anthony is 17, and Ambrose has not yet graduated from college. Both of them are lead developers on highly technical projects within the Clojure community, and both did a great job of speaking in front of 300+ people who were mostly older than them. When I worked at OSAF, I worked with Stuart Parmenter, who started working in open source when he was 14. It’s great to work with these young, very gifted people, and I love seeing the community welcome and make a space for them.
The flip side of this is that like many open source, programming language oriented conferences, there were very few women in attendance. Perhaps the Clojure community could take a page from the very successful work that my friend Sarah Allen has done on RailsBridge.
Learning More
O’Reilly has finally recanted and is doing a Lisp book. Clojure Programming should be done soon, and Manning has Clojure in Action and The Joy of Clojure. If you are looking for an interactive way of learning Clojure, there is Try Clojure. Those looking to sharpen their Clojure skills can look at the Clojure Koans and 4Clojure .
The speaker slides from the Clojure/Conj 2011 are available on GitHub.
Update: corrected the name of Indiana University – thanks to Lindsey Kuper
Update: linked to a more up to date Overtone repository – thanks to Sam Aaron