Category Archives: programming languages

Strange Loop 2013

It’s been a while since I have written a post, or been to a conference. I wish that I had time to write more and that I could write about what I am up to. In lieu of that, here is a report on Strange Loop 2013, which is the only conference that I am attending all year.

Emerging Languages Camp

One does not attend Strange Loop without attending Emerging Languages Camp (ELC). I view the camp as a kind of roulette. Of course it’s unlikely that many, or perhaps any of the languages presented in the forum will ever see widespread adoption. That’s not really the point. The camp is about getting language designers and implementors together, and giving them a forum to exchange ideas. Much of what has happened in languages recently is that we’ve been taking old ideas and trying to recast them in a modern context, either because computing platforms are now fast enough to bear the inefficiencies of those ideas, or because the computing community as a whole appears to be of a mind to accept them. ELC is also just a place for people who want to experiment with languages. Here was some of what stood out to me:

Gershwin – an embedding of Forth semantics in Clojure – apparently this can be helpful when one is using the Clojure threading macros all over the place
Noether – a highly stratified approach to language design. Unfortunately this talk dwelt too much on the philosophy and mathematical semantics and ran out of time before covering the details of the actual language. It’s a shame that we didn’t get to see the full content.
Qbrt bytecode – This was an interesting look at a byte code system for a different design space, where the byte code was representing somewhat high level functionality as opposed to machine level instructions.
J – J is cool because it’s basically APL 2.0. But now we also have R, Julia, and other languages. This is a space that has many contenders but no clear leader. My problem with J is that it looks even more write only than Perl.
BODOL – This talk wasn’t interest so much for the language, which the presenter acknowledged was “another toy lisp”, but for the presentation, which involved HTML based slides being displayed inside of Emacs. I felt like I was back in 6.001 watching the construction of a lisp, but the presentation quality was really high, which made for a great talk.

In addition to the talks I had a number of dinner and hotel lobby discussions around ELC related topics. The ELC attendees are a great bunch.

Sessions

Jenny Finkel from Prismatic gave a great overview of how they use machine learning in the product. As a user this was a great explanation of how the system really works. Machine learning in keynotes is fough because it’s not something that everyone has studied so it is hard to get the level right. I definitely enjoyed it. The most useful information in the talk was when she said that coding on an iPad version has begun. I will definitely be using the living daylights out of that when it comes out.

This year there was definitely a theme around making it easier to deal with asynchronous / event-driven systems. I was able to make it to two of the talks – there were several more. Matthew Podwysocki and Donna Malayeri from Microsoft presented on the Rx extensions, which I’ve written plenty about on this blog already. This time they came with some cool Kinect based demos. Nice to see something other than web systems. The other talk that I saw was Rich Hickey’s talk on core.async. As usual, Rich brought a great mix of theoretical concerns around core.async’s CSP based model, while melding it with treal world system building. I consider this to be a hallmark of Rich’s style, and he is one of the very very few people who is really able to fuse theory and practice into a whole. And of course, Clojure is the manifestation of that fusion. I’ve got a bunch of David Nolen’s posts on core.async in tabs in my browser, but just haven’t had the time to sit down and read them. I feel a little more caught up now.

Another talk that I really enjoyed was Martin Odersky’s keynote on “The Trouble with Types“. The beginning of the talk was about the usual comparison between static and dynamic typing, and the end of the talk was about his work on Dotty, which is his new work on types based on “projecting higher kinded functional types onto names in modules”. He hopes that Dotty will eventually become the foundation for a future version of Scala. The interesting part for me happened in the middle of the talk, because that was the part where he admitted to some of the problems with types. Like needing a debugger for your type system because it had become so computationally powerful. The aha moment for me was around his discussion of how orthongonality in the type system design had contributed to the problems that he saw with Scala’s type system. It is a tenet among computer scientist that orthogonality is desirable. It was one of the foundational arguments in the CISC vs RISC computer architecture wars, and it is a badge of honor among language designers to have as much orthogonality as possible. However, orthogonality leads to a potentially large surface area of possibilities and interactions, which users need to master and be aware of, and which implementors need to implement completely and efficiently. On reflection, this should be obvious, but the lights went on for me during the talk.

I stopped into to see the latest installment of Friedman and Byrd’s journey with MiniKanren. I was very interested to see their exploration of the Chomsky hierarchy (of computational complexity). As usual, this is related to my interest in Clojure’s core.logic. They “cheated” a little bit in what they showed, but it was still interesting.

Avi Bryant gave a great talk on the applicability of abstract algebra, well mostly monoids, to the type of streaming calculations that are needed in many analytics systems. He showed how this provides a foundation for solutions like hyperloglog, min-hash, and count-min sketch.

Crista Lopes gave a talk called Exercises in Style which was about styles of programming. She observed that within teams, one can often figure out who wrote a piece of code just by looking at it. Another observation was that in art, students are taught what comprises the various styles and how to produce them. I thought that this was leading to a very promising discussion. She then presented the same program written in 9 different styles that she had observed. The first 3-4 styles (which have numbers and not names yet) were really what I was expecting to see. As we moved to more styles, they started to look like language paradigms, which I think is less interesting. Lopes is working on a book on this topic and has 33 styles that she’s identified so far. I’ll be keeping my eye out for that.

Another theme at Strange Loop this year was diversity. You saw it both in the speakers roster and in the session content. I didn’t get a chance to ask him, but I am pretty sure that Alex Miller made a very concerted effort to invite women speakers, because there was a much higher number of women speakers than last year, and also much higher than any other conference that I can remember. On the content side, there were several sessions. There was a good presentation on the history of women in computing. I definitely learned a bunch of things. The focus was on the early history of computing which was great, but I was disappointed that several prominent women in recent history were omitted. That’s the problem with histories, invariable someone gets left out. Some history is better than no history, especially on this topic. Alex also invited Jen Myers to do one of the keynotes. I’m not sure how to summarize this presentation because there are too many different angles. There was the angle about diversity, there was the angle about boosting education, there was the angle of making something good because we are good as people. So rather than try, I’ll just insert a Ray Bradbury quote that Jen used in her talk. This version is longer than the version in the talk, but it speaks to me.

I think it’s part of the nature of man to start with romance and build to a reality. There’s hardly a scientist or an astronaut I’ve met who wasn’t beholden to some romantic before him who led him to doing something in life.

I think it’s so important to be excited about life. In order to get the facts we have to be excited to go out and get them, and there’s only one way to do that — through romance. We need this thing which makes us sit bolt upright when we are nine or ten and say, ‘I want to go out and devour the world, I want to do these things.’ The only way you start like that is with this kind of thing we are talking about today. We may reject it later, we may give it up, but we move on to other romances then. We find, we push the edge of science forward, and I think we romance on beyond that into the universe ever beyond. We’re talking not about Alpha Centauri. We’re talking of light-years. We have sitting here on the stage a person who has made the film* with the greatest metaphor for the coming billion years. That film is going to romance generations to come and will excite the people to do the work so that we can live forever. That’s what it’s all about. So we start with the small romances that turn out to be of no use. We put these tools aside to get another romantic tool. We want to love life, to be excited by the challenge, to life at the top of our enthusiasm. The process enables us to gather more information. Darwin was the kind of romantic who could stand in the middle of a meadow like a statue for eight hours on end and let the bees buzz in and out of his ear. A fantastic statue standing there in the middle of nature, and all the foxes wandering by and wondering what the hell he was doing there, and they sort of looked at each other and examined the wisdom in each other’s eyes. But this is a romantic man — when you think of any scientist in history, he was a romancer of reality.

Alex has historically done a great job of getting great speakers for Strange Loop, and just just recent stars, but pioneers and old timers. This year we had Chuck Moore, Dan Friedman, and Douglas Hofstader. This year’s closing keynote was Hofstader, whose book “I am a Strange Loop” was the inspiration for the name of the conference. Hofstader’s talk was an exploration of that topic, and was everything that you could have hoped for given Hofstader’s amazing works of literature. What one could not have hoped for, however, was what followed. Alex commissioned David Stutz to do a multimedia performance based on Hofstader’s work. “Thrown for a Loop: A Carnival of Consciousness” was a performance that involved theater, a 5 piece brass quintet, Macintosh driven multimedia (including forays into Emacas and Clojure’s nREPL), and an aerialist. You will have to go and watch the video when it comes out, because I don’t have the words to describe it.

Miscellanea

Thursday night of Strange Loop we were treated to a conference party at the City Museum in St. Louis, which can only be described as part architectural museum and part playground, or as @petrellic put it “A habit rail for adults”. This was one of the most amazing venues that I have ever been to for a conference party. The three hours that we were allotted vanished quickly as we all explored the mysteries of the museums and its paths, trail, tunnels, stairways, and slides.

I’ve always had a little trouble describing what Strange Loop is to my coworkers and friends. I found a tagline, courtesy of @samberan: “Strange Loop is TED for programmers”.

Conference organizers take note: Strange Loop has seriously raised the already high bar on you.

The Sun sets on me

21 Replies

On Friday I was notified that I will not be making the transition from Sun to Oracle. Sun was a company filled with talented and energetic people, and I am grateful for the chance to work with them.

Pythonistas (and others) may be wondering what this means for dynamic languages at Oracle. I wish I knew. I don’t have any direct knowledge of this, since I’ve never actually spoken to anyone at Oracle about the topic.

I am definitely looking for another opportunity. During my time at Sun I’ve worked on a bunch of Python related stuff, as well as a few things related to cloud computing. Other skills in my repertoire include server side development (Java and Python), open source community work, and engineering management. I’m definitely open to different possibilities. The about page of this blog has my contact information, and my LinkedIn profile is a pretty good summary of my credentials.

Concurrency => Parallelism

3 Replies

I wanted to clarify a point from my post The Cambrian Period of Concurrency.

I made the statement

From where I sit, this is all about exploiting multicore hardware

because I’ve seen a pile of actor and other concurrency libraries which have not taken parallel execution of the concurrent program seriously. If I am going to go to the trouble of writing a concurrent program, then I want that execution to be parallel, especially in a multicore world.

Simon Marlow from the GHC team said that if programming multicore machines is the only goal we ought to be looking at parallelism first and concurrency only as a last resort. Haskell has some nice features for taking advantage of parallelism. However, I explicitly stated that I was not as interested in highly regular or data parallel computations, which is what Haskell’s parallelism tools are aimed at. These are fine ways to get parallelism, but I am interested in problems which are genuinely concurrent, not just parallel. In a Van Roy hierarchy, these are the problems with observable nondeterminism. I also specifically called out reduction of latency as one of my goals, something which Marlow says is a possible benefit of concurrency. The GHC team is interested in a different mix of problems than I am.

Van Roy in short

I also forgot to mention Peter Van Roy’s paper Programming Paradigms for Dummies: What Every Programmer Should Know, which includes an overview of his stratification of concurrency and parallelism (and other stuff). If you don’t have time to read his book, the paper is shorter and more digestible.

The Cambrian Period of Concurrency

20 Replies

Back in July, I gave an OSCON talk that was a survey of language constructs for concurrency. That talk has been making the rounds lately. Jacob Kaplan-Moss made referred to it in a major section of his excellent keynote Snakes on the Web, and Tim Bray has cited it as a reference in his Concur.next series. It seems like a good time for me to explain some of the talk content in writing and add my perspective on the current conversations.

The Cambrian

The Cambrian period was marked by a rapid diversification of lifeforms. I think that we are in a similar situation with concurrency today. Although many of the ideas that are being tossed around for concurrency have been around for some time, I don’t think that we really have a broad body of experience with any of them. So I’m less optimistic than Tim and Bruce Tate, at least on time frame. I think that we have a lot of interesting languages, embodying a number of interesting ideas for working with concurrency. I think that some of those languages have gained enough interest/adoption that we are now in a position to get a credible amount of experience so that we can start evaluating these ideas on their merits. But I think that the window for doing that is pretty large, on the order of 5 to 10 years.

What kinds of problems

The kinds of problems I am interested in are general purpose programming problems. I’m specifically not interested in scientific, numeric, highly regular kinds of computations or data parallel computations. Unlike Tim, I do think that web systems are a valid problem domain. I see this being driven by the need to drive down latency to provide good user response time, not to provide additional scalability (although it probably will).

It’s not like Java

Erik Engbrecht, one of Tim’s commenters said:

To get Java, you basically take Smalltalk and remove all of the powerful concepts from it while leaving in the benign ones that everyday developers use.

I think there’s something to be learned from that.

This presupposes that you know what all the good concepts are and what the benign ones are. It doesn’t seem like we are at that point. When Java was created, both Lisp and Smalltalk had existed for quite sometime and it was possible to do this kind of surgery. I don’t have a clear sense of what actually works well, much less what is powerful or benign.

The hardware made me do it

From where I sit, this is all about exploiting multicore hardware, and when I say this I mean machines with more than 4 or 8 hardware threads (I say threads, not cores – actual parallelism is what is important). The Sun T5440 is a 256 thread box. Intel’s Nehalem EX will let you build a 128 thread box later this year. Those are multicore boxes. If you look at experiments, you see that systems that seem to work well at 2 or 4 threads don’t’ work well at 16 or 64 threads. Since there’s not a huge amount of that kind of hardware around yet, it’s hard for people to run experiments at larger sizes. Experiments being run on 2 thread MacBook Pro’s are probably not good indicators of what happens at even 8 threads.. This is partially because dealing with more hardware threads requires more administrative overhead, and as the functional programming people found out, that overhead is very non-trivial. The point is, you have to run on actual hardware to have believable numbers. This makes it hard for me to take certain kinds of systems seriously, like concurrency solutions running on language implementations with Global Interpreter Locks. See David Beazley’s presentation on Python’s Global Interpreter Lock, for an example.

Comments on specific languages

At this point I am more interested in paradigms and constructs as opposed to particular languages. However, the only way to get real data on those is for them to be realized in language designs and implementations.

Haskell – Functional Laziness aside, the big concurrency thing in Haskell is Software Transactional Memory (STM). There are other features in Haskell, but STM is the big one. STM is an active research field in computer science, and I’ve read a decent number of papers trying to make heads from tails. Among the stack that I have read, it seems to be running about even between the papers touting the benefits of STM and the the papers saying that STM cannot scale and will not work in practice. The jury is very much out on this one, at least in my mind.
Erlang – I like Erlang. It’s been in production use for a long time, and real systems have been built using it. In addition to writing some small programs and reviewing some papers by Erlang’s designers, I spent a few days at the Erlang Factory earlier this year trying to get a better sense of what was really happening in the Erlang community. While there’s lots of cool stuff happening in Erlang, I observed two things. First, the biggest Erlang systems I heard described (outside of Facebook’s) are pretty small compared to a big system today. Second, and more importantly, SMP support in Erlang is still relatively new. Ulf Wiger’s DAMP09 presentation has a lot of useful information in it. On the other hand, BEAM, the Erlang VM is architected specifically for the Erlang process/actor model. This feels important to me, but we need some experimental evidence.
Clojure – Clojure as a ton of interesting ideas in it. Rich Hickey has really done his homework, and I have a lot of respect for the work that he is doing. Still it’s the early days for Clojure, and I want to see more data. I know Rich has run some stuff on one of those multiple hundred core Azul boxes, but as far as I know, there’s not a lot of other data.
Scala – The big thing in Scala for concurrency is Actors, but if you compare to Erlang, Actors are the equivalent of Erlang processes. A lot of the leverage that you get in Erlang comes from OTP, and to get that in Scala, you need to look at Jonas Boner’s highly interesting Akka Actor Kernel project. Akka also includes an implementation of dataflow variables, so Akka would give you a system with Actors, supervision, STM, and Dataflow (when it’s done).
libdispatch/Grand Central Dispatch – Several of Tim’s commenters brought up Apple’s Grand Central Dispatch, now open sourced as libdispatch. This is a key technology for taking advantage of multicore in Snow Leopard. GCD relies on programmers to create dispatch queues which are then managed by the operating system. Programmers can send computations to these queues via blocks (closures), which are a new extension to Objective-C. When I look at Apple’s guide to migrating to GCD from threads, I do see a model that I prefer to threads, but it is not as high level as some of the others. Also, the design seems oriented towards very loosely coupled computations. It will be several years before we can really know how well GCD is working. I am typing this post on a 16 thread Nehalem Mac Pro, and I rarely see even half of the CPU meters really light up, even when I am running multiple compute intensive tasks. Clearly more software needs to take advantage of this technology before we have verdict on its effectiveness in production.
.Net stuff like F#/Axum, etc – There is some concurrency work happening over on the CLR, most notably in F# and Axum. I spent some time at Lang.NET earlier this year, and got a chance to learn a bit about these two technologies. If you look at paradigms, the concurrency stuff looks very much like Erlang or Scala, with the notable exception of join patterns, which are on Martin Odersky’s list for Scala. I will admit to not being very up to speed on these, mostly for lack of Windows and the appropriate tools.

Other thoughts

Jacob’s take away from my talk at OSCON was “we’re screwed”. That’s not what I wanted to convey. I don’t see a clear winner at the moment, and we have a lot of careful experimentation and measuring to do. We are quite firmly in the Cambrian, and I’m not in a hurry to get out – these things need to bake a bit longer, as well as having some more experimentation.

In addition to my talk, and Tim’s wiki page, if you are really interested in this space, I think that you should read Concepts, Techniques, and Models of Computer Programming by Peter van Roy and Seif Haridi. No book can be up to date with the absolute latest developments, but this book has the best treatment that I’ve seen in terms of trying to stratify the expressiveness of sequential and concurrent programming models.

DjangoCon 2009

7 Replies

Last week I attended DjangoCon 2009 in Portland. Due to scheduling conflicts, I wasn’t able to attend DjangoCon last year, and I was disappointed that I missed that inaugural event. I’ve seen some Django stuff at PyCon, and I’ve written some Django code, being at a conference like DjangoCon helps me to understand the community and technology in a way that just reading the documentation doesn’t.

Talk Highlights

Here are some of the talks that I found notable:

Shawn Rider and Nowell Strite of PBS gave a talk titled: Pluggable, Reusable Django Apps: A Use Case and Proposed Solution. I think that very few people in the Django community had any idea that Django was being used extensively in PBS. That definitely falls into the category of pleasant surprises. One of the strengths of Django is the focus on building small single purpose applications which can be used to build up larger applications. Doing this is harder than it sounds, and Shawn and Nowell described some of the problems that they ran into, as well as showing some ways of dealing with those issues. There was some crosstalk between the PBS guys and the Pinax developers, who are also doing a lot with reusable apps. I hope that these folks will work to share and combine their knowledge and then disseminate that to the broader Django community.

There were a number of talks which followed the theme of “how not to use parts of Django”. It’s interesting because people like Django, and even if they don’t like some parts, they want to use the rest, and are willing to work to make that possible. You would expect people to just walk away from the framework in cases like these.

Eric Ocean, one of the developers of SproutCore gave what I considered to be a pretty interesting talk. Unfortunately, his talk didn’t have much of a connection to Django, other than to suggest some things that Django could do to support SproutCore better. I know from watching the IRC and Twitter back talk, that people were put off by the style of presentation (a little too like a commercial), and the weak connection to Django. SproutCore is interesting to me because it’s at a different level than most of the Javascript frameworks. It’s at a higher level, which I think will be necessary as browser based applications become more sophisticated. I know that I am going to be taking a closer look at SproutCore, and I hope that a useful Django/Sproutcore collaboration will emerge from the sprints.

Simon Willison gave a keynote about Cowboy programming. The big piece for me was his description of how the Guardian built an application to help the public scrutinize the expenses of British MP’s. There’s something about these situations that appeals to me, against my knowledge and better judgment of “sound software engineering practices”. I guess it’s a guilty pleasure of sorts.

Ian Bicking gave a keynote which might be described as “a free software programmer’s midlife crisis”. Ian was very philosophical and reminded us that free software (as opposed to open source software) was rooted in a set of moral (not economic or process) imperatives. It was a very thoughtful speech, and I think that its worth several reads of his text (something which is hard to do on a train ride with an iPhone) and some additional ponderings.

Avi Bryant’s keynote took its root in his experiences building Trendly. As one might expect, Avi started building Trendly using Seaside. But by the time he finished, he noticed that very little of Seaside was actually being used. He attributed this to the fact that Trendly’s architecture involves loading a single HTML, with a ton of Javascript. That Javascript then manages all of the interaction with the server, which consists of snippets of JSON data. This range true to me because we used a similar architecture for Chandler Hub, the web based version of Chandler (our interaction with the server was based on atom and atompub, not JSON), and it’s the kind of architecture that GMail is based on. Avi also treated us to a demonstration of Clamato, his Smalltalk dialect that compiles to Javascript. Again, another attempt to deal with the challenges of engineering large Javascript applications in a web browser.

There were plenty of other good talks, and many of the slides are already available.

My keynote

I’m afraid that I am not equal to the task of writing out my presentation text in full as Ian and Jacob have done, so you will have to settle for the highlights and wait until the video appears.

My keynote was organized around two major sections.

The first section was a look at what I see in the Django community at present. This includes a look at some pseudo statistics around job postings and a poll of web frameworks being used by startups in an effort to get some view into whether and how much adoption of Django is happening. The short answer is that things look promising, but there is still plenty of room to grow. On the technology side, I pointed out the emphasis on combining applications and the work of the Pinax and PBS folks. The other major technology thing that I called out was GeoDjango, which is undoubtedly the most sophisticated GIS functionality in any web framework in any language or platform. This is going to be very attractive to people building location aware mobile apps, and I showed two examples of augmented reality applications as illustrations. This section ends with some observations about the Django community, using the PyCon sprints as an example. Ok, there are also some lighthearted slides about Django’s mascot, the djangopony.

The remainder of the talk was about the ways that web applications are changing and how Django might adapt to them. There are (at least) three groups of people that will be impacted by these changes. From the view point of users, the two big things are richer, more interactive applications, and access from location enable devices. Developers are going to need help in dealing with these new requirements, and the people who operate web applications need much more support than they currently have.

I see several technologies that will be important in facilitating these changes. The first of these is some Rich Internet Application technology. The second is API’s to web applications. A digression on this point. When the iPhone was introduced, the only way to develop applications was using web technologies. This made a lot of people very angry, and Apple followed up with the ability to build native platform applications. It should be possible to build rich web interfaces on the iPhone. My observation is that given the choice beween a rich web interface and a native iPhone application, users pick the native application. Look no further than the furor over the native Google Voice application. The native applications are talking to the servers using API’s. Those API’s are not just cool Web 2.0 frosting. The last technology is cloud computing, which started out as a deployment/operations technology and is now moving up to impact application development at many levels.

In light of this, what are framework developers to do? I did a quick survey of several web frameworks which have interesting ideas or approaches in them, so that the Django folks could see what their “competitors” are up to. The frameworks that I included are Rails, Lift (Scala), Webmachine (Erlang), Nitrogen (Erlang), CouchApps (CouchDB + JavaScript – this isn’t quite a framework in the traditional sense, but it met the spirit of my criteria), and Javascript. In the case of Javascript, the observation is that the rapid increase in Javascript performance coupled with a good Javascript framework leads to something which is economically attractive (same technology in the server and client).

The talk finishes with a set of proposals for “science projects” that might be attempted in the context of Django. Some of what I outlined is emerging, and in some cases speculative. Django doesn’t need to blow itself up and start over. Instead, what’s needed is for people with Django sensibilities to look some of these problems and see if a Django flavored solution can be found. Here’s the list of projects:

Asynchronous Messaging – if there’s any use of messaging, it’s typically to do jobs in the background. What would happen if we made the use of messaging pervasive throughout the framework?
Comet – I think that the Django+orbited approach to Comet is limited in comparison to what you see in Lift or Nitrogen. Can Django do Comet support at the same level (or better) than these frameworks? What would happen if the Comet stuff were hooked up directly to the messaging stuff I just described? Imagine the equivalent of urls.py that routed Comet requests to messaging.
REST – There are several good packages for dealing with REST in Django. It would be nice to have this all packaged up neatly and made available for people.
Deployment – This is really a mess. Are there changes that could be made to Django to make it easier to deploy, or to work better with tools like Puppet, Chef, Fabric, etc?
Monitoring – Typically frameworks provide very little monitoring information. It seems like there is a lot that could be done here.
Analytics – Once you have raw monitoring information, the next step is to do some analytics on it. Django is famous for creating admin UI’s with a very small amount of effort. What if we applied that same thinking to analytics?
Cloud – If you add up the first 6 items, you are well on your way to what might be a cloud friendly framework. There is still some work to do in terms of making applications on the framework adapt to elastic deployment scenarios, but it would be a good step.
Stacks – A very basic step towards cloud stuff would be to build a preconfigured stack of software to run/develop a Django app. This is a controversial idea, because everyone has their own idea of what software should be in such a stack, and how all the configuration switches should be set. I still that having one (or more) such stack would help more than it would hurt. In my ideal world this stack would be delivered as a virtual machine image that could also be uploaded to cloud providers.

Here are the slides:

All of the talks were videotaped so those of you who were unable to attend will be able to catch up soon.
I had a great time hanging out with the Djangonauts. My thanks to Leah Culver and Robert Lofthouse for inviting me to speak.

Re: Snakes on the Web

1 Reply

Jacob Kaplan-Moss, one of the BDFL’s for Django has published the text and slides for his upcoming talk(s) “Snakes on the Web” at PyCon Argentina and PyCon Brazil. Jacob says that he’s trying to answer three questions:

What sucks, now, about web development?
How will we fix it?
Can we fix it with Python?

There’s some good stuff in here, and it’s definitely a worthwhile read.

Jacob is in South America, which means he won’t be at DjangoCon next week. I was disappointed about that before I saw his talk, and I’m even more disappointed now. I’ll be giving the last keynote at DjangoCon, and I’ll be discussing some thoughts and ideas for where Django (and other web frameworks) might go next. It would have been a great opportunity to carry on the conversation in person. I guess we’ll be doing by blog instead.

One of the topics that Jacob covered in his talk was concurrency, and he pointed to my OSCON talk on concurrency constructs as something that has influenced his thinking. I do think that he got the wrong idea from my conclusion. At the moment I don’t see a clear solution for concurrency, but I don’t believe that the situation is hopeless, either. I think that we are looking at a period where we have a lot of experimentation, and I think that’s a good thing. It is way premature to say that we have a solution, and I’d rather people keep experimenting.

I have some more thoughts on some of Jacob’s points, but those are in the keynote, so I’ll save those until after I’ve actually given the presentation at DjangoCon on Thursday. I suspect that the other keynote speakers, Avi Bryant, and Ian Bicking to have some thoughts in this general direction as well. I think it would be great to have an open space on these topics sometime on Thursday.

CommunityOne / JavaOne 2009

1 Reply

This was my second year attending these events as a Sun employee. Everything in software at Sun seems to revolve around these two events. For quite some time before the show, people are working away furiously getting things ready to be unveiled, myself included.

CommunityOne

For me, the big theme at CommunityOne was cloud computing. Sun itself was emphasizing cloud stuff and the latest release of OpenSolaris, 2009.06, which were the main topics of the CommunityOne general session. The Sun Cloud is due out sometime this summer, so much of the cloud part of the session was having partners come up and tell about their experiences working with our cloud. The OpenSolaris team has done a huge amount of work in 2009.06. The feature that stuck out to me the most is “Crossbow“, which is a completely rewritten networking stack. Solaris already had CPU virtualization technology built into it via the zones feature. Crossbow makes it possible to virtualize networking configurations. This means that you could run an instance of OpenSolaris on your laptop (either natively or via VirtualBox, VMWare or whatever) and actually have a virtualized data center configuration running right there. That’s pretty interesting stuff.

I went to several cloud sessions, and I’d have to say that the current state of cloud computing is pretty rough. At least that’s true at the Infrastructure as a Service level where Amazon and the Sun Cloud are. As an example, I went to a good presentation by fellow Sun employees on cloud computing patterns. I happened to be sitting with James Governor and Stephen O’Grady of Redmonk, and I turned to Stephen and said “these patterns are all at a level that I never want to have to worry about”. The patterns themselves were fine, but I personally don’t want to have to deal with things at that level in a cloud platform. There is lots of room for improvement and innovation in this space.

My CommunityOne talk was called “Programming Languages for the Cloud”.

Programming Languages For The Cloud

The talk is based on my experience as a language guy who has been asked to work on cloud computing stuff. As such, I’m really trying to raise questions (for which I don’t yet have answers) about places where work on programming languages might usefully intersect with cloud computing. I figured that this would be a niche kind of talk, so I was very surprised to find myself in one of the larger rooms at Moscone, complete with a live video feed. I was even more surprised to see that the room was pretty full. After the presentation, one of the Salesforce.com engineers working on Apex (their domain specific language for the cloud) came up to the front. We ended up having lunch and I learned a bunch interesting stuff about their experience withe Apex. This sort of thing is what makes conferences worthwhile.

JavaOne

I spent the first day of JavaOne prepping for my presentation, “Seeding the Cloud”, which was about some ways that tools could help developers who choose to build applications in the cloud.

Seeding The Cloud

Ashwin Rao and I had some pretty interesting demos lined up, but we had problems with the internet connection in the room so a number of the demos failed. I learned later that the internet connection for all of Moscone Center had gone out, which made me feel slightly better. As someone commented to me, it was a good illustration of some of the weak points of the cloud (web, really) model.

The demonstration that I really wanted to show was an extension of some work that the Kenai team has done. Kenai is going to have support for doing continuous integration via Hudson, and the machines for doing that can be allocated as cloud instances. This is great if you have a project in Java or some other language that has major build steps. Another use for a dynamically allocated farm of machines is to do web UI testing on browser combinations. Back at PyCon, I put a bug into Adam Christian and Mikeal Roger‘s ears about this. Adam and Mikeal are the primary guys behind the Windmill web UI testing framework. Adam has been working with Hudson author Kohsuke Kawaguchi, and between the two of them they came up with a way for Hudson to start up a bunch of different browsers on different operating systems. If my demo had worked, people would have seen me kick off a Hudson build from inside of Netbeans 6.7, and then we would have watched (via RDC) the various browsers running though some UI tests on a web application. Oh well.

I spent the rest of JavaOne ducking into various language and concurrency talks. Jonas Boner gave a very nice talk comparing some of the concurrency mechanisms that are available on the JVM. Alex Miller gave a talk on Java concurrency gotchas. The net effect of Alex’s talk was to reinforce the fact that we need one or more of the mechanisms that Jonas covered in his talk. Also in the concurrency vein, I stopped in on Philipp Haller and Frank Sommer’s ‘s talk on Scala Actors. Probably the most fun concurrency thing was a random conversation with Clojure author Rich Hickey and Jonas Boner in the speaker room.

JavaOne is big on the keynote / general sessions. I only went to two, the opening session, and Bob Brewin’s technical keynote. The big news (to me) in Bob’s keynote was Mark Reinhold’s demonstration of a modularized JDK. This is cool for a variety of reasons, like reducing the footprint of the downloads, ability to build distribution packages trivially, and so forth. But the thing that made me happies was news that the CLASSPATH is finally going way, to be replaced by a module-info.java file.

The opening general session was very subdued. There were a variety of partner / sponsor segments, but things were really running at a low energy level until the end when Scott McNealy took the stage and then introduced Oracle CEO Larry Ellison. Despite Ellison’s reassurances to the Java community, it was a sad moment. I’ve only been at Sun for a little over a year, but my history with the company is pretty long. When I was in grad school, Brown was on of the first large installations of Solaris (replacing SunOS). Like many developers, I’ve used Java over the years. Sun has made a number of very important contributions to the computer industry, and it’s sad to me that a company so full of innovation was unable to remain independent.

Photography

This year things were so busy and frenetic that I really didn’t have much time to pull out the camera. Between presentations and meeting up with Sun people from all over the world, there just wasn’t time. Here are a few from the few times that my camera escaped its bag:

Bob Brewin’s Technical Keynote

The Extra Action Marching Band on the CommunityOne Expo Floor

The CommunityOne Party

DSLDevCon 2009

2 Replies

I’ve been having trouble coming up with a good summary of the (Domain Specific Language) DSL DevCon. That’s partly because there was a lot of information to absorb between Lang.NET and the DevCon. Even more so, I’m finding it hard to distill what I saw, what I didn’t see, what I wanted to see, and what I think we need to see next. That’s odd because I’ve accepted the notion of DSL’s should be a part of the programmer’s toolbox ever since I sat through the “metalinguistic abstraction” section of Sussman and Abelson’s MIT class in 1984.

Reporting

I’m going to call out four talks that really stood out for me. There were more than just these four, but it was either these four or all of them, and all of them is too much work.

Guillaume LaForge’s talk on Groovy DSL’s was important because he not only showed how to build DSL’s using Groovy, but he’s actually working with real customers, like Mutual of Omaha, who are using those DSL’s in production.
I was happy to hear Markus Voelter’s talk Textual DSL’s and Code Generation with Eclipse Tools because a lot of the noise that I’ve heard on the DSL front has been coming from the Ruby and .NET side of the world. One thing that got my attention at the DevCon was the importance of tooling, so it was good to see that there are some tooling efforts in the Java space. It’s too bad that no one from JetBrains was there to present on MPS.
Brad Cross and Ted Neward did a talk entitled “Functional vs. Dynamic DSLs: The Smackdown”. I came away from this talk wanting more, and not in a good way. Brad and Ted really needed about 2 hours in order to give all the relevant background a chance to settle in. During the talk they presented a set of things which differentiated the functional programming and dynamic language styles of creating “Internal” (I really dislike the Internal/External terms) DSLs. Unfortunately, there wasn’t enough time to really dig in and explore the meat of what they said. I think that a deep addressing of the points that they made would be a very important contribution to the DSL topic. Maybe we’ll get to see a series of blog posts, developerWorks articles, or even an academic paper of some kind.
I view Intentional Software as one of those grand computer science projects. Having worked on Chandler, I have an appreciation for the perils of large, grand efforts. This is the first time that I had a chance to see a presentation by anyone from Intentional Software, and it is just as well that it was a demo of their just shipped product. I took note when the Intentional Software project was started back in 2002, but I’ve not heard a lot about their progress since then. What we saw was a demonstration of a production version of their “Domain Workbench” which is a system for allowing domain experts and programmers to work together to build a system which domain experts can then use to write software. Instead of writing programs, the programmers write the generator which takes the domain language (which can be visual) and then generates code. The system represents the domain information in a way that allows multiple, editable, “projections” (views). The demonstrations that we saw included an actuarial workbench, complete with mathematical notation, and an electronics workbench, expressed as circuit diagrams. If you are interested, your best bet is to watch the video when the videos are posted.

I am pretty impressed with what I saw, but there are lots of questions. How many domains can this actually work for? How hard is it to write generators? What’s the business model for domain workbenches? It seems pretty clear to me that for the domains and organizations where this can work, this approach is going to have a pretty sizable impact. Perhaps not this year, but within the next 5 years. I have to hand it to the Intentional Software guys. Their presentation was pretty low key, and they are going out of their way to not hype their stuff. They plan to work with a small number of customers to gradually prove out their approach. In an area which is highly susceptible to hype, it was refreshing to see people trying to keep expectations to a reasonable level.

The DevCon (and Lang.NET) were also my chance to meet two people who I’ve followed for sometime from afar: Ted Neward, and Larry O’Brien. Ted is well known and I’ve been following his blog for some time. He’s local to the Puget Sound area, and it’s probably just bad timing that we never met before this week. Larry O’Brien has been a commenter on my blog, as well as a responder on Twitter. I’ve appreciated his blog as well as the columns that he’s written over the years. It was great fun to run to the back of the room after each talk and see what the Twitter cabal (which included Larry) had to say about the material we had just seen.

Analysis

I think that DSL’s are inevitable. It’s remarkable to me how prescient Abelson and Sussman were when they defined three categories of abstraction: control abstraction, data abstraction, and metalinguistic abstraction. If you look at some of the recent frenzies in languages, you’ll see that we are mostly improving the ability of various languages to perform various kinds of abstraction. These concepts are not new, but they are appearing in languages which are approachable by today’s practitioners. Object oriented programming? Data abstraction. Closures? Control Abstraction. Pattern Matching/Algebraic datatypes? Data and control abstraction. DSLs and the capabilities needed to enable them? Metalinguistic abstraction.

Language as an abstraction is very powerful, and requires support from the underlying language as well as the tools. These two topics (as well as specific examples of domain specific languages) were the focus of the DevCon. The audience makeup appeared to be mostly language and compiler geeks. There were a few people (mostly consultants as far as I could tell) who write business applications, but this group was pretty small. This is important because most of the DSL’s presented were aimed at very computer science kinds of domains. If DSL’s are to have a broader impact, then it would be great to see more business people at events like this.

One thing which was not addressed at all was the process end of this. In order to build DSL’s for non computer domains, there has to be a collaboration between developers and domain experts. The Intentional Software guys recognize this via some “groupware” to facilitate this process. However, tooling alone is not enough to bridge this gap. I hope that we’ll be hearing reports on the process of collaboration between developers and domain experts as more and more people build DSL’s.

This is an interesting space, from a technical point of view. There is lots of cool language design and compiler stuff, some of my favorite topics. On the business end, it seems like there are some decent sized opportunities here, and that tooling is going to play a very large role — language support for DSL’s will be important, but may be overshadowed by the importance of good tools.

Update: the videos are now avaiable

Lang.NET 2009

5 Replies

Back in September, I attended the JVM Language Summit. This week I’ve been doing the Bainbridge Island to Redmond ferry commute in order to attend the Lang.NET symposium. Here’s some of the stuff that stood out to me.

Hopscotch

Gilad Bracha talked about Hopscotch, the IDE for Newspeak. Newspeak is one of those things that’s on my very long list of things to look at. For a Smalltalk or Lisp developer, much of the talk was familiar — an illustration of the power of writing the IDE for a dynamic language in a dynamic language. In fact Bracha asserted that the true power of Smalltalk is the dynamism in the IDE itself. I think that’s probably true today, since almost all popular IDE’s are written in a static language. The other interesting feature of Hopscotch is the pervasive use of a web browser style UI in the IDE. This looks promising, and gives me hope that a web browser based IDE might be feasible.

Newspeak is implemented atop the Squeak VM, and when questioned about this choice, Bracha said that it would be much harder to implement Newspeak on either the CLR or JVM (and he’d be a position to know about the JVM in particular). He was hopeful that John Rose’s work on dynamic extensions for the JVM would take out some (but not all) of the pain.

Powershell

At the JVM Language Summit, Rich Hickey gave a 15 minute talk that left me wishing he had been given 30 minutes. Jeffrey Snover’s 15 minute talk on Powershell left me feeling the same way. Powershell may not be a monad toting concurrency friendly uber functional language on the CLR, but it was compelling to me because of it’s sheer practicality and usefulness. I’ve been following Powershell via blog posts since it was announced, but seeing it demoed by its designer is another thing altogether. The Powershell folks have done some nice work to present shell users with a very consistent user interface. A side effect of this work makes it easy to make GUI’s that can output Powershell commands, and apparently in future versions of Windows, the GUI admin tools will do this. In addition to rationalizing the user visible behavior of the shell, Powershell is able to do some very impressive stuff in terms of remote execution, limited/secure execution, single machine transactions, and more.

I spent some time talking with Snover later in the day, and he seemed to think that UNIX shells could gain a fair amount of PowerShell’s capabilities by recognizing that pipes ship bytestreams, adopting a data format (like JSON or XML) for those byte streams, and proceeding from there. That might be true technically, but that would be a huge cultural change for that community.

Lars Bak on V8

Lars Bak talked about his work on V8. V8 is the nth VM that Bak has worked on, and my main takeaway from his talk was that the V8 team has tried to do a lot of careful measurement of real (and anticipated future) Javascript programs. They’ve then turned around and used those measurements to guide them in selecting from the techniques developed for Self, StrongTalk, and Hotspot. None of those techniques would be a mystery to people familiar with the literature. But the insight into this all applies to Javascript was pretty interesting.

Since this is all about performance, it’s interesting to note that V8 has just about doubled its performance during its one year life. Bak predicted that by 3-4 years in perrformance will have quadupled. There was a funny moment where someone asked about comparison’s with Firefox’s TraceMonkey tracing JIT. Bak’s reply was “let’s run the (Google V8) benchmarks“, which he then proceeded to do (V8 won). Bak is clearly competitive about this, which can only mean good things for Javascript users.

F# is something that’s been at the edge of my radar because of asynchronous workflows. Lang.NET was a chance for me to get a quick education on what that’s all about. It turns out that async workflows are an application of computation expressions, which are a way of doing monadically inspired language extension. Async workflows end up being a way of using CPS style tasking but with a nice syntax. Most of this became clear during Tomas Petricek’s talk on Reactive pattern matching for F#. The code that Petricek showed was very evocative of actor style code and the use of F# pattern matching was evocative of Erlang’s use of the same. One interesting point was the use of join patterns, something that doesn’t have a direct analog in either Erlang or typical actor implementations.

Amanda Laucher‘s talk Concurrency for F# was a discussion of a client engagement that she did using F# and the concurrency features of the language. The application in question was an insurance application, and the speedup numbers looked pretty interesting. This is notable because most of the concurrency examples that you can find are outside the domain of “business programming”. Having this kind of data is really useful for people interested in the concurrency space. Amanda’s work was done as a Microsoft case study, and published version will be available at some point.

Erik Meijer

Erik pretty much gets his own section because he’s that entertaining/interesting. This time he was talking about the Livelabs Reactive Framework, where he described a way to use LINQ for distributed computations, using the specific example of an AJAX dictionary based suggestion function. Erik started by using (throwing) coins to illustrate the IEnumerable and IObservable interfaces. He then used mindless symbol pushing to demonstrate that these two interfaces were duals of each other, and that they obeyed the rules for monads. That being true, he then showed how you could define observable collections which could be used with LINQ.

Tidbits

Herman Venter talked about a Common Compiler Infrastructure for the CLR. There’s some nice stuff in here if you are implementing a language on the CLR. One user of this infrastructure is the Code Canvas, a “spatial development environment”. It’s good to see that people are stretching the notion of what a development environment should be. The thing that stuck out to me most from this talk was not the technology (which is good), but Venter’s call to action at the end. The CCI is being open sourced, and Venter’s call for participation and contribution was indistinguishable from the same end of talk call at any open source conference. A sign of changes underway.

Joshua Goodman did a product rundown on What’s new in CLR V4 for Languages. There’s several years worth of changes to the CLR queued up for V4, which will be a big update. Most interesting to me was the inclusion of a Hill Climbing algorithm for allocating / managing threads. This strikes me as a kind of policy decision, something which you might want to plug into the CLR as opposed to having it installed under the hood. It will be interesting to see how this ends up working out in production settings.

Deja Vu

In a previous life, I did some work on database programming languages, and I experienced deja vu twice during the conference. The first was during Philip Wadler’s Links: Web programming without tiers. The goal of Links is enable front end and back end web programming using a single language. Wadler didn’t discuss the front end part during his talk. He focused on the back end, particularly on the ability to translate programming language code into SQL, in a fashion slightly beyond LINQ. This is done using a combination of the monadic techniques used for LINQ, as well as a type/effect system which is used to help determine whether a piece of code is SQLizable or not. The work was eerily familiar, and when Wadler quoted Limsoon Wong and Leonid Libkin’s work on Kleisi, I knew why. We looked at their work on structural recursion (and comprehension notation) for our work on list and tree queries.

The last talk(s) of Lang.NET was a double header talk on Oslo/M by Paul Vick and David Langworthy. David and I were officemates while we were graduate students at Brown. Even more deja vu was induced when I saw David demoing how the M toolchain could generate SQL schemas and queries. As he put it to me afterwards – “this is all that stuff we were talking about 10 years ago”. Well, closer to 20 now, but who’s counting. M is in its very early days – I think it will be more interesting when it can generate CLR code to match the SQL.

Last bits

Thanks to Harry Pierson for making sure that I got an invitation to Lang.Net. People were very welcoming, even though I had to ask questions about various bits of Microsoft alphabet soup. I suspect that this is the only conference I will go to all year where Macs are the minority. It’s probably good for me to get out of the bubble every once in a while…

Overall, I found the topics to be a little more broad ranging than the JVM Summit. There were quite a few talks on tools, while there were comparatively fewer at the JVM summit, and those tools discussed at the JVM summit were really aimed at language implementors. One thing that worked well with the JVM Summit was allowing for some open space sessions so that attendees could get together and discuss topics of mutual interest, often based on what had been presented during the sessions. I think that some similar time slots would have enhanced the experience.

I definitely came away impressed by some of the work going on in the CLR ecosystem, and I hope that the exchange between the JVM and CLR communities will continue and expand.

Update: Videos of the talks are now available.

Best PyCon Evar

5 Replies

I probably should have chosen a different title for this post, because at the rate things are going for PyCon, I’ll just have to use the same title again for the next few years. This year, PyCon happened during the same week as ApacheCon EU (the 10th anniversary of the ASF), and EclipseCon. I have a slight bit of regret that I wasn’t at ApacheCon for the 10 year anniversary, but I’m planning to be at the 10th anniversary celebration at ApacheCon US in Oakland, in November. That roughly corresponds to the time when first got involved with Apache and open source, so it will be pretty meaningful. Beyond that, it was hands down for PyCon, my favorite conference. Even if the PyCon organizers hadn’t invited me to speak on a topic of my choosing, there are just so many things to love about PyCon.

The Talks

Despite a very active and fun hallway track, I did go to a number of talks.

I went to Adam Christian and Mikeal Rogers‘ talk on Windmill mostly for moral support. We worked together at OSAF, and I like Windmill, and it’s really good to see Windmill picking up steam in the Python and other communities. If you are looking for a web testing framework, particularly one that is string at AJAX applications, you owe it to yourself to look at Windmill.

There were a few tools talks that I attended. I use IPython, so I was curious to see how Reinteract: a better way to interact with Python, would improve on IPython. I like the Visicalc/TkSolver like worksheet that allows you to change values in a Python interpreter history and have values propage forward. I’d love to see all these REPL tools come together in an integrated way. We might finally get back to the functionality of the Lisp Machine REPLs someday. I also attended How AlterWay releases web applications using “zc.buildout“ since Jacob Kaplan-Moss warned me that the zc.buildout documentation was sorely lacking. Even that talk wasn’t enough to get me going, but the sprints produced some great new documentation for buildout. I’m looking forward to digging into that.

Some talks dealt directly with topics that are relevant to work, particularly now that the dynamic languages folks at Sun are now a part of the Cloud Computing division. These talks included:

Twisted AMQP and Thrift: Bridging Messaging and RPC for building scalable distributed applications – Twisted bridges to AMQP and Thrift.
Concurrency and Distributed Computing with Python Today – Jesse Noller did a great job surveying the various offerings available in Python today. There’s a lot of stuff there, but I think that there’s still quite some way to go yet. That’s not picking on Python, that’s just my general view of this space.
Drop ACID and think about data – Bob Ippolito did a really nice survey of the various non-relational/non-transactional data storage options out there. Bob actually tried many of these, so the survey is useful for weeding out systems aren’t really ready for prime time. A must view if you haven’t been paying attention to this space.
Pinax: a platform for rapidly developing websites – I’ve been following Pinax via Twitter for some time now, and James Tauber and I were involved at the beginning of the Apache XML project almost 10 years ago. Despite all that, we’ve never actually met in person until this week. James had a tough job with his talk. Pinax is very new, so he could either talk for the people who didn’t know what Pinax is, or he could talk to people wanted to know where things were. James knew this was going to be a problem and said so in his talk. And it was, at least for me. Fortunately, I managed to sit down with James at the sprints and get my questions answered. Zed Shaw recently wrote a (very positive) review of Django. That’s interesting since Zed was a hard core Rails guy. It’s also interesting because he called out Django’s emphasis on modularity and Pinax as an example of that modularity. My questions about Pinax were mostly about what (if anything) Pinax has done to build on the modularity provided by Django. At the moment, the various Pinax components cooperate mostly via conventions. Things are still early in Pinax, and I wasn’t surprised to hear this. James did say that some conventions were close to getting codified/documented/supported by the framework, which is what I am really interested in. In some ways, the data representation and modularity problems are similar to the kinds of problems that we were trying to solve for Chandler. Pinax is in the social application domain and Chandler is in the PIM domain, so while there are some similarities there are also differences. I’ll definitely be sticking my nose a bit deeper into the Pinax checkout that’s been sitting on my hard disk.

The most entertaining talk that I attended was Ian Bicking’s Topics of Interest. Ian took the invitation to speak on something of interest quite literally which created an air of mystery. In the end, Ian prepared some slides (some of which were quite thoughtful and introspective), used an instance of the new Google Moderator to queue up some audience questions, and created an IRC backchannel which he kept on the screen during his talk. The result has to be watched (and the video is already up) to be understood. It was quite hilarious, with the exception of some unpleasant commentary after someone in IRC asked “why aren’t there more women at PyCon”. The resulting IRC conversation only serves as an explanation for why. Many people felt this way, and discussion of this spilled out into Twitter, and I hope that perhaps we can change things for the better.

I gave my talk, Challenges and Opportunities for Python, and got a pretty good reception. I had a number of hallway and other conversations with people based on the content. I think that I was successful in giving people a perspective on the dynamic language world as a whole, on Python’s place in it, and some things that we might be able to do in order to grow. You can watch the video and make your own assessment, and decide if there are actions worth taking.

This year the conference is benefitting from a great new website (built in Django), and you’ll find the slides and video for each talk on the links. The video team is doing a great job of cranking out the video, so all of them should be up soon, or you can go to pycon.blip.tv to see them all together. Here are some talks that I am going to be checking out:

About Python Namespaces (and Code Objects)
Coverage testing, the good and the bad. (And for the nth PyCon, I managed to not meet Ned Batchedler! – I miss the CrowdVine that was used last year, along with “meet the author” button that would send you to CrowdVine)
Behind the scenes of EveryBlock.com – plug GeoDjango
Lack of design patterns in Python
Class Decorators: Radically Simple
Abstraction as Leverage
Paver: easy build and deployment automation for Python projects – (I didn’t meet up with Kevin Dangoor either)

The Lightning Talks

I put the lightning talks in a separate category from the talks because they are a phenomenon at PyCon. This year there were two lightning talk sessions, one at the beginning of each day and one at the end of each day. That’s 6 sessions of lightning talks! Jacob Kaplan-Moss only allowed signups for the next session, and it was truly first come first serve (without last year’s arrangement with the sponsors). There were a number of really good lightning talks. There really isn’t a good record of what got presented except perhaps on Twitter. A search for #pycon should get most of it.

Update: the lightning talks were also video’ed and will be posted on pycon.blip.tv

The Sprints

The PyCon sprints remain a phenomenon. While I don’t think quite as many people stayed this year as last year, there were still a lot of people — enough to fill the basement conference rooms at the Crowne Plaza hotel, and enough to need one of the ballrooms to serve lunch and dinner in. Once again, I hung at the Jython sprint, and wandered in and out of the Django and Pinax sprints. During the two days of sprints that I stayed for, I observed the folks working on ctypes for Jython actually crashing the JVM. SQLAlchemy started to really run on Jython and so did Twisted. Four days of hacking with the core developers of a project generally tends to produce results. So does spending time to bring new people from the community into your project.

I reported a bug in Django as I tried to get buildout setup to do Django on MySQL. I’m talking about Python and MySQL at the MySQL conference in a few weeks, so I was working on my example code. Turns out that MySQLdb doens’t build cleanly on the Mac. The trunk version almost builds cleanly, so I used that, but that version chokes something in Django. Before I discovered that I had done some gymnastics involving a git-svn clone of MySQLdb, a push of that to github, and a git recipe for buildout. I never quite got the git/buildout part working and I decided that it was overkill and that’s when I finally discovered that the trunk didn’t work with Django.

Of course, the sprints are also a time to catchup with/meet people in the community. It’s a time when there are friendly rivalries, joking, and alcohol. One of the momentous occasions during 2008, was that Django got a pony.

The exuberant Django people decided to bring the pony to PyCon…

Guido decided that he wanted the pony…

This all made for great fun and entertainment, which then spilled over into the sprints as a three way Python Core/Django/Pinax feud, which lead to things like this and this. This is hard core fun, people.

Overall Conference Commentary

The organizers estimated the attendance for this year’s PyCon at around 900 people. That’s a slight decline from last year, but the economic situation is much much worse than it was last year. I think that a 10% decline is a huge success, and a testament to the growth of interest in Python and it’s surrounding ecosystem.

From an organizational point of view, PyCon is continuing its tradition of being a mostly volunteer organized conference. It this respect it is a tour de force, at least in the space of open source conferences. PyCon is using a production company to assist, just as ApacheCon is, but the on site footprint of that company is much smaller than the on site footprint of the company for ApacheCon. Moreover, the number of volunteers helping with things is just enormous. Session chairs, runners to escort speakers from the green room to their sessions, a web site builder, lightning talk coordinator, open spaces coordinator, greeters at the conference desk, photographers, and I’m sure there are a bunch more people whose roles I didn’t even get to hear about. Absent a fancy lighted stage display for keynotes, production value wise, I feel that PyCon is operating at the same level of quality as any of the O’Reilly conferences. The program was excellent – tutorials, keynotes, invited talks, regular talks, open spaces, and lightning talks.

With PyCon, the Python community is getting way more mileage out of its face to face time than any other open source community. The combination of lightning talks, open space, and sprints creates a powerful feedback loop within the conference proper, which then extends into the sprint days. This dynamic has evolved over the years as PyCon attendees have come to understand the role of these vehicles. Here’s how it works:

The lightning talks allow anyone, regardless of stature, influence, or reputation to get in front of the entire conference. People now recognize that some of the most interesting, surprising, and entertaining moments of PyCon take place during the lightning talks. It’s a measure of the influence of the lightning talks that even the 8AM morning lightning talk sessions were well attended. At other conferences the morning sessions are reserved for keynote presentations by paying sponsors. I usually skip these because the content value is low. But I definitely got up to make sure that I hit those 8AM lightning talks. If you’ve gotten in front of the community with a lightning talk, you can extend your reach by scheduling an open space session.

Above is a shot of the open space board for Saturday. Note that the time slots go from 10AM to 10PM. There were a few prank type sessions, but for the most part, that board really is full all day long with 10 rooms available during each one hour time slot. Consider that there were 4 ballrooms for the talks, and that the talks went from 10:20AM till 5PM. There was way more air time in the open space sessions, and people certainly made use of it. This is why PyCon is a working conference – it’s not only about transfer of information, real work gets done there.

The only tricky thing with open space is that it would be great to have electronic access to the contents of the open space board during the conference. That would help make the open spaces a first class citizen in the minds of attendees. This is an interesting problem, because part of the value of the open space is the physical board, so turning it all electronic wouldn’t be a good idea. I wonder if Kaliya Hamlin has an experience with this sort of thing.

Used well, the open space sessions are great for organizing your little (or big) slice of the world wide Python community. They are also great as a prelude to a sprint once the conference has finshed. And as I’ve already mentioned, the sprints are a great time to reinforce a project’s community as well as move it forward.

PyCon 2009

All of this notwithstanding, the PyCon organizers are not sitting on their laurels. They keep on looking for ways to improve the conference. The buckets you see above are an example of this. Instead of paper or electronic surveys, attendees were asked to vote for talks by taking a red chip and tossing it a bucket on their way out the door. Green for good, yellow for ‘meh’, and red for bad. This is way less effort than the surveys, and I observed a decent number of people putting in their chips. Doug Napoleone has more on the origins of this system, as well as a pointer to the raw data on the results.

Twitter is now in the mainstream at PyCon. Guido mentioned Twitter during his keynote, and used it to ask questions during the conference. One of James Tauber’s first slides told people which hashtag to use when covering his talk. I’d guess that I got at least 20 new followers each day of PyCon, and I think that I might even be trained to use hashtags now. #pycon was in the top 10 Twitter during the days of the conference. The takeway is that if you are going to a conference and you are not on twitter, you are missing out. The corollary is that if you are a conference, and you aren’t making use of twitter, you need to pay attention. Ian Skerrett has an interesting post on how they used Twitter during EclipseCon. One thing that was missing was a video display of the search for #pycon. I know from talking with Doug Napoleone that he has some wonderful ideas for taking all the social networking stuff to the next level. I’m really looking forward to seeing that next year.

Photography

I’ve been to a lot of conferences over the last few years, always with a camera in hand. At each conference I shoot less and less. There are now lots of people swarming around with cameras, and I feel a bit done out with shots of people speaking from the front of a room, rows of white male attendees listening to a talk, and the rest of the usual conference shots. The same thing happpend with me and liveblogging conferences. Also, it’s hard to do the hallway track and do decent photography. Last year, the PyCon organizers asked me to take some official pictures, which I was happy to do. This year they didn’t (which was fine by me), but I had planned to bring the camera anyway, because PyCon is PyCon, and photographing there is one way that I try to give back to the community.

It turns out that the organizers were way more organized about photography this year. They actually had someone to coordinate the photography for the conference. Steven Wilcox had a last minute emergency and couldn’t make it. I found out about all of this just a half an hour before I left for the airport. Steven had planned to do headshots of Pythonistas, and was planning to get studio lighting equipment and so on. All of that was now up in the air. Since I had done a bunch of headshots of ASF people at ApacheCon, I tossed some Strobist lighting gear into my suitcase, just in case. By the time I landed in O’Hare, Erich Heine had stepped up to replace Steven, and I joined the “Python Paparazzi” or “pyparazzi”, along with Erich, Jason Samsa, Dan Ryder, and Stéphane Jolicoeur-Fidelia.

Since PyCon was in Chicago last year, I was familiar with the Crowne Plaza Hotel, which is a decent hotel, but nothing to write home about. This year the conference proper moved to the Hyatt Regency down the street. PyCon has a tradition of trying to keep costs low in order to keep the conference accessible to the community, so I was expecting something like the Crowne Plaza. I couldn’t have been more wrong. The Hyatt is a photographer’s paradise. There are lots of interesting colors, textures, and some areas with beautiful overhead natural light. If you were going to photograph a wedding, you would die for settings like these for the bridal portraits.

This tiled inset in wall turned into the backdrop for James Tauber’s headshot.

It doesn’t have to be strobe(ist) to be a good headshot!

This orange lit panel behind a bench seat turned into the backdrop for Jim Baker’s headshot.

In addition to the pyparazzi, there were plenty of other cameras floating around the conference. Andy Smith decided to do a photographic project called the “Beards of Python“. When this set was announced on Flickr, it caused some Twitter buzzing amongst some of the female attendees of the conference. One thing about photographers is that we (or at least I) are always willing to take some interesting photos. So when the Twitter buzzing reached me, I offered to photograph any interested Geek Girls. James Duncan Davidson and I have discussed the value of trying to photograph female attendees at technology conferences. Since our photographs are often used for advertising, this can be a way of helping women feel more comfortable about attending — knowing that there will be other women there can be a help. So not only did I get to shoot more pictures of interesting people, I hope that in some small way this will contribute to making PyCon friendlier to women.

This is Catherine Devlin, a contributor to sqlpython. Go read her post “Five minutes at PyCon change everything” for an actual example of the lightning talk/open space/sprint scenario that I described above.

The entire set of Pythonista headshots, as well as the rest of my conference coverage are up on Flickr. Who knows what we’ll come up with for next year in Atlanta…

Travel

Regular readers will know that a trip to PyCon traditionally involves some kind of travel mishap. This year was pretty minor compared to previous years. United lost my luggage for the flight from Seattle to O’Hare, despite the fact that I arrived 2.5 hours early, and checked in at the “Premier” checkin line. I got my bag the next day, so it wasn’t really that bad. Maybe next year will be the PyCon with no travel glitches.