Category Archives: concurrency

Strange Loop 2010

Leave a reply

Last week I was in Saint Louis for Strange Loop 2010. This was the second year of Strange Loop, which is a by hackers for hackers conference. I’m used to this sort of conference when it’s organized by a single open source community – I’d put ApacheCon, PyCon, and CouchCamp in this category. At the same time, Strange Loop’s content was very diverse, and had some very high quality speakers. It’s sort of like a cross between ApacheCon and OSCON. One difference is that there isn’t a community that’s putting on Strange Loop, and the fun community feel of ApacheCon or PyCon is missing.   

One of the reasons that I was interested in attending Strange Loop was Hilary Mason’s talk on data science / machine learning. This is an area that I am starting to delve into, and I did study a little machine learning right around the time that it was starting to shift away from traditional AI and more towards the statistical approach that characterizes it now. Hilary is the chief scientist at bit.ly, and as it turns out, a Brown alumnae as well. Her talk was a good introduction to the current state of machine learning for people who didn’t have any background. She talked about some of the kinds of questions that they’ve been able to answer at bit.ly using machine learning techniques. Justin Bozonier used Twitter to ask Hilary if she would be wiling to sit down with interested people and do some data hacking, so I skipped the session block (which was painful because I missed Nathan Marz’s session on Cascalog, which was getting rave reviews). We ended up doing some simple stuff around the tweets about #strangeloop. Justin has a good summary of what happened, complete with code, and Hilary posted the resulting visualization on her blog. It was definitely useful to sit and work together as a group and get snippets of insight into how Hilary was approaching the problem.

Another area that I am looking at is changes in web application architecture due to the changing role of Javascript on both the client and the server. I went to Kyle Simpson’s talk on Strange UI architecture, as well as Ryan Dahl’s talk on node.js. Kyle has built BikechainJS, another wrapper around V8, like Node.js. There’s a lot of interest around server side javascript – the next step is to think about how to repartition the responsibilities of web applications in a world where clients are much more capable, and where some code could run on either the client or the server.

Guy Steele gave a great talk, and the number of people who can give such talk is decreasing by the day. As a prelude to talking about abstractions for parallel programming, Guy walked us through an IBM 1130 program that was written on a single punch card. He had to reverse engineer the source code from the card, which was complicated by the fact that he used self modifying code as well as some clever value punning in order to get the most out of the machine. The thrust of his comments on parallel programming was that the accumulator style of programming which pervades imperative programs is bad when it comes to exploiting parallelism. Instead, he emphasized being able to find algebraic properties such as associativity or commutativity which would allow parallelism to be exploited via the map/reduce style of programming pioneered decades ago in the functional programming community, and popularized by systems like Hadoop. Guy was proposing that mapreduce be the paradigm for regular programming problems, not just “big data” problems. For me, the most interesting comment that Guy made was about Haskell. He said that if he know what he knew now when he had started on Fortress, he would have started with Haskell and pushed it 1/10 of the way to FORTRAN instead of starting with FORTRAN and pushing it 9/10 of the way to Haskell.

I’m not generally a fan of panel sessions, because the vast majority of them don’t really live up to their promise. Ted Neward did a really good job of moderating the panel on “Future of Programming Languages”. At the end of the panel, Ted asked the panelists which languages they though people should be learning in order to get new ideas. The list included Io (Bruce Tate), Rebol (Douglas Crockford), Forth and Factor (Alex Payne), Scheme and Assembler (Josh Bloch), and Clojure (Guy Steele). Guy’s comments on Clojure rippled across Twitter, mutating in the process, and causing some griping amongst Scala adherents. The panel appears to have done it’s job in encouraging controversy.

Also in the Clojure vein, I attended Brian Marick’s talk “Outside in TDD in Clojure“. Marick has written midje, a testing framework that is more amenable to the a bottom up style of programming that is facilitated by REPL’s. It’s an interesting approach relying on functions that provide a simple way to specify placeholders for functions that haven’t been completed yet. This also serves as a leverage point for the Emacs support that he has developed.

Doug Crockford delivered the closing keynote. I’ve heard him speak before, mostly on Javascript. His talk wasn’t about Javascript at all, but it was very engaging and entertaining. If you have the chance to see him speak in that kind of setting, you should definitely do it.

A few words on logistics. The conference was spread out across three locations. I feared the worst when I heard this, but it turned out to be fine – OSCON in San Jose was much more inconvenient. The bigger logistical issue was WiFi. None of the three venues was really prepared for the internet requirements of the Strange Loop attendees. WiFi problems are not a surprise at a conference, but the higher quality conferences do distinguish themselves on their WiFi support.

All in all, I think that Strange Loop was definitely worthwhile. The computing world is becoming “multicultural”, and it’s good to see a conference that recognizes that fact.

Concurrency => Parallelism

3 Replies

I wanted to clarify a point from my post The Cambrian Period of Concurrency.

I made the statement

From where I sit, this is all about exploiting multicore hardware

because I’ve seen a pile of actor and other concurrency libraries which have not taken parallel execution of the concurrent program seriously. If I am going to go to the trouble of writing a concurrent program, then I want that execution to be parallel, especially in a multicore world.

Simon Marlow from the GHC team said that if programming multicore machines is the only goal we ought to be looking at parallelism first and concurrency only as a last resort. Haskell has some nice features for taking advantage of parallelism. However, I explicitly stated that I was not as interested in highly regular or data parallel computations, which is what Haskell’s parallelism tools are aimed at. These are fine ways to get parallelism, but I am interested in problems which are genuinely concurrent, not just parallel. In a Van Roy hierarchy, these are the problems with observable nondeterminism. I also specifically called out reduction of latency as one of my goals, something which Marlow says is a possible benefit of concurrency. The GHC team is interested in a different mix of problems than I am.

Van Roy in short

I also forgot to mention Peter Van Roy’s paper Programming Paradigms for Dummies: What Every Programmer Should Know, which includes an overview of his stratification of concurrency and parallelism (and other stuff). If you don’t have time to read his book, the paper is shorter and more digestible.

The Cambrian Period of Concurrency

20 Replies

Back in July, I gave an OSCON talk that was a survey of language constructs for concurrency. That talk has been making the rounds lately. Jacob Kaplan-Moss made referred to it in a major section of his excellent keynote Snakes on the Web, and Tim Bray has cited it as a reference in his Concur.next series. It seems like a good time for me to explain some of the talk content in writing and add my perspective on the current conversations.

The Cambrian

The Cambrian period was marked by a rapid diversification of lifeforms. I think that we are in a similar situation with concurrency today. Although many of the ideas that are being tossed around for concurrency have been around for some time, I don’t think that we really have a broad body of experience with any of them. So I’m less optimistic than Tim and Bruce Tate, at least on time frame. I think that we have a lot of interesting languages, embodying a number of interesting ideas for working with concurrency. I think that some of those languages have gained enough interest/adoption that we are now in a position to get a credible amount of experience so that we can start evaluating these ideas on their merits. But I think that the window for doing that is pretty large, on the order of 5 to 10 years.   

What kinds of problems

The kinds of problems I am interested in are general purpose programming problems. I’m specifically not interested in scientific, numeric, highly regular kinds of computations or data parallel computations. Unlike Tim, I do think that web systems are a valid problem domain. I see this being driven by the need to drive down latency to provide good user response time, not to provide additional scalability (although it probably will).

It’s not like Java

Erik Engbrecht, one of Tim’s commenters said:

To get Java, you basically take Smalltalk and remove all of the powerful concepts from it while leaving in the benign ones that everyday developers use.

I think there’s something to be learned from that.

This presupposes that you know what all the good concepts are and what the benign ones are. It doesn’t seem like we are at that point. When Java was created, both Lisp and Smalltalk had existed for quite sometime and it was possible to do this kind of surgery. I don’t have a clear sense of what actually works well, much less what is powerful or benign.

The hardware made me do it

From where I sit, this is all about exploiting multicore hardware, and when I say this I mean machines with more than 4 or 8 hardware threads (I say threads, not cores – actual parallelism is what is important). The Sun T5440 is a 256 thread box. Intel’s Nehalem EX will let you build a 128 thread box later this year. Those are multicore boxes. If you look at experiments, you see that systems that seem to work well at 2 or 4 threads don’t’ work well at 16 or 64 threads. Since there’s not a huge amount of that kind of hardware around yet, it’s hard for people to run experiments at larger sizes. Experiments being run on 2 thread MacBook Pro’s are probably not good indicators of what happens at even 8 threads.. This is partially because dealing with more hardware threads requires more administrative overhead, and as the functional programming people found out, that overhead is very non-trivial. The point is, you have to run on actual hardware to have believable numbers.   This makes it hard for me to take certain kinds of systems seriously, like concurrency solutions running on language implementations with Global Interpreter Locks. See David Beazley’s presentation on Python’s Global Interpreter Lock, for an example.

Comments on specific languages

At this point I am more interested in paradigms and constructs as opposed to particular languages. However, the only way to get real data on those is for them to be realized in language designs and implementations.

  • Haskell – Functional Laziness aside, the big concurrency thing in Haskell is Software Transactional Memory (STM). There are other features in Haskell, but STM is the big one. STM is an active research field in computer science, and I’ve read a decent number of papers trying to make heads from tails. Among the stack that I have read, it seems to be running about even between the papers touting the benefits of STM and the the papers saying that STM cannot scale and will not work in practice. The jury is very much out on this one, at least in my mind.
  • Erlang – I like Erlang. It’s been in production use for a long time, and real systems have been built using it. In addition to writing some small programs and reviewing some papers by Erlang’s designers, I spent a few days at the Erlang Factory earlier this year trying to get a better sense of what was really happening in the Erlang community. While there’s lots of cool stuff happening in Erlang, I observed two things. First, the biggest Erlang systems I heard described (outside of Facebook’s) are pretty small compared to a big system today. Second, and more importantly, SMP support in Erlang is still relatively new. Ulf Wiger’s DAMP09 presentation has a lot of useful information in it. On the other hand, BEAM, the Erlang VM is architected specifically for the Erlang process/actor model. This feels important to me, but we need some experimental evidence.
  • Clojure – Clojure as a ton of interesting ideas in it. Rich Hickey has really done his homework, and I have a lot of respect for the work that he is doing. Still it’s the early days for Clojure, and I want to see more data. I know Rich has run some stuff on one of those multiple hundred core Azul boxes, but as far as I know, there’s not a lot of other data.
  • Scala – The big thing in Scala for concurrency is Actors, but if you compare to Erlang, Actors are the equivalent of Erlang processes. A lot of the leverage that you get in Erlang comes from OTP, and to get that in Scala, you need to look at Jonas Boner’s highly interesting Akka Actor Kernel project. Akka also includes an implementation of dataflow variables, so Akka would give you a system with Actors, supervision, STM, and Dataflow (when it’s done).   
  • libdispatch/Grand Central Dispatch – Several of Tim’s commenters brought up Apple’s Grand Central Dispatch, now open sourced as libdispatch. This is a key technology for taking advantage of multicore in Snow Leopard. GCD relies on programmers to create dispatch queues which are then managed by the operating system. Programmers can send computations to these queues via blocks (closures), which are a new extension to Objective-C. When I look at Apple’s guide to migrating to GCD from threads, I do see a model that I prefer to threads, but it is not as high level as some of the others. Also, the design seems oriented towards very loosely coupled computations.   It will be several years before we can really know how well GCD is working. I am typing this post on a 16 thread Nehalem Mac Pro, and I rarely see even half of the CPU meters really light up, even when I am running multiple compute intensive tasks. Clearly more software needs to take advantage of this technology before we have verdict on its effectiveness in production.
  • .Net stuff like F#/Axum, etc – There is some concurrency work happening over on the CLR, most notably in F# and Axum. I spent some time at Lang.NET earlier this year, and got a chance to learn a bit about these two technologies. If you look at paradigms, the concurrency stuff looks very much like Erlang or Scala, with the notable exception of join patterns, which are on Martin Odersky’s list for Scala. I will admit to not being very up to speed on these, mostly for lack of Windows and the appropriate tools.

Other thoughts

Jacob’s take away from my talk at OSCON was “we’re screwed”. That’s not what I wanted to convey. I don’t see a clear winner at the moment, and we have a lot of careful experimentation and measuring to do. We are quite firmly in the Cambrian, and I’m not in a hurry to get out – these things need to bake a bit longer, as well as having some more experimentation.

In addition to my talk, and Tim’s wiki page, if you are really interested in this space, I think that you should read Concepts, Techniques, and Models of Computer Programming by Peter van Roy and Seif Haridi. No book can be up to date with the absolute latest developments, but this book has the best treatment that I’ve seen in terms of trying to stratify the expressiveness of sequential and concurrent programming models.

Erlang Factory 2009

4 Replies

I spent Thursday and Friday of last week at the Erlang Factory in San Francisco (although the event was actually in Palo Alto).

Why did I go?

I’ve written about Erlang in this space before. Erlang is having a major influence on other languages, such as Scala on the JVM side and Axum on the CLR side. In addition every language seems to have several implementation of Erlang style “actors” (despite the fact that this is historically incorrect). Erlang has been around for a long time, and has seen industrial usage in demanding telecom applications. As a dynamically typed functional language with good support for concurrency and distribution, it is (if nothing else) a source of interesting ideas. Earlier this year, my boss asked me to start doing some thinking about cloud computing in addition to the stuff that I was already doing around dynamic languages — another good match for Erlang. This was the first large scale gathering of Erlang people in the US (at least that I am aware of), so I wanted to drop in and see what was going on, what the community is like, and so on.

Talks

The program at the Erlang Factory was very strong. In many of the slot sessions, there were 3 excellent talks to choose from. Every single talk that I went to was of very high quality. It was so bad that I wasn’t able to explore all the areas that I wanted to. Fortunately, the sessions were videotaped and are supposed to be made available on the web. Also, there was a decent amount of twittering going on, so a Twitter search for #erlangfactory will turn up some useful information.

I attended a number of “experience” talks by companies / individuals. There were experience talks from Facebook, SAP, Orbitz, and Kreditor (the fastest growing company in Sweden). I made it to the Facebook talk and the Kreditor talk. Facebook’s usage/deployment is on the order of 100 machines, which provide the chat facility for Facebook. Erlang is doing all the heavy lifting, and PHP is doing the web UI part. There was a lot of this kind of architecture floating around the conference. It seemed like the most popular combination was Ruby/Erlang, but there was definitely Python and PHP as well. The Kreditor talk was interesting because their site has been running for 3 years with very small amounts of downtime. Unfortunately, their entire deployment is probably less than 10 machines, so that blunts the impressiveness of what they have done. Still it was interesting to hear how they accomplished this using features of Erlang. In addition to the talks, I spoke with many attendees who are using Erlang in their companies. One such person was eBay founder Pierre Omidyar, who is running Ginx, a web based Twitter client. Pierre is doing the coding and deployment of the site, and was well versed in the Erlang way of doing things. An interesting data point.

The Erlang community (like all communities) has it’s old guard. These are folks who have worked with Erlang for years, before its recent burst of interest. There were a pair of keynotes by Erlang long-timers Robert Virding (The Erlang Rationale) and Ulf Wiger (Mulitcore Programming in Erlang). Both of these talks shared a common trait — the speakers were pretty honest about what was good about Erlang, and where there were problems. Given how prone the computing business is to fashion, I found this to be refreshing. Virding talked about the reasons why Erlang is designed the way it is. He accepted the blame for inconstencies in the libraries, talked about the need to avoid the process dictionary, and agreed that “a char type is probably not wrong”. Wiger’s talk was about why parallelizing code is hard (even with Erlang). He used the example of parallelizing map to demonstrate this, and showed the use of the QuickCheck testing tool to aid in finding parallelism bugs. The Erlang version of QuickCheck was inspired by the Haskell version of QuickCheck, and it’s a very very useful tool. The adaptations for parallelism look very nice. It’s a shame that the Erlang version is commercial software. I don’t grudge the authors the right to charge money for their software, but I do think that this will hold back adoption of this important tool.

There were many talks on what I would describe as “cloud problems”. For example, Ezra Zygmutowicz’s “You got your Erlang in my Ruby” was really about how he built a self assembling cluster of Ruby daemon’s (Nanite), Dave Fayram and Abhay Kumar’s “Building Reliable Distributed Heterogenous Services with Katamari/Fuzed“, and Lennart Ohman’s “A service fail over and take-over system for Erlang/OTP”. Like PyCon, there was a lot of interested in eventually consistent databases/key-value stores/non-relational databases. Cliff Moon’s talk on dynomite (a clone of Amazon’s Dynamo system), was particularly encouraging because he was reaching out to other people in the audience (and there were a decent number of them) to try an consolidate all their efforts into a single project. From what I could tell, people seemed receptive to that idea.

CouchDB also fits into that last category of non-relational databases, but it gets it’s own paragraph. One reason is that I helped mentor the project through the Apache Incubator (and chauffeured those CouchDB committers who were present). Another is that CouchDB creator Damien Katz got a keynote. Third is that there was basically a CouchDB track on the second day of the conference. There was a lot of interest in CouchDB, and a lot of activity as well. I was told that some of the people who took the CouchDB training during the training days had actually submitted patches on the project already. Damien’s talk was not about the technical details of CouchDB, but about his personal journey to CouchDB, which included selling his house and living off his savings in order to see CouchDB come to life.

Activity has really picked up in the Erlang web framework space. In addition to Erlang Web, and Yariv Sadan’s Erlyweb, there is also Rusty Klophaus’ Nitrogen. Nitrogen focuses more on the UI side of the web framework, omitting any kind of data storage. It’s very easy to create an AJAX based user interface using Nitrogen, and there is nice support for Comet. As part of his presentation, Rusty showed his slides on a Nitrogen based webcast reflector. You specify the UI using Erlang terms, which then causes HTML/Javascript/etc to be generated, which caused a stir in part of the Twitter peanut gallery. I was mostly happy to see people focusing on solving the current generation of problems. My favorite web space talk was probably Justin Sheehy’s talk on Webmachine. I think that I prefer the description of WebMachine as a REST or HTTP toolkit. Webmachine gives you what you need to implement any HTTP method correctly, and then provided a set of callback functions that can be implemented to customize that processing to do actual work. One of the coolest things about Webmachine is it’s ability to visually show you that path taken in processing a particular HTTP request, and being able to inspect/dump data at various points in the diagram. It makes for a very nice demo.

There were not that many “language geek” talks. This contrasts with the early years of PyCon (at least for as long as I have attended) where there were quite a number. I missed Robert Virding’s talk on Lisp Flavored Erlang (but I saw some example usage in a CouchDB talk), because it overlapped the dynomite talk. I was able to attend Tony Arcieri’s talk on “Building Languages on Erlang (and an introduction to Reia)”. During the first part of his talk, Tony showed how to construct an Erlang module on the fly in the Erlang shell. He then discussed some tools which are useful to people trying to build languages on top of BEAM, the Erlang virtual machine:

  • Robert Virding has written leex, a lexical analyzer generator
  • yecc, a Yacc style parser generator is included in the Erlang distribution
  • the erl_syntax_lib library aids in constructing Erlang abstract syntax trees, which can then be compiled to Erlang bytecode.
  • Erlyweb contains the smerl (simple metaprogramming) library for creating and manipulating Erlang modules at runtime.

After that, he launched into a description of REiA. I’m not sure that I agree with some of the choices that he has made, but I am happy to see people experimenting with languages on top of BEAM, and in keeping with Erlang’s process model and the OTP infrastructure. One of the things that Tony mentioned was abandoning indentation based syntax. He wrote an entire postmortem on that experience in his blog. Python’s indentation based syntax has won me over and made me a fan, and I am sad to see that indentation syntax, blocks/closures, and expression orientation continue to be at odds.

Coda

It looks like Erlang is starting to find a home. Companies are using it in production. There are books starting to be written about it. Many (not all) of the things which make Erlang seem odd to “mainstream” programmers also appear in languages like Scala, Haskell, and F#. At the same time, Erlang has a long history of industrial deployment, albeit in a single (large) market segment. Many of the problems which we now face in large web systems (and the cloud): concurrency, distribution, high availability, and scalability are strengths for Erlang. Indeed, many of the people that I heard from or talked to basically said that they couldn’t solve their problem with any other technology, or that their solutions were dramatically simpler than the technologies that they already knew. Will that be enough to propel Erlang into the mainstream? I don’t know. I also don’t know if our current state of mainstreamness is going to remain. More and more I’m seeing an attitude of “let’s use the best tool for the job”, not only in languages, but in all parts of (web) applications.

There’s also the issue of the Erlang community itself.   Around 120 people showed up for the conference. As I mentioned previously, there are the folks who have been doing Erlang for years. Then there are the relative newcomers, who are web oriented/web savvy, and solving problems in very different domains than the original problem domain of Erlang and it’s inventors. Thus far, the two segments seem to be getting along fine. I hope that will continue — success or the potential for success has a tendency to bend relationships.

OSCON 2008

3 Replies

Another OSCON has come and gone, and as usual, I am exhausted in the aftermath. I’ve developed a love-hate relationship with OSCON over the years. The diversity of the OSCON community is one of the huge pluses of the conference. I got involved in open source via Apache, and OSCON was where I really started to get more of a sense of the open source community as a whole. That’s led to friendships with people doing all sorts of open source stuff, which makes the conference a natural place to reconnect with many of those folks. Which leads to the primary downside of OSCON, which is that there is just no way to keep up with, never mind see all the people that you’d like to see. Combine that with the sheer scale of the event, and you have recipe for burnout. This year is no exception, which is why this post is delayed by a few days.

Languages

It’s fitting to start a review of OSCON with programming languages, since OSCON began as a Perl conference. There are still lots of Perl hackers running around, and by the distribution of the program (the Python track was 1 day shorter than the Perl, PHP, and Ruby tracks), it seems that Perl is not going anywhere anytime soon. I think that we are going to need to drum up some more Python talks for OSCON next year. Then again, with PyCon topping 1000 people this year, maybe all the Python folks are going there. It certainly is cheaper than going to OSCON. Despite all of this, I saw lots of people that I knew from the Python community, as well as plenty of people who had affixed a yellow Python ribbon to their badge. The ribbons are a nice way of helping people find their tribe at a big show like OSCON – a lower tech version of what the Pathable folks are doing.

I spent a lot of time nosing around various concurrency oriented sessions. I attended Steven Parkes’ tutorial on Actors, which was pretty well attended. Steven has implemented a version of Actors as a set of Ruby and Python libraries. During the tutorial I was able to meet Debasish Ghosh, who has a great blog and Twitterstream on high-level languages, and concurrency topics in general. I also took in a BOF on Actors, which had some really interesting conversation. There were a lot of Erlang folks in the room for that one, which made the discussion pretty interesting.

Databases

OSCON 2008

There was lots of non-traditional database stuff happening at OSCON this year. I am one of the mentors for the CouchDB project at Apache, and I was finally able to meet my first CouchDB commiter, Jan Lehnardt, at the show. Jan gave a nice high level overview talk on CouchDB, which was well attended, and I was interested to see Brian Aker of MySQL/Drizzle in the audience and among the throng of questioners after the talk.

OSCON 2008

I also went to a talk on Prophet, which is a peer to peer database that is being done by some of the folks that brought us SVK. I’m not sure that I quite recovered from my initial reaction to that revelation, but Jan was sitting next to me during the entire talk, and was saying something about stealing some ideas from the Prophet guys. In open source we call that standing on the shoulders of giants, or something like that.

“Memes”

The XMPP folks had a three day summit during the conference, which I gather was well attended. There was a decent amount of XMPP buzz floating around in the hallways, so I expect the blogosphere to be full of XMPPness during the next week or so. I’ve done a bunch of blogging on XMPP in the past, and while things have improved, they haven’t improved to the point where XMPP is taking over the world. Things like Twitter are definitely helping, but there is still a long way to go before XMPP achieves world domination. But we can hope. And at least XMPP makes a great advertisement for Erlang.

Along with XMPP, we had the microblogging meme. I made heavy use of Twitter throughout the week, and it definitely played a useful part in making connections with people. Well, except for the times when it was down. I was able to spend a little time with Leah Culver, the founder of Pownce, which has the virtue of being written in Python, and of having a very nice API for dealing with the service. It’s interesting to get additional perspectives on a problem, and since I had already talked some with the Twitter guys, it definitely helped to hear Leah describe Powce’s take on the problem(s) and solutions. O’Reilly was not to be outdone, and did some very active boostering for identi.ca. I’ve got very mixed feelings on identi.ca. One the one hand, I should love identi.ca, because it’s open source. On the other hand, it’s written in PHP, which means I won’t be touching the code, and more importantly, my network is not there. Actually, it was kind of annoying to have to explain to lots of identi.ca zealots that it’s the network that’s the value, not the software, or ironically, the quality of the service. Still if another microblogging service can convince my network to move, and remain up, and even deliver some new functionality, I would definitely switch. I think I could probably write another post about “microblogging”, but I’ll refrain for now.

Theo Schlossnagle gave an amazing presentation called “Full-stack introspection crash course”, which is code for “let me show you some amazing stuff that you’ll only be able to do with DTrace”. This was a brilliant choice of title on Theo’s part, because it didn’t scare away all of those people whose preconceptions about DTrace or Sun would prevent them from coming to such a talk. Instead, Theo played to a very full room, and I would say that about one-third of the audience actually uttered the phrase “Oh My God” out loud at some point during the presentation. This was certainly true for thetwo gentlemen sitting directly to my right and directly behind me. I later heard from people at the Sun booth, that a bunch of people came to the booth having heard about DTrace (I assume at Theo’s talk), asking for whatever CD’s they needed in order to be able to use it. Theo clearly understands how to communicate about DTrace. We at Sun need to learn that lesson.

Open Source

Of course, you can’t have a conference on open source without meta stuff about open source itself. I was fortunate to attend the morning session of Microsoft’s Participate08 event, which was an interesting case study led by Karim Lakhani from the Harvard Business School. The case was on threadless.com and involved a lot of issues which are very relevant to injecting corporate involvement into an existing community based organization. I’ve been following Karim’s work over the years (he studied under Eric von Hippel, whose work I am also fond of), so I was happy for the chance to meet him and participate in an activity with him. I also met Siobhan O’Mahony, who is also doing great work studying open source communities. I’m not sure what direct value Microsoft got out of sponsoring Participate, other than being able to say that they did an event around OSCON, but I know that I definitely appreciated the chance to interact with a bunch of people.

OSCON 2008

Microsoft was all over the news by the end of OSCON, having announced that they would become a Platinum sponsor of the Apache Software Foundation. This was not a complete surprise to me: Justin Erenkrantz, the current ASF president told me what was happening the night before at a party. I think that this is an interesting step for Microsoft, an it’s definitely a step in the right direction. However, as one questioner pointed out, Microsoft has a long history of incendiary rhetoric towards the open source community, and that’s going to mean that just about everything happens in steps. I do find it interesting that one of the reasons that the ASF has taken donations is to build up a legal defense fund against what we regarded as inevitable legal attacks. It’s somehow ironic to think of Microsoft’s $100,000 going into that pool. I think that the next interesting milestone in Microsoft’s relationship with the ASF will be when the first Microsoft sponsored project shows up at the front door of the Apache Incubator.

I also contributed to the metaness with a talk titled “Open Source Community Antipatterns” (slides are now available on the O’Reilly slide page). The talk was decently attended, but I suspect that the all-star antipatterns panel immediately following my talk drew off some of the audience that might have come to my talk. The people track expanded a great deal this year, which I think is a good thing.

Photography

I always have photographic memories associated with OSCON. I got my first digital SLR, right before OSCON 2005, and I’ve shot a bit a each OSCON, and even won the OSCON photo contest one year. This year I found myself shooting less. There were too many other things that I needed to do, and between knowing that Duncan is making is covering stuff and some artistic blockage, I lacked both time and motivation to crank out the shots.

Duncan has been a great friend and photographic mentor, and I always look forward to catching up with him during OSCON. This time was no exception. We did a bunch of stuff together, ranging from hanging out, having a wide angle shootout (well he was wide) to Duncan putting one of his cards into my D3 and giving the pixels a once over. Probably the most fun thing that we did was an impromptu photoshoot. Duncan was shooting headshots of the OSCON staff for a thank you slide for the closing keynotes. Only problem was that he needed one of himself, so he drafted me. With the safe shot in hand, we spent a few more minutes doing something a little edgier and fun.

OSCON 2008


Fin

That’s it for another OSCON. I hope we’ll be back in Portland again next year.

Thoughts on MagLev – VM’s for everybody!

6 Replies

One of the most visible presentations from last weeks RailsConf was Avi Bryant’s demonstration of MagLev, which is a RubyVM that is based on Gemstone’s S/64 VM for Smalltalk. This caused a stir because the micro benchmark performance of MagLev looks really good because S/64 has been out in production for a while and because it appears to have some really interesting features (an OODB, shared VM’s, etc). MagLev is a reminder that the world of production quality, high-performance virtual machines is bigger than many of us remember at times.

I believe that over the next few years we will see a flourishing of virtual machines, as well as languages atop existing virtual machines. Take for example Reia, a Ruby/Pythonesque experiment atop Erlang’s BEAM VM. As we return to a multi language world, we will also necessarily return to a multiple implementation world. Before Java, there were many languages and many implementations of those languages. You could argue that there were probably too many, and I think that’s probably true. I would argue that we need to enter a new period of language and runtime experimentation. A big driver, but not the only driver, for this is the approaching multi-core world. When you don’t know how to solve something, more attempts at solutions is better.

Notes on A History of Erlang

9 Replies

Joe Armstrong wrote a paper for last year’s HOPL-III conference on the history of Erlang. For some reason, I didn’t get a paper copy of those proceedings, and was too busy to notice their absence. Fortunately Lambda the Ultimate picked it up and supplied links to the paper and the accompanying presentation. Digging into the history of something like Erlang is always fascinating, and Armstrong has done a good job of explaining how Erlang came to be.

Here are a bunch of quotes on topics that I found interesting. I’ve grouped them into categories, but searching the PDF of the paper shouldn’t be hard if you want to know where they originated.

Sources of inspiration

Those familiar with Prolog will not find it at all surprising that Erlang has its roots in Prolog (mostly due to implementation reasons). What I did find interesting was the origin/history/viewpoint on the concurrency model

The explanations of what Erlang was have changed with time:

1. 1986 – Erlang is a declarative language with added concurrency.

2. 1995 – Erlang is a functional language with added concurrency.

3. 2005 – Erlang is a concurrent language consisting of communicating components where the components are written in a functional language.

…

Today we emphasize the concurrency.

Note that the word actor never appears in those descriptions. Indeed, the word actor does not appear in the paper at all. So for all the discussion about Erlang’s usage of the actor model, it appears that the Erlang folks independently duplicated many of the ideas for Hewitt’s Actors. I think that is kind of interesting.

Lisp and Smalltalk are cited as inspirations, but more for the implementation of the runtime than for any features in the language. I came away from the paper with the impression that Armstrong and his colleagues are not paradigm ideologues. They are trying to get the job done.

Reliability

There is a huge emphasis on reliability throughout the paper, supporting Steve Vinoski’s remarks about Erlang. I’l just include a series of quotes, which you can interpret as you see fit:

Erlang was designed for writing concurrent programs that “run forever”

At an early stage we rejected any ideas of sharing resources between processes because of the difficulty of error handling. In many circumstances, error recovery is impossible if part of the data needed to perform the error recovery is located on a remote machine and if that remote machine has crashed.

In order to make systems reliable, we have to accept the extra cost of copying data structures between processes and always make sure that processes have enough data to continue by themselves if other processes crash

The key observation here is to note that the error-handling mechanisms were designed for building fault-tolerant systems, and not merely for protecting from program exceptions. You cannot build a fault-tolerant system if you only have one computer. The minimal configuration for a fault-tolerant system has two computers. These must be configured so that both observe each other. If one of the computers crashes, then the other computer must take over whatever the first computer was doing.

This means that the model for error handling is based on the idea of two computers that observe each other. Error detection and recovery is performed on the remote computer and not on the local computer.

Links in Erlang are provided to control error propagation paths for errors between processes.

It was about this time that we realized very clearly that shared data structures in a distributed system have terrible properties in the presence of failures. If a data structure is shared by two physical nodes and if one node fails, then failure recovery is often im-possible. The reason why Erlang shares no data structures and uses pure copying message passing is to sidestep all the nasty problems of figuring out what to replicate and how to cope with failures in a distributed system.

In our world, we were worried by software failures where replication does not help.

Design criteria

Here are some quotes related the design criteria for Erlang.

Changing code on the fly was an initial key requirement

the notion that three properties of a programming language were central to the efficient operation of a concurrent language or operating system. These were:

1) the time to create a process

2) the time to perform a context switch between two different processes

3) the time to copy a message between two processes

The performance of any highly-concurrent system is dominated by these three times.

One of the earliest design decisions in Erlang was to use a form of buffering selective receive

Pipes were rejected in favor of messages

In the concurrent logic programming languages, concurrency is implicit and extremely fine-grained. By comparison Erlang has explicit concurrency (via processes) and the processes are coarse-grained.

The final strategy we adopted after experimenting with many different strategies was to use per-process stop-and-copy GC. The idea was that if we have many thousands of small processes then the time taken to garbage collect any individual process will be small.

…

Current systems run with tens to hundreds of thousands of processes and it seems that when you have such large numbers of processes, the effects of GC in an individual process are insignificant.

The BEAM compiler compiled Erlang programs to BEAM instructions.

On functionalness

This next series of quotes will probably make the pure functional language people shake their heads, but i think that it’s important to understand Erlang in contrast with pure functional languages.

Erlang is not a strict side-effect-free functional language but a concurrent language where what happens inside a process is described by a simple functional language.

Behaviors in Erlang can be thought of as parameterizable higher-order parallel processes.

… the status of Erlang as a fully fledged member of the functional family is dubious. Erlang programs are not referentially transparent and there is no system for static type analysis of Erlang programs. Nor is it relational language. Sequential Erlang has a pure functional subset, but nobody can force the programmer to use this subset; indeed, there are often good reasons for not using it.

An Erlang system can be thought of as a communicating network of black boxes.

In the Erlang case, the language inside the black box just happens to be a small and rather easy to use functional language, which is more or less a historical accident caused by the implementation techniques used.

History and Usage

One thing that I was looking for in the paper was more details on how long Erlang had been around (besides before Java), how big the largest programs/systems were, and so forth. Here is what I found.

This history spans a twenty-year period…

(The history starts in 1986)

The largest ever system built in Erlang was the AXD301. At the time of writing, this system has 2.6 millions lines of Erlang code.

…

The AXD301 is written using distributed Erlang. It runs on a cluster using pairs of processors and is scalable up to 16 pairs of processors.

…

In the analysis of the AXD reported in [7], the AXD used 20 supervision trees, 122 client-server models, 36 event loggers and10 finite-state machines. All of this was programmed by a team of 60 programmers.

…

As regards reliability, the AXD301 has an observed nine-nines reliability [7]—and a four-fold increase in productivity was observed for the development process [31].

The AXD 301 is circa 1998.

Perhaps the most exciting modern development is Erlang for multicore CPUs. In August 2006 the OTP group released Erlang for an SMP.

This corroborates something that David Pollak told me at the RedMonk unconference during CommunityOne, namely that SMP support in Erlang had not been there very long. Of course, Erlang was running on systems with 16 physical (pairs, no less) of processings in a distributed environment. So while the runtime might not be that mature on SMP, the overall runtime for concurrency is probably a bit more mature than that. Nonetheless, worthwhile to know the precise facts.

All in all, I found the paper to be a very worthwhile read – (and a nice change from my usual intake of blog posts and tweets). One of my pet peeves about the computer business is the lack of awareness of the history of the field. At least I’ve removed a bit of my own ignorance as relates to Erlang.

The Scala vs Erlang whirlwind

11 Replies

Over the last week or two there’s been a bit of commotion with various parties in the blogosphere making the case for Scala against Erlang or for Erlang against Scala. Here’s a see spot run summary of the main writers and their positions / content:

Ted Neward (1, 2) – Ted (how confusing) is in the Scala camp, and thinks that the library approach of Scala’s actor library is preferable to Erlang’s VM (BEAM). He cites managability as a major concern. He also thinks that adapting a process style model into the JVM would be easier than adding SNMP monitoring to BEAM. The length of the Barcelona project bibliography suggests otherwise, but we’ll never know unless some brave soul goes and tries to do this. Fortunately, the JDK is open source now. One has to wonder whether such a change could make its way through the JCP, though. Unfortunately for Ted, I found that many of his arguments were weakened by his lack of knowledge about Erlang.

Steve Vinoski (1, 2, 3) – Steve’s articles are more about the reliability aspects of Erlang, and he’s mostly trying to correct Ted’s facts on Erlang. He thinks that Erlang had proven its reliability chops by running for years non-stop. Given the frequency with which Java app servers need to be (or are) bounced, this doesn’t seem that incredible to me.

Patrick Logan (1, 2, 3) – Patrick piled on after Steve and has spent most of his writing trying to correct/challenge Ted’s assertions about Erlang. Patrick thinks that the conventional (i.e. JVM and CLR) runtimes will have problems implementing an Erlang style shared-nothing model, since the pre-existing libraries for those runtimes are not engineered in a shared-nothing manner.

Barry Kelly was an observer of the Neward-Vinosk-Logan discussion, and added some commentary on the impact of VM primitives on things like lift. This is a point which resonates with me, because it seems to me that both languages and language runtimes will need some work to meet the challenges of large scale multicore computing.

Yariv Sadan has done a pile of stuff in Erlang, and supplied his own summary of the differences between Scala and Erlang. There is a very informative exchange between Yariv and lift author David Pollak in the comments of this one.

That’s the short rundown. This is a very interesting problem space — before I turned into database programming language guy in graduate school, I was angling to be a concurrent programming language guy. Along the way to that I got pretty good doses of functional and logic programming, as well as actor programming. That work was in the context of people planning to build (for the day) highly concurrent computers, on the order of 1000’s of processors. Today, multicore hardware is not quite up to that level, but it is approaching it pretty quickly. If there is any force in computing that is likely to precipitate the need for a new programming ecosystem (language, runtime, libraries), I think concurrent programming is it. I also think there is just not enough experience with this problem to have a real sense of what is really going to work. Cliff Click and Brian Goetz were right when they said that we just don’t have a good programming model for this stuff. Absent a model, I don’t know how we can think that we really understand what the runtime needs to deliver.

Scala liftoff

1 Reply

I stayed around in San Francisco for one more day after JavaOne, in order to attend the Scala liftoff. The liftoff was an open space style conference (which has a more specific meaning than “unconference”, at least to me). My friend Kaliya Hamlin did a great job of facilitating the day.

Scala liftoff 2008

Scala has steadily been gaining attention, and hasn’t yet hit (at least in my eyes) the hype part of the classic Gartner hype cycle. I’ve been poking about with Scala, mostly because of the type inferencing, the Actor library, and lift. I have great respect for the work that Martin Odersky has done over the years, which also has me interested. Couple that with what I learned about closures in Java at JavaOne, and the list of reasons to look more deeply at Scala is getting long, especially if you are determined to have a statically typed languages.

Scala liftoff 2008

I wasn’t able to make it to any of sessions on lift. It just worked out that other sessions overlapped them in a pathological way. While this is unfortunate, I am sure that I’ll be able to pick up anything that I need from the mailing lists and other documentation. I was able to attend two sessions on actors. One of the sessions had people with questions about actors, but no Scala actor experts were in that group. There was some discussion of Pi-calculus and the join calculus, but no discussion of the actual actor theory.

Steve Yen’s session on actor-d was pretty useful. Steve set out to build a version of memcached using Scala’s actors. He spent most of his slot talking about Scala/Java isms that he ran into – this was important since he was comparing to the C memcached. By the time he got to the actor related stuff, he was almost out of time. Steve found that he had to remove actors from the main loop of his server in order to get sufficient performance. He wanted to get statistics from the server in the background and discovered that he main loop actor was always processing messages and was never idle long enough to report statistics. He ended up replacing the actor with plain old Java Threads (POJT?). This was in addition to all the fact that he ran into many of the standard Java problems as well. I’m not sure what to conclude from this. I don’t recall what kind of hardware he was on, and I am not convinced that he had the right architecture for an actor based system. Some of his experience also seemed contrary to what the lift folks have been claiming. I think that we are in for a decent amount of investigation here. One of Martin’s statements about Scala is that it is possible (and better) to extend the language via libraries than via actual language constructs. For the most part, I agree with this, but there are certain extensions which have interactions with the runtime – like concurrency. In those cases, I don’t see how the library approach allows taking advantage of runtime features. The current version of Scala actors is implemented as a library.

One of the things that I am currently working on is support for Python in NetBeans, so I dropped into the session on IDE support for Scala. With the exception of IntelliJ, none of the IDE plugin principals were present, so it was hard to have a really productive discussion. Martin did attend the session and we talked about the possibiliy of getting hooks into the existing Scala compiler, particularly the parser and the type inferencer. That could yield some big dividends for people working on IDE support. One IDE feature that I would like to see is the ability to hit a key, and have the IDE “light up” all the inferred types, overlaid on the existing program code. This would allow developers to see if their intuition about the types actually matched that of the type inferencer. I’d like a feature like this for Python/Ruby/Groovy/Javascript code as well. Further discussion was deferred to the scala-tools mailing list.

Scala liftoff 2008

The other session that I participated in was the session on Scala community and governance. Several people wondered about this during Kaliya’s “What questions do you have about Scala” portion of the schedule building. When nobody else put up a session in this area, I grabbed a slot, hoping to spur some conversation – if for no other reason than my own education. Fortunately, Martin had already been thinking about the problem. He is going to adopt a Python style governance, with him (and EPFL) having the final say on language design matters. There will be Scala Enhancement Proposals (SEPs), like the Python PEPs. I’m very happy with this. I think that Python has done very well at maintaining the balance between (lots) of community input on the language design, while still retaining that “quality without a name”. One of the things that I said during the CommunityOne general session panel was that particular individuals in the right place, at the right time, matter at great deal. After watching Martin for the day, and seeing his interactions on the mailing list over the last few months, I think that the design of Scala is in very good hands.

We also talked about the evolution of the Scala libraries. The Scalax project is working to build a set of utility libraries for Scala. Martin views scalax as a place where anyone can submit a library, have it tested, vetted, reworked, etc. Eventually some code in scalax would be candidates for addition to the Scala standard libraries. This also seems like a sane approach to me. I like the idea of having a place for libraries to shakeout before going into the standard libraries. Martin also mentioned a LINQ in Scala project. I need to track that one down too.

It is good to be in a multi-language world again. There’s room for Scala, Python, Ruby, and others. Another language that I am keeping my eye on is Newspeak.