Category Archives: programming

Notes on A History of Erlang

Joe Armstrong wrote a paper for last year’s HOPL-III conference on the history of Erlang. For some reason, I didn’t get a paper copy of those proceedings, and was too busy to notice their absence. Fortunately Lambda the Ultimate picked it up and supplied links to the paper and the accompanying presentation. Digging into the history of something like Erlang is always fascinating, and Armstrong has done a good job of explaining how Erlang came to be.

Here are a bunch of quotes on topics that I found interesting. I’ve grouped them into categories, but searching the PDF of the paper shouldn’t be hard if you want to know where they originated.

Sources of inspiration

Those familiar with Prolog will not find it at all surprising that Erlang has its roots in Prolog (mostly due to implementation reasons). What I did find interesting was the origin/history/viewpoint on the concurrency model

The explanations of what Erlang was have changed with time:

1. 1986 â€“ Erlang is a declarative language with added concurrency.

2. 1995 â€“ Erlang is a functional language with added concurrency.

3. 2005 â€“ Erlang is a concurrent language consisting of communicating components where the components are written in a functional language.

…

Today we emphasize the concurrency.

Note that the word actor never appears in those descriptions. Indeed, the word actor does not appear in the paper at all. So for all the discussion about Erlang’s usage of the actor model, it appears that the Erlang folks independently duplicated many of the ideas for Hewitt’s Actors. I think that is kind of interesting.

Lisp and Smalltalk are cited as inspirations, but more for the implementation of the runtime than for any features in the language. I came away from the paper with the impression that Armstrong and his colleagues are not paradigm ideologues. They are trying to get the job done.

Reliability

There is a huge emphasis on reliability throughout the paper, supporting Steve Vinoski’s remarks about Erlang. I’l just include a series of quotes, which you can interpret as you see fit:

Erlang was designed for writing concurrent programs that “run forever”

At an early stage we rejected any ideas of sharing resources between processes because of the difficulty of error handling. In many circumstances, error recovery is impossible if part of the data needed to perform the error recovery is located on a remote machine and if that remote machine has crashed.

In order to make systems reliable, we have to accept the extra cost of copying data structures between processes and always make sure that processes have enough data to continue by themselves if other processes crash

The key observation here is to note that the error-handling mechanisms were designed for building fault-tolerant systems, and not merely for protecting from program exceptions. You cannot build a fault-tolerant system if you only have one computer. The minimal conï¬guration for a fault-tolerant system has two computers. These must be conï¬gured so that both observe each other. If one of the computers crashes, then the other computer must take over whatever the ï¬rst computer was doing.

This means that the model for error handling is based on the idea of two computers that observe each other. Error detection and recovery is performed on the remote computer and not on the local computer.

Links in Erlang are provided to control error propagation paths for errors between processes.

It was about this time that we realized very clearly that shared data structures in a distributed system have terrible properties in the presence of failures. If a data structure is shared by two physical nodes and if one node fails, then failure recovery is often im-possible. The reason why Erlang shares no data structures and uses pure copying message passing is to sidestep all the nasty problems of ï¬guring out what to replicate and how to cope with failures in a distributed system.

In our world, we were worried by software failures where replication does not help.

Design criteria

Here are some quotes related the design criteria for Erlang.

Changing code on the fly was an initial key requirement

the notion that three properties of a programming language were central to the efï¬cient operation of a concurrent language or operating system. These were:

1) the time to create a process

2) the time to perform a context switch between two different processes

3) the time to copy a message between two processes

The performance of any highly-concurrent system is dominated by these three times.

One of the earliest design decisions in Erlang was to use a form of buffering selective receive

Pipes were rejected in favor of messages

In the concurrent logic programming languages, concurrency is implicit and extremely ï¬ne-grained. By comparison Erlang has explicit concurrency (via processes) and the processes are coarse-grained.

The ï¬nal strategy we adopted after experimenting with many different strategies was to use per-process stop-and-copy GC. The idea was that if we have many thousands of small processes then the time taken to garbage collect any individual process will be small.

…

Current systems run with tens to hundreds of thousands of processes and it seems that when you have such large numbers of processes, the effects of GC in an individual process are insigniï¬cant.

The BEAM compiler compiled Erlang programs to BEAM instructions.

On functionalness

This next series of quotes will probably make the pure functional language people shake their heads, but i think that it’s important to understand Erlang in contrast with pure functional languages.

Erlang is not a strict side-effect-free functional language but a concurrent language where what happens inside a process is described by a simple functional language.

Behaviors in Erlang can be thought of as parameterizable higher-order parallel processes.

… the status of Erlang as a fully ï¬‚edged member of the functional family is dubious. Erlang programs are not referentially transparent and there is no system for static type analysis of Erlang programs. Nor is it relational language. Sequential Erlang has a pure functional subset, but nobody can force the programmer to use this subset; indeed, there are often good reasons for not using it.

An Erlang system can be thought of as a communicating network of black boxes.

In the Erlang case, the language inside the black box just happens to be a small and rather easy to use functional language, which is more or less a historical accident caused by the implementation techniques used.

History and Usage

One thing that I was looking for in the paper was more details on how long Erlang had been around (besides before Java), how big the largest programs/systems were, and so forth. Here is what I found.

This history spans a twenty-year period…

(The history starts in 1986)

The largest ever system built in Erlang was the AXD301. At the time of writing, this system has 2.6 millions lines of Erlang code.

…

The AXD301 is written using distributed Erlang. It runs on a cluster using pairs of processors and is scalable up to 16 pairs of processors.

…

In the analysis of the AXD reported in [7], the AXD used 20 supervision trees, 122 client-server models, 36 event loggers and10 ï¬nite-state machines. All of this was programmed by a team of 60 programmers.

…

As regards reliability, the AXD301 has an observed nine-nines reliability [7]â€”and a four-fold increase in productivity was observed for the development process [31].

The AXD 301 is circa 1998.

Perhaps the most exciting modern development is Erlang for multicore CPUs. In August 2006 the OTP group released Erlang for an SMP.

This corroborates something that David Pollak told me at the RedMonk unconference during CommunityOne, namely that SMP support in Erlang had not been there very long. Of course, Erlang was running on systems with 16 physical (pairs, no less) of processings in a distributed environment. So while the runtime might not be that mature on SMP, the overall runtime for concurrency is probably a bit more mature than that. Nonetheless, worthwhile to know the precise facts.

All in all, I found the paper to be a very worthwhile read – (and a nice change from my usual intake of blog posts and tweets). One of my pet peeves about the computer business is the lack of awareness of the history of the field. At least I’ve removed a bit of my own ignorance as relates to Erlang.

The Scala vs Erlang whirlwind

11 Replies

Over the last week or two there’s been a bit of commotion with various parties in the blogosphere making the case for Scala against Erlang or for Erlang against Scala. Here’s a see spot run summary of the main writers and their positions / content:

Ted Neward (1, 2) – Ted (how confusing) is in the Scala camp, and thinks that the library approach of Scala’s actor library is preferable to Erlang’s VM (BEAM). He cites managability as a major concern. He also thinks that adapting a process style model into the JVM would be easier than adding SNMP monitoring to BEAM. The length of the Barcelona project bibliography suggests otherwise, but we’ll never know unless some brave soul goes and tries to do this. Fortunately, the JDK is open source now. One has to wonder whether such a change could make its way through the JCP, though. Unfortunately for Ted, I found that many of his arguments were weakened by his lack of knowledge about Erlang.

Steve Vinoski (1, 2, 3) – Steve’s articles are more about the reliability aspects of Erlang, and he’s mostly trying to correct Ted’s facts on Erlang. He thinks that Erlang had proven its reliability chops by running for years non-stop. Given the frequency with which Java app servers need to be (or are) bounced, this doesn’t seem that incredible to me.

Patrick Logan (1, 2, 3) – Patrick piled on after Steve and has spent most of his writing trying to correct/challenge Ted’s assertions about Erlang. Patrick thinks that the conventional (i.e. JVM and CLR) runtimes will have problems implementing an Erlang style shared-nothing model, since the pre-existing libraries for those runtimes are not engineered in a shared-nothing manner.

Barry Kelly was an observer of the Neward-Vinosk-Logan discussion, and added some commentary on the impact of VM primitives on things like lift. This is a point which resonates with me, because it seems to me that both languages and language runtimes will need some work to meet the challenges of large scale multicore computing.

Yariv Sadan has done a pile of stuff in Erlang, and supplied his own summary of the differences between Scala and Erlang. There is a very informative exchange between Yariv and lift author David Pollak in the comments of this one.

That’s the short rundown. This is a very interesting problem space — before I turned into database programming language guy in graduate school, I was angling to be a concurrent programming language guy. Along the way to that I got pretty good doses of functional and logic programming, as well as actor programming. That work was in the context of people planning to build (for the day) highly concurrent computers, on the order of 1000’s of processors. Today, multicore hardware is not quite up to that level, but it is approaching it pretty quickly. If there is any force in computing that is likely to precipitate the need for a new programming ecosystem (language, runtime, libraries), I think concurrent programming is it. I also think there is just not enough experience with this problem to have a real sense of what is really going to work. Cliff Click and Brian Goetz were right when they said that we just don’t have a good programming model for this stuff. Absent a model, I don’t know how we can think that we really understand what the runtime needs to deliver.

Scala liftoff

1 Reply

I stayed around in San Francisco for one more day after JavaOne, in order to attend the Scala liftoff. The liftoff was an open space style conference (which has a more specific meaning than “unconference”, at least to me). My friend Kaliya Hamlin did a great job of facilitating the day.

Scala has steadily been gaining attention, and hasn’t yet hit (at least in my eyes) the hype part of the classic Gartner hype cycle. I’ve been poking about with Scala, mostly because of the type inferencing, the Actor library, and lift. I have great respect for the work that Martin Odersky has done over the years, which also has me interested. Couple that with what I learned about closures in Java at JavaOne, and the list of reasons to look more deeply at Scala is getting long, especially if you are determined to have a statically typed languages.

I wasn’t able to make it to any of sessions on lift. It just worked out that other sessions overlapped them in a pathological way. While this is unfortunate, I am sure that I’ll be able to pick up anything that I need from the mailing lists and other documentation. I was able to attend two sessions on actors. One of the sessions had people with questions about actors, but no Scala actor experts were in that group. There was some discussion of Pi-calculus and the join calculus, but no discussion of the actual actor theory.

Steve Yen’s session on actor-d was pretty useful. Steve set out to build a version of memcached using Scala’s actors. He spent most of his slot talking about Scala/Java isms that he ran into – this was important since he was comparing to the C memcached. By the time he got to the actor related stuff, he was almost out of time. Steve found that he had to remove actors from the main loop of his server in order to get sufficient performance. He wanted to get statistics from the server in the background and discovered that he main loop actor was always processing messages and was never idle long enough to report statistics. He ended up replacing the actor with plain old Java Threads (POJT?). This was in addition to all the fact that he ran into many of the standard Java problems as well. I’m not sure what to conclude from this. I don’t recall what kind of hardware he was on, and I am not convinced that he had the right architecture for an actor based system. Some of his experience also seemed contrary to what the lift folks have been claiming. I think that we are in for a decent amount of investigation here. One of Martin’s statements about Scala is that it is possible (and better) to extend the language via libraries than via actual language constructs. For the most part, I agree with this, but there are certain extensions which have interactions with the runtime – like concurrency. In those cases, I don’t see how the library approach allows taking advantage of runtime features. The current version of Scala actors is implemented as a library.

One of the things that I am currently working on is support for Python in NetBeans, so I dropped into the session on IDE support for Scala. With the exception of IntelliJ, none of the IDE plugin principals were present, so it was hard to have a really productive discussion. Martin did attend the session and we talked about the possibiliy of getting hooks into the existing Scala compiler, particularly the parser and the type inferencer. That could yield some big dividends for people working on IDE support. One IDE feature that I would like to see is the ability to hit a key, and have the IDE “light up” all the inferred types, overlaid on the existing program code. This would allow developers to see if their intuition about the types actually matched that of the type inferencer. I’d like a feature like this for Python/Ruby/Groovy/Javascript code as well. Further discussion was deferred to the scala-tools mailing list.

The other session that I participated in was the session on Scala community and governance. Several people wondered about this during Kaliya’s “What questions do you have about Scala” portion of the schedule building. When nobody else put up a session in this area, I grabbed a slot, hoping to spur some conversation – if for no other reason than my own education. Fortunately, Martin had already been thinking about the problem. He is going to adopt a Python style governance, with him (and EPFL) having the final say on language design matters. There will be Scala Enhancement Proposals (SEPs), like the Python PEPs. I’m very happy with this. I think that Python has done very well at maintaining the balance between (lots) of community input on the language design, while still retaining that “quality without a name”. One of the things that I said during the CommunityOne general session panel was that particular individuals in the right place, at the right time, matter at great deal. After watching Martin for the day, and seeing his interactions on the mailing list over the last few months, I think that the design of Scala is in very good hands.

We also talked about the evolution of the Scala libraries. The Scalax project is working to build a set of utility libraries for Scala. Martin views scalax as a place where anyone can submit a library, have it tested, vetted, reworked, etc. Eventually some code in scalax would be candidates for addition to the Scala standard libraries. This also seems like a sane approach to me. I like the idea of having a place for libraries to shakeout before going into the standard libraries. Martin also mentioned a LINQ in Scala project. I need to track that one down too.

It is good to be in a multi-language world again. There’s room for Scala, Python, Ruby, and others. Another language that I am keeping my eye on is Newspeak.

The Open Screen project

5 Replies

Around this time last year, Adobe open sourced its Flex framework for rich internet applications. Today Adobe announced the Open Screen project, which encompasses a number of things, probably most importantly, the removal of the license restrictions on the SWF file format used by Flash. The other aspects of the announcement relate to Adobe’s Flash Player, and while they are steps towards openness, Adobe’s player will remain closed. The importance of opening Adobe’s player has decreased because dropping the file format licensing should make things easier for the Gnash folks. The worry then is that we’ll end up with incompatible versions of Flash, which is in almost nobody’s interest. That’s probably the next problem that needs addressing.

Erlang == CGI?

2 Replies

Jay Nelson, in the comments to Damien Katz’s Lisp as Blub:

The two relevant issues are system granularity and garbage collector behavior (if it is related to memory and garbage collection).

Erlang encourages an architecture of many small-granularity processes. To the extent that this approach is followed, failures are localized. It is possible to do this with other languages, but erlang does encourage the approach more so than other languages.

The other difference is that erlang uses a single-threaded garbage collector per process. This makes the garbage collection process simpler, more finely grained and distributed. Smaller processes mean less complicated memory structures, and thus the language encourages a simpler model with localized garbage collection failure. Determining the cause of overburdened memory usage (or any other resource because of the localized nature of small processes) becomes easier.

An erlang system can get wedged, but following the principle of many small processes makes it less likely to happen than in other languages which encourage large processes with shared memory structures.

It strikes me that this is a sort of CGI’ish view of the world (well except for the garbage collector). CGI scripts run, use (non-shared) resources, release them all and die. The entire post and comment thread is worth some pondering.

Python at CommunityOne

1 Reply

CommunityOne is a free and open developer conference that is run by Sun on the day before JavaOne. This year, there will a space at CommunityOne dedicated to the Python community, complete with whiteboards and wifi. If you are in the Bay Area for JavaOne, or in the Bay Area, or just plain interested in Python, please register for CommunityOne — space is limited.

Registering for CommunityOne gets you a bag of swag, a free lunch the day of CommunityOne, access to all the CommunityOne events and sessions, and a free pass for Day 1 of JavaOne. When you register, put “Python/Jython” in for the referral code.

I will be on a panel on community models during the general session from 9:30AM – 10:45AM, and Frank Wierzbicki and I will be doing a Python/Jython panel. In addition to the usual developer stuff, there will also be a two day Startup Camp, and the folks from RedMonk will be back to do their day long unconference thing.

Dynamic language jobs?!

PyCon 2008

2 Replies

It’s been 2 years since I’ve been to PyCon, and things have definitely changed. The last time I went to PyCon (2006 in Dallas), it was still a relatively small conference (3-400), if I remember what I was told), with a familiar feel, especially if you had attended in previous years, or were a part of the Python community. This year, there were over 1000 people (double the 500 people that came in 2007, apparently). I spent a sizable portion of the conference days feeling like “I miss a year, and you guys go and get 1000 people”. It’s a great thing that so many people are interested in Python.

The talks

I went to a reasonable number of talks – talk quality at PyCon has historically been pretty good, and I was a little out of date on the latest on things like Django and Turbogears. The best talk that I went to was Raymond Hettinger’s talk “Core Python Containers — Under the Hood”. This was a great talk for several reasons: Raymond was a good and entertaining speaker. There was significant technical meat – explanations of the implementation choices for all the core containers in Python, lists, sets, and dicts. We heard about doubling factors and amortized big-oh time. Most importantly, there was significant practical applications for Python programmers. Raymond’s talk gives a cost model for the core containers, and having an understanding of that model is important for folks who are writing Python programers. It’s also useful for developers of alternate Python implementations because it allows them to follow suit or to diverge and (hopefully) document the places where the cost model is different. My next favorite talk was Jim Baker’s “More Iterators in Action”. I missed the talk given last year, but I liked this one. Jim hit two of my favorite topics, language integrated query (LINQ) (albeit without the DSL), and concurrency.

Concurrency

There was a lot of interest in concurrency this year, which warms my heart, because I see high-level/dynamic languages and concurrency as the chocolate and peanut butter in the old Reese’s peanut butter cup commercials. There were 2 open space sessions and 1 lightning talk, and the topic entered many of the conversations that I had.

Sun, Jython, and JRuby

People were generally positive to learn about Sun’s interest in Python and Jython. A number of people stopped me to congratulate me on the new job, and we had a nice turnout at the open space session, where people were free with ideas, comments, and a few not so easy to answer questions. I hope that Sun can live up to the goodwill that people extended towards me and Frank.

If I was surprised about the jump in size of PyCon, I was even more surprised by the amount of energy around Jython. At most of the previous PyCon’s that I attended, people would mention Jython, and either be sorry that it was too out of date to consider, or be just plain dismissive of it. This year there was none of that. People were very interested in Jython. I was really surprised by how much interest there was, and by some of the people who were interested. It was certainly a nice feeling to sit in the sprint room and occasionally have people pop in to ask if such and such was running in Jython yet, or did Jython support X because package Y needed it.

This was the first time that I had met Frank Wierzbicki in person — I think he’s the happiest person at Sun right now. I was also able to spend some time hanging out with various folks from the Jython community. It seemed to me that the community was doing quite nicely. If you looked at some of the community metrics that we would use at the ASF to allow a project to graduate from incubation, almost all of those criteria have already been fulfilled. One of my goals for Sun’s Python efforts is for as many of them as possible to be highly community oriented, so it was nice to see that Jython is well on it’s way in that regard. The folks working on Jython are very sharp (including the aforementioned Jim Baker, who it turns out was a classmate of mine at the Brown CS dept – although neither of us can remember meeting the other), and have one of the those (in my mind) essential community ingredients, a community sense of humor.

Jim snuck this bit of commentary on Jython’s lack of a global interpreter lock into his talk.

There were several Ruby related surprises at PyCon this year. David Heinemeier Hansson, create of Ruby on Rails, made an appearance for one day, and a number of the JRuby committers made a road trip down from Minnesota, to hang out, meet the Jython folks, and generally display their hacker prowess. Which they totally did. Charlie and Tom powered their way to JRuby 1.1RC3 during the conference. Meanwhile Nick Sieger demonstrated what a happens when you stick a bunch of hackers, an EVDO card, and an EVDO hub into a car. The Jython guys (if any of them lived in the same state) need to get some of that – The best thing since the Adobe AIR Bus Tour, and at a fraction of the cost.. The JRuby folks and Jython folks are already starting to talk and share experiences, and I am sure that this will only result in even better dynamic language stuff for the JVM.

Other Cool stuff

On one of the sprint days, I did a bit of wandering and stopped to talk to my friend Brian Dorsey, who is doing some cool stuff here in Seattle. Brian was working with Richard Jones on pyglet and Bruce. Pyglet is a set of Python libraries for writing games and doing other kinds of multimedia. There’s pygame, which I am aware of because of Armin Rigo’s infamous use of pygame to deliver talks about PyPy. Richard has created Bruce, a presentation tool based on pyglet. In addition to being able to do cool multimedia presentation effects, there are some really cool things that you can do. Perhaps the coolest is that you can have a slide which is essentially an embeddd Python interpreter, so no more switching out of your presentation to demo your Python code at work. Really slick.

On a different note, on several evenings, conference goers who stuck around to hang out in the hotel’s common area were treated to musical performances by a dynamic (as in constantly changing set of members) band of Pythonistas:

The Sprints

Perhaps the most amazing way in which the conference has changed is this picture.

That is a picture of a part of the lunch crowd on the first day of the sprints following the conference. When I talked to David Goodger about it, he said that he had taken a count and there were over 250 people at the sprints. Visually, that sprint lunch room looked to be about the size of the room for the first PyCon that I attended (PyCon 2004). Simply amazing. The ASF has a hackathon before every ApacheCon, but I can’t remember one ever reaching this kind of size or scale. Another thing about the PyCon sprints is that they are aimed at growing the community — you don’t have to be a committer on any project in order to attend, and experienced project members will take time to sit and help new people get started. There were several people like that in the Jython sprint room. I was more impressed by what happened with the sprints than any other part of the conference. The only central organization here was that the conference planners obtained sprint space, and in a few cases got some sponsors to cough up money for lunch. Everything else was organized by the projects themselves (I heard that the Django folks closed 100 bugs in a single day). If you want to get a sense of what kinds of things got accomplished at the sprints, you can look at this page on the wiki — it’s not exhaustive, but it’s a start.

Travel

In the past, I’ve had some travel nightmares getting home from PyCon. This year I am happy to report that I didn’t have any problems at all, except for a fight that I had with the Sun internal travel system (and lost).

Conclusion

It was great to be back at PyCon. Interest in Python is growing (as measured by attendance), as is interest in Jython, and interested people are also rolling up their sleeves and pitching in (as measured by sprint attendance growth).

If you think my job is all about Jython, you are confused.

5 Replies

Apparently people are confused about what I am working on at Sun, and with PyCon starting tomorrow, this is not a good thing. I am not going to be working on Jython directly, although I will certainly be poking my nose in to see what’s going on. The Python related part of my job (which will be the majority in the short term) is to figure out what Sun should be doing in the Python space, across all of the relevant platforms at Sun, including but not limited to: the JVM and JEE, Netbeans, and Solaris.

Sun isn’t done in the dynamic language space, and I will also be looking for opportunities with other dynamic languages and related technologies.

Lazyweb: Virtualization software

12 Replies

I am looking at building a bunch of virtualized machines, and I have no idea what software I should be using.

I want to create and run images on Linux and OS X
I want those images to be runnable on Linux, OS X, and Solaris (optional)
I want to make images that run Linux, Windows, and Solaris
I want to run one set of images on a box connected to the internet, and have those images appear as separate machines.
Bonus round: I want there to be some ISP/hosting provider that I can send images to and have them hosted.

I am assuming that my choice are: VMWare, Parallels, and VirtualBox.

Useful pointers and advice appreciated.