EuroPython 2008

8 Replies

Next week I’ll be at EuroPython 2008, in Vilnius, Lithuania, where I’ll be moderating a panel on Jython. If you have questions that you’d like to put to the Jython committers, leave them as comments on this post. I’ll make sure that I post answers for any of those questions.

After the conference ends, I’ll be spending a few days at Sun’s office in Prague, which is where much of the NetBeans team is located. I’ve never been to Vilnius or Prague, so suggestions for things to see, etc., would also be great comments.

DTrace on Linux?

5 Replies

I’ve been meaning to write a post about DTrace, and Tim Bray’s tweet finally got me moving. It looks like some people are trying to make DTrace a topic for this year’s Linux Kernel Summit. I hope they succeed. I also hope that those folks pushing for user level tracing have their voices heard. I was amused to read one of the messages which claimed that DTrace is:

DTrace is more a piece of sun marketing coolaid which they use to beat us up at every opportunity.

My experience at Sun thus far is that people generally don’t really appreciate the benefits of DTrace. It stems from a view that I also saw in the LKS threads, which is that DTrace (and tools like Systemtap) is a tool for system administrators, because it reports on activity on the kernel. That’s not how I look at it. DTrace is a tool for dealing with full system stack problems, which initially manifest themselves as operating system level problems. The fact that DTrace can trace user land code as well as kernel code is what makes it so important, especially to people building and running web applications. Because of all the moving parts in a complicated web application (think relational database, memcached or other caching layers, programming language runtime, etc), it can be hard to debug a web application that has gone awry in production. Worse, sometimes the problems only appear in production. Tools which cut across several layers of the system are very important, and DTrace provides this capability, if all the layers have probes installed. When a web application goes wrong in production, you see it at the operating system level – high usage of various system resources. That’s where you start looking, but you will probably end up somewhere else (unless you are ace at exercising kernel bugs). Perhaps a bad SQL query or perhaps a bad piece of code in part of the application. A tool that can help connect the dots between operating system level resource problems and application level code is a vital tool. That’s where the value is.

One of the cooler features of DTrace is that you can register a user level stack helper (a ustack helper), which can translate the stack in a provider specific manner. One cool example of this is the ustack helper that John Levon wrote for Python, which annotates the stack with source level information about the Python file(s) being traced. On an appropriately probed system, this would mean that you could trace the Python code of a Django application, memcached, and your relational database (PostgreSQL and soon MySQL). That would be very handy.

I’d love to see DTrace on Linux, because I have it on OS X and it’s in OpenSolaris and FreeBSD, but I’d also be happy to see SystemTap get to the point where it could do the same job.

Book Review: The Creative Digital Darkroom

2 Replies

There seem to be growth cycles that photographers go through. One of them is related to postprocessing of photographs. When I started taking pictures, I didn’t really do much to my pictures, on the belief that a good photographer ought to get things right straight out of the camera. I only shot film as a consumer, and not for very long. While I had a brief exposure to a photographic darkroom, I didn’t leave with the right impression about the role of the developing and printing process. Until I got Aperture, I never adjusted a picture. After I got Aperture, I mostly made small exposure, contrast or saturation bumps, never more than that. Now I am using Lightroom rather than Aperture, and I am still doing mostly the same sorts of things, although I’ve started to work more with adjusting the black point and contrast curves of pictures. In the last 6-7 months, I’ve started to use Photoshop on pictures. I was able to do a bit here and a bit there. I checked out books from the library, I bought a few books on Photoshop CS3 when it came out. My friend Ogalthorpe, sat with me once and showed me how he works some of his magic on his pictures.

It seemed like things were going in one ear and out the other, partially because I didn’t have a good idea of what I was trying to do or why. That made retaining the “how” pretty difficult.

I recently picked up The Creative Digital Darkroom by Katrin Eismann and Sean Duggan. This is the first Photoshop book that actually tries to walk you though the reasoning behind why you are doing what you are doing, and that does it in language that can be understood by someone with zero darkroom experience. I really appreciated the emphasis on the creative aspects in the middle of all the pictures of curves, layers, layer masks, and all the usual Photoshop stuff. The book is very recent, so it covers Photoshop CS3, and in places where Lightroom can do the same thing, there is coverage of Lightroom as well.

My skill level is such that the two chapters (out of 10!) “Toning and Contrast” and “Dodging, Burning, and Exposure Control” will probably keep me busy for a good long time. I am sure that as I start to apply some of these principles, I will grow into material in the other chapters. But for now, I am happy to have what feels like a basic footing that I can work from. Now all I need to do is spend some time making images good enough to process a lot.

What I am hoping for at WWDC

2 Replies

1. 3G iPhone with hardware GPS – I am dying to put my Nokia 6600 to rest

2. An emphasis on stability and performance in 10.6. – 10.5 just seems less reliable than it should. I am having problems with Firewire disks and with the WindowServer freaking out and consuming all available cores.

3. ZFS – my photo hard disk situation is a mess.

And that’s it. If there are other goodies, and I am sure there will be, that’s fine, but I’d be happy to check off those three items and call it a day.

Thoughts on MagLev – VM’s for everybody!

6 Replies

One of the most visible presentations from last weeks RailsConf was Avi Bryant’s demonstration of MagLev, which is a RubyVM that is based on Gemstone’s S/64 VM for Smalltalk. This caused a stir because the micro benchmark performance of MagLev looks really good because S/64 has been out in production for a while and because it appears to have some really interesting features (an OODB, shared VM’s, etc). MagLev is a reminder that the world of production quality, high-performance virtual machines is bigger than many of us remember at times.

I believe that over the next few years we will see a flourishing of virtual machines, as well as languages atop existing virtual machines. Take for example Reia, a Ruby/Pythonesque experiment atop Erlang’s BEAM VM. As we return to a multi language world, we will also necessarily return to a multiple implementation world. Before Java, there were many languages and many implementations of those languages. You could argue that there were probably too many, and I think that’s probably true. I would argue that we need to enter a new period of language and runtime experimentation. A big driver, but not the only driver, for this is the approaching multi-core world. When you don’t know how to solve something, more attempts at solutions is better.

OS X Scripting

12 Replies

John Gruber followed up on Daniel Jalkut’s suggestion that Apple replace AppleScript with Javascript:

I agree with this wholeheartedly. Or maybe even make a clean break and scrap OSA and introduce a new system.

I’ve been talking up the benefits of scripting apps on the Mac since the 1990’s. The sad fact of it is that Apple has never really supported scripting to the level that it deserves. It’s even more important now in the days of a UNIX based MacOS. I have a bunch of scripts that I rely on daily to help me get things done more efficiently. I’d write more of them, but two things hold me back. AppleScript is really a funky language. I’ve partially solved that by switching to using Python (via appscript) to do the scripting, but that’s only half the problem. The other half of the problem is that the API exposed via OSA is also pretty funky. If Apple cleaned all that up, in say, 10.6, I’d be happy to rework my existing body of scripts.

Even if that happened, the big problem is that developer’s don’t really support scripting that well, so a good scripting system overhaul needs to look at making it easy for developers to expose application functionality to scripts. Unless that part happens, improvements in the scripting language, and OSA’s API’s will not be enough to push scripting to the level where it belongs.

Book Review: Practical Artistry: Light & Exposure for Digital Photographers

1 Reply

I’m not sure how it happened, but somehow I got onto the list for a review copy of Practical Artistry: Light & Exposure for Digital Photographers by Harold Davis. I’m not complaining in the slightest bit, just mystified. I’m a reader of Harold’s blog, and his choice of nature subjects is pretty close to mine, so I’ve always appreciated his photographs. This book is aimed a people who are trying to learn (and subsequently master) the basics of exposure, which puts it squarely in competition with Bryan Peterson’s Understanding Exposure, which is a book that every photographer really ought to read.

So how does Davis’ work stand up when compared to Peterson’s? The basic sections are strong, and either book is fine in this respect. I think that Understanding Exposure has more of an emphasis on the creative aspects of photography, and does a better job of helping the reader understand how to achieve particular creative effects by manipulating exposure. In particular, for beginners, Peterson gives stronger guidance on specific values for apertures and shutter speeds, that can be helpful to people that don’t have much of a background.

On the other hand, I think that Davis has a much better section on understanding and using the in-camera light meters. I have the revised version of Understanding Exposure, which is updated for digital cameras, but Light & Exposure for Digital Photographers is really much more up to date for the digital age. There is a good discussion on ISO, and how to use ISO as part of the exposure control. Along with that there is an excellent discussion on noise and the use of noise in the creative process. I was glad to find some treatment of white balance. One of the easiest ways to pick out inexperienced photographer’s pictures is to look at the white balancing of the pictures. There is also a chapter on post-processing and RAW processing, a topic to which entire books are devoted, but I think it is helpful for people to understand the role of post-processing in the digital age.

As you would expect with a photography book, there are lots of pictures. As I mentioned, Davis’ preference in nature subjects is close to mine, so I really enjoyed the pictures, and there is lots of commentary accompanying each photograph. The photos in Peterson’s book are more diverse in subject matter, which is probably better from a teaching point of view.

At the end, though, I like both books. For the stage that I am at photographically, I have a slight preference for Peterson’s Understanding Exposure. I am working hard on the creative aspects of my photography, so I am in a frame of mind to be biased towards Peterson’s treatment. Light & Exposure for Digital Photographers is also a very good book, and I would have no problem recommending it to a beginning photographer. I certainly got something out of reading it (and probably could stand to read each book a few more times). The subject matter in these books is so important that one of these two books should be included in the box with every DSLR sold.

Tim and Twitterbucks

3 Replies

So Tim Bray wants to make sure that Twitter stays around, and therefore wants a business model for Twitter. I’d like it to stay around too, which means there has to be a business model for it, but I’m not sure that directly charging people for it is the right model. I don’t have any visibility into Twitter’s economics, but I do have some decent visibility into my usage of the service. All of Tim’s proposals for Twitter are predicated on the notion of wanting to “reach people”. He also cited the classification of Twitter as microblogging, which might be sort of accurate, but which doesn’t capture the whole situation, at least not for me.

My usage of twitter breaks down into several categories:

Reaching people in the sense that Tim means. This breaks down by category into several groups, some overlapping: technologists, photographers. These are tweets of links, facts, ideas and so forth. This is the most blogging/microblogging usage of Twitter

Random spicy commentary about nothing This is just random information about me, the virtual equivalent of the water cooler at work. These tweets add color, but probably are devoid of directly useful information. Alhough you never know how people might use intimate knowledge of your lunch habits.

Social banter One of the twitter tribes that I am in is the local Seattle Flickr tribe. This group is one of the reasons that Facebook became sticky for me, at least for a time. That pretty much stopped when a critical mass of those folks discovered twitter. These tweets are where people are, what they are having for lunch, dinner, etc. They play the role of building a social fabric which is essential for that group to be as successful as it has become.

Social arranging This happens because of the SMS Twitter gateway and accessibility of Twitter via mobile devices. Twitter killed whatever usage I might have had on Dodgeball. When I am at conferences, Twitter has become an essential part of the hallway/after hours track. So much so that this usage will drive me to buy a 3G class web enabled telephone, as soon as Jobs announces it.

So there are many usages besides “reaching people” in a blogging like sense, and it’s not clear to me that some of these usages would continue if Twitter raised the bar by charging for usage. For the social connections part, reducing the ubiquity of the service is a real negative. The value of Twitter would definitely be reduced by cutting out people who couldn’t/wouldn’t afford to pay for it, like starving aspiring photographers.

Notes on A History of Erlang

9 Replies

Joe Armstrong wrote a paper for last year’s HOPL-III conference on the history of Erlang. For some reason, I didn’t get a paper copy of those proceedings, and was too busy to notice their absence. Fortunately Lambda the Ultimate picked it up and supplied links to the paper and the accompanying presentation. Digging into the history of something like Erlang is always fascinating, and Armstrong has done a good job of explaining how Erlang came to be.

Here are a bunch of quotes on topics that I found interesting. I’ve grouped them into categories, but searching the PDF of the paper shouldn’t be hard if you want to know where they originated.

Sources of inspiration

Those familiar with Prolog will not find it at all surprising that Erlang has its roots in Prolog (mostly due to implementation reasons). What I did find interesting was the origin/history/viewpoint on the concurrency model

The explanations of what Erlang was have changed with time:

1. 1986 – Erlang is a declarative language with added concurrency.

2. 1995 – Erlang is a functional language with added concurrency.

3. 2005 – Erlang is a concurrent language consisting of communicating components where the components are written in a functional language.

…

Today we emphasize the concurrency.

Note that the word actor never appears in those descriptions. Indeed, the word actor does not appear in the paper at all. So for all the discussion about Erlang’s usage of the actor model, it appears that the Erlang folks independently duplicated many of the ideas for Hewitt’s Actors. I think that is kind of interesting.

Lisp and Smalltalk are cited as inspirations, but more for the implementation of the runtime than for any features in the language. I came away from the paper with the impression that Armstrong and his colleagues are not paradigm ideologues. They are trying to get the job done.

Reliability

There is a huge emphasis on reliability throughout the paper, supporting Steve Vinoski’s remarks about Erlang. I’l just include a series of quotes, which you can interpret as you see fit:

Erlang was designed for writing concurrent programs that “run forever”

At an early stage we rejected any ideas of sharing resources between processes because of the difficulty of error handling. In many circumstances, error recovery is impossible if part of the data needed to perform the error recovery is located on a remote machine and if that remote machine has crashed.

In order to make systems reliable, we have to accept the extra cost of copying data structures between processes and always make sure that processes have enough data to continue by themselves if other processes crash

The key observation here is to note that the error-handling mechanisms were designed for building fault-tolerant systems, and not merely for protecting from program exceptions. You cannot build a fault-tolerant system if you only have one computer. The minimal configuration for a fault-tolerant system has two computers. These must be configured so that both observe each other. If one of the computers crashes, then the other computer must take over whatever the first computer was doing.

This means that the model for error handling is based on the idea of two computers that observe each other. Error detection and recovery is performed on the remote computer and not on the local computer.

Links in Erlang are provided to control error propagation paths for errors between processes.

It was about this time that we realized very clearly that shared data structures in a distributed system have terrible properties in the presence of failures. If a data structure is shared by two physical nodes and if one node fails, then failure recovery is often im-possible. The reason why Erlang shares no data structures and uses pure copying message passing is to sidestep all the nasty problems of figuring out what to replicate and how to cope with failures in a distributed system.

In our world, we were worried by software failures where replication does not help.

Design criteria

Here are some quotes related the design criteria for Erlang.

Changing code on the fly was an initial key requirement

the notion that three properties of a programming language were central to the efficient operation of a concurrent language or operating system. These were:

1) the time to create a process

2) the time to perform a context switch between two different processes

3) the time to copy a message between two processes

The performance of any highly-concurrent system is dominated by these three times.

One of the earliest design decisions in Erlang was to use a form of buffering selective receive

Pipes were rejected in favor of messages

In the concurrent logic programming languages, concurrency is implicit and extremely fine-grained. By comparison Erlang has explicit concurrency (via processes) and the processes are coarse-grained.

The final strategy we adopted after experimenting with many different strategies was to use per-process stop-and-copy GC. The idea was that if we have many thousands of small processes then the time taken to garbage collect any individual process will be small.

…

Current systems run with tens to hundreds of thousands of processes and it seems that when you have such large numbers of processes, the effects of GC in an individual process are insignificant.

The BEAM compiler compiled Erlang programs to BEAM instructions.

On functionalness

This next series of quotes will probably make the pure functional language people shake their heads, but i think that it’s important to understand Erlang in contrast with pure functional languages.

Erlang is not a strict side-effect-free functional language but a concurrent language where what happens inside a process is described by a simple functional language.

Behaviors in Erlang can be thought of as parameterizable higher-order parallel processes.

… the status of Erlang as a fully fledged member of the functional family is dubious. Erlang programs are not referentially transparent and there is no system for static type analysis of Erlang programs. Nor is it relational language. Sequential Erlang has a pure functional subset, but nobody can force the programmer to use this subset; indeed, there are often good reasons for not using it.

An Erlang system can be thought of as a communicating network of black boxes.

In the Erlang case, the language inside the black box just happens to be a small and rather easy to use functional language, which is more or less a historical accident caused by the implementation techniques used.

History and Usage

One thing that I was looking for in the paper was more details on how long Erlang had been around (besides before Java), how big the largest programs/systems were, and so forth. Here is what I found.

This history spans a twenty-year period…

(The history starts in 1986)

The largest ever system built in Erlang was the AXD301. At the time of writing, this system has 2.6 millions lines of Erlang code.

…

The AXD301 is written using distributed Erlang. It runs on a cluster using pairs of processors and is scalable up to 16 pairs of processors.

…

In the analysis of the AXD reported in [7], the AXD used 20 supervision trees, 122 client-server models, 36 event loggers and10 finite-state machines. All of this was programmed by a team of 60 programmers.

…

As regards reliability, the AXD301 has an observed nine-nines reliability [7]—and a four-fold increase in productivity was observed for the development process [31].

The AXD 301 is circa 1998.

Perhaps the most exciting modern development is Erlang for multicore CPUs. In August 2006 the OTP group released Erlang for an SMP.

This corroborates something that David Pollak told me at the RedMonk unconference during CommunityOne, namely that SMP support in Erlang had not been there very long. Of course, Erlang was running on systems with 16 physical (pairs, no less) of processings in a distributed environment. So while the runtime might not be that mature on SMP, the overall runtime for concurrency is probably a bit more mature than that. Nonetheless, worthwhile to know the precise facts.

All in all, I found the paper to be a very worthwhile read – (and a nice change from my usual intake of blog posts and tweets). One of my pet peeves about the computer business is the lack of awareness of the history of the field. At least I’ve removed a bit of my own ignorance as relates to Erlang.