The Architecture of Open Source Applications, Volume 1 – Free Online Book (Brown and Wilson 2011) BSSE

BSSE and BSCS students can read The Architecture of Open Source Applications, Volume 1 free online. Edited by Amy Brown and Greg Wilson, it is a collection of 25 chapters in which the developers of real open source projects explain how their software is built and why. It fits Software Architecture and Design and Software Construction courses. There is no official PDF: the editors publish the book free in the browser, and print and e-book editions are sold separately.

Each chapter covers one project, from the Bash shell, LLVM and Berkeley DB to HDFS, Eclipse, Selenium and the game Battle for Wesnoth. The authors describe the main components, the trade-offs behind them and what they would change today. Every chapter stands alone, so you can pick the systems that match your course. The book assumes you can already read code and know basic object-oriented ideas.

Book Overview

CourseSoftware Architecture & Design / Software Construction
Degree ProgramsBSSE, BSCS
LevelUniversity / Undergraduate (intermediate to advanced)
EditionVolume 1 (2011), the first book of the AOSA series
EditorsAmy Brown and Greg Wilson
Chapter Authors43 developers of the projects described, one team per chapter
LanguageEnglish
Total Chapters25 chapters plus an introduction; no declared parts
Projects CoveredAsterisk, Audacity, Bash, Berkeley DB, CMake, Eclipse, Graphite, HDFS, Jitsi, LLVM, Mercurial, Riak, Selenium WebDriver, Sendmail, VTK and more
ExercisesNone (each chapter ends with lessons learned)
LicenseCC BY 3.0 Unported — Model: Link-only
FormatFree online book (HTML chapters); no PDF; paperback, ePub and Kindle editions sold separately

Chapter List

Chapter 1: Asterisk

Difficulty: Medium · Key topics: channels, frames, bridging, channel drivers, dialplan, codec translators, threads

Asterisk is an open source telephony platform that handles VoIP, analog and digital phone calls and lets developers customize how calls are processed. The chapter starts with three core ideas: a channel is one leg of a call, bridging joins two channels, and frames carry voice, video or signalling between them. It then describes the pluggable component types (channel drivers, dialplan applications, dialplan functions and codec translators) and the thread model, with monitor threads for new calls and one thread per channel. Two walkthroughs, checking voicemail and bridging a call, tie it together, and scaling across servers is named as the main open problem.

Key Points:

  • A channel is one leg of a call, so a call between two phones uses two channels.
  • A frame is a typed unit of media or signalling, such as voice, video, DTMF or control.
  • A generic bridge passes all media through the core; a native bridge lets two channels of the same technology hand media off directly and is faster.
  • A small core loads modules at runtime, and each channel driver implements one common interface for its protocol.
  • Dialplan applications do the work of a call; dialplan functions only read or set channel data.
  • Each active channel runs in its own thread, while separate monitor threads wait for new incoming calls.

Memory Tip: Picture Asterisk as a telephone exchange built from plug-in cards: the core is the switchboard, and every driver, application and translator is a card you can add or remove.

Common Mistake: Assuming one phone call is one channel. A call between two phones has two channels joined by a bridge.

Important Questions:

  • What is a channel in Asterisk? One leg of a call. A call between two phones uses two channels, and Asterisk bridges them together.
  • Why does Asterisk have both generic and native bridges? A generic bridge works for any pair of technologies but sends all media through the core, while a native bridge lets two channels of the same technology exchange media more directly and faster.

Chapter 2: Audacity

Difficulty: Medium · Key topics: wxWidgets, PortAudio, ShuttleGui, TrackPanel, BlockFiles, threads, scripting

Audacity is a desktop audio editor, and this chapter shows how it is layered on wxWidgets for the interface and PortAudio for recording and playback. The author walks through the ShuttleGui helper that builds dialogs and moves preference data, the TrackPanel that draws tracks itself instead of creating many widgets, and the three threads that share audio data. He then explains BlockFiles, which split audio into reference-counted blocks with summary blocks for fast redraws, followed by the scripting pipe and the plan for real-time effects. The chapter is open that the code is a mix of well-structured and less well-structured parts.

Key Points:

  • Audacity is layered on PortAudio for audio input and output and wxWidgets for the GUI, with optional libraries loaded dynamically.
  • ShuttleGui is a facade over wxWidgets: one code path both builds a dialog and moves values to and from preferences.
  • The TrackPanel draws its own lightweight components to avoid the cost of many window handles.
  • Audio is stored as blocks of about 1 MB, plus summary blocks of minimum and maximum values that make waveform drawing fast.
  • Blocks are reference counted, so copied audio can share the same block until it changes.
  • Three execution contexts run side by side: a PortAudio callback, an audio thread that fills buffers, and the GUI thread.

Memory Tip: A BlockFile is like a book split into chapters with a one-line summary on each cover. The summary lets Audacity draw the waveform without reading the audio itself.

Common Mistake: Moving or deleting files in a project’s data folder by hand. The project file only points to the block files, so missing blocks mean lost audio.

Important Questions:

  • What are BlockFiles and why does Audacity use them? They are fixed-size pieces of audio stored in separate files, so edits such as cut and paste work on references instead of copying huge files, and summary blocks speed up display.
  • Which threads does Audacity use for audio? A PortAudio callback thread, an audio thread that moves data between disk, tracks and buffers, and the wxWidgets GUI thread.

Chapter 3: The Bourne-Again Shell

Difficulty: Medium · Key topics: Readline, parser, word expansion, globbing, redirection, builtins, job control

Chet Ramey follows a command line through the stages of Bash, the GNU shell: input processing, parsing, word expansion and command execution. He covers how Readline handles line editing, how the Yacc-based parser and the lexer share state, and how expansions such as braces, tildes, variables, command substitution and filename globbing are applied in a fixed order. The chapter then explains how builtins, external commands, redirections, job control and compound commands are carried out. It closes with lessons from decades of maintenance, including regression tests, standards work and documentation, and what the author would do differently.

Key Points:

  • A command moves through four stages: input, parsing, expansion and execution, and each stage hands its result to the next.
  • Bash passes words between stages in WORD_DESC structures linked in WORD_LIST lists, and stores variables in hash tables.
  • Readline is a separate library that reads keystrokes, resolves them through keymaps and redisplays the line.
  • Word splitting (using IFS) and globbing happen after the other expansions.
  • External commands run in a child process after fork, while builtins run inside the shell process itself.
  • Redirections use dup2 and are undone afterwards, and job control relies on process groups.

Memory Tip: Remember the pipeline as read, parse, expand, execute. The shell never runs your text directly; it first turns the text into words.

Common Mistake: Expecting a program to expand *.txt itself. The shell expands the glob before the program starts, so the program only sees the list of names.

Important Questions:

  • In what order does Bash process a command line? It reads input, parses it into commands, expands the words (braces, tilde, variables, command substitution, word splitting, globbing) and then executes the command.
  • Why are some commands builtins? A builtin runs inside the shell process, so it can change the shell’s own state, such as the current directory with cd. A child process could not do that.

Chapter 4: Berkeley DB

Difficulty: Hard · Key topics: embedded library, Btree, Hash, Recno, Queue, Mpool, locking, write-ahead logging, transactions

Margo Seltzer and Keith Bostic describe Berkeley DB as a library that links into the application instead of running as a separate database server. The chapter explains the modular design: four access methods (Btree, Hash, Recno and Queue), a library interface layer, and underlying subsystems for buffer management (Mpool), locking, logging and transactions. It shows how write-ahead logging, log sequence numbers and checkpoints combine to give ACID guarantees and two-pass recovery. The authors finish with a list of design lessons, including enforcing module boundaries, because architecture decays with every change.

Key Points:

  • Berkeley DB is an embedded library: the application links it in, and there is no server process to run.
  • The four access methods are Btree, Hash, Recno (variable-length records) and Queue (fixed-length records).
  • Mpool is the page cache, and log sequence numbers on pages enforce write-ahead logging.
  • The lock manager supports hierarchical locking with intention locks and a configurable conflict matrix.
  • The log is an append-only series of files, and a log sequence number is a file number plus an offset.
  • Recovery makes two passes: undo uncommitted work, then redo committed work.

Memory Tip: Hold the transaction stack in four verbs: Mpool caches, Lock isolates, Log remembers, Txn ties them together.

Common Mistake: Assuming an embedded database needs no recovery planning. The log and checkpoints are what make a crash survivable.

Important Questions:

  • What does write-ahead logging mean? The log record describing a change is written to disk before the changed page may be written, so after a crash the change can be undone or redone.
  • How is Berkeley DB different from a client-server database? It runs inside the application’s own process as a library, so there is no network round trip and the application chooses which components to use.

Chapter 5: CMake

Difficulty: Medium · Key topics: CMakeLists.txt, configure and generate, CMakeCache.txt, targets, dependency analysis, CTest, CPack, policies

CMake is a cross-platform build system that generates native Makefiles and IDE project files from one description. The chapter gives the history and requirements, then walks through the two phases of a run: configure, which parses the CMakeLists.txt files, and generate, which writes the build files. It covers the main internal objects (commands, makefiles and targets), automatic dependency analysis, and the companion tools CTest, CDash and CPack. It ends with lessons on backwards compatibility through policies, the weaknesses of a custom language, why plugins did not work, and why exposing fewer APIs is safer.

Key Points:

  • CMake needs only a C++ compiler to build; it avoids depending on an external scripting runtime.
  • A run has two phases: configure (parse and check the system) and generate (write the native build files).
  • CMakeCache.txt stores persistent variables in the build tree instead of fragile environment variables.
  • Each language command is a cmCommand subclass, and targets live in cmMakefile objects that mirror the source directories.
  • Out-of-source builds keep generated files away from the source tree.
  • Policies, introduced in CMake 2.6, let a project declare which behavior it expects so new releases do not break old projects.

Memory Tip: CMake does not build anything itself; it writes the files another tool builds with. Describe once, generate many.

Common Mistake: Editing the generated Makefiles by hand. They are overwritten the next time CMake runs, so change CMakeLists.txt instead.

Important Questions:

  • What are the two phases of a CMake run? Configure reads the CMakeLists.txt files, runs checks and fills the cache; generate then writes the native build files for the chosen tool.
  • Why does CMake have a policy system? It keeps old projects building as CMake changes. Each project states the behavior version it expects, so fixes can be added without breaking existing builds.

Chapter 6: Eclipse

Difficulty: Medium · Key topics: plug-ins, extension points, OSGi, Equinox, SWT and JFace, p2, Eclipse 4 dependency injection

Kim Moir tells the story of Eclipse through its releases. Early Eclipse was a platform plus the Java Development Tools and the Plug-in Development Environment, built on a plug-in model with extension points and lazy loading. Version 3.0 moved the runtime to OSGi through Equinox and opened the platform to the Rich Client Platform; 3.4 introduced the p2 provisioning system; and 4.0 added a model workbench, CSS styling and dependency injection. The conclusion argues that a published API is a long-term promise, and that an API that grows too large becomes a cost for everyone.

Key Points:

  • Eclipse is a small runtime plus plug-ins; each plug-in declares its dependencies and extension points in a manifest.
  • Plug-ins are loaded lazily, so their classes load only when they are first used.
  • Exported packages form the public API, and everything else is private.
  • SWT wraps native widgets, and JFace adds higher-level helpers such as wizards and preference pages.
  • Eclipse 3.0 adopted OSGi bundles through Equinox, with a compatibility layer for older plug-ins.
  • p2 (Eclipse 3.4) installs software as installable units in profiles, and Eclipse 4 uses dependency injection through hierarchical contexts.

Memory Tip: Remember the four generations in order: platform, OSGi (3.0), p2 (3.4), model workbench (4.0). Each one fixed a problem the previous one exposed.

Common Mistake: Treating internal packages as API just because they are visible in the code. Only the exported packages are a supported contract.

Important Questions:

  • What is an extension point? A place where one plug-in declares that others may add behavior. Other plug-ins contribute extensions to it without changing its code.
  • Why does the chapter say that API is forever? Once an API is released, other projects depend on it, so changing it breaks them. Eclipse had to keep compatibility layers, and its API grew larger than most consumers needed.

Chapter 7: Graphite

Difficulty: Medium · Key topics: whisper, carbon, Django web app, time-series data, write batching, Memcached, clustering

Chris Davis shows how a small monitoring tool grew into a large-scale graphing system. Graphite has three parts: the whisper time-series database library, the carbon daemon that receives metrics over the network, and a Django web application that renders graphs on demand. The chapter explains how storage I/O quickly became the bottleneck, and how batching writes, buffering points in memory and limiting queue sizes fixed it. It then covers keeping recent data visible, protecting the kernel cache from catastrophic failures, and clustering with remote nodes and relays, before reflecting on design and on becoming open source.

Key Points:

  • Metrics are named with dot-separated paths that map directly to files on disk.
  • whisper stores each metric in a fixed-size time-series file, similar to RRDtool.
  • Clients send plain text lines containing a metric name, a value and a timestamp to carbon.
  • Disk I/O, not CPU, was the bottleneck, so the speed of Python was not the limit.
  • update_many combines contiguous points into a single write.
  • Memcached caches rendered graphs and lookup results to reduce repeated work.

Memory Tip: Three words cover the system: whisper stores, carbon receives, the web app draws.

Common Mistake: Optimizing code speed before finding the real bottleneck. In Graphite the slow part was disk writes, so batching I/O mattered more than faster code.

Important Questions:

  • Why was I/O the bottleneck in Graphite? Each metric lives in its own file, so many metrics meant a huge number of small random writes. The disk fell behind long before the CPU did.
  • How are recent data points shown while writes are batched? Points wait in in-memory queues, and the web application can ask carbon for them and merge them with the data read from disk.

Chapter 8: The Hadoop Distributed File System

Difficulty: Hard · Key topics: NameNode, DataNode, block replication, image and journal, rack awareness, balancer, federation

The five Yahoo! authors describe the Hadoop Distributed File System, built to store very large files across commodity servers. A single NameNode keeps the namespace and block locations in memory, while DataNodes store replicated blocks and report back through heartbeats and block reports. The chapter explains the image and journal, the CheckpointNode and BackupNode, the read and write pipeline, rack-aware block placement and background services such as the balancer. It then covers experience running HDFS at Yahoo!, including federation, and lessons about keeping the design simple.

Key Points:

  • Metadata and data are separated: the NameNode holds the namespace and block map, and DataNodes hold the blocks.
  • Blocks are replicated, by default three times, to get durability and read bandwidth instead of using RAID.
  • The NameNode’s persistent state is a checkpoint (the image) plus a journal of changes.
  • DataNodes send regular heartbeats and block reports, and the NameNode replies with instructions.
  • Rack-aware placement puts the first replica on the writer’s node and the others on a different rack.
  • HDFS allows one writer and many readers per file, enforced with leases.

Memory Tip: Think of a library: the NameNode is the catalog that says where each book is, and the DataNodes are the shelves that actually hold the books.

Common Mistake: Believing the NameNode stores file contents. It holds only metadata, and clients read the actual data directly from DataNodes.

Important Questions:

  • Why does HDFS replicate blocks? Servers are cheap and fail often, so copies on different nodes and racks protect the data and let more clients read in parallel.
  • What is the role of a DataNode heartbeat? It tells the NameNode that the node is alive, and the reply can carry instructions such as replicating or deleting a block.

Chapter 9: Continuous Integration

Difficulty: Medium · Key topics: Buildbot, CDash, Jenkins, pony-build, build recipes, master and slave, trust

C. Titus Brown and Rosangela Canino-Koning compare how continuous integration systems are built. After explaining what CI software does and how it connects to version control, build tools and notifications, they examine four architectural models: Buildbot, where a master controls build slaves; CDash, where clients run builds and submit reports to a server; Jenkins, a hybrid with plugins; and pony-build, a decentralized design. They then discuss build recipes, the trust problem of running code from a repository, and how to choose a model. The chapter ends with future directions such as shared recipe languages and common report formats.

Key Points:

  • CI software checks out the code, builds it, runs the tests and reports the result for every change.
  • Buildbot uses a central master that schedules builds and controls connected build slaves.
  • CDash clients build on their own schedule and submit XML reports to a reporting server.
  • Jenkins supports both master-driven and client-initiated builds and extends through plugins.
  • pony-build splits the results server, reporting server and clients, which talk over XML-RPC.
  • Build recipes run arbitrary code, so who is trusted to write them becomes an architecture question.

Memory Tip: Ask who controls the schedule: the master (Buildbot), the client (CDash), either (Jenkins) or nobody in particular (pony-build).

Common Mistake: Comparing CI tools only by feature lists. The chapter’s point is that the architecture decides which features are easy to build later.

Important Questions:

  • What is the main difference between Buildbot and CDash? In Buildbot a master schedules and controls builds on its slaves. In CDash the clients run builds themselves and only submit reports to a server.
  • Why is trust an issue in continuous integration? Build recipes execute code from the repository on the build machines, so a faulty or malicious recipe can damage them or leak data.

Chapter 10: Jitsi

Difficulty: Medium · Key topics: OSGi, bundles, services, protocol provider, operation sets, media service, SIP, XMPP, RTP

Emil Ivov describes Jitsi, a Java application for voice, video and chat over many protocols such as SIP and XMPP. The design goals were multi-protocol support, cross-platform operation and easy extension, so the team built Jitsi on the OSGi framework instead of writing its own plugin system. The chapter shows how features are packaged as bundles that expose services, how the Protocol Provider Service and operation sets hide protocol differences, and how the Media Service and UI Service work. It closes with lessons on where Java fell short for audio and video and how native libraries filled the gaps.

Key Points:

  • Every feature is an OSGi bundle (a JAR file), with service interfaces kept separate from implementation packages.
  • A BundleActivator registers and looks up services when a bundle starts and stops.
  • One protocol provider instance represents one account on one protocol.
  • Operation sets let the GUI ask what a protocol supports instead of assuming every protocol has every feature.
  • Signalling protocols such as SIP and XMPP negotiate the call, and RTP carries the media.
  • Java was fast enough, but weak for audio device selection, video capture and encoding, which needed native help.

Memory Tip: Service in front, Impl behind: a bundle publishes interface packages and hides its classes in implementation packages.

Common Mistake: Coding against another bundle’s implementation classes. Other bundles should depend only on the service interface.

Important Questions:

  • What is an operation set? An interface that groups related features, such as instant messaging or presence. The GUI checks which sets a protocol supports and only offers those features.
  • Why did Jitsi use OSGi? It gave the project a ready-made framework for modular, dynamically loaded features, so the team did not have to write and maintain its own plugin system.

Chapter 11: LLVM

Difficulty: Hard · Key topics: three-phase compiler, LLVM IR, passes, PassManager, code generator, TableGen, bitcode, BugPoint

Chris Lattner begins with the classical three-phase compiler: front end, optimizer and back end. He explains why a shared intermediate representation lets many languages and many targets combine without a separate compiler for every pair, then introduces LLVM IR, a typed, low-level, RISC-like language that exists as text, in memory and as bitcode. The chapter shows how LLVM is a set of libraries, how optimizer passes are run by a PassManager, and how the retargetable code generator uses target description files. It ends with benefits of the modular design, such as link-time optimization, unit testing with opt and automatic test case reduction with BugPoint.

Key Points:

  • A compiler has three phases (front end, optimizer, back end) joined by a common intermediate representation.
  • LLVM IR has three equivalent forms: text (.ll), in-memory data structures and binary bitcode (.bc).
  • Each optimization is a Pass class, and the PassManager runs passes in an order that respects their declared dependencies.
  • LLVM is a collection of libraries, so a program links only the passes it actually uses.
  • Target description files (.td) processed by tblgen generate instruction matching and encoding code.
  • BugPoint automatically shrinks a failing test case to a small one.

Memory Tip: With a common IR you write N front ends plus M back ends, not N times M compilers.

Common Mistake: Thinking LLVM is only a compiler for C. A language front end such as Clang is just one client of the same optimizer and code generator.

Important Questions:

  • Why does LLVM use an intermediate representation? So that any front end can feed any back end through one shared optimizer, which removes the need for a separate compiler for every language and target pair.
  • What is a pass? A unit of analysis or transformation that works on the IR, such as a function pass, scheduled and run by the PassManager.

Chapter 12: Mercurial

Difficulty: Medium · Key topics: changesets, DAG, revlog, changelog, manifest, dirstate, branches, extensions, hooks

Dirkjan Ochtman starts with a short history of version control, from centralized systems to distributed ones. He then explains the data structures behind Mercurial: history as a graph of changesets identified by hashes, stored in revlogs that pair an index with delta-compressed data. Three revlogs, the changelog, the manifest and the filelogs, are layered together, and the dirstate tracks the working directory. The chapter then covers branches, tags, the overall code structure, and the extension and hook systems, and ends with lessons about Python, rewriting history and usability.

Key Points:

  • History is a directed acyclic graph of changesets, with merges where branches join.
  • A changeset ID is a SHA1 hash covering content, parents and metadata, so history can be verified.
  • A revlog is an index of fixed-length records plus a data file of deltas, with delta chains capped to limit reconstruction cost.
  • The changelog points to a manifest, which maps file names to filelogs holding each file’s history.
  • Data is written in a fixed order, filelogs then manifest then changelog, to keep the repository consistent.
  • Extensions add or change commands, and hooks run shell scripts or Python functions on events.

Memory Tip: Changelog lists the changes, manifest lists the files, and each filelog lists one file’s history, so three logs answer three different questions.

Common Mistake: Assuming local revision numbers are the same in every clone. They are local shorthand, so share the hash instead.

Important Questions:

  • What is a revlog? The storage structure for the history of one item: an index file of fixed-length entries plus a compressed data file of deltas, which allows fast lookup by revision number.
  • How does Mercurial keep a repository consistent while writing? It writes in a fixed order and takes locks, so references only ever point to data that has already been written.

Chapter 13: The NoSQL Ecosystem

Difficulty: Hard · Key topics: data models, durability, sharding, consistent hashing, range partitioning, CAP, quorum, eventual consistency

Adam Marcus surveys the design choices behind NoSQL systems rather than a single product. After explaining the origins of the term, he compares data models (key-value, key-data structure, document, column family and graph) and the query and transaction limits that come with them. The chapter then covers single-server and multi-server durability, scaling with sharding, consistent hash rings and range partitioning, and consistency from the CAP theorem through strong and eventual models. The conclusion urges readers to understand the underlying design decisions instead of memorizing product features.

Key Points:

  • Most NoSQL stores look up data by key and leave joins to the application.
  • Key-value, document and column family models each trade query power for predictable performance.
  • Durability depends on how often data is flushed to disk, on write-ahead logs and on replication.
  • Consistent hash rings with virtual nodes spread keys evenly, while range partitioning supports range scans.
  • With N replicas, W write acknowledgments and R read responses, R + W > N gives strong consistency.
  • Eventual consistency relies on tools such as vector clocks, read repair, hinted handoff and anti-entropy.

Memory Tip: Network partitions will happen, so CAP really asks one question: when the network splits, do you keep consistency or availability?

Common Mistake: Choosing NoSQL as simply a faster SQL. The chapter says to shard only when you must, and reminds you that you give up guarantees such as joins and transactions.

Important Questions:

  • What does R + W > N guarantee? The set of replicas read and the set written must overlap, so every read sees at least one copy of the latest write. That is strong consistency.
  • When would you pick range partitioning instead of consistent hashing? When you need range scans over ordered keys. Consistent hashing spreads keys evenly and handles nodes joining and leaving, but it cannot serve range queries efficiently.

Chapter 14: Python Packaging

Dated chapter (describes 2011) · Difficulty: Medium · Key topics: Distutils, setup.py, metadata, PEP 345, PEP 376, PEP 386, PyPI mirrors

Tarek Ziadé explains why packaging Python software was painful and how a set of standards was meant to fix it. The chapter reviews Distutils, its single setup.py script, its metadata and PyPI, and how tools such as Setuptools and Pip grew around its flaws. It then describes the improved standards proposed in PEPs for metadata, version comparison, the record of installed distributions and data files, along with the architecture of PyPI mirrors. The lessons stress that standards change slowly through PEPs, that a package in the standard library is hard to change, and that backward compatibility is essential.

Key Points:

  • Distutils uses one setup.py script for building, publishing and installing.
  • Metadata generated by a script reflects the platform that produced it, which is why static, declarative metadata was proposed.
  • PEP 386 defined a version ordering, and PEP 345 added project-level fields such as Requires-Dist.
  • PEP 376 defined a database of installed distributions (dist-info with METADATA and RECORD files), which makes uninstalling possible.
  • PyPI mirrors synchronize through the changelog and verify authenticity with signed files.
  • Today Distutils is gone from the standard library (removed in Python 3.12) and projects use pyproject.toml, so read this chapter for the ideas.

Memory Tip: Declare, do not execute: metadata you can read without running code is safer and easier for tools to understand.

Common Mistake: Following this chapter as a how-to for packaging today. It records the 2011 situation and the reasoning behind later standards, not current commands.

Important Questions:

  • Why is a setup.py script a problem for packaging? You must run code to learn a project’s metadata, and the result can depend on the platform that ran it. Declarative metadata can be read statically.
  • What did PEP 376 add? A standard record of installed distributions, so tools can list what is installed and uninstall it cleanly.

Chapter 15: Riak and Erlang/OTP

Difficulty: Hard · Key topics: Erlang processes, message passing, gen_server, gen_fsm, supervisors, consistent hashing, gossip

Francesco Cesarini, Andy Gross and Justin Sheehy use Riak to show why Erlang and OTP suit fault-tolerant distributed systems. The chapter begins with a short introduction to Erlang processes and message passing, then builds up the process skeleton and the OTP behaviors: generic servers, finite state machines and event handlers. It explains supervisors, restart strategies and applications, which form supervision trees, and then looks at how Riak replicates data and communicates between nodes using consistent hashing and a gossip protocol. The authors conclude that the “let it crash” approach and simple message passing made Riak quick to build and robust.

Key Points:

  • Erlang processes are lightweight and communicate only by sending asynchronous messages.
  • A process skeleton is: spawn, optionally register a name, initialize state, run a receive loop, then clean up and stop.
  • gen_server, gen_fsm and gen_event are OTP behaviors for servers, state machines and event handlers.
  • Synchronous calls with a timeout act as serialization points that regulate load.
  • Supervisors restart crashed child processes according to a strategy such as one-for-one.
  • Riak follows the Dynamo design: consistent hashing decides where replicas go, and gossip shares who owns which partition.

Memory Tip: Workers work and supervisors watch. If a worker dies, its supervisor starts a fresh one in a clean state.

Common Mistake: Writing defensive code for every possible error. The Erlang approach is to let the process crash and have its supervisor restart it.

Important Questions:

  • What is a supervisor in OTP? A process that starts, monitors and restarts child processes under a restart strategy. Supervisors can be nested into a supervision tree.
  • How does Riak share cluster membership without a central server? Nodes use a gossip protocol, exchanging information about the ring and partition ownership with each other.

Chapter 16: Selenium WebDriver

Difficulty: Medium · Key topics: browser automation, atoms, remote protocol, Firefox driver, IE driver, Selenium RC, RPC

Simon Stewart explains how Selenium WebDriver automates real browsers for testing. He traces the merger of Selenium and WebDriver, then lists architectural themes: keep costs down, emulate the user, prove the drivers work, share code between drivers and treat every call as a remote procedure call. The chapter describes the layered design with browser automation atoms written in JavaScript, the JSON-over-HTTP remote protocol, the Firefox and Internet Explorer drivers, and the older Selenium RC. It ends with reflections on browser quirks and the future of the project.

Key Points:

  • Selenium Core and Selenium RC merged with WebDriver, which began as an adapter-style API.
  • Browser automation atoms are small JavaScript functions compiled with Google’s Closure tools and reused by every driver.
  • Every WebDriver call is an RPC: the remote protocol sends JSON over HTTP.
  • Drivers prefer native operating system events over synthesized JavaScript events, to behave like a real user.
  • The Firefox driver is an extension with an embedded HTTP server, while the IE driver is a C++ library with a flat API.
  • Selenium RC used an HTTP proxy and long polling to send commands to the browser.

Memory Tip: Test like a user, talk like an RPC: the script acts like a person at the browser, but every action travels as a request and response.

Common Mistake: Assuming a click or a visibility check behaves the same in every browser. Browsers differ in many edge cases, and the shared atoms exist to hide those differences.

Important Questions:

  • What are browser automation atoms? Small shared JavaScript functions that implement behaviors such as checking whether an element is displayed. Every driver reuses them, so a fix in one place helps all browsers.
  • Why does WebDriver treat every call as an RPC? The test script and the browser are often in different processes or on different machines, so each command travels as a request and a response.

Chapter 17: Sendmail

Difficulty: Medium · Key topics: mail transfer agent, rewriting rules, m4 configuration, SMTP, mail queue, milter, security

Eric Allman, the author of sendmail, tells how the mail transfer agent evolved from delivermail in the early Internet. He lists his design principles: accept that one programmer is finite, do not redesign user agents or the local mail store, adapt to the world, change as little as possible and think about reliability early. The chapter then covers the development phases, configuration through rewriting rules and m4, the embedded SMTP and queue design, the milter filter interface and security. It ends with what he would do differently if he started today.

Key Points:

  • Sendmail handles mail transport only; it leaves user agents and the local mail store alone.
  • The earlier delivermail routed mail with compiled-in rules based on characters such as @, : and !.
  • Runtime configuration uses token-based rewriting rules, and m4 macros generate the raw configuration file.
  • The queue keeps separate control and data files per message, and directory scans slow down when the queue grows very large.
  • Milter runs mail filters as separate processes, so a crashing filter cannot take sendmail down.
  • Sendmail gives up root privileges after opening its SMTP socket, to limit damage from bugs.

Memory Tip: The guiding rule is do as little as possible: each piece of work you leave to another program is code you never have to maintain.

Common Mistake: Putting everything into one program. Allman regrets exposing a configuration language to end users and delaying a redesign of it.

Important Questions:

  • What was sendmail’s first design principle? Accept that one programmer is finite. It pushed Allman to reuse existing user agents and the local mail store instead of redesigning them.
  • Why was milter added? It lets filtering run in separate processes, which adds features such as spam checks without risking the stability or security of the main mail daemon.

Chapter 18: SnowFlock

Difficulty: Hard · Key topics: VM cloning, lazy replication, copy-on-write, multicast, memtap, page presence bitmap, Xen

Roy Bryant and Andrés Lagar-Cavilla describe SnowFlock, a system that clones a running virtual machine into many copies in about a second, like fork for whole VMs. The chapter explains the choice of lazy over eager state replication, the small architectural descriptor sent first, and the parent side with the memory server, copy-on-write and the mcdist multicast protocol. It then covers the clone side with memtap and a page presence bitmap, tricks that avoid unnecessary fetches, and the API for cloning applications. The lessons praise simplicity and warn that scale exposes new bottlenecks.

Key Points:

  • VM fork is the VM-level analogue of Unix fork and can start many clones in parallel, including on other hosts.
  • Lazy replication sends only the state a clone needs to start and fetches memory pages on demand.
  • The architectural descriptor is a small seed holding page tables, virtual registers and device metadata.
  • Copy-on-write freezes the parent’s memory so the memory server can serve a consistent snapshot.
  • The mcdist protocol uses IP multicast, so one transmission of a page can reach many clones.
  • A page presence bitmap, updated with atomic bit operations, lets Xen, memtap and the guest coordinate without heavy locking.

Memory Tip: Think lazy loading for whole machines: copy a tiny seed first, then bring in memory only when the clone actually touches it.

Common Mistake: Assuming a design that works for a few clones will work for hundreds. The chapter shows a simple unicast method that failed at scale and a narrowly scoped multicast design that fixed it.

Important Questions:

  • What is lazy state replication? The clone starts with minimal state and fetches the rest of the parent’s memory pages only when it first needs them, which makes cloning much faster than copying everything.
  • Why does SnowFlock use multicast? Many clones need the same pages from the parent, so sending a page once to all of them avoids the load of many separate unicast streams.

Chapter 19: SocialCalc

Difficulty: Medium · Key topics: WikiCalc, JavaScript spreadsheet, MVC, command run-loop, undo, save format, real-time collaboration

Audrey Tang explains how SocialCalc, a spreadsheet that runs in the browser, was rewritten from the earlier WikiCalc, which kept most work on the server. In SocialCalc the server only serves saved sheets, while calculation, editing and rendering happen in JavaScript on the client. The chapter describes the layered model, view and controller classes, the command run-loop that makes every edit a command with an inverse for undo, and the text-based save format. It then covers rich-text cells and real-time collaboration, and ends with lessons about running a distributed project.

Key Points:

  • WikiCalc was server-centric; SocialCalc moves calculation and interaction into the browser.
  • Sheet is the sparse data model, RenderContext draws the view and TableControl handles input.
  • Every edit is a command string that runs through a sequential loop of execute, recalculate and render.
  • Each executed command generates an inverse command, which is the basis of undo.
  • The save format is a human-readable MIME multipart text document with meta, sheet, edit and audit parts.
  • Real-time collaboration broadcasts commands to peers and resolves conflicts by undoing and redoing pending commands.

Memory Tip: Edit equals command. If every change is a command, undo, audit trail and collaboration all come from the same list.

Common Mistake: Thinking real-time collaboration means sending the whole sheet again. SocialCalc sends small commands instead.

Important Questions:

  • Why does SocialCalc represent edits as commands? The same command can be executed, logged in the audit trail, inverted for undo and broadcast to collaborators.
  • How does SocialCalc resolve conflicts between collaborators? Local pending commands are undone, the remote commands are applied, and then the pending commands are replayed on top.

Chapter 20: Telepathy

Difficulty: Medium · Key topics: D-Bus, connection managers, channels, handles, channel dispatcher, observers, approvers, handlers, sidecars

Danielle Madeley describes Telepathy, a framework that lets desktop applications share instant messaging, voice and video connections through separate cooperating processes. After a short introduction to D-Bus, the chapter lays out the components: connection managers for each protocol, the account manager, the channel dispatcher and the client applications. It explains handles, connections and channels, how channels are requested and dispatched to observers, approvers and handlers, and how to reduce D-Bus traffic. It closes with robustness, sidecars for protocol-specific extensions and lessons about API design.

Key Points:

  • Telepathy treats communication as a shared service, much like printing, so many applications can use one connection.
  • A connection manager implements one protocol, and components talk to each other over D-Bus.
  • Handles are connection-scoped integers that identify contacts and rooms, so clients compare numbers instead of strings.
  • A client can be an Observer (watches), an Approver (asks the user) or a Handler (does the work).
  • Fetching all properties at once, and sending immutable properties with channel signals, reduces D-Bus round trips.
  • Sidecars add protocol-specific interfaces without changing the core API.

Memory Tip: Remember the dispatch order of a new channel: Observers see it, Approvers ask the user, and one Handler takes ownership.

Common Mistake: Writing pseudo-synchronous code against D-Bus. Blocking calls can freeze the interface, reorder messages or deadlock, so use asynchronous callbacks.

Important Questions:

  • What is a Telepathy handle? A connection-scoped integer that identifies a contact or room, so a client can compare handles quickly instead of comparing normalized strings.
  • What roles can a Telepathy client play? Observer (watch a channel), Approver (ask the user to accept it) and Handler (take over and handle the channel).

Chapter 21: Thousand Parsec

Difficulty: Medium · Key topics: game framework, rulesets, objects and orders, binary protocol, TPCL, persistence, metaserver, BattleXML

Alan Laudicina and Aaron Mavrinac describe Thousand Parsec, a framework for turn-based space empire strategy games rather than a single game. The chapter explains the game model of objects, orders, resources and designs, and the binary protocol that lets any compatible client talk to any compatible server. It then covers supporting parts: server persistence with database plugins, the Thousand Parsec Component Language for validating designs on both client and server, BattleXML, the metaserver and single-player mode. The authors finish with what worked and what did not, including the difficulty of debugging a binary protocol.

Key Points:

  • The server holds game state and enforces the rules, and clients, including AI clients, receive the same information.
  • Rulesets define the objects, orders and resources of a particular game on top of the shared protocol.
  • TPCL, a Scheme-like language, checks ship designs on both the client and the server.
  • The protocol is binary and versioned, and is built from request and response frames.
  • Persistence is an abstract class with MySQL and SQLite plugins.
  • The metaserver lists game servers, which must re-announce themselves regularly or expire from the list.

Memory Tip: One protocol, many games: change the ruleset and the same server and clients run a different game.

Common Mistake: Putting game logic in the client. Logic and validation stay on the server so every client sees the same, fair game.

Important Questions:

  • What is a ruleset? The set of rules a server loads that defines the objects, orders and resources of one game, so a single server engine can run different games.
  • Why does Thousand Parsec use TPCL on both client and server? The client can check a design instantly and give feedback, while the server still validates it as the final authority.

Chapter 22: Violet

Difficulty: Easy · Key topics: graph framework, Prototype pattern, JavaBeans, XMLEncoder, Java WebStart, Java 2D, undo and redo, ServiceLoader

Cay Horstmann describes Violet, a small UML diagram editor written in Java, and how it leans on standard Java platform features instead of custom frameworks. The chapter covers the graph framework with nodes and edges created from prototypes, JavaBeans properties for editing, long-term persistence with XMLEncoder, Java WebStart, Java 2D drawing, the lack of a Swing application framework, undo and redo, and a plugin architecture based on ServiceLoader. Its conclusion is that using the JDK’s own mechanisms kept the program small and easy to maintain.

Key Points:

  • Diagrams are built from Node and Edge interfaces, and new elements are cloned from prototype objects.
  • JavaBeans getter and setter pairs, such as getColor and setColor, expose properties to the editor with no extra code.
  • XMLEncoder saves a diagram as construction statements rather than Java serialization, so old files stay readable.
  • Every element returns a Shape from Java 2D, which is used for drawing, shadows and vector printing.
  • Undo and redo are driven by model events, using Swing’s UndoManager and CompoundEdit.
  • Plugins are JAR files discovered with ServiceLoader, which the author chose over OSGi.

Practice Tip: Write a tiny class with getColor and setColor, save it with XMLEncoder and open the XML file. You will see construction statements, not binary data.

Common Mistake: Driving undo from user interface actions. Undo is more reliable when it listens to changes in the model instead.

Important Questions:

  • How does Violet add new diagram elements? It uses the Prototype pattern: the program keeps template node and edge objects and clones one whenever the user adds that element to a diagram.
  • Why does Violet save with XMLEncoder rather than Java serialization? It writes the calls needed to rebuild each object, so files remain readable even when the classes change, as long as the method meanings stay the same.

Chapter 23: VisTrails

Difficulty: Hard · Key topics: workflows, provenance, version tree, dataflow execution, caching, packages, analogies, visual spreadsheet

The six authors describe VisTrails, a scientific workflow and visualization system that records how every result was produced. After defining workflows and provenance, the chapter shows the version tree, which stores a workflow’s history as a series of change operations rather than full copies. It explains dataflow execution with caching of intermediate results, versioned serialization, and the package system for extending the tool in Python. Features such as the visual spreadsheet, workflow analogies, provenance queries and upgrades follow, and the lessons discuss the learning curve and documentation.

Key Points:

  • A workflow is a dataflow graph of modules, and provenance records how a result was produced.
  • The version tree stores edit operations between workflow versions, like a transaction log, instead of full copies.
  • The execution engine caches intermediate results, which assumes modules are functional and free of side effects.
  • Modules that use randomness or have side effects must be flagged as non-cacheable.
  • New modules are added through packages written in Python, with ports for inputs and outputs.
  • An analogy applies the difference between two workflows to a third workflow.

Memory Tip: Store the changes, not the copies: any version can be rebuilt by replaying the edits from the root of the tree.

Common Mistake: Caching the output of a module that is not repeatable. If a module gives different answers for the same input, reusing its cached result gives wrong science.

Important Questions:

  • What is the version tree? The history of a workflow stored as change operations between versions. Any earlier version can be reproduced, and users can branch from it.
  • Why does VisTrails cache intermediate results? So that unchanged upstream steps are not computed again when a user tweaks part of a workflow, which makes exploration much faster.

Chapter 24: VTK

Difficulty: Hard · Key topics: C++ library, reference counting, object factories, data objects, demand-driven pipeline, executive, rendering, observers, BSD licence

Berk Geveci and Will Schroeder explain the design of the Visualization Toolkit, a large C++ library for scientific visualization that is automatically wrapped for Python, Java and Tcl. They cover core features such as reference counting and object factories, the classes that represent data, the demand-driven pipeline of algorithms and executives, the rendering subsystem and the observer-based event system. The chapter explains why a permissive BSD licence and a CMake-based build helped the community grow, and closes with lessons on managing growth, adopting new technologies and open science.

Key Points:

  • The C++ core is wrapped automatically for Python, Java and Tcl, and built, tested and packaged with CMake, CTest and CPack.
  • Reference counting frees large datasets promptly, with a garbage collector only for reference cycles.
  • An object factory lets New() return a device-specific subclass without changing client code.
  • Data objects such as vtkPolyData, vtkUnstructuredGrid and vtkImageData hold the datasets.
  • The pipeline is demand-driven: a filter runs only when output is requested and something upstream has changed.
  • Observers react to events, and widgets combine an interaction observer with a visible prop.

Memory Tip: Pull, do not push: asking for the final output pulls the request back up the pipeline, and only stale stages recompute.

Common Mistake: Expecting a filter to run as soon as you connect it. In VTK nothing executes until some output is requested, for example when you render.

Important Questions:

  • What does demand-driven pipeline mean in VTK? Algorithms execute only when their output is needed, and they re-execute only if their input or settings have changed since the last run.
  • Why does VTK use reference counting? Datasets can be very large, and reference counting releases memory as soon as nothing uses it, instead of waiting for a garbage collector.

Chapter 25: Battle For Wesnoth

Difficulty: Easy · Key topics: game engine, Wesnoth Markup Language, preprocessor, units and abilities, multiplayer server, dependencies, accessibility

Richard Shimooka and David White describe the architecture of Battle for Wesnoth, a turn-based fantasy strategy game whose goal was to be easy for non-programmers to extend. The chapter gives an overview of the C++ engine and its few dependencies, then explains the Wesnoth Markup Language (WML) used for most game content, with its preprocessor and caching. It shows how units are built from unit types, abilities and attacks rather than class inheritance, and how multiplayer works through a thin server that relays commands. The conclusion says accessibility drove the decisions and attracted many contributors.

Key Points:

  • The engine is C++ and relies on SDL for video, input and portability, with a deliberately short list of other dependencies.
  • WML is a simple attribute-based markup language that most game content and campaigns are written in.
  • A preprocessor handles file inclusion and macros, and its output is cached and invalidated by file timestamps.
  • A unit is an instance that points to an immutable unit type, so new units need no new C++ code.
  • Abilities that the engine understands are combined in WML, and they stay hard-wired so the AI can use them.
  • Player actions are stored as command nodes, which supports replays, and the server wesnothd is a thin relay.

Memory Tip: Data, not code: when a game is built so content lives in a simple markup file, anyone with a text editor can add a unit.

Common Mistake: Choosing the cleanest design without asking who has to use it. Wesnoth accepted a less efficient syntax so beginners could contribute.

Important Questions:

  • Why did Wesnoth use WML instead of plain XML or code? It was designed to be friendly for non-programmers, so players could create campaigns and units with only a text editor.
  • How are units defined in Wesnoth? A unit instance refers to a shared unit type and combines abilities, attacks and traits described in WML, instead of using a class for each unit.

How to Study This Book

This is a collection, not a textbook with a story from page one. The editors’ introduction explains the aim, and then each of the 25 chapters describes one project, written by the people who built it. The chapters are listed alphabetically by project, the book declares no parts, and each chapter stands alone. You can read in any order, so choose by topic instead of by chapter number. The introduction has no block on this page because it is a short preface and the contributor list.

A good first chapter is Chapter 22 (Violet). It is the smallest program in the book and shows familiar design ideas clearly: the Prototype pattern, JavaBeans, undo through model events and a plugin loader. From there, Chapters 6 (Eclipse), 10 (Jitsi) and 20 (Telepathy) are a natural trio on plug-in and service architectures, using OSGi twice and D-Bus once. Read them side by side and ask what each uses to keep modules independent.

For a Software Architecture and Design course, group the chapters by theme. Modular and plug-in design: 1, 6, 10, 11, 20 and 22. Data and storage: 4, 7, 8, 12 and 13. Distributed and concurrent systems: 8, 13, 15 and 18. Build, test and delivery tools: 5, 9 and 16. Language, shell and mail infrastructure: 3, 11 and 17. Scientific and creative applications: 2, 19, 23 and 24, with games in 21 and 25. Chapters 4, 8, 11, 13, 15, 18, 23 and 24 are the hardest, so save them for after you are comfortable with the lighter ones.

Each chapter ends with lessons learned, and these are the most reusable part of the book. Pay special attention to the retrospectives in Chapters 3 (Bash), 4 (Berkeley DB), 6 (Eclipse), 17 (Sendmail) and 21 (Thousand Parsec), where the authors say plainly what they regret. These sections work well as material for design discussions and viva questions on trade-offs.

The book describes each system as it was in 2011, and the details have moved on. The most dated chapter is Chapter 14 (Python Packaging), which reflects a Distutils-era proposal, and other projects have changed releases and tooling since. Treat the version-specific facts as history, and read for the architecture decisions and reasoning, which are still valid. Check each project’s own documentation if you need current commands or features.

Keep expectations realistic. The chapters do not end with exercise sets, so the Important Questions on this page are built from each chapter’s own content. The book also assumes you can already read code, so it does not teach C, Java, Python or Erlang from scratch. It will not give you a formal catalogue of design patterns either; it shows real systems and leaves you to see the patterns in them.


Used In These Programs

This book is used for the Software Architecture and Design course in: BS Computer Science · BS Software Engineering. Browse all Software Engineering books or all Computer Science books.

Who Should Read This

This book is written for BSSE and BSCS students who already know at least one programming language and the basics of object-oriented design, and who are taking Software Architecture and Design, Software Construction or a related course, usually after the first two years. It is also useful alongside Operating Systems, Database Systems and Distributed Systems courses, because many chapters (Bash, Asterisk, Berkeley DB, HDFS, NoSQL, Riak) explain those ideas in working software. It is not for complete beginners and it is not an entry-test guide. For the operating system theory behind the shell, telephony and virtual machine chapters, pair it with Operating Systems: Three Easy Pieces, and for the Java background behind Eclipse, Jitsi and Violet, read Think Java first. For the storage and transaction ideas in Chapters 4, 8 and 13, the Database Management System book is a useful companion.


Applicable Universities

This book is useful for students at Pakistani universities offering BS Software Engineering and BS Computer Science, including Punjab University, Virtual University, COMSATS, FAST, UET, NUST, and other HEC-recognized institutions.

Official Copy of Architecture of Open Source Applications Volume 1

This book is free from its official site. There is no official PDF, so the legitimate free way to read it is the online edition. Click the button below to open aosabook.org, then choose Volume 1 to read each chapter. Print and e-book editions are sold separately, and all royalties go to Amnesty International.

→ Visit Official Page

FAQs

Is The Architecture of Open Source Applications Volume 1 free?

Yes. The editors publish every chapter free to read on aosabook.org, and the work is made available under the Creative Commons Attribution 3.0 Unported licence (CC BY 3.0). That licence lets you share and adapt the text, even commercially, as long as you credit the authors. All royalties from sales of the printed book are donated to Amnesty International.

Is there a PDF of this book?

The official site does not offer a PDF of Volume 1. The legitimate free way to read it is the online edition, chapter by chapter. A paperback and e-book editions (ePub and Kindle) are sold separately. This page links only to the official site and does not host any file.

Which edition is this, and is it still current in 2026?

This is Volume 1, published in 2011. It describes each project as it was then, so version numbers, tools and some details have changed. The architecture ideas and trade-offs are still worth studying. The most dated chapter is Python Packaging, because Distutils, which it describes, has since been removed from Python’s standard library.

How many chapters does it have, and are there exercises?

Volume 1 has an introduction by Amy Brown and Greg Wilson and 25 chapters, each about a different project, from Asterisk to Battle for Wesnoth. The chapters were written by 43 developers of those projects. The book has no exercise sets, so the Important Questions on this page are built from each chapter’s own content.

Is it good for complete beginners?

Not really. It assumes you can read code and know basic object-oriented ideas, and it does not teach C, Java, Python or Erlang. Start with the smaller chapters such as Violet (Chapter 22) and Battle for Wesnoth (Chapter 25), then move to the harder ones like Berkeley DB, LLVM and HDFS.

Are there other volumes in the series?

Yes. The same site publishes AOSA Volume 2 (24 chapters), The Performance of Open Source Applications (12 chapters) and 500 Lines or Less (22 chapters) as separate books. This page covers Volume 1 only.

Related Books

The Architecture of Open Source Applications, Volume 1 is a free way to see how real systems, from Bash and LLVM to HDFS and Eclipse, were designed, and what their authors would change. It is not a design patterns catalogue and it has no exercises, but its lessons-learned sections are ideal for architecture discussions. Browse more Computer Science books for the rest of your semester.

The Architecture of Open Source Applications, Volume 1, edited by Amy Brown and Greg Wilson, with chapters by the project authors named on aosabook.org. Licensed under the Creative Commons Attribution 3.0 Unported License (CC BY 3.0). Official source: https://aosabook.org/. This page links to the official site and does not host or copy the book.

Leave a Comment