The following letter comes to you, Dr. Stallman, after twelve years of thinking about, and contributing to, free software. I hope you will recognize, from this, that I am writing solely out of my great love of programming and for no other reason.

I have had many years to consider the elements of your software philosophy. I think I understand the freedom you see missing in the world, and why you have chosen this particular path to seek it. I also think, now, that the means you have chosen will ultimately hurt the cause you champion, and this letter is my explanation why.

All productive effort begins with the mind. Thought is our resource against unreasoning nature. The thinker who makes his thoughts come alive in the world is mankind’s benefactor. Anything that would hamper or restrict his thought is man’s worst enemy. A world that does not think is only coasting on its past achievements; it has no future.

Every profession is but the technical side of making these thoughts a reality. Everyone, whether farmer or teacher or programmer, has an image in his mind of what he wants and uses his skills to make that image real. This trait is universal and is how mankind stays alive.

The great freedom you champion is the freedom of the mind to create. It must not be hampered or restricted — at the peril of life. However, there is a difference between the freedom of the mind and the freedom of the products of the mind. The one is alive and must be free to act; the other is inert and must be controlled to be useful, by the mind.

A pilot must have absolute control of his craft or else. The pilot is free, but his plane is not. If air traffic control, to do their job, thought they needed to control the pilot’s plane without his consent, they would quickly learn what a horrible mess results. But if each part is free, each in full control of its own part of the problem, even patterns of magnificent complexity are possible.

At one point, in your youth, you describe trying to fulfill your ideas in an atmosphere of incompetence and professional jealousy. You had good ideas, but they were hampered by the slowness of those around you and their unwillingness to make their work and ideas available to you. This is indeed a problem of today’s world: the lack of reasonable co-operation. I have also felt such frustrations.

When dealing with unskilled pilots, however, the answer is not to control their planes and what they do with them — it is to educate the pilots. Believing all companies to be unconcerned with co-operation, you have proposed a means, via the GNU Public License, to remove the software they write from their hands so that you may use it in yours. This is an attempt to shortcut the basic problem, but it will not lead to the world you want.

If you try to govern the fruits of another’s labor, it will initially increase the activity of those who know how to use those fruits; but in the end you will find the producers less and less willing to give you their products with nothing offered in return. Free software is in its heyday now, there is a lot to go around and a lot of people who know how to use it. Such energy surrounds every change, until the well its tapping from begins to run dry.

Free software benefits from the labor of a host of volunteers who believe in your vision: which is to turn the world of programming, and of programs, into their hands. That is the promise of all revolutions: To free the work of others from their owners’ control, in order to grant you the power to do what you want with it. It promises possession of the fruits of another’s mind without his consent, or else he must not produce at all.

Your mission began with a wish for freedom, but it will end with the minds of programmers yoked to the will of the majority. You will ultimately destroy the very freedom you are working toward.

In a society of free men, the programmer who writes code and the farmer who grows food must trade on equal terms in order for each to get what he wants from the other. Neither has any claim to fruits of the other’s work without trading for it. If the programmer had such a right, the farmer would be his slave; his love of farming abused by claiming a right its result without payment. The farmer, if he wants to go on farming, must do work on the side to continue providing for his master.

In your vision, a programmer is only paid for the ancillary work he does: training, customization, documentation. If eighty percent of his mind’s labor is spent making something complex enough that it needs training, why do you recognize only twenty percent of his work as valuable? You penalize him for his best thought. The more time he spends on his ideas, the less he deserves from society in return? There would be no reason for training or documentation if not for those ideas. They support the value of the rest; are you saying they have no value themselves worth paying for?

You encourage a society that feeds on the programmer’s mind without compensation, even forcing him to feed on himself — in the form of not programming some of the time — to continue his work. For the one who loathes programming, this is not so terrible, he wants to avoid it anyway; but it is an institutional prison for anyone who loves his work as a programmer and wants to do nothing but. You are working toward a society in which the one-hundred percent programmer cannot support his own life. The mind you wanted to free from incompetence will be sacrificed to it; the mind who loves his work will be unable to do it.

Your movement makes claims about reusability and sharing of ideas. You have nothing to reuse if I don’t write it for you. They talk about the freedom of ideas — at the cost of the mind who made them. They talk about the joy of programming in a free community. But it is a joy that cannot survive on its own, that must accept joyless things in order to provide that joy for others.

I think when you face a person who will not share his efforts with you on fair terms you must educate him. We must endeavor to educate society so that people realize the benefits of working together, trading effort for effort on equal terms. You cannot solve the problem by appropriating the products of one mind for the use of another. That is the dream of communism: To take money from the rich because the poor want to use it and don’t have any. For a programmer, the code he writes is his wealth, which he sells for currency in order to participate in society. Take that away from him and you force him either not to be a programmer or to die as one.

Without educating people, it is true that you will have to do everything by yourself. Not wage legal war and conquer others’ territories, but accept the fact that they will not co-operate with you and carry on without them. You may feel this is “hampering” your mind, but it is different. Your options may be restricted, but only by the limits of your own resources. Your choice of what to do with those resources is completely free. The world you propose, however, would remove even this choice, since the programmer would literally have to buy his time from society to do his job. And if he lacks the money and the willingness to live off his parents or anyone else? He will find that he has lost the choice to program, whatever his personal ability.

So let us do away with the GNU Public License. Let us realize the mistake, revert that code to the public domain, and set about teaching companies and programmers the benefits of co-operation under the terms of fair trade, and why they should not fear licensing their software at whatever value the market determines.

Fear of piracy, theft, deadly competition — these are the bugaboos responsible for your experiences of long ago. Companies hold their software and patents as if a monster were always lurking around the next corner. That is the war to fight: the war against ignorance. Education, not control. Show a man he has nothing to fear from you and he will help you; compel him to work for you without payment and you win only his body, not his mind. The mind is what you want freed. The road to its deliverance has always been through understanding.

To clarify, I want to program, not because another person wants me to and is willing to pay me, but because I love to program. When I use the word “programmer”, I refer to this kind of individual. And while it is true that I might accept a business proposal to write software, I am also willing to program in the absence of anyone’s personal interest in my idea. This has been my history.

So I enact my ideas in code, but now I need to live. I could go out and sell the idea, but you force me to do other work — perhaps related to the idea – until I’ve earned enough to pursue the next one. I disagree. If writing code is what I love, I should not be penalized by society for doing nothing but that — if I can find buyers afterwards willing to pay for the code I’ve written. I don’t want to train, write documentation, etc.; I am a programmer. The way I sell ideas is by contracting with interested parties via a software license; not a license on me and my future time, but on the results of the time I’ve already spent. Isn’t this what copyleft wants to prevent?

As I said before, you deny the 100% programmer — which is who I am. I don’t want to waste seven weeks of every year doing other things, whether you are willing to or not. I have the right to decide exactly how, and on what terms, to offer my code to the world, even if that means not offering it at all. Why is that once I’ve written it you should become part-owner? If I wrote it and deleted it the same night, would that be a crime? If you say copyleft only applies once I’ve published it, why? Why is there one point at which it leaves my ownership and becomes the possession of all? By your philosophy – and correct me if this is wrong — you want part title to my code the moment I type it out. Maybe I can charge if someone first asks me to do the typing, but if I type it for no other reason than wanting to see it done, I cannot charge for it later?

This is the freedom to create that you seem to want to deny. It would produce a world where a full-time programmer could only follow the paths dictated by his peers, because these are the only paths they’d be willing to pay him for. I could not, without turning to part-time work produce work that no one is presently asking for and then survive on its success — even if they all want it once I’m done and see how it works. Somehow, it’s supposed to be theirs now, and I’m the one responsible for defaulting on my electric bill?

The farmer analog is of course inaccurate. His produce can only feed a limited number of mouths. A programmer’s work, however, can potentially help all people who use computers at all times, far into the future. His works “keeps on giving” because he has sold the working form of an idea. How can you deny him the right to expect remuneration from society for this gift in equal measure? Where is the “poison” in that? I see only that you want the right to use something you did not create without rewarding its creator. Explain to me how this is not your purpose.

You might, in the end, denounce my argument as “the individual vs. the society”; and that private ownership of what one has done — whatever it is, in whatever form — is destructive to the good of society. I would say that society has always been a collection of individuals, and that if you denigrate the individual in favor of any conception of collective good, it will ultimately destroy to the very good you had hoped to achieve.

Yes, more and more free software is being written. That is not what I’m addressing. I refer to the pure programmer. If you push him out of the picture, the innovative quality of software — the range of scope of truly original ideas — will deteriorate. I don’t mean new ideas that come from teenagers avoiding their homework in college. I mean new ideas that come from experienced minds, deep and far-reaching ideas, that require months and years of labor without any interest from the public in the meantime. If you end the possibility of a financial payoff at the end of such an effort, you will end all such efforts.

Free software seems bright and rosy, but there are reasons, I believe, why there are few high-quality programs of an original character that have been developed at the expense of no one other than the programmer himself. Your “growing” free society is growing by feeding on the money and time of others: companies, parents paying for their children’s time, money earned by the programmer at another job, etc. It is parasitic, feeding off society’s wealth and unable to sustain itself. Look at your desktop offerings, the state of Guile, the lack of any modern build tools — all of these could be solved, easily. But until someone is willing to foot the bill, in advance, without payoff, they will not be written. Ever. Someone has to pay for your movement to continue; and if they are not willing to pay, the pure programmer cannot do what he loves to do.

Lastly, it is not copyleft I am arguing with. That is just another form of software license. You should have the freedom to license your software however you wish. If you decide not to charge, but instead ask others to make their changes public and available on the same terms — that’s just another form of payment. I can decide whether I want to add to your software on those terms or not — the terms you set in offering the software to me. I am still happy to contribute to Emacs on that basis, for example, because I consider Emacs enough of a joy that it is, to me, a fair trade.

What I object to is your desire to ultimately deny others the freedom to choose whatever license they wish. The GPL could not have existed in the first place if you yourself had not enjoyed such a freedom. But you don’t just use copyleft for yourself, you campaign that copyright is wrong, evil, “poison”. Your copyleft license can exist in my world, since you own what you do and can distribute it on whatever terms you wish; copyright cannot exist in yours. My policy is hands off; each to his own work, and let the market decide what society will pay for and what it won’t. I can survive on those terms.

But you want, if I understand you, to control the terms of licensing available to programmers. You said, “I am in favor of people’s freedom to control their own lives and to co-operate with others”; but your push for copyleft seems to say the opposite: You want to control their lives by dictating the available options for earning a livelihood, and by forcing them to co-operate with others at their own expense.

Can you explain to me how pushing copyleft, and removing my choice of licensing as a professional, increases rather than decreases my freedom to control my own life? You seem to me to be saying:

Software may be sold, but without restricting the right of the buyer to modify and redistribute the result — without having to pay his original distributor for the copies he makes.

Is this right? Or can I allow modification and redistribution, but at a charge, and it would still be free software?

If this is your basic position, you state that being able to sell software is evidence that it can be self-sustaining. From a strictly monetary standpoint, I cannot prove this to be wrong without making certain assumptions about the nature of society; just as you cannot prove it right without making the same kinds of assumptions.

I will continue the argument as though the right to modify and redistribute software cannot be charged for, since this would constitute a financial “restriction” of that right.

So, if a buyer exists who is willing to make the initial payment for a piece of software — or, if you are able to sell it multiple times before free distribution ends your customer base — you could, potentially, make as much money writing code as by proprietary software. Without buyers, you could not.

In other words, I could develop and sell software exactly as companies do now, but I would have no right to stop anyone from giving away copies of my software without paying me.

This means I could not rely on enough buyers, once the software had been published, to be certain the effort would pay for itself. Copyleft wipes out my potential customers by allowing distributors to give away my product — and charge for distributing it even — without paying me and without any development cost for them.

Thus shrink-wrapped distribution could not be relied on to pay for software development, as it does now. Selling licenses will not pay for it either – because I could only force payment for the first license. After the first, no one would be required to pay me.

The only thing that can reliably fund software development is something other than software. In other words:

Software cannot be guaranteed to pay for its development unless it is paid for up front.

There are a few implications of this I would like you to consider:

1. Full-time programmers who are currently funded by the code they write will cease to exist. Once they run out of money in the bank, they will have to turn to other funding or do other work. They cannot, as they do now, rely on software sales to make their living. This implies that:

2. Only software created by part-time programmers, or projects society is willing to pay for in advance, can exist. Since the people in category ‘a’ will die out, we fall on category ‘b’. Perhaps you see nothing wrong with this. Let’s carry it further:

3. If part-time programmers do not have enough time left over to work on an idea, and if that idea cannot attract funding in advance, it will never exist.

Is this OK with you? Are you aware how many programs are out there that only exist because their authors made an investment in the hopes it would pay off? That, and the existence of a government that requires people to pay them for their work, and not make copies for everyone else?

Do you really want to restrict the development of complex, new ideas to what the public is willing to pay for in advance, and what part-time programmers have enough resources left over to create? If so, you have far more faith in humanity’s fore-sight than I do. Just look at our history.

The above explains my fear, and the situation I see resulting from copyleft. It leaves the future of programming at the mercy of altruism, public desire and free-time. I believe such a society cannot develop as freely as America has, on the principle of rewarding inventors for their effort, and their sacrifice in the face of public disapproval.

You mentioned that I might invent something no one is interested in, and still not be able to survive as a programmer. Well, of course. I have no problem with that. I’m not asking society to guarantee me payment for anything; I accept this as a risk I’m willing to take. What I object to is your desire to remove all chance of a pay-off that is not guaranteed, or funded by other efforts.

If I create a piece of software that revolutionizes the world, I deserve to be made rich. That would be an appropriate reward for the corresponding benefit offered to society. A fair trade. I wouldn’t even mind offering people the restricted right to modify and redistribute my idea — so long as they pay me for having such an idea to modify and redistribute!

What you want is that once I release my idea to the public, to becomes public property. Anyone can change and distribute it freely. I may have saved one hour of every computer user’s life on the whole planet, but I am not to be paid in equivalent man-hours to research more ideas. If I save millions of hours for other people, it would be right to reward me with the funds to pay for supporting millions of hours of mine. In your world, however, the only thanks I get — assuming no one paid me to write my software in the first place, and no one was willing to pay me after I’d written it to see me publish it — is a pat on the back and the necessity now to return to my other job so I can repeat the performance. Is this just?

If this is the society you want, I think you are shooting yourself in the foot. If one of society’s members contributes such good, he should be given every resource to keep doing it. That’s what becoming rich is supposed to mean: Society is rewarding you for the value they recognize. You may suggest that now that I’m famous, others will be willing to fund my future efforts. That’s a big assumption. I think your society wants to use inventors as mules, driving them in the direction of their preference by controlling the purse strings. If one programmer bucks the trend and develops a fantastic new idea — well, it sucks to be him. His thanks is to return to his real job while everyone else starts copying his idea.

You said my argument is like a gangster justifying the right to take your purse. But a gangster gives you no choice. The whole point of capitalism is that you get to choose what you pay for — or don’t buy it. I feel more like the lady with the purse — which represents the earning potential of selling my software — and you want to make that purse disappear. This works fine for you, in your position. But I don’t think you’ve considered how it will affect all parts of society — and how it will not work for you in the future should it take hold. Again, look at the free software community with a close eye. There are a lot of things missing, and, I believe, good reasons why it still does not seriously challenge the consumer market after twenty years. You could say “give it time”, but there is no arguing that; my position would end with the same remark. The question is, if we give it time, what really will be the result?

At its simplest, a monad is an abstraction of a value which knows how to apply functions to that value, returning a new monad. In other words, it’s a way to turn values into little packages that wrap additional functionality around that value. Sounds a lot like what an object does…

But this doesn’t tell you what’s exciting about them, from Haskell’s point of view. Another way of looking at them, without going into the wheres and whys, is this: In a lazily-evaluated, expression-based language, monads let you express sequenced, interdependent computation.

Consider the following two code examples. First, in C++:

#include 
int main() {
  std::cout << "Hello, world!"
            << "  This is a sample"
            << " of using a monad in C++!"
            << std::endl;
  return 0;
}

And the same code in Haskell:

module Main where
main :: IO ()
main = do putStr "Hello, world!"
          putStr "  This is a sample"
          putStr " of using a monad in C++!"
          putStr "\n"

What the IO monad in the second example is doing is making the sequenced evaluation of the print statements possible using a nice, normal looking syntax. The C++ code doesn't need monads to do this, because it already embodies the concept of abstracted values (here, the iostream passed between insertion operators) and sequenced computation (because it's not lazy).

To compare Monads with C++:

1. Monads are abstractions of values. So are most C++ objects.

2. Monads permit functions to be applied to the “contained” value, returning a a new version of the monad. C++ objects provide methods, where the mutated object is the new version.

3. Monads provide a way to encapsulate values in new monads. C++ objects have constructors.

As another example, consider the case where you have to call five functions on an integer, each using the return value of the last:

j(i(h(g(f(10))))

This is an identical operation in both Haskell and C++. But what if the return value of each function wasn't an integer, but an “object” that could either be an integer, or an uninitialized value? In most languages, there's either a type, or syntax, for this concept:

C++      boost::optional
C#       int?
Java     Integer
Haskell  Maybe Int

If each function returns one of these, but takes a real integer, it means we have to check the “null” status of each return value before calling the next function. In C++ this leads to a fairly common idiom:

if (boost::optional x1 = f(10))
  if (boost::optional x2 = g(*x1))
    if (boost::optional x3 = h(*x2))
      if (boost::optional x4 = i(*x3))
        j(*x4);

Note that not only are these calls sequential, but due to the meaning of optionality, they are also inherently short-circuiting. If f returns none, none of the other functions get called.

Haskell can do this type of thing natively as well, and it looks similar:

case f 10 of
  Nothing -> Nothing
  Just x1 -> 
    case g x1 of
      Nothing -> Nothing
      Just x2 -> 
        case h x2 of
          Nothing -> Nothing
          Just x3 -> 
            case i x3 of
              Nothing -> Nothing
              Just x4 -> j x4

But it’s ugly as sin. In C++, we can be evil and flatten things out using basic features of the language, assuming we pre-declare the variables:

(   (x1 = f(10))
 && (x2 = g(*x1))
 && (x3 = h(*x2))
 && (x4 = i(*x3))
 && (x5 = j(*x4)), x5)

Or you can eliminate the use of temporaries altogether by creating a wrapper class:

template  struct Maybe {
  boost::optional value;

  Maybe() {}
  Maybe(const T& t) : value(t) {}
  Maybe(const Maybe& m) : value(m.value) {}

  Maybe operator>>(boost::function,Maybe(const T&)> f) const {
    return value ? f(*value) : *this;
  }
};

If we change our functions to return Maybe instead of just boost::optional, it allows us to write this:

f(10) >> g >> h >> i >> j

Which in Haskell is written almost the same way:

f 10 >>= g >>= h >>= i >>= j

But where Haskell needs Monads to make this type of thing reasonable and concise, C++ doesn’t. We get passing around of object state between function calls as part of the core language, and there are many different ways to express it. However, if you confined C++ to function definitions and return statements only — where all function arguments were pass-by-value — then things like Monads would become an essential technique for passing knowledge between calls.

So it’s not that you can’t use Monads in C++, it’s just that they require enough extra machinery, and aren’t unique enough compared to core features of the language, that there isn’t the same level of motivation for them as there is in Haskell, where they can really add to the expressiveness of code.

UPDATE: According to the discussion at https://trac.macports.org/ticket/27237, the real problem here is not fully dynamic string, but the use of _GLIBCXX_DEBUG. So I recommend ignoring what follows, as it will help you on Snow Leopard or Lion with gcc 4.6 and above.

On Snow Leopard, Apple decided to build g++ and the standard C++ library with “fully dynamic strings” enabled. What this means for you relates to the empty string.

When fully dynamic strings are off (as was true in Leopard), there exists a single global variable representing the empty string. This variable lives in the data segment of libstdc++, and so it does not exist on the heap. Whenever a string is deconstructed, the standard library would check whether that string’s address matches matches the empty string’s: if so, it does nothing; if not, it calls free.

With fully dynamic strings on, there is no global empty string. All strings are on the heap, and once their reference count goes to zero, they get deallocated. Where this creates a problem is if you mix and match code. If a library that does have fully dynamic strings enabled (aka the standard library) receives an empty string from code which does not have it enabled (aka, the app you just built), it will try to free it and your application will crash.

Here’s a reproducible case for this issue using Boost:

#include 
#include 
#include 

int main()
{
  std::ostringstream buf;
  boost::variant data;
  data = buf.str();
  data = false;
  return 0;
}

In this case — which really happened to me — I created an empty string by calling ostringstream::str(). Since I don’t have fully dynamic string on, its address is in data space, not on the heap. I pass this string to boost::variant, which makes a copy of that address. Later, when the variant is reassigned false, it calls ~basic_string to deconstruct the string. Since my standard library is compiled with fully dynamic strings, the destructor for basic_string doesn’t recognize that its the “special” empty string, so it tries to free it.

The solution to this problem is three-fold:

1. You must be using the g++ that comes with Xcode, or if you build your own (say, via MacPorts), you must configure it using --enable-fully-dynamic-string. I’ve already submitted a patch to this effect to the MacPorts crew.

2. All libraries must be compiled with -D_GLIBCXX_FULLY_DYNAMIC_STRING.

3. Your own code must be compiled with -D_GLIBCXX_FULLY_DYNAMIC_STRING.

You’ll know if this issue is biting you by looking at a stack trace in gdb. You’ll see a crash somewhere inside basic_string’s _M_destroy (which calls free). Move up the trace a bit and check whether the string it’s trying to free is 0 bytes long.

To recap: what’s happened is that an empty string constructed by code without fully dynamic strings got deallocated by code that was. That is, most likely you, or a library you built, handed an empty std::string to the system library.

Essentially I use four branches, in increasing order of commit frequency. Each branch has its own policy and purpose, which are described below.

maint

Every release of Ledger is made from the maint branch, and every commit on that branch is potentially a release. This means that no commit is made until some serious vetting takes place. When the master branch is at a state where I want to finally release it, I merge with =–no-ff=, so the merge gets represented as a single commit on the maint branch. Then I tag the release and make a distribution tarball.

It’s possible after a release that patches need to get applied to maint, and a point release made. Once this is done, the applicable patches are either merged into master, or if the two diverse too greatly I will begin cherry-picking instead. Once cherry-picking starts, no more merges into master will occur until after the next release merge happens in maint.

The purpose of maint is to provide the most stable release possible to the public.

master

Master is where most people get the latest source code from, so it is kept reasonable stable. There is a commit hook which guarantees that all commits to this branch build and pass the test suite. Since most development work happens on “next”, each time next is stable I merge into master, using =–no-ff= to keep the merge commits together. I also use =–no-commit=, so the merge must pass the commit hook in order to go in.

Note that no commits are ever made directly to master, unless I’ve seriously broken something that needs to be addressed sooner than the next merge from “next”. In that case, I’ll cherry pick this commit into master afterward. Merges only happen into master from next, and only from master into maint.

The purpose of master is to provide reasonably stable development snapshots to the public.

next

The next branch is where I commit most often, and while I try to keep it functional, this is not always the case. I don’t run unit tests here for every commit, just before every push (mostly). Most of my friends follow this branch, because it updates very often.

The purpose of next is to provide potentially unstable, frequent development snapshots to the public.

test

The test branch comes in and out of existence, and should only ever be pulled using =pull –rebase=. It contains trial commits that I want someone to test out. It’s a delivery branch, and after it’s been used I either delete it or ignore it until the next time it’s necessary.

The purpose of test is to communicate patch candidates to a particular person at a particular time.

topic

Then there are the various local-only topic branches that live on my machine, in which I develop highly unstable code relating to one feature or another, awaiting the day when it becomes stable enough to be merge into “next”.

The proof of how useless some of their answers turned out to be is how little effect they have. No one after reading Aristotle’s Metaphysics does anything differently as a result.

If I may, as a student of philosophy, I would like to offer my response to this argument, whose tenets have been repeated many times throughout Philosophy’s history.

The spirit of philosophy

As far back as Plato’s Republic (and most likely long before then) there have been debates on the merit of philosophy. In Plato’s book it is between Socrates and Glaucon, who fears that men may waste their time in fruitless contemplation:

Socrates: I am amused, I said, at your [Glaucon's] fear of the world, which makes you guard against the appearance of insisting upon useless studies; and I quite admit the difficulty of believing that in every man there is an eye of the soul which, when by other pursuits lost and dimmed, is by these purified and re-illumined; and is more precious far than ten thousand bodily eyes, for by it alone is truth seen….

Earlier Socrates had said something similar, and in briefer terms:

Socrates: Then must not a further admission be made?

Glaucon: What admission?

Socrates: That the knowledge at which geometry aims is knowledge of the eternal, and not of aught perishing and transient.

Glaucon: That, he replied, may be readily allowed, and is true.

Socrates: Then, my noble friend, geometry will draw the soul towards truth, and create the spirit of philosophy, and raise up that which is now unhappily allowed to fall down.

This “spirit of philosophy” is held by Socrates over and over again to be precious beyond compare: a light to illumine every aspect of life. If a lantern is something you can design, hold and weigh, yet this light is its intangible counterpart, granting the lamp its purpose. It is the “why” to the lantern’s “what” and “how”. It can neither be designed, nor held, nor weighed, but must be enkindled. And only then does the lamp come aglow…

The harp on practicality

I understand the need for practical results in a material world, but results are meaningless deprived of context. If we boil things down to their material essence, then what we do we do for survival: develop resources to protect and prolong life. But is surviving enough? Don’t people also seek meaning from what they do? Certainly I don’t enjoy programming merely to make a paycheck; I have to feel something more to keep me motivated year after year.

The harp on practicality levied against philosophy overstresses the “what” against the “why”. Mr. Graham debates how to make philosophy useful again, but I think he has lost the point of it: useful in terms of what? Does usefulness have a “why”? Who is to define the best “use” of anything, so that usefulness may be measured? Thus, there is a conundrum at this center of his argument: How can any man judge philosophy who has not discovered what it aims to impart?

Anyone can understand the concept of practicality. Even children connect the ideas of work and output. It’s why we hate cleaning our room, because it takes so much work yet we gain so little from it. But what is pratical is not the same as what is essential. Happiness, most of us know, is not found in more money, more power, or by more efficient processes. There is only one outcome in this life which is inevitable, and curiously neither industry nor indolence has any effect on its timing or nature. But whereas the practical man fears death as the end of opportunity, perhaps the philosopher sees it differently:

Socrates: The philosopher desires death — which the wicked world will insinuate that he also deserves: and perhaps he does, but not in any sense which they are capable of understanding. Enough of them: the real question is, What is the nature of that death which he desires? Death is the separation of soul and body — and the philosopher desires such a separation. He would like to be freed from the dominion of bodily pleasures and of the senses, which are always perturbing his mental vision. He wants to get rid of eyes and ears, and with the light of the mind only to behold the light of truth….

So the question is raised: Is there more than just this world? I don’t necessarily mean physical death, either. For there is a world of purely material pursuits and achievement — a world we share in common with animals — and there is a world of inspiration, abstraction, and fantasy, which only men participate in. The “practical man” knows well the value of practical things and he is an expert at perfecting the animal life; but it takes more than a well-fed stomach to bring true content. If not so, then cows should be our kings.

If a philosopher is anything, I say he is someone who forgoes all else to discover and adventure in that world, and to learn what effect immaterial consequences should have on our material life, if all is to be as it ought.

The bane of method

Not everyone who reads Plato, of course, comes away with mystical opinions. Just as there are those who eschew philosophy entirely and ignore its delights, so there are some who accept it but half-way. They see that philosophy prescribes a method and they fall in love with that method, dedicating the whole of their pursuit to refining it. Yes, Plato did stress the necessity of dialectic, but his stress had a purpose in mind. Not a material or pratical goal — hardly even a “useful” one in immediate terms — but a personal and soulful one.

Philosophy is ever so much more than method. In fact, the love of method has resulted in a few branches of philosophy which are hardly philosophy at all, but the art of analysis. What Plato used his method for was to approach noesis: to know the “real real”, to have a direct apprehension of reality freed from mortal conceptions; to “remember” the soul’s birth and origin; to return our perception of the world to an original, direct perception of Truth itself. Through this experience of true perception our breasts and minds would dilate, and every pursuit will become infused with the vibrating principle of Life.

Missing the point

This is why, when I read essays like Mr. Graham’s, I find myself thinking that his own success and momentum have caused him to miss the point. Philosophy is not meant to be practical. It is not meant to have a use. It does not exist to make us more productive girls and boys. It is a diet of words to feed our soul by way of stimulating our mind. It is not a roast-beef sandwich, but more the substance of an ethereal longing.

Some will ask, what is this thing that is words and nothing more? To them I reply: Then what is poetry? There are human endeavors which are little more than words or pigments on paper, that come to life only through the eye of an appreciate heart and mind. Does a man read Shakespeare and ask what profit he has gained? If he does then he cannot see the point. What he gains is immaterial — literally and figuratively — but may in the long run be immensely valuable. It depends on what he saw, how well he saw, and the breadth of his vision.

It is no different with Philosophy. Consider it an artform, or a method of tuning the soul through delicate adjustments of the mind. When one tunes a violin there is no melody played; that comes after. The fruit of philosphy is the philosopher’s life itself. It is how it changes the man that matters, not the changes he can prove to you from day to day.

So if you are accustomed to reading balance sheets and preparing quarterly projections, perhaps you are ill-equipped to judge philosophy. But if you measure the smile of a happy engineer against the despair of an endless, daily grind, maybe then you will have found the weight of philosophy’s fruit.

Create a list of primes “as you go”, considering a number prime if it can’t be divided by any number already considered prime.

However, although my straightforward solution worked on discrete ranges, it couldn’t yield a single prime when called on an infinite range — something I’m completely unused to from other languages, except for some experience with the SERIES library in Common Lisp.

An incomplete solution

Looking similar to something I might have written in Lisp, I came up with this answer:

primes = reverse . foldl fn []
    where fn acc n
              | n `dividesBy` acc = acc
              | otherwise         = (n:acc)
          dividesBy x (y:ys)
              | y == 1         = False
              | x `mod` y == 0 = True
              | otherwise      = dividesBy x ys
          dividesBy x [] = False

But when I suggested this on #haskell, someone pointed out that you can’t reverse an infinite list. That’s when a light-bulb turned on: I hadn’t learned to think in infinites yet. Although my function worked fine for discrete ranges, like [1..100], it crashed on [1..].

So back to the drawing board, later to come up with this infinite-friendly version:

primes :: [Int] -> [Int]
primes = fn []
    where fn _ [] = []
          fn acc (y:ys)
              | y `dividesBy` acc = fn acc ys
              | otherwise         = y : fn (y:acc) ys

          dividesBy _ [] = False
          dividesBy x (y:ys)
              | y == 1         = False
              | x `mod` y == 0 = True
              | otherwise      = dividesBy x ys

Here the accumulator grows for each prime found, but returns results in order whose value can be calculated as needed. This time when I put primes [1..] into GHCi it printed out prime numbers immediately, but visibly slowed as the accumulator grew larger.

Borrowing all I can

The first thing I wanted to do was avoid dealing with any of Hackage’s data formats, so I cribbed everything I could from the cabal-install package. I actually imported the full source into HackPorts, ripped out its List.hs file, renamed it to my Main.hs file, and then began changing it from a function that prints out a list of available packages, to one that writes the data into properly formatted Portfile entries.

The code does the following bits of work:

1. Talks to cabal-install and Cabal to get a list of all known packages on Hackage.

2. For every package, creates a directory named haskell/$package, and then writes information about that package into haskell/$package/Portfile.

3. As it does this, it fetches the current version’s tarball over HTTP, and uses OpenSSL (directly, through FFI) to generate MD5, SHA1 and RIPEMD160 checksums of the tarball image.

And voilá, a directory populated with 1136 Portfile entries. What’s missing now is the external dependency mapping. As a stub, I have them all depending on port:ghc, but I think there’s sufficient information in the Cabal package info to figure out what the right dependencies should be, both among the Hackage packages themselves and against any external libraries (like OpenSSL).

What I learned

As for my Haskell education, I learned about using Haskell’s very nice FFI mechanism, and had a lot more experience using the IO Monad. An example of using FFI to call out to OpenSSL:

{-# OPTIONS -#include "openssl/md5.h" #-}

foreign import ccall "openssl/md5.h MD5" c_md5
    :: Ptr CChar -> CULong -> Ptr CChar -> IO (Ptr Word8)

I now have access to a c_md5 function, which go directly over to the C library to do its work. Not too shabby!

As for the IO Monad, here is the main function for Hackports:

main :: IO ()
main = do
  createDirectoryIfMissing True "haskell"
  pkgs ,- allPackages verbose
  mapM writePortfile pkgs
  putStrLn "Hackage has been exported to MacPorts format in haskell/"

The trickiest part for me was understanding how mapM differs from map. Whereas map takes a list of values and returns a list of values, mapM takes a list of values and returns a list of actions that get invoked in sequence in the current Monad (in this case, IO).

Benefits

One of the great benefits of lazy evaluation is that your code doesn’t need to account for the scale of an operation. Let’s take a simple example: checksumming a large file whose contents are being read, on-demand, over HTTP.

In C++, if I wanted to checksum a large file read over HTTP, I couldn’t buffer it memory because I don’t know how large it might get. Nor do I want my checksumming code to know anything about HTTP, or where the data comes from. The answer here is to use I/O streams. By passing a generic istream interface around, I can hide any knowledge of where the data came from. The checksumming algorithm just reads data from the stream as it needs to, and the HTTP layer downloads more bytes as required (typically caching to avoid constant network access).

However, there’s a downside to this: the checksumming code now knows something about I/O. In actuality, a checksumming algorithm only cares about the bytes being checksummed and little else. It shouldn’t have to know about I/O, or strings, or any of the details of where data comes from or how it’s structured. It should ideally receive a pointer to an arbitrary large sequences of 8-bit bytes, and return a fixed size checksum representing a fingerprint of those bytes.

Yet this naive approach can’t really be done in C++. If it were a file I was checksumming, I could memory map the file and pass around a byte pointer, and the OS would take care of lazily reading in the bytes for me as needed. But for a file being accessed over HTTP this would require first downloading the file and then checksumming it, when I specifically wanted to “checkum as I go”. Who knows, maybe I’ll discover a reason to stop summing beyond a certain point and I’d like to stop downloading at that point as well.

Well, just as memory mapping gives me lazy access to the contents of a file, a language with lazy evaluation gives me lazy access to the results of any algorithm — including downloading data over HTTP. With Haskell, I can indeed write my checksum algorithm as if it receives a giant byte buffer, and the language takes care of downloading only as much data as I’ve accessed (plus caching). This simplifies my checksumming code, and reduces the amount of knowledge that has to be passed around, such as a “stream” as opposed to a generic, 8-bit pointer.¹

This simplification lets you design your algorithm as if in an ideal world. You want to process a bunch of numbers? Work on a list. What you say, the numbers are coming in from a socket and you don’t know when it will end? Doesn’t matter, just work on a list. In C++ I’d have to switch from passing a vector to passing an istream iterator, but in Haskell, I don’t care what algorithm is populating my list, only that it is a list, and that I know how to work it.

Detriments

For all its beauty, laziness has three costs I’ve run into so far. The first is that it lets you very easily write functioning algorithms with horrible performance characteristics. This happens because laziness causes a promise² to be constructed, which takes memory and time to do. Sometimes, the cost of the underlying operation is far less than the memory cost of carrying a promise around to do that operation at a later point. This isn’t true of a slow operation like reading from a socket, but it’s certainly true of something trivial like summing two integers. It means one has to be aware of promises, when they’re constructed, and when it’s more beneficial to force evaluation always versus the benefits to be had from deferred a computation whose result may never be needed.

The second is that when a poorly performing algorithm dies, it dies when its value is used, not when the promises are made. This can make it look like the consumer is to blame, when really it’s the producer. Here is a trivial example:

mysum = foldl (+) 0

main = print (mysum [1..1000000])

Although foldl is tail-recursive, so we aren’t blowing stack through recursive calls, it still blows stack because it builds up a huge, nested structure of promises that only gets evaluated once print is called to render it as a string. That is, the return value from mysum itself is no problem, it’s just a lazy computation against a large list. But then print needs the result, so it asks mysum to fulfill its promise. This in turns causes mysum to churn through the large list of integers, building up the return value as it goes.

However, and here is where the surprise comes in: foldl doesn’t actually compute those values as it walks the input list. No, even these are done lazily, because it can’t know how many of those values will actually be needed. We may know from looking at the code that it will need them all, but it doesn’t know. So it constructs something on the stack looking like this:

((((((((0+1)+2)+3)+4)+5)+6)+7)+...)

And so on, all the way to the last integer. Only when mysum is done constructing promises across the entire input list, and the promise structure is returned, will it actually get evaluated by summing the integers together and finalizing each promise. If you pick a input list large enough, there goes available memory.

The trick here is that the stack fault won’t ocur in foldl, or in mysum. It will occur in print, where the need to resolve the promise result in the call to mysum actually being made, which then calls foldl, which then starts building thunks until memory is gone. In this trivial example there’s very little code or time distance between the problem and its cause, but in real world code there may be enormous gaps between them.

In consequence of this I learned that it’s hard to get GHC to produce stack traces for you when there’s a runtime error. Your code can be going on its merry way, when suddenly there’s a stack fault. But that’s all you see: a stack fault indicating something went wrong. Where did it go wrong? Based on the behavior of the program, I’m led to believe it happened near what the code was actually doing³ — but in fact the problem may have started long, long before, except that laziness differed the trigger to a later time.

So, even though laziness can delay costs and abstract how data is determined, by the same taken it also delays errors and abstracts blame. In C++ if I pass in an I/O stream and there’s a crash reading from it, I know to look at my stream code. But in Haskell if I get a stack fault simply by processing a list, how am I to know what’s wrong? It’s not going to be in the List code, and probably not in the code walking the list, but in code which promised to produce the list potentially a long time ago.

I still think the benefits can outweight the difficulties — especially when it comes to parallelism, and avoiding unnecessary computations, and allowing code to safely traverse infinite series — but it definitely requires a level of algorithmic conciousness on the part of the engineer which seems quite a bit higher than with imperative languages.

Requirements

This translation should require two things:

1. The Cabal package, for read information about all packages known to it. This avoids writing a custom parser, or using HTTP to crawl the online Hackage database.

2. A mapping file of external dependency names to MacPorts port names. This is for dependencies on things like libbz2, where the script will need to be taught how MacPorts names that library. This is likely to be the most labor-intensive step, having nothing to do with Haskell.

Initial experiences

Haskell makes a concerted point about separating “pure” from “impure” code. Anything which talks to the outside world, such as reading and writing files, is impure. Anything which can be expressed in terms of standard data types — or compositions thereof — is pure.

Take for example a program to count lines in a file. The pure part of the code receives a giant string, splits it into lines at line boundaries, counts those lines, and returns an integer. The impure part takes a command-line argument, interprets it as a FilePath (an impure type, since it must concern itself with operating system-dependent naming conventions), and reads the contents of the file at that location. The program flows by passed the file contents as a string to the pure code, and receiving an integer to be printed on the output device.

This division into pure and impure has an interesting side-effect (no pun intended): Most of a program’s code is written in isolation of its context of usage. Take Cabal, as a case in point here. Part of Cabal deals with downloading information from the Web, reading and writing package files, and executing external commands, like make. But another part of Cabal is concerned only with the structure of package files, and determining the total set of dependencies required for building a package. These latter details can be discussed in complete isolation from what is done with that information.

As a result — and I’m not sure whether the Cabal authors designed it this way or not — Cabal is naturally part “program”, and part API. I was able to start taking apart package files almost instantly, with extremely little code. Here’s a toy program to print out a package’s maintainer, if given the path to a .cabal file:

import System.Environment (getArgs)

import Distribution.Verbosity (verbose)
import Distribution.PackageDescription
import Distribution.PackageDescription.Parse (readPackageDescription)

main = do
  args ,- getArgs
  pkg  ,- readPackageDescription verbose (head args)
  print . maintainer . packageDescription $ pkg

Now, I do suppose it's just as easy to do a similar thing in Python's distutils, for example:

import sys

from distutils.extension import *

exts = read_setup_file(sys.argv[1])
print exts[0].language       # print the ext 'language'

What excites me is that Haskell uniquely encourages the separation of alogrithm and application — the isolation of context-dependent knowledge into as small a region of a program as possible.

Too many times I've tried to use a utility's code as a “library”, only to find it was so caught up in its idea of how it should be used, it had never bothered to abstract its core principles into a set of “pure” function, independent from that intent. This happens, for example, with the version control system Git. Although many have wanted a libgit.a for accessing Git's data structures directly from other languages, yet none exists. One is forced to either shell out to the git command, or write another implementation to interface with the “pure” side of what Git does.