Part of what Ableson and Sussman keep saying in these lovely lecture videos is that a good way to solve a problem is to divide it into layers of abstraction, where one layer of abstraction gets its work done by using a "language" that you wrote in the abstraction layer immediately below it. They say that a good way to solve a problem is to write something powerful enough to solve the general KINDS of problems that you're trying to solve, because you're probably going to need something that powerful later anyways. In these videos they tend to say in a very clear, direct, rigorous way things that I've heard said many times in other places by other people.
In particular, in the videos they stress trying to keep your "custom languages" written in such a way that they can be used recursively. My grand PHP project that I've been working on for some time at work is disturbingly similar to some of the examples they use in those lectures; namely, my project seems to be a good example of how NOT to do things.
My project is a set of scripts to build forms to conduct an interview. It's basically an enormous HTML form, but the form is split into pages, and the pages contain questions, which are groupings of HTML controls and text labels and Javscript and other things. To borrow language from SICP: questions are really my primitives. In addition to questions there are "groups" of questions that behave like single questions in certain circumstances but not others. And some questions have "sub-questions" that depends on them in hard-to-handle ways. So in SICP terminology: Questions can be combined by forming groups, or forming question/sub-question pairs, or otherwise just by listing them sequentially. The groupings can be into rows, or into columns, or into both, or into other kinds of patterns.
The program works by having someone write the survey in YAML, which is a very easy plaintext-ish way to write config files. Then my scripts parse the YAML, build my question objects and render them. The end. What I have though is essentially a custom language, with YAML as its syntax, used for building arbitrary surveys. It works fine and I was able to write our 20-page interview in this survey-language very easily. Ableson and Sussman would hopefully be proud (if highly disgusted by the details of my implementation in PHP).
When I was designing these scripts I do remember having the thought of whether I should make the questions in my language behave fully recursively. In other words I could simply make EVERY question have the ability to store references to one or more sub-questions. Then there would be only one kind of question, which may or may not have sub-questions. When I render a question, I'd just have it render itself and then render all its children recursively.
I gave that a try but gave up. In practice (so I thought at the time), questions should never really need to nest more than one or two levels deep, and the questions should only be combined into groups in a very limited couple of ways. So really, a fully recursive language wasn't necessary. So I wrote my language such that grouping questions together was a special case handled with extra code. Likewise sub-questions for normal questions were a special case handled with even more extra code. This worked OK.
If only I'd have seen these videos a month ago. This week I came upon a situation where the survey-building language I came up with isn't powerful enough to handle a certain type of grouping construct which I need. My language isn't arbitrary enough to let me do what I need to do. To make matter worse, looking at my code now, I realized that all the special-case code required to handle all the things I thought wouldn't be a problem has accumulated far beyond what I'd have ever expected.
Live and learn, I suppose. I think I was wandering down the right path, even if I fell off into the ditch before I reached the end of it.
Some other guy built a similar survey system for a different group at work a while back, and I had a chance to look at his code. That code consisted of one long listing of intermingled PHP, SQL and HTML, hard-coded throughtout to work specifically for the survey the group was conducting. I wanted to use those scripts myself but they were essentially unusable because they were so non-generic. I at least did better than that.