Module system for T                  -*- outline -*-

* Issues that must be dealt with

(1) Simplicity & semantic cleanliness
  The module system **must** be simple to use & understand.  If not, it has
  failed entirely.  It should also not be semantically unclean; e.g., it
  shouldn't load code into some sort of 'global environment.'  (Monolithic
  locales as currently in T will be expunged.  No CL packages, either.)

(2) Environment control
  It must be possible to finely control the lexical environment, with both
  variable and syntactic bindings.  (No syntax table kludgery.)  Imports
  should be explicitly written; exports should be what is generally
  available for another to import, not a specific set of names to inject
  into another module.

(3) Compilation control & macro semantics
  The system must be able to finely control compilation without stooping to
  the low levels of loading sequences.  (There should also not need to be a
  separate 'compilation manager' like that of SML/NJ or MLton.)  It should
  easily support both incremental compilation (for varying increment sizes)
  and also whole-system compilation.  Macros and phase separation must be
  able to fit coherently into the system.

(4) Feature control/system distribution
  While no particular structure of distribution should be imposed by the
  module system, the module system will need to provide some mechanism for
  accessing external systems/packages.

(5) Interactive development
  There must be facilities for rapidly reloading code into the system.  If
  the system requires nesting, it should not be difficult to selectively
  reload nested components of configurations.  In general, prototypes must,
  absolutely must, be able to be constructed trivially & rapidly.  It is a
  crucial aspect of the module system to support exploratory programming.

(6) Separation of interface from implementation
  Interfaces must be separate entities from implementations; not only must
  they be separate for reasons of abstraction, but it should be possible to
  have more than one implementations of a given interface or have a single
  body of code implement several different interfaces.

(7) Abstraction at the module level: parameterized modules
  It must be possible to create bodies of code parameterized by modules to
  import.  (This is another reason for separating interfaces from specific
  implementations: module combinators need to enforce that their inputs
  satisfy certain interfaces, but they need not know anything about the
  implementations of their inputs.)

(8) Mutual reference
  Components of code must be able to be mutually recursive, at least within
  certain well-defined partitions of module groups.  A theory for mutual
  reference and its interaction with syntactic hygiene must be developed.

(9) Purely declarative language
  The language for defining modular components and systems must be purely
  declarative and easily statically analyzed.  Certain elements must be
  statically decidable, most notably various aspects of interfaces, such as
  the precise interface (ignoring static type analysis) of every component
  without processing the component's code.  It also must be possible to
  decide whether the binding for a certain name exported by a module is a
  syntactic binding or a variable binding.

(10) Dynamic loading support
  While the module system should be easily statically analyzable, it should
  also easily support dynamic loading, not just in interactive development
  but also for dynamically extensible applications.  The programmatic
  interface to dynamic loading must be lightweight and usable.

(11) Language generality & reader customization
  The module system should support modules written in entirely different
  languages from T and even with source code read using a reader other than
  the standard one.  A related matter, though not quite the same, is that
  the module system should support foreign modules, not just not written in
  T but built using an entirely different programming system (like C).

(12) OS-independence without OS-clashing
  The module system should support distribution of libraries in such a way
  that it does not impose the model of a particular OS, such as Unix, but
  also allows for generality in the model used, so that it can be portably
  used across many operating systems.

(13) File system interaction
  Similarly, the module system should mesh easily with the file system on
  which the source is stored, but the file system should impose nothing on
  the module system; e.g., there should absolutely not be a 1-1 mapping
  between files and modules.

(14) No meta-modular complexity
  It should not be necessary for there to be a tower of increasingly meta
  modules in order to work with large software involving many smaller
  components.

(15) Interaction with compiler optimization
  The module system must support the transmission of information between
  modules that the compiler can use for optimization and extensive flow
  analysis.  This should be able to work both in incremental modes and
  whole-system modes.

* Existing module systems

Many Scheme systems support some sort of module system; Common Lisp has its
package system, and there is also ASDF.  None of the module systems I have
seen, however, satisfies all of the requirements listed above; few satisfy
more than a couple of them.  In this listing of what is satisfied and what
is not, along with a description every entry is marked blank (for fully
satisfied), S (for semi-satisfied), U (for unsatisfied), or ? (for unknown
satisfaction).

** T's status quo (locales & loading sequences)

T's locale mechanism is very rudimentary, and loading sequences are much
too painful & low-level to work reasonably with.  While T does support
reloading code easily in debugging, it really has no complete & coherent
module system.  (Even T's original authors admitted that, and they tried to
create a module system to fix the situation.  Nothing seemed to come out of
that effort, however.)  COMFILE doesn't cut it for compilation control.

** MIT Scheme's CREF (cross-reference)

CREF is really useful only for building cross-referencing information and
very rudimentary compilation control.  It satisfies essentially none of the
above requirements, and it is very difficult to use in a rapid-prototyping
environment.  Packages export names _into_ other packages, also, which is
quite backwards -- but even so, it makes no difference to the compiler,
which is not integrated with the system at all.  The compiler just compiles
individual modules at a time.

** PLT's module & unit/signature systems

PLT has two different module systems: modules and units/signatures.  The
former is extremely static; the latter is too dynamic to be very useful.
Many of the design decisions for modules were made to support silly tricks
like info.ss and FrTime, but impede the usefulness of the system so much
that units & signatures exist.  Here is a list of what it does & doesn't
satisfy, along with why it doesn't satisfy it, with U marking unsatisfied &
S marking semi-satisfied:

  (1)  Limitingly simple, but simple nevertheless.
S (2)  Limited control over the environment, but some control still.
S (3)  Limited control.  Phase separation is odd: modules are instantiated
       differently depending on how they're used.  Coherent macro semantics
       is still supported, though.
  (4)  PLaneT & the collection system with (LIB ...)
U (5)  Not at all.  No convenient dynamic reloading.  DrScheme forces that
       _entire_ modules be reloaded from scratch, and there is no updating
       of other modules when they are.
U (6)  The PROVIDE special form within module bodies just lists specific
       names for the module.  There is no separation of interface from
       implementation; use units & signatures for that.  (See below).
U (7)  No parameterized modules.  Use units & signatures.
U (8)  Modules cannot be mutually referential.
U (9)  The language is purely declarative, but it requires processing the
       bodies of modules to find out, for instance, what a module exports.
U (10) Dynamic loading support is incoherent.
S (11) The language can be customized more than with, say, Scheme48, but
       the reader is not customizable at all.  (Note that MODULE forms wrap
       their code, so it is fundamentally impossible to customize the
       reader with the PLT module system.)
  (12) The collection system seems to work well in this regard.
U (13) 1-1 mapping between files & modules.  Bletch.
  (14) No meta-module system at all.
  (15) While the current compiler doesn't seem to use the capability, it
       seems like it would be possible to extract early-binding information
       from modules in order to compile later modules.

The unit/signature system is very different (and somewhat older).  It works
entirely at run-time, so it doesn't handle macros or anything.

U (1)  Horribly complicated.  Naming conventions are a mess, too.
S (2)  Fine-grained control over variable bindings.  Nothing for syntactic.
U (3)  No compilation control.  No macros.
U (4)  No feature control or system distribution.  Use the module system.
U (5)  No easy interactive development reloading hooks.
  (6)  Signatures are separate from units.  More than one unit can be
       defined with a given signature.  However, units have exactly one
       signature, if I recall correctly; you can't make multiple views on a
       single unit.  This can be worked around, though.
  (7)  Units are first-class values; simply use Scheme procedures.
  (8)  Units can be mutually referential through a complicated linking
       scheme.
S (9)  Signatures are statically decidable.  However, units are run-time
       values: you operate on them with Scheme, so the language isn't
       purely declarative.
U (10) No simple dynamic loading support that I'm aware of.
U (11) No language generality or reader customization.  (Run-time values.)
U (12) Orthogonal to package distribution.  Use the module system.
U (13) Ditto.
  (14) Since units are first-class values, you can export them from other
       units, but there's no necessity of doing so.
U (15) Units are run-time values, so no compiler optimization involved.

** Scheme48's module system

Scheme48 has a sophisticated module system based on a fairly simple
'configuration language.'  It is based mostly on SML's module system.  It
consists of structures, interfaces (what SML calls signatures), and regular
procedures (which act as SML's functors).  Packages underlie structures:
structures are really just views on a package, i.e. a package plus the set
of names it exports (the interface).

  (1)  The system is pretty simple, though it looks complex initially.
  (2)  Sophisticated environment control.
  (3)  Compilation control is high-level, possibly somewhat too high-level.
       No loading sequences, however: the module system figures out the
       dependency graphs itself.  Clean macro semantics.
U (4)  No distribution or feature control mechanism.
  (5)  Very well-integrated with the interactive development environment.
       Mutation just works for just about everything (even at the module
       system level, for reloading interfaces & structures).
  (6)  Interface & implementation are entirely separate.  Not only can an
       interface be reused arbitrarily many times, but a single module can
       have several views on it (structures) with different interfaces.
  (7)  Support for parameterized modules.
U (8)  Modules may not be mutually referential.  (It 'works' only sometimes
       because of the mutation semantics.)
  (9)  Purely declarative language.  Easily statically analyzable.
  (10) Dynamic loading support.  Perhaps somewhat more complicated than
       necessary, but not too much so, and there nevertheless.
S (11) Language generality is fully supported, but reader customization
       was taken out some time ago.  (I don't know why.)
U (12) The module system doesn't deal with distribution or anything.
  (13) Works well with the file system, but doesn't allow any imposition of
       arbitrary things by the file system.
S (14) While meta-modules are sometimes necessary, it's usually not hard to
       avoid getting too complex with them.
S (15) The compiler is well-integrated with the module system for what
       little optimization it does.  No module groups, however, within
       which compiler information flows freely: the flow is one-way.

** Dylan

Dylan has a two-level module system: there are libraries & modules.  The
two are very similar, only libraries import & export modules whereas
modules import & export variables, constants, & macros.

S (1)  Fairly simple in use, but the interface & implementation stuff, or
       the binding creation & export weirdness, is semantically confusing.
  (2)  Adequate control of imports & rename-on-import & such.
? (3)  Not sure about compilation control, but macro semantics aren't an
       issue, because there are no programmatic macros.
  (4)  Seems to work OK.
? (5)  There don't seem to be any really good REPL-type systems for Dylan,
       and I don't think any of them support convenient code reloading. I'm
       not certain, however, whether or not it would work.
S (6)  Interfaces & implementations work a little weirdly, and there isn't
       enough space here to describe it.  Read _Dylan Programming_ or the
       DRM if you're curious.
U (7)  No parameterized modules that I'm aware of.
U (8)  No mutually referential modules.
  (9)  I believe the module language is purely declarative.
? (10) Don't know, but I don't think it's supported by either of the two
       main Dylan implementations.  However, it might nevertheless be
       possible to create a lightweight dynamic loading interface.
S (11) I think it supports some generality of language, but not in the
       core syntax.  (There's a DYLAN library & module.)
  (12) Seems sufficiently OS-independent without clashing with the OS.
S (13) Interacts with the file system reasonably well, although the
       association between module & source is not clear. from the module
       definition itself.
  (14) No meta-module system at all: strictly two levels.
  (15) Probably supports compiler optimization decently.

** Common Lisp's packages & ASDF

(Elaborate on this quick comment.)

ASDF works well given CL's package system, but that combination is entirely
inapplicable to T & Scheme.

** psyntax's module system

psyntax, the portable implementation of the monstrosity of SYNTAX-CASE, has
a simplistic module system for controlling lexical scoping of bindings.  It
is intrinsically built into the syntax expander mechanism.

  (1)  Quite simple.
S (2)  Rudimentary environment control, but it's possible to build better
       control using SYNTAX-CASE.
S (3)  I've heard claims that it supports compilation control mechanisms,
       but I am extremely doubtful.  Only recently was support for macro
       semantics added; before then it was extremely ad-hoc.  The merit of
       the mechanism now is questionable, but it works.
U (4)  No feature control or system distribution.
U (5)  No interactive development environment integration.
S (6)  Nothing built-in for separating interface from implementation, but
       again it can be built using SYNTAX-CASE.
S (7)  Similarly, nothing built-in for parameterized modules, but it's
       buildable using SYNTAX-CASE.
? (8)  Not sure whether mutually referential modules are supported.
U (9)  The language is not purely declarative; it is intertwined heavily
       into Scheme.
U (10) No dynamic loading -- modules are inherently purely static to all
       code.
U (11) No language generality or reader customization.
U (12) No distribution, so orthogonal to the OS.
S (13) Use an INCLUDE macro for separate files.
  (14) No meta-module system at all.
? (15) Chez is wicked fast, but I don't think the module system is able to
       help in that regard.
