Camlp5 sources

Information for developpers of the Camlp5 program.

  1. Kernel
  2. Compatibility
  3. Tree structure
  4. Fast compilation from scratch
  5. Testing changes
  6. Before committing your changes
  7. If you change the main parser
  8. Switching between transitional and strict mode

Kernel

The sources are composed of:

Some other directories contain configuration files, tools, documentation and manual pages.

The kernel is sufficient to make the core system work: it is possible to compile and bootstrap only it. All sources being in revised syntax, the first compilation of Camlp5 is done by a version of this kernel in pure OCaml syntax, located in the directory ocaml_src.

These sources in pure OCaml syntax are not modified by hand. When changes are made to the kernel, and a check is done that it correctly compiles and bootstraps, the kernel in pure OCaml syntax is rebuilt using Camlp5 pretty print. This is done by the command "make bootstrap_sources".

Compatibility

This distribution of Camlp5 is compatible with several versions of OCaml. The definition of OCaml syntax trees may change from OCaml version to version, which can be a problem. Since OCaml does not install the sources nor the compiled versions of its syntax tree, a copy of the necessary source files, borrowed from the source of the OCaml compiler is in the directory 'ocaml_stuff', in subdirectories with the OCaml version number.

If the present distribution of Camlp5 is not compatible with the version of OCaml you have (the command 'configure' tells you), it is possible to add it. For that, you need the sources to your specific OCaml distribution. If you have them then a 'configure' telling you that camlp5 is not compatible, do:

  make steal OCAML_SRC=<path-to-OCaml-sources>

This creates a new directory in 'ocaml_stuff' with sources of the syntax tree of your OCaml compiler.

If you want to check that the sources of the syntax tree of OCaml are up-to-date (e.g. if this is the current OCaml developpement), do:

  make compare_stolen OCAML_SRC=<path-to-OCaml-sources>

The compatibility is also done with the file 'main/ast2pt.ml', which is the module converting Camlp5 syntax tree into OCaml syntax tree.

In the directory 'ocaml_src' which contains the pure OCaml sources of the Camlp5 core (see chapter TREE STRUCTURE below), there are as many versions of this files as versions of OCaml. They are named 'ast2pt.ml_<version>'. If you are adding a new version of OCaml, you need this file. As a first step, make a copy from a close version:

  cd ocaml_src/main
  cp ast2pt.ml_<close_version> ast2pt.ml_<version>

Then, you can rerun "configure" and do "make core". If the file 'ocaml_src/main/ast2pt.ml' has a compilation problems, fix them and to 'make core' again.

Later, the same file 'main/ast2pt.ml' in Camlp5 syntax may have similar compilation problems. There is only a single version of this file, thanks to IFDEF constructs used here or there.

While compiling with some specific version of OCaml, this file is compiled with 'OCAML_vers' defined where 'vers' is the version number form the beginning to the first space or charcter '+' with all dots converted into underscores. For example, if your OCaml version is 7.04.2+dev35, you can see in the compilation process of ast2pt.ml that OCAML_7_04_2 is defined, and you can add statements defined by the syntax extension 'pa_macro.cmo', for example IFDEF OCAML_7_04_2. Add statements like that in 'main/ast2pt.ml' to make it compile successfully.

Tree structure

The directory 'ocaml_src' contains images in pure OCaml syntax of the directories odyl lib main and meta. This allows the creation of a core version of Camlp5 with only the OCaml compiler installed.

You can decompose the building of the Camlp5 core into:

1. make library_cold
just makes the directory 'ocaml_src/lib' and copy the cmo and cmi files into the directory 'boot'
2. make compile_cold
makes the other directories of ocaml_src
3. make promote_cold
copies the executables "camlp5", "camlp5r" and the syntax extensions (cmo files) into the directory 'boot'

From this point, the core Camlp5 is in directory 'boot'. The real sources in the top directories odyl, lib, main and meta, which are written in revised syntax with some syntax extensions (grammars, quotations) can be compiled. To achieve their compilation, you can do:

  make core

Or to compile everything do:

  make all

or just:

  make

Notice that doing "make core" or "make all" from scratch (after a make clean), automatically starts by making the core files from their pure OCaml versions.

Fast compilation from scratch

  ./configure
  make clean core compare
  make coreboot
  make all opt opt.opt

Testing changes

1. do your changes

2. do:

  make core compare

if it says that the bootstrap is ok, you can do:

  make all
  make opt
  make opt.opt

otherwise, to make sure everything is ok, first do:

  make coreboot

sometimes two bootstraps ('make coreboot' twice) are necessary, in particular if you change things in the directory 'lib'. It is even possible that three bootstraps are necessary.

If things go wrong, it is possible to return to the previous version by typing:

  make restore clean_hot

then you can change what is necessary and continue by typing:

  make core

and test the bootstrap again:

  make coreboot

After several bootstraps (by 'make coreboot' or 'make bootstrap'), many versions are pushed in the directory 'boot' (you can type 'find boot -type d -print' to see that). If your system correctly bootstraps, you can clean that by typing:

  make cleanboot

which keeps only two versions. (The command 'make clean' also removes these stack of versions.)

Before committing your changes

Make sure that the cold start with pure OCaml sources work. For that, do:

  make compare_sources | less

This shows you the changes that would be done in the OCaml pure sources of the directory ocaml_src.

To make the new versions, do:

  make new_sources
  make promote_sources

Notice that these pure OCaml sources are not supposed to be modified by hand, but only created by the above commands. Although their source is pretty printed they are usually not easy to read, particularly for expanded grammars (of the statement 'EXTEND').

If these sources do not compile, due to changes in the OCaml compiler, it is possible however to edit them. In this case, similar changes may need to be performed in the normal sources in revised syntax.

After doing 'make new_sources' above, and before doing 'make promote_sources' below, it is possible to do 'make untouch_sources' which changes the dates of the newly created files with the dates of the old files if they are not modified. This way, the "svn commit" will not need to compare these files, which may be important if your network is not fast.

The 'make new_sources' builds a directory named 'ocaml_src.new'. If this directory still exists, due to a previous 'make new_sources', the command fails. In this case, just delete it (rm -rf ocaml_src.new) without problem: this directory is not part of the distribution, it is just temporary.

The 'make clean_sources' deletes old versions of ocaml_src, keeping only the last and the before last ones.

The command:

  make bootstrap_sources

is a shortcut for:

  make new_sources
  make untouch_sources
  make promote_sources
  make clean_sources

If there are changes in the specific file 'main/ast2pt.ml', do also:

  make compare_all_ast2pt

and possibly:

  make bootstrap_all_ast2pt

because this file, in 'ocaml_src/main' directory has different versions according to the OCaml version.

After having rebuilt the pure OCaml sources, check that they work by rebuilding everything from scratch, starting with "configure".

If you change the main parser

If you change the main parser 'meta/pa_r.ml', you should check that the quotations expanders of syntax tree 'meta/q_MLast.ml' match the new version. For that, do:

  cd meta
  make compare_q_MLast

If no differences are displayed, it means that 'q_MLast.ml' is ok, relatively to 'pa_r.ml'.

Otherwise, if the displayed differences seem reasonable, update the version by typing:

  make bootstrap_q_MLast

Then returning to the top directory, do 'make core compare' and possibly 'make coreboot' (one of several times) to check the correctness of the file.

And don't forget, if you want to commit, to re-create the pure OCaml sources like indicated above.

Switching between transitional and strict mode

If Camlp5 is compiled in some mode, it is possible to change its mode in two boostrapping steps. Type:

  make MODE=T coreboot

to switch to transitional mode, or:

  make MODE=S coreboot

to switch to strict mode.

After two (necessary) bootstraps, the kernel is compiled in the new mode. Complete the compilation by:

  make MODE=T all opt opt.opt

or:

  make MODE=S all opt opt.opt

according to the new mode you want to use.

Another solution is, of course, recompile everything from scratch:

  make clean
  ./configure -transitional
  make world.opt

or:

  make clean
  ./configure -strict
  make world.opt

Copyright 2007 Daniel de Rauglaudre (INRIA)

Valid XHTML 1.1