_____

Fortran90 dependencies and make

A well known problem when writing Fortran90 programs is: how to deal with the dependencies based on the usage of modules. A summary of the problems:

  • The names of the generated module files are chosen by the programmer and bear no fixed relation with the name of the source file
  • Fortran compilers have, as far as I know, no option to generate dependencies like the -M flag of most C compilers
  • Module files can change during compiling in a, for Make, unpredictable manner
  • For most compilers: the time stamp of a module file changes, regardless if the interface of the module file has changed or not
  • Even when using a Fortran90 compiler (gfortran for example) that does not over-write not-changed interfaces of module files: Make makes it plan at the start and cannot take care of changing module files
  • Even if in the Makefile the source files are presented in a correct order, a parallel make will often fail

Include files are more simple to handle: include files normally do not change during compiling, so it is easy to tell Make about these dependencies.

In this article I propose a partial solution for these problems, that can be easily incorporated in a standard Makefile or a Makefile.am.

I am sure that other people have other, and better, solutions, but I could not find them on the Internet.

Partial solution: makedepo.py

Makedepo.py reads Fortran source files, and tries to find dependencies based on module files and include files. It contains a very simple parser to filter out USE, MODULE and INCLUDE statements. The program writes to standard output dependencies in Make syntax, to be included in a Makefile or Makefile.am. Always trying to get an optimal solution, makedepo.py can output various kinds of dependencies, but the most useful and default are dependencies of the form:

  a.o: a.f90 inc1.h inc2.h b.o c.o

So, the dependencies of object files are based on:

  1. source
  2. include files
  3. other object files

The determination of the first 2 kinds of dependencies are trivial, the third one is based on the USE and MODULE statements found in the sources.

In practice, this works very reliable:

  • the programmer does not have to care about the order of compilation
  • parallel makes run without problems
  • the solution is easily included in existing projects

Better solutions?

The solution above ensures that every source that ultimately depends on a just edited source will be recompiled, even if the module interfaces are not changed. Until now, I was not able to take care of this, at the same time allowing parallel make and allowing only minor changes to the Makefile(.am). See also f90_mod_deps.

A nearly complete solution

Here is a nearly complete solution, but it requires the following from the Fortran compiler:

  1. it must be able to produce quickly the module files, without generating the object files
  2. the time stamp of not-changed module files must not change when compiling the source

Gfortran, version 4.3 and better, fulfills both requirements. Intel Fortran fulfills the first requirement. Both compilers understand the flag -fsyntax-only.

Using gfortran, a not-changed source file will only be recompiled if one or more of the module interfaces ahve been changed.

Using Intel Fortran, the behaviour is the same as in the partial solution above, but see 'wrappers' below.

How it works

We create two kinds of dependencies:

  1. the one mentioned above, but '.o' replaced by '.n' (or another suitable suffix)

  2. dependencies expressing the dependence of the module files that are needed to produce the object file, like:

    a.o: a.f90 inc1.h inc2.h bmodule.mod cmodule.mod

Example Makefile, creating 'hello' from 'hello.f90', 'sub1.f90' and 'sub2.f90':

  all: mods
          $(MAKE) hello

  SRCS  = hello.f90 sub1.f90 sub2.f90
  OBJS  = $(SRCS:.f90=.o)
  NOBJS = $(OBJS:.o=.n)

  FC = gfortran

  .SUFFIXES:
  .SUFFIXES: .f90 .o .n
  .PHONY:    clean dep

  hello: $(OBJS)
          $(FC) -o $@ $(OBJS)

  .f90.o:
          $(FC) -c -o $@ $<

  .f90.n:
          $(FC) -fsyntax-only $<
          touch $@

  mods: $(NOBJS)

  include .deps

  dep .deps:
          ./makedepo.py -s .n $(SRCS) > .deps
          ./makedepo.py -m O:FHM $(SRCS) >> .deps

  nclean:
          rm -f *.n .deps

  clean: nclean
          rm -f *.o *.mod

Also this solution is not perfect: for example, there are no rules to create a module file. If a module file gets lost, the system is in an inconsistent state, and a simple 'make' will not work. However, it seems that most of the time one can repair this kind of failures in a short time by removing the '*.n' files and dependency file and calling make again:

  make nclean
  make -j

Download

Find here a tarball with the program makedepo.py and three example folders demonstrating the partial solution:

  1. make: example for use in a Makefile.
  2. auto: example for usage with GNU autotools
  3. autolib: example for usage with GNU autotools, creating a program, objectfiles and a library

Moreover, there are three folders demonstrating the more complete solution:

  1. nmake: example for use in a Makefile.
  2. nauto: example for usage with GNU autotools
  3. nautolib: example for usage with GNU autotools, creating a program, object files and a library

In every directory is a script 'doit', which is supposed to do it all, and a script 'cleaner' which removes all generated stuff.

Making it work with other compilers using wrappers

To use the 'complete' solution with other compilers than gfortran requires in general a wrapper script, that takes care of:

  • choosing the fastest mode of the compiler to generate modules
  • find out which modules will be generated (the output of makedepo.py -m M:F could probably be used), and take appropriate action if the modulefiles are not changed.

Wrapper for ifort

Something like this for ifort:

  #!/bin/bash
  # compiler wrapper for ifort
  # not-changed module files will keep their time stamp
  # arguments: these will be passed unchanged to the compiler

  options="$@"
  src=""
  for p in $options ; do # put fortran sources in src
     case "$p" in
        *.[fF] | *.[fF]9[05] | *.[fF]0[38] | \
           *.for  | *.fpp    | *.ftn       | *.i | *.i90 | \
           *.FOR  | *.FPP    | *.FTN        ) src="$src $p" ;;
     esac
  done
  dir=`mktemp -d`

  # find out which modules will be generated
  modules=$(./makedepo.py -m M:F $src | cut -d: -f1 -s )

  # move the modules to $dir
  for module in $modules ; do
     if [ -f $module ] ; then
        mv $module $dir/$module 
     else
        touch $dir/$module
     fi
  done

  ifort $options || { rm -r $dir; exit 1; }

  # Compare freshly made modules with saved ones.
  # Problem is that ifort places a timestamp in the 
  # module. Solution: we skip the first 56 bytes while comparing.
  # If the module did not change, move the original back

  for module in $modules ; do
     if cmp -s <(tail -c +57 $dir/$module ) <(tail -c +57 $module) ; then
        mv $dir/$module $module
     fi
  done

  rm -r $dir

If only modules are to be generated, call this wrapper like:

  ./wrapper -c -fsyntax-only ${FCFLAGS} prog.f90

Wrapper for portland group fortran

Something like this for pgf90:

  #!/bin/bash
  # compiler wrapper for portland group fortran (pgi)
  # not-changed module files will keep their time stamp
  # arguments: these will be passed unchanged to the compiler

  options="$@"
  src=""
  for p in $options ; do # put fortran sources in src
     case "$p" in
        *.[fF] | *.[fF]9[05] | *.[fF]0[38] | \
           *.for  | *.fpp    | *.ftn       | *.i | *.i90 | \
           *.FOR  | *.FPP    | *.FTN        ) src="$src $p" ;;
     esac
  done
  dir=`mktemp -d`

  # find out which modules will be generated
  modules=$(./makedepo.py -m M:F $src | cut -d: -f1 -s )

  # move the modules to $dir
  for module in $modules ; do
     if [ -f $module ] ; then
        mv $module $dir/$module 
     else
        touch $dir/$module
     fi
  done

  pgf90 $options || { rm -r $dir; exit 1; }

  # Compare freshly made modules with saved ones.
  # Problem is that pgi places a timestamp in the 
  # module. Solution: filter out the time stamp line before compiling.
  # If the module did not change, move the original back

  for module in $modules ; do
     if cmp -s <(sed '/[0-9][0-9]\/[0-9][0-9]\/[0-9]\{4\}/d' $dir/$module ) \
        <(sed '/[0-9][0-9]\/[0-9][0-9]\/[0-9]\{4\}/d' $module) ; then
        mv $dir/$module $module
     fi
  done

  rm -r $dir

If only modules are to be generated, call this wrapper like:

  ./wrapper -c -o /dev/null -O0 prog.f90

I could not find a way to instruct the pgf90 compiler to disable the generation of object files, hence -O0 (fastest compilation) and -o /dev/null (to prevent the generation of object files).