=========================================== CBE Part 10: Translation Units and GNU Make [TOC] =========================================== Even medium sized projects require that you use more than just one translation unit. This requires to have some understanding how code needs to be organized, how to provide header files. It also requires some understand on how tools like the compiler and linker work. And it also requires to use tools for implementing a build system. IDEs (Integrated development environments) provide easy to use build systems. You click on a button and after some miracle happen your build is complete. Or you get some strange error and have no glue how to fix it if you don't know how such a build system works in principle. In this class we will use __GNU Make__ for implementing our own build system. Sooner or later you should invest some rainy afternoon in reading the complete __GNU Make Documentation__. However, it can be of advantage to do that with some background about how GNU Make works in general, what kind of problems it can solves and what kind of features it provides. Giving you some of this background is the purpose of the material provided here. ---- VIDEO ------------------------------ https://www.youtube.com/embed/3O3clFF5hxw ----------------------------------------- :links: GNU Make -> https://www.gnu.org/software/make/ GNU Make Documentation -> https://www.gnu.org/software/make/manual/html_node/index.html#Top Provided Material ================= The Makefile that was developed in the video: :import: session13/using_make/Makefile [noet] Note that in HTML you do not have tabs in the character set. So if you copy and paste the content of the Makefile you will not have tabs in the Makefile. As you know this is not what you want in a Makefile. So use `wget` or `curl` if you want to download the Makefile file directly. GNU Make Features Used in the Example ===================================== Simply Expanded Variables ------------------------- GNU Make has different __flavors of variables__. In all cases the value of a variable is just a string. If `foo` is the identifier of a variable then `$(foo)` is its value. Here so called _simply expanded variables_ are used. You specify this flavor if you initialize a variable with '`:=`' (and not with '`=`'). That means in an expression `$(foo)` immediately gets expanded to the string stored in the variable. Unlike build rules the assignments of variables are processed line by line. After this lines ---- CODE (type=mk) ------------ foo := x y z bar := $(foo) foo := a b c -------------------------------- variable `foo` has the value `a b c` and `bar` the value `x y z`. Wildcard Function and Text Functions ------------------------------------ Using the __wildcard function__ you can get a list of existing files that match a given pattern. For example `$(wildcard *.c)` gives a list of all files in the current directory that have the suffix '`.c`' and after ---- CODE (type=mk) ------------ src.c := $(wildcard *.c) -------------------------------- this list is the value of variable `src.c`. Note that in GNU Make a list means that you have a string of whitespace-separated words. So "`x y z`" is in this sense a list with words "`x`", "`y`" and "`z`". Because all variables are string a rich set of __text functions__ are provided to conveniently manipulate strings. Function `patsubstr` ---- CODE (type=mk) ---------------- $(patsubst pattern,replacement,text) ------------------------------------ can be used to replace `foo.c bar.c` with `foo.o bar.o` as follows: ---- CODE (type=mk) --------------------------------- text := foo.c bar.c replacement := $(patsubst %.c,%.o,text) ----------------------------------------------------- Here "`%.c`" was used for the pattern. It matches if a word in the list `text` has the suffix "`.c`". In the replacement the "`%`" is replaced with the part of the pattern that matched '`%`' in the pattern. Pattern Rules and Automatic Variables ------------------------------------- It would be tedious to write rules like ---- CODE(type=mk) ----------------------------- foo.o : foo.c gcc -c foo.c bar.o : bar.c gcc -c foo.c ------------------------------------------------ where you basically repeat the same rule for every source file. More conveniently you can express the same with a __pattern rule__ ---- CODE(type=mk) ----------------------------- %.o : %.c gcc -c $< ------------------------------------------------ where "`%.c`" in the dependency list can match any file with suffix "`.c`". For each match a target "`%.o`" is considered where '`%`' is replaced with the matched pattern (in this case the _base name_ of a C source file). In the command you can use __automatic variables__ to obtain values for the matched target or match dependency. For example, with "`$@`" you get the target and with the "`$<`" name of the first word in the dependency list. In this example (this is a complete Makefile, so try it out): ---- CODE(type=mk) ----------------------------- print: foo bar echo $< ------------------------------------------------ you would see after `make print` just the word "`foo`" (because "`$<`" gives _just_ the first word of the dependecies). To get all dependencies you can use "`$^`": ---- CODE(type=mk) ----------------------------- print: foo bar echo $< print_all: foo bar echo $^ ------------------------------------------------ Now with `make print_all` you get `foo bar`. Including Files --------------- With the __include directive__ you can include the content of another file which than gets processed like a regular part of the makefile (so like when you include files with the C preprocessor). If in this example ---- CODE(type=mk) ----------------------------- include dummy ------------------------------------------------ no file `dummy` exists _and_ no rule was defined to create it, you will get an error. Using instead `-include` prevents such an error in this case, the include then just gets ignored. :links: flavors of variables -> https://www.gnu.org/software/make/manual/html_node/Flavors.html#Flavors wildcard function -> https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html#Wildcard-Function text functions -> https://www.gnu.org/software/make/manual/html_node/Text-Functions.html#Text-Functions pattern rule -> https://www.gnu.org/software/make/manual/html_node/Pattern-Rules.html automatic variables -> https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html#Automatic-Variables include directive -> https://www.gnu.org/software/make/manual/html_node/Include.html Experiments with the Makefile ============================= Here the source files from the video: :import: session13/using_make/bar.h [fold] :import: session13/using_make/bar.c [fold] :import: session13/using_make/foo.h [fold] :import: session13/using_make/foo.c [fold] :import: session13/using_make/main.c [fold] And here the dependency graph that we have to consider in the build system: --- TIKZ ----------------------------------------------------------------------- \begin{tikzpicture} \input{flowchart.tex} \renewcommand\BoxWidth {5} \renewcommand\FlowColDist {3} \renewcommand\BoxHeight {1.2} \renewcommand\BoxDistance {1.8} \SetMargin{1}{2}{1}{3} \renewcommand\FlowCol {0} \PutStatement{2}{main.c } \PutStatement{4}{main.o} \renewcommand\FlowCol {2} \PutStatement{0}{foo.h } \PutStatement{2}{foo.c } \PutStatement{4}{foo.o} \renewcommand\FlowCol {4} \PutStatement{0}{bar.h } \PutStatement{2}{bar.c } \PutStatement{4}{bar.o} \renewcommand\FlowCol {2} \PutStatement{6}{a.out} \AddDepPath[above right=0.8cm and 0.1cm]{0}{2}{2}{0}{depends on} \AddDepPath[above right=0.8cm and 0.1cm]{2}{2}{2}{0}{depends on} \AddDepPath{2}{2}{4}{0}{} \AddDepPath{4}{2}{4}{0}{depends on} \AddDepPath{0}{4}{0}{2}{depends on} \AddDepPath{2}{4}{2}{2}{depends on} \AddDepPath{4}{4}{4}{2}{depends on} \AddDepPath{2}{6}{0}{4}{} \AddDepPath[above right=0.8cm and 0.1cm]{2}{6}{2}{4}{depends on} \AddDepPath{2}{6}{4}{4}{} \end{tikzpicture} -------------------------------------------------------------------------------- Test the makefile by "touching" certain files and then use _make_ to rebuild: ---- SHELL (path=session13/using_make) ----------------------------------------- make touch bar.h make touch bar.c make touch main.c -------------------------------------------------------------------------------- Advantage of the Current Makefile ================================= The following can be done without modifying the makefile: - Adding or removing include directives. Also header files can include other header files. The build system will keep track on how the dependencies have changed. - You also can add new source files to the project as long as these do not contain a _main()_ function. Currently we only support the generation of one executable (see _Disadvantages and How to Fix Them_ below). Try things on your own, for example: Add a source file _dummy.c_ that includes _dummy.h_. Let _foo.h_ include _bar.h_ and let _bar.h_ include _dummy.h_ etc. Disadvantages and How to Fix Them ================================= We certainly want to give the executable a name different from _a.out_. If we think of the executable as a test program for functions implemented in the other source files we probably would like to have more than just one test program. For every bug that we find in the implementation we want to add a new test so that we don't run into the same problem again. Many build system used in IDEs have problems when you delete or rename files in your projects. You haven find tips in the internet like "just create a new project, import the files you still need and delete the old project". Currently we certainly can do better than that. The Clean target ---------------- In the makefile developed in the video I simply forgot that _make clean_ should also delete the dependency files. This is easy to fix. Change the rule _clean_ as follows: ---- CODE (type=mk) ------------------------------------------------------------ clean: rm -f a.out $(obj) $(deps) -------------------------------------------------------------------------------- Even better use the variable _RM_ that GNU make predefines (see __variables used in implicit rules__) and has the default value _rm -f_. Note that the option '_-f_' is important so that you do not get an error if you call _make clean_ twice: ---- CODE (type=mk) ------------------------------------------------------------ clean: $(RM) a.out $(obj) $(deps) -------------------------------------------------------------------------------- There is actually another problem. Try the following: - Run _make_ so that we have complete build. - Create a file _clean_, e.g. with _touch clean_. - Run _make clean_ In this case _make clean_ will not delete the generated files: ---- SHELL (path=session13/using_make2,hide) ----------------------------------- rm -f clean make clean -------------------------------------------------------------------------------- ---- SHELL (path=session13/using_make2) ---------------------------------------- make touch clean make clean -------------------------------------------------------------------------------- For _make clean_ we hoped to take advantage how GNU make handles rules with an empty list of dependencies: - If a target has no dependencies the rule get always triggered *if the target does not exists* - However, if the target exists the rule *never* gets triggered. Fortunately GNU make supports this hack by declaring _clean_ as a __phony target__: ---- CODE (type=mk) ------------------------------------------------------------ .PHONY: clean clean: $(RM) a.out $(obj) $(deps) -------------------------------------------------------------------------------- Now the existence of a file named _clean_ gets ignored and the rule always gets triggered. Compiler and Compilation Flags ------------------------------ Currently we explicitly compile with _gcc_. It should be possible to select a different compiler without rewriting the makefile. It also should be possible to specify some compiler flags, e.g. _-Wall_ for getting more warings or _-O3_ to turn on aggressive optimization. In __variables used in implicit rules__ you find the following predefined variables that are interesting in this context: - _CC_ for selecting a compiler. The default value is _cc_ which is on Unix-like systems the default C compiler. For example, on MacOS it is alink to _clang_, on Linux to _gcc_. - _CFLAGS_ for compiler flags - _CPPFLAGS_ for preprocessor flags - _LDFLAGS_ for linker flags We will use these variables in the rule commands as follows: ---- CODE (type=mk) ------------------------------------------------------------ a.out: $(obj) $(CC) $^ $(LDFLAGS) %.o: %.c $(CC) -c $(CFLAGS) $(CPPFLAGS) -MT $@ -MMD -MF $<.d $< -------------------------------------------------------------------------------- Here the complete code of the makefile with the changes applied so far: :import: session13/using_make3/Makefile [fold] ---- SHELL (path=session13/using_make3, hide) ---------------------------------- make clean -------------------------------------------------------------------------------- You now can specify the compiler when invoking make or by defining and exporting a shell variable: ---- SHELL (path=session13/using_make3) ---------------------------------------- make CC=gcc CFLAGS="-Wall -O3" make clean export CC=gcc make -------------------------------------------------------------------------------- Unfortunately on _theon_ this version of the makefile will not work with the default values because _cc_ is not set to a C compiler: ---- SHELL (path=session13/using_make3) ---------------------------------------- make clean make -------------------------------------------------------------------------------- To deal with platform that do not comply to all the Unix conventions is unfortunate but what you have to expect in the real world. Here some workaround to select _gcc_ in the case no compiler was explicitly selected by the user and the defaul Unix compiler does not exists: ---- CODE (type=mk) ------------------------------------------------------------ ifeq ($(origin CC),default) cc_check := $(shell $(CC) -v > /dev/null 2>&1 && echo "sane") ifneq ($(strip $(cc_check)),sane) CC := gcc endif endif -------------------------------------------------------------------------------- The value of _$(origin CC)_ is _default_ if _CC_ was not set before make was invoked. With _$(shell $(CC) -v > /dev/null 2>&1 && echo "sane")_ we try to invoke the default compiler and if that succeeds print "sane". If we succeed the value of ~cc_check~ is "sane". If this is not the case we set _CC_ to _gcc_. Here the patched makefile :import: session13/using_make4/Makefile [fold] and here a test ---- SHELL (path=session13/using_make4) ---------------------------------------- make clean make -------------------------------------------------------------------------------- Multiple Executable Test Programs --------------------------------- Often projects have one directory that just contains the source files with a _main()_ functions, i.e. the test programs. And another library directory with all the other source files. We will keep it simple and will just use one directory. But we also want to keep the makefile simple. For achieving that we require that only source files with a prefix _xtest_ contain a _main()_ function. Clash with Implicit Rules ~~~~~~~~~~~~~~~~~~~~~~~~~ We can use the _filter_ __text function__ together with _patsubst_ text function to get from the list of lists of targets: ---- CODE (type=mk) ------------------------------------------------------------ # all targets target := $(filter xtest%,$(patsubst %.c,%,$(src))) -------------------------------------------------------------------------------- Because nesting functions easily looks cluttered I __split lines__ by escaping the newline with a backslash: ---- CODE (type=mk) ------------------------------------------------------------ # all targets target :=\ $(filter xtest%,\ $(patsubst %.c,%,\ $(src))) -------------------------------------------------------------------------------- We will also need a list of the object files that are required by the target. These are the object file with out the prefix _xtest_ and we get them from the _obj_ variable using the _filter-out_ text function: ---- CODE (type=mk) ------------------------------------------------------------ # objects that are required by the targets lib.o :=\ $(filter-out xtest%,\ $(obj)) -------------------------------------------------------------------------------- The new default rule should now generate or update all test programs. We therefore define a new rule _all_ that depends on the file list in _target_ and all object files. This target is also a phony target. To be the default rule it has to be the first rule in the makefile. The __special variable__ _.DEFAULT_GOAL_ one can explicitly specify the default rule: ---- CODE (type=mk) ------------------------------------------------------------ .DEFAULT_GOAL := all .PHONY: all all: $(target) $(obj) -------------------------------------------------------------------------------- We also change the _clean_ rule to delete the target list instead of _a.out_. ---- CODE (type=mk) ------------------------------------------------------------ .PHONY: clean clean: $(RM) $(target) $(obj) $(deps) -------------------------------------------------------------------------------- So far we had only one target _a.out_ and used this rule for building it: ---- CODE (type=mk) ------------------------------------------------------------ a.out: $(obj) $(CC) $^ $(LDFLAGS) -------------------------------------------------------------------------------- Now we have eventually multiple executables so we need a __pattern rule__ to specify how to generate for example ~xtest_main~ from its object file ~xtest_main.o~ and all the other object files: ---- CODE (type=mk) ------------------------------------------------------------ %: %.o $(lib.o) $(CC) -o $@ $^ $(LDFLAGS) -------------------------------------------------------------------------------- So we end up with the following makefile: :import: session13/using_make5/Makefile [fold] It just has this little flaw of not working: ---- SHELL (path=session13/using_make5) ---------------------------------------- make clean make -------------------------------------------------------------------------------- The problem is that GNU make has predefined __implicit rules__, or built-in rules, that often makes it easier to write makefiles. In this case it causes this little problem. With the option _-r_ all predefined implicit rules are canceled. So this would be a quick and dirty fix: ---- SHELL (path=session13/using_make5) ---------------------------------------- make clean make -r -------------------------------------------------------------------------------- Of course this is not a nice solution. With _make -p_ you can see all rules that are considered by make. This list will be long. But with ---- CODE (type=sh) ------------------------------------------------------------ _make -p | less_ -------------------------------------------------------------------------------- you will easily find the relevant one: ---- CODE (type=mk) ------------------------------------------------------------ %: %.c # recipe to execute (built-in): $(LINK.c) $^ $(LOADLIBES) $(LDLIBS) -o $@ -------------------------------------------------------------------------------- You will also find in the list ---- CODE (type=mk) ------------------------------------------------------------ LINK.c = $(CC) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH) -------------------------------------------------------------------------------- So this rule states that ~xtest_main~ depends on ~xtest_main.c~ and that the target can be generated basically with _gcc xtest_main.c -o xtest_main~. So this shortcut bypasses our rule that the target depends on all these object files. Deleting Implicit Rules ~~~~~~~~~~~~~~~~~~~~~~~ As we found out that just one predefined implicit rules is causing the trouble we simply delete it (see __Canceling Implicit Rules__): ---- CODE (type=mk) ------------------------------------------------------------ %: %.c -------------------------------------------------------------------------------- So we end up with this makefile :import: session13/using_make6/Makefile [fold] And this solves our problem: ---- SHELL (path=session13/using_make6) ---------------------------------------- make clean make -------------------------------------------------------------------------------- Deleting Header Files --------------------- ---- SHELL (path=session13/using_make7,hide) ----------------------------------- rm -f dummy.* make cp dummy/dummy.* . -------------------------------------------------------------------------------- We now even can add more files to our project and make takes care of everything. Say we add the header _dummy.h_ :import: session13/using_make7/dummy.h and the source file _dummy.c_ :import: session13/using_make7/dummy.c If previously everything was already build just _dummy.o_ get created and the target is created by linking the object files: ---- SHELL (path=session13/using_make7) ---------------------------------------- make -------------------------------------------------------------------------------- So far so good. But then for some reason we just wanna delete the header file _dummy.h_ and not the source file. Afterwards we get an error from make: ---- SHELL (path=session13/using_make7) ---------------------------------------- rm dummy.h make -------------------------------------------------------------------------------- The origin of that problem is the dependency file that was generated by the compiler for us: :import: session13/using_make7/dummy.c.d As you can read from the error message the object files _dummy.o_ depends on _dummy.h_. But the file _dummy.h_ does not exist *and there is no rule to build it*. So the solution is also found in the error message. The dependency file should be extended by a rule for _dummy.h_: ---- CODE (type=mk) ------------------------------------------------------------ dummy.o: dummy.c dummy.h dummy.h : -------------------------------------------------------------------------------- Because the rule for _dummy.h_ has no dependencies it never fails (like the rule for _clean_). Luckily the compiler will generate with the addition option _-MP_ here the relevant part from the gcc man page: ---- CODE (type=txt) ----------------------------------------------------------- -MP This option instructs CPP to add a phony target for each dependency other than the main file, causing each to depend on nothing. These dummy rules work around errors make gives if you remove header files without updating the Makefile to match. This is typical output: test.o: test.c test.h test.h: -------------------------------------------------------------------------------- So we add this option to the rule for generating object files: ---- CODE (type=mk) ------------------------------------------------------------ %.o : %.c $(CC) -c $(CPPFLAGS) $(CFLAGS) -MT $@ -MMD -MP -MF $<.d $< -------------------------------------------------------------------------------- This is the makefile after the change: :import: session13/using_make8/Makefile [fold] Cleaning Up After Deleting Source Files --------------------------------------- ---- SHELL (path=session13/using_make7,hide) ----------------------------------- rm -f dummy.* cp dummy/dummy.* . make clean -------------------------------------------------------------------------------- There is still a flaw in our build system. Assume we begin with a clean directory and build everything: ---- SHELL (path=session13/using_make8) ---------------------------------------- make -------------------------------------------------------------------------------- So far so good. Everything was build. Now we delete _dummy.h_ and _dummy.c_ and rebuild again: ---- SHELL (path=session13/using_make8) ---------------------------------------- rm dummy.h dummy.c make -------------------------------------------------------------------------------- This also looks good. The target needs to be updated by linking the remaining object files. So all code from _dummy.o_ is now out of the system, forever forgotten. Or is that so? Well, the dummy code is gone but there are some leftovers. Its dependency is still there and _make clean_ will never remove it: ---- SHELL (path=session13/using_make8) ---------------------------------------- make clean ls *.c.d -------------------------------------------------------------------------------- A quick solution to that is to great a list of obsolete dependency files. We simply use the wildcard function to get all existing dependency files and filter out what is still needed: ---- CODE (type=mk) ------------------------------------------------------------ # dependency file leftovers of gone source files obsolete.deps:=\ $(filter-out $(deps),\ $(wildcard *.c.d)) -------------------------------------------------------------------------------- In the _clean_ rule we add this list to the remove command: ---- CODE (type=mk) ------------------------------------------------------------ .PHONY: clean clean: $(RM) $(target) $(obj) $(deps) $(obsolete.deps) -------------------------------------------------------------------------------- So this completly erases _dummy.*_ from history the next time invoke _make clean_: ---- SHELL (path=session13/using_make9,hide) ----------------------------------- rm -f dummy.* cp dummy/dummy.* . make clean -------------------------------------------------------------------------------- ---- SHELL (path=session13/using_make9) ---------------------------------------- make rm dummy.h dummy.c make make clean ls *.c.d || echo "all dependency files are gone" -------------------------------------------------------------------------------- Here the complete makefile :import: session13/using_make9/Makefile [fold] But why wait for the next _make clean_? We could also cleanup the next time we generate an object file or target. Types of Prerequisites ---------------------- We can add another phony rule which will remove all obsolete dependency files when it gets triggered: ---- CODE (type=mk) ------------------------------------------------------------ # rule for removing obsolete dependency files .PHONY: $(obsolete.deps) $(obsolete.deps) : $(RM) $(obsolete.deps) -------------------------------------------------------------------------------- And we want to trigger this rule the next time a target or object file gets created. For that we can change the rules to ---- CODE (type=mk) ------------------------------------------------------------ # our rule: to build target link its object file against library object files %: %.o $(lib.o) | $(obsolete.deps) $(CC) $^ $(LDFLAGS) # our rule to build objects: also generate a dependency file %.o: %.c | $(obsolete.deps) $(CC) -c $(CPPFLAGS) $(CFLAGS) -MT $@ -MMD -MP -MF $<.d $< -------------------------------------------------------------------------------- where after the pipe symbol '|' the dependency rule for cleaning up the left overs was added. We added a so called _order-only prerequisite_ (see __Types of Prerequisites__). It will considered before the normal prerequisites i(the one on the left hand side of the pipe symbol) ---- SHELL (path=session13/using_make10,hide) ---------------------------------- rm -f dummy.* cp dummy/dummy.* . make clean -------------------------------------------------------------------------------- So now we can save the _make clean_: ---- SHELL (path=session13/using_make10) --------------------------------------- make rm dummy.h dummy.c make ls *.c.d || echo "all dependency files are gone" -------------------------------------------------------------------------------- :import: session13/using_make10/Makefile [fold] Is That All? ============ Certainly not. Currently we still have the problem that each target gets linked against all object files. But in general a target just needs to be linked against a few of these object files. We will later solve this problem by using _static libraries_ where the linker can pick what is actually needed. :links: variables used in implicit rules -> https://www.gnu.org/software/make/manual/html_node/Implicit-Variables.html#index-rm phony target -> https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html text function -> https://www.gnu.org/software/make/manual/html_node/Text-Functions.html#Text-Functions split lines -> https://www.gnu.org/software/make/manual/html_node/Splitting-Lines.html special variable -> https://www.gnu.org/software/make/manual/html_node/Special-Variables.html pattern rule -> https://www.gnu.org/software/make/manual/html_node/Pattern-Rules.html implicit rules -> https://www.gnu.org/software/make/manual/html_node/Catalogue-of-Rules.html#Catalogue-of-Rules Canceling Implicit Rules -> https://www.gnu.org/software/make/manual/html_node/Canceling-Rules.html Types of Prerequisites -> https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html -