Reduce Travis CI build matrix

Currently we build all 64 possible configurations for gcc/linux, clang/linux and clang/osx.
This leads to a long build time for automatic builds.
Now we build all 64 configuration only for one combination (clang/linux) and only selected reference combinations for gcc/linux and clang/osx.