Makefile overhaul

[magnumripper]

Add real auto-dependency stuff (gcc's own) and drop the incompetent perl script we had for plugins only. The old support for "makedepend" is kept for non-gcc systems.

Fix bugs in format's plugin stanzas.

Have "make distclean" also clear kernel caches (the ones within our tree only).

Move plugin logic back from configure to Makefile. Having them in configure was a major brainfart. Also enhance it so we can actually move plugins (formats or not) into or out of the source tree and just "make" regardless of their timestamps. I made the logic very quick for the case of nothing changed - blindly running the preprocessor over 470 files takes time so we do not want that unless actually needed.
For the same reason, I did not add the plugins themselves as dependencies for the "format register" stuff in john.c. That would mean very slow handling of repeated edit-make with a specific plugin and it's only needed if we do things like change the format struct name.

See #3566 and #3842.

[solardiz]

Thanks. This is a lot. I don't expect to have time to review it in the next 2 weeks.

[magnumripper]

Notes to self as well as for your consideration, here's what I have tested:

• Configure + build after make distclean.
• Configure with OpenCL disabled + build.

With no re-configure:

• make with no changes.
• make after touching a plugin.
• make after adding a plugin.
• make after removing a plugin.
• make after touching color.h.
• make after touching params.h.
• make after changing git HEAD.
• make emulating a non-gcc-compatible compiler (auto dependency generation are turned off). (obsolete, see below)
• make after deleting ALL plugin files. I had to add a commit promoting sm3_plug.c to a non-plug because the dynamic format needs it present, but that would have been needed even without this entire PR.
• make after putting all the plugin files back.

Instead of simply disabling auto dependency generation for compilers that MAY be incompatible, we could have the configure script detect if the needed options are supported. We do that for several other things. I'll give that a try and possibly add it as a separate commit.

[magnumripper]

Instead of simply disabling auto dependency generation for compilers that MAY be incompatible, we could have the configure script detect if the needed options are supported. We do that for several other things. I'll give that a try and possibly add it as a separate commit.

Done, trivial, works like a champ.

[solardiz]

I still have no time for a proper review, but:

I successfully built from this PR's branch in a fresh Docker container with Alpine Linux (so with musl) as a non-root user by ./configure --without-openssl and make -sj2, after having run as root apk add make gcc git libc-dev linux-headers.

Change sm3_plug.c to a non-plugin

It's needed for the dynamic format. We're supposed to be able to delete
every single plugin and build should still succeed.

Hmm, but doesn't our CircleCI no-plugs job test that? If so, why is it succeeding before these changes?

[magnumripper]

Hmm, but doesn't our CircleCI no-plugs job test that? If so, why is it succeeding before these changes?

I didn't look now but possibly it only deletes plugin formats, not other files. Or maybe there's just an exception but then we should have done this earlier.

Now, when I do delete all plugin sources, there will be two "errors" from building (reason being we preprocess using gcc -E *_plug.c resulting in "no input files") . While they do stick out they aren't fatal: The build continues and succeeds. I left that as-is but it can be avoided if we want to. OTOH I think such a build is extremely rare, not sure why anyone would do it except for testing the plugin gearing.

[magnumripper]

This is a lot.

It isn't really, especially if you review commit for commit. But take your time.

On another note, I realise we could make the whole dynamic format beast plugin(s) now without much effort (there are lots of #if DYNAMIC_DISABLED left from a period when it was defunct and Makefile could define that unless the plugs are present). Could be a separate PR though, if we want it at all.

[magnumripper]

Oh BTW I also tried with gcc 3.3.6 but that fails for other reasons than this PR.

argon2_core_plug.c: In function `argon2_validate_inputs':
argon2_core_plug.c:478: warning: comparison is always false due to limited range of data type
blake2b_plug.c: In function `blake2b_compress':
blake2b_plug.c:197: warning: dereferencing type-punned pointer will break strict-aliasing rules
gost3411-2012-core_plug.c: In function `g':
gost3411-2012-core_plug.c:126: warning: implicit declaration of function `_mm_cvtsi64_si128'
gost3411-2012-core_plug.c:126: error: incompatible types in assignment

EDIT: See #4271

Maybe we should fix that but I'm not sure I'm interested myself. Also, according to ChatGPT gcc has supported the flags needed for dependency generation since 3.0 but it may well be hallucinating.

[magnumripper]

Added another commit, for building particular targets (such as unit-tests, found by a buildbot) from a distclean source directory (i.e. with .deps/ missing).