Milter-greylist

Hello all,

   I have been fixing milter-greylist to build on my farm of assorted
Solaris machines (from Sol8 to OI), and a generic problem popped up:
if I execute a parallel build (i.e. "gmake -j all") I am almost always
receiving an error (except on one very fast computer). Tracing the
problem got me to the POSIX-based usage of yacc/bison, which causes
an output file to have a fixed name. When two build jobs write into
it in parallel - bad things happen. Maybe this is also augmented by
my boxes using the same build homedir over NFS (taking turns, of
course - but lags/unsyncs can be assumed to happen in comparison
to local HDD filesystems).

   So, I came up with a solution which works for me with GNU gmake and
Sun CCS make, but before trying to commit it into the source repo it
should be checked that this solution does not break builds (sequential
and parallel) on other OSes and build platforms (i.e. with other make
programs). Solution is attached, a patch that delivers fixes to the
Makefile.in template and a minor one to configure.ac (run "autoconf"
to build the new configure script).

   Ultimately I've made the attached patches to reinvoke the make
in non-parallel mode for those targets while not having to rebuild
milter-greylist itself which no longer directly depends on those
object files - by the virtue of touching a flag-file which for
make is an okay dependency.

   The sequential invokation of make is a Makefile variable MAKESEQ
which should ideally be set by configure - contributions of detection
for different make programs are welcome. So far value "make -j 1"
is hard-coded, and an example of another (which also works for gmake)
which zeroes out MAKEFLAGS is offered in comments. The latter is
not gracious IMHO, because other flags could be in use - like VPATH,
etc. With such a simple hack they would be lost.

   Also, while my tests ran over NFS-mounted home, there were some
lags which caused the first run to fail (sync/timing issues) while
the second sequential one passed ok. I don't think this can be worked
around well, had this for years - sequential builds over NFS also
fall prey to this issue, though much more rarely.

   One way or another, I also made a "rebuild" target which does
"clean" and then reinvokes make to build all - in original mode
(may be paral or seq as user requests) and if that fails, retries
in seq mode. In part this target is to test the routine, and may
be practical - i.e. if sources to those objects which are now
hidden by the lockfile are updated, a "rebuild" may ease the...
um... rebuilding of the project properly

   Also I added "sync || true" to fence off some parts which broke
for me over NFS (i.e. cleans must complete before builds begin).

   I've tested this with gmake and Sun (CCS) make, works ok for me
for local and NFS builds.

   If possible, this should be tested with more "make" programs and
perhaps the "sync" call should also be detected by configure - and
invokations skipped if the program is not present. Likewise, the
exact spelling of make-sequential invokations (-j1, -j 1, something
else) should also be configure-able, which they are not now in my
patch.

HTH, thanks in advance for verifications, and good luck,
//Jim Klimov

Bump... If anyone has got around to checking out the Makefile patch -
whether it breaks milter-greylist builds on their systems or not in
both parallel and sequential make routines, in local FS and over NFS -
please let the list know so the patch can be further fixed or already
integrated into the repository.

Tell us the details, so we can have a checklist what works and what
does not - which OS name/release/platform, which make program. So far
this has been tested with Solaris derivates with GNU gmake and with
Sun CCS make, locally and over NFS - all successfully:
* Solaris 10u8 SPARC and x86/x86_64
* Solaris 8 x86
* OpenIndiana 151a4 x86/x86_64
* OpenSolaris SXCE 117 x86/x86_64

Likewise, if anyone is ready to contribute further evolution to this
patch (autoconf methods to detect proper parallel/sequential make
invokations on a particular system) - please do so, too ;)

http://xa.yimg.com/kq/groups/12763546/1390623730/name/milter-greylist-4%2E4%2E2-parallel-gmake-fix%2Epatch

Thanks,
//Jim Klimov

On 2013-01-20 04:47, Jim Klimov wrote:
> [Attachment(s) <#TopText> from Jim Klimov included below]
>
> Hello all,
>
> I have been fixing milter-greylist to build on my farm of assorted
> Solaris machines (from Sol8 to OI), and a generic problem popped up:
> if I execute a parallel build (i.e. "gmake -j all") I am almost always
> receiving an error (except on one very fast computer). Tracing the
> problem got me to the POSIX-based usage of yacc/bison, which causes
> an output file to have a fixed name. When two build jobs write into
> it in parallel - bad things happen. Maybe this is also augmented by
> my boxes using the same build homedir over NFS (taking turns, of
> course - but lags/unsyncs can be assumed to happen in comparison
> to local HDD filesystems).
>
> So, I came up with a solution which works for me with GNU gmake and
> Sun CCS make, but before trying to commit it into the source repo it
> should be checked that this solution does not break builds (sequential
> and parallel) on other OSes and build platforms (i.e. with other make
> programs). Solution is attached, a patch that delivers fixes to the
> Makefile.in template and a minor one to configure.ac (run "autoconf"
> to build the new configure script).
>
> Ultimately I've made the attached patches to reinvoke the make
> in non-parallel mode for those targets while not having to rebuild
> milter-greylist itself which no longer directly depends on those
> object files - by the virtue of touching a flag-file which for
> make is an okay dependency.
>
> The sequential invokation of make is a Makefile variable MAKESEQ
> which should ideally be set by configure - contributions of detection
> for different make programs are welcome. So far value "make -j 1"
> is hard-coded, and an example of another (which also works for gmake)
> which zeroes out MAKEFLAGS is offered in comments. The latter is
> not gracious IMHO, because other flags could be in use - like VPATH,
> etc. With such a simple hack they would be lost.
>
> Also, while my tests ran over NFS-mounted home, there were some
> lags which caused the first run to fail (sync/timing issues) while
> the second sequential one passed ok. I don't think this can be worked
> around well, had this for years - sequential builds over NFS also
> fall prey to this issue, though much more rarely.
>
> One way or another, I also made a "rebuild" target which does
> "clean" and then reinvokes make to build all - in original mode
> (may be paral or seq as user requests) and if that fails, retries
> in seq mode. In part this target is to test the routine, and may
> be practical - i.e. if sources to those objects which are now
> hidden by the lockfile are updated, a "rebuild" may ease the...
> um... rebuilding of the project properly
>
> Also I added "sync || true" to fence off some parts which broke
> for me over NFS (i.e. cleans must complete before builds begin).
>
> I've tested this with gmake and Sun (CCS) make, works ok for me
> for local and NFS builds.
>
> If possible, this should be tested with more "make" programs and
> perhaps the "sync" call should also be detected by configure - and
> invokations skipped if the program is not present. Likewise, the
> exact spelling of make-sequential invokations (-j1, -j 1, something
> else) should also be configure-able, which they are not now in my
> patch.
>
> HTH, thanks in advance for verifications, and good luck,
> //Jim Klimov
>
> 
>
> Attachment(s) from Jim Klimov
>
> 1 of 1 File(s)
>
> milter-greylist-4.4.2-parallel-gmake-fix.patch
> <http://xa.yimg.com/kq/groups/12763546/1390623730/name/milter-greylist-4%2E4%2E2-parallel-gmake-fix%2Epatch>