The nbproject and nohup.out files are peculiarities of your own build environment. Ignoring them, via .gitignore and .hgignore likely will not be helpful for others, but neither should it inconvenience anyone, so I've no objection to making this change.
OTOH, we do not need to incorporate build-aux, either as a sub-directory, or as a repository sub-module. What we do need is install-sh, which we may conveniently source from our existing build-aux source repository; alternatively, we could just as easily source it from any autoconf package distribution. In either case, it belongs in the ${top_srcdir} directory of wtklite, not in any build-aux sub-directory, and, as install-sh, it is already enumerated within both .gitignore and .hgignore. Thus, your proposed references to build-aux, in .gitignore, and as an AC_CONFIG_AUX_DIR reference in configure.ac, do not seem to be appropriate.
We need to ignore incidental project hierarchy files in VCS and add build-aux sub-module to configure.ac.