待辦事項 #41070

Please include libgccjit with MinGW GCC distribution

啟用日期: 2020-12-24 02:28 最後更新: 2021-01-28 04:42

回報者:
負責人:
類型:
狀態:
開啟 [Owner assigned]
元件:
里程碑:
(無)
優先權:
5 - 中
嚴重程度:
5 - 中
處理結果:
檔案:
9
Vote
Score: 0
No votes
0.0% (0/0)
0.0% (0/0)

細節

ease add libgccjit to the binaries included in the MinGW GCC distributions. This is required to be able to build projects that use libgccjit for JIT compilation of code. One example of this is "gccemacs", a branch of GNU Emacs development (soon to land on the master branch of Emacs) that compiles Emacs Lisp programs into native x86 code for faster runtime performance.

Thank you.

Ticket History (3/40 Histories)

2020-12-24 02:28 Updated by: eliz
  • New Ticket "Please include libgccjit with MinGW GCC distribution" created
2020-12-29 05:52 Updated by: keith
  • 負責人 Update from (無) to keith
評語

A preliminary discussion, related to this request, may be found in the MinGW-Users mailing list archives; see the topic thread commencing with this post.

2020-12-30 22:41 Updated by: keith
評語

As noted in the referenced mailing list thread, I've chosen to explore this, with respect to the existing GCC-9.2.0 package set.

Having perused the relevant online GCC documentation, I note that I should configure with --enable-host-shared, when adding jit to the --enable-languages set. Since the principal effect of --enable-host-shared is to force generation of position independent code, within shared object libraries, this seems kind of redundant on MS-Windows, where code in DLLs is always position independent anyway; however, without --enable-host-shared, the configure script rejects the addition of jit to --enable-languages, so I configured as instructed. This immediately led into the first jit build issue: with --enable-host-shared, the generated libiberty object code is expected to reside in a libiberty/pic build directory, but no such directory is ever built for --target=mingw32!

Initially, as stated in my first response to the mail thread, I kludged around this, (on my GNU/Linux build host), by running 'cd libiberty; ln -s . pic' after the make aborted, and then restarted make. Obviously, such kludges are not satisfactory in any ultimate solution, so, drawing inspiration from this patch set, as suggested in Eli's follow-up mail, I've implemented the attached patch, to resolve this particular aspect of the issue.

2020-12-31 08:57 Updated by: keith
評語

Having resolved the libiberty issue described in my preceding comment, my libgccjit build subsequently succumbed to the compilation failures, outlined in this mail posting, viz.:

- GCC's jit sources gratuitously use dlopen(), but neglect to include dlfcn.h; jit-common.h needs a patch, to work around this.
- availability of fchmod() is gratuitously assumed (in jit-playback.c); a further patch is needed, to use chmod() instead.

The attached patches, 22-libgccjit-win32-dlfcn.patch, and 21-libgccjit-win32-fchmod.patch, respectively address each of these issues, allowing the jit code to compile, and thus taking us a step closer to a successful build.

2021-01-04 07:05 Updated by: keith
評語

With the previously posted series of patches, (20...22), augmenting the pre-existing series for MinGW GCC-9.2.0, the libgccjit shared library sources compile; however, they do not link, because they introduce an unresolved dependency on pthreads mutexes.

While it would be technically feasible to include -lpthread within the link operation, and so resolve this dependency, but, as noted in this mailing-list message:

The dependency on pthreads is also unfortunate, as that many times conflicts with w32api headers and creates build problems. It would be good to avoid that dependency, if possible.

It arises because the JIT context is gratuitously guarded by a POSIX mutex lock. Perhaps that could be converted to a Windows alternative, (critical section, maybe); perhaps GCC-10 already does so, because the patches you pointed me to do nothing to address this.

I think MinGW64 simply requires pthreads in all the GCC-related builds, we had our share of problems stemming from that in Emacs development, when MinGW64 users complained about that.
It should be very simple to replace a pthreads mutex lock with a critical section, if GCC 10 didn't.

the dependency on a third-party pthreads mutex implementation does seem kind of alien, on the Windows platform, when a native alternative would appear to be entirely feasible.

FWIW, a cursory examination of the GCC-10 sources suggests that the pthreads mutex dependency is still present, without change. Consequently, while not entirely trivial, I have engineered a relatively simple replacement, in terms of a native Windows CRITICAL_SECTION; it may be found in the attached 23-libgccjit-win32-no-pthreads.patch.

2021-01-05 00:11 Updated by: keith
評語

All patches, up to and including #23, are sufficient to both compile, and link, the libgccjit shared library. However:

  • The build fails immediately thereafter, due to an improperly specified dependency, in the GCC Makefile; this is corrected by the trivial 24-libgccjit-full-driver-name.patch.
  • Although the generated libgccjit shared library is, in fact, a Windows DLL, it is named according to ELF convention, with a .so extension, and multiple symbolic links to variously version-annotated alternatives; furthermore, no accompanying libgccjit.dll.a, (as we would normally expect for a Windows build), is created.

The latter failing is corrected by the attached 25-libgccjit-mingw-link-options.patch; this modifies the link command, for the Windows build-case, to name the generated shared library as libgccjit-0.dll, while also creating the libgccjit.dll.a import library, as a linker side effect. Additionally, it modifies the installation commands, to install both the DLL, and its import library, into appropriate locations.

2021-01-05 02:33 Updated by: eliz
評語

Although this is still WIP, AFAIU, I would like to thank you for your efforts to make this happen. Thanks!

2021-01-05 04:32 Updated by: keith
評語

Since all Windows code is position independent anyway, and the reason that JIT requires configuration with --enable-host-shared appears to be to ensure position independence of generated code, it seems likely that the --enable-host-shared option is not actually required for a MinGW build of libgccjit-0.dll; however, the current behaviour of GCC's configure script enforces the requirement that it be specified. The attached 26-libgccjit-mingw-host-shared.patch overrides such enforcement, in the specific case of building for a MinGW target, allowing the whole caboodle to be built in a single operation, without requiring specification of --enable-host-shared.

2021-01-05 05:33 Updated by: keith
評語

Having successfully cross-built a MinGW-GCC, inclusive of libgccjit-0.dll, but uncertain how to effectively test it, I followed Eli's suggestion from this mailing-list posting:

Is the produced libgccjit functional?

I don't know. I suspect that the testsuite will be inadequate, when cross compiling, and I don't know how to test it, otherwise.

This tutorial has a simple text program, it seems:
https://gcc.gnu.org/onlinedocs/jit/

and copied the sample program from part 1 of the libgccjit tutorial, for use as a rudimentary test program ... alas, it fails spectacularly, with a segmentation fault which will require further investigation.

2021-01-05 21:24 Updated by: keith
評語

alas, it fails spectacularly, with a segmentation fault ...

Identification of the cause is far from straightforward. The actual point of failure appears to lie deep within the bowels of Windows system DLLs, and the GDB backtrace bears no obvious relationship to the source code. Seems like a rebuild with -g -O0, followed by stepping through the code, will be necessary to locate the origin of the failure.

(Edited, 2021-01-06 08:57 Updated by: keith)
2021-01-06 08:56 Updated by: keith
評語

After a rebuild with -g -O0, GDB yields a more meaningful backtrace:

(gdb) bt
#0  0x01765861 in pp_format (pp=0x0, text=0x22fcac)
    at ../../src/gcc-9.2.0/gcc/pretty-print.c:1016
#1  0x0175bb77 in diagnostic_report_diagnostic (
    context=0x238bfa0 <global_diagnostic_context>, diagnostic=0x22fcac)
    at ../../src/gcc-9.2.0/gcc/diagnostic.c:1015
#2  0x0175bfe7 in diagnostic_impl (richloc=0x22fd00, opt=-1,
    gmsgid=0x1eee273 <trim_filename(char const*)::this_file+211> "in %s, at %s:%d", ap=0x22fd68, kind=DK_ICE)
    at ../../src/gcc-9.2.0/gcc/diagnostic.c:1159
#3  0x0175cc66 in internal_error (
    gmsgid=0x1eee273 <trim_filename(char const*)::this_file+211> "in %s, at %s:%d")
    at ../../src/gcc-9.2.0/gcc/diagnostic.c:1540
#4  0x0175cdbd in fancy_abort (
    file=0x19aa0dc <NUM_GCC_JIT_TYPES+64> "../../src/gcc-9.2.0/gcc/jit/jit-tempdir.c", line=54,
    function=0x19aa0be <NUM_GCC_JIT_TYPES+34> "make_tempdir_path_template")
    at ../../src/gcc-9.2.0/gcc/diagnostic.c:1607
#5  0x00459c9f in make_tempdir_path_template ()
    at ../../src/gcc-9.2.0/gcc/jit/jit-tempdir.c:54
#6  0x00459dc3 in gcc::jit::tempdir::create (this=0xd2084b0)
    at ../../src/gcc-9.2.0/gcc/jit/jit-tempdir.c:90
#7  0x004553f1 in gcc::jit::playback::context::compile (this=0x22fe9c)
    at ../../src/gcc-9.2.0/gcc/jit/jit-playback.c:1784
#8  0x004474da in gcc::jit::recording::context::compile (this=0xd207db0)
    at ../../src/gcc-9.2.0/gcc/jit/jit-recording.c:1379
#9  0x00443e32 in gcc_jit_context_compile (ctxt=0xd207db0)
    at ../../src/gcc-9.2.0/gcc/jit/libgccjit.c:2698
#10 0x0040163a in main (argc=1, argv=0xd201470)
    at /z/jit/tut01-hello-world.c:103
(gdb)
The segmentation fault is due to that pp = 0x0 argument in pp_format(): it is supposed to be a (non-NULL) pointer to a pretty_printer object, and is immediately dereferenced, to acquire a pointer to the associated output buffer. Since the pretty_printer object pointer is itself NULL, which it clearly isn't expected to be, the segmentation fault is inevitable. It appears that, wherever it should have been initialized, that pretty_printer object pointer has not been; I have no idea where that initialization should have occurred, nor how to fix it.

Where I *can* identify a clear defect is at the origin of the diagnostic call, at line 54 of jit-tempdir.c, where I see

gcc_assert (tmpdir_buf[tmpdir_len - 1] == DIR_SEPARATOR);
in which gcc_assert is a macro, invoking fancy_abort(), and where DIR_SEPARATOR has been explicitly defined as '/'. This follows assignment of tmpdir_buf, and tmpdir_len, within the code fragment:
/* The result of choose_tmpdir is a cached buffer within libiberty, so
   we must *not* free it.  */
tmpdir_buf = choose_tmpdir ();

/* choose_tmpdir aborts on malloc failure.  */
gcc_assert (tmpdir_buf);

tmpdir_len = strlen (tmpdir_buf);
/* tmpdir_buf should now have a dir separator as the final byte.  */
gcc_assert (tmpdir_len > 0);
gcc_assert (tmpdir_buf[tmpdir_len - 1] == DIR_SEPARATOR);
in which choose_tmpdir() returns a path with Windows-style '\\' directory name separators, and thus, the assertion tests an invalid assumption. This I can fix, but the potential for some other assertion triggering a segmentation fault would remain.

(Edited, 2021-01-06 09:01 Updated by: keith)
2021-01-08 03:41 Updated by: keith
評語

I filed GCC bug #98586, to report the segmentation fault.

2021-01-08 05:15 Updated by: keith
評語

Attached 27-libgccjit-invalid-assertion.patch corrects the invalid assumption, in the assertion which validates the final DIR_SEPARATOR of the temporary directory path name. With all patches applied, up to and including this one, I am able to successfully compile and link libgccjit-0.dll, and subsequently compile, link, and run the tut01-hello-world.exe example program.

I have posted the resulting libgccjit packages, to augment the GCC-9.2.0 collection in our FRS; currently published as hidden packages, you will need to access them via this direct link — they will not be visible, in the regular package list. Additionally, I have updated the source code tarball, to incorporate the additional patches, and configuration options, which are required to achieve this additional package build.

Please test.

2021-01-08 22:41 Updated by: eliz
評語

I've succeeded in compiling and linking the hello-world test program, but I cannot run it: it looks like libgccjit-0.dll depends on libintl-8.dll, and the version of libintl you used is different from what I have here: the libintl_setlocale entry point is missing from the version I have here. (Yes, my libintl is very old, but it was never a problem until now.)

Why is libgccjit need to depend on libintl anyway? gcc.exe and cc1.exe, for example, don't depend on it. Is that dependency really needed?

Thanks again for working on this.

2021-01-09 00:41 Updated by: eliz
評語

Btw, judging by the Info manual that comes with libgccjit, a more appropriate name for the DLL would be libgccjit-10.dll, since it implements LIBGCCJIT_ABI_10 (and the next versions of GCC advance the ABI by several more notches).

MinGW64 also calls it libgccjit-0.dll, so we could adopt the same name, regardless. But then we will need to invent ABI numbers out of thin air for future releases, which might be sub-optimal.

(Edited, 2021-01-09 00:41 Updated by: eliz)
2021-01-09 00:48 Updated by: davidmalcolm
評語

Reply To eliz

Btw, judging by the Info manual that comes with libgccjit, a more appropriate name for the DLL would be libgccjit-10.dll, since it implements LIBGCCJIT_ABI_10 (and the next versions of GCC advance the ABI by several more notches). MinGW64 also calls it libgccjit-0.dll, so we could adopt the same name, regardless. But then we will need to invent ABI numbers out of thin air for future releases, which might be sub-optimal.

Upstream notes on API/ABI compatibility: https://gcc.gnu.org/onlinedocs/jit/topics/compatibility.html

libgccjit hides all data layouts, and only ever adds entrypoints, so I've avoiding bumping the SONAME, using symbol versioning instead, to allow for backwards ABI compatibility. I'm not sure if that's going to work with mingw.

2021-01-09 01:05 Updated by: eliz
評語

Reply To davidmalcolm

Reply To eliz

Btw, judging by the Info manual that comes with libgccjit, a more appropriate name for the DLL would be libgccjit-10.dll, since it implements LIBGCCJIT_ABI_10 (and the next versions of GCC advance the ABI by several more notches). MinGW64 also calls it libgccjit-0.dll, so we could adopt the same name, regardless. But then we will need to invent ABI numbers out of thin air for future releases, which might be sub-optimal.

Upstream notes on API/ABI compatibility: https://gcc.gnu.org/onlinedocs/jit/topics/compatibility.html libgccjit hides all data layouts, and only ever adds entrypoints, so I've avoiding bumping the SONAME, using symbol versioning instead, to allow for backwards ABI compatibility. I'm not sure if that's going to work with mingw.

I'm not sure I understand the documentation to which you pointed: it seems to hint that detecting the ABI needs the end-user to have objdump installed? That's not gonna happen for users of MinGW programs.

FWIW, I don't see such symbols using the objdump command described in the docs. Which doesn't surprise me.

And in any case, how would detecting the required ABI help? all you can do when you detect an incompatible ABI version (e.g., a program linked with libgccjit whose ABI level is N, but finds only lower ABI level at run time) is refuse to run. Likewise, if you try running a program which imports a symbol from a DLL, but the DLL found at run time has no such symbol, the OS loader on Windows refuses to start the program and pops up a dialog showing the offending missing function(s). And linking the program on Windows against libgccjit-NN.dll ensures that cleanly and easily. So I don't really see how checking for those symbols would help. What am I missing?

(Edited, 2021-01-09 01:08 Updated by: eliz)
2021-01-09 01:54 Updated by: davidmalcolm
評語

Sorry about the confusion. The information I posted is more aimed at packagers and maintainers than end-users.

For example, on my RPM-based system, I can run:

$ rpm -q --provides libgccjit
libgccjit = 10.2.1-1.fc32
libgccjit(x86-64) = 10.2.1-1.fc32
libgccjit.so.0()(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_0)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_1)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_10)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_11)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_12)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_13)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_2)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_3)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_4)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_5)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_6)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_7)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_8)(64bit)
libgccjit.so.0(LIBGCCJIT_ABI_9)(64bit)
where the versions get detected by rpmbuild and "baked into" the package metadata. Similarly, packages that use libgccjit can record exactly which ABI they were built again in their package metadata, ensuring that RPM can enforce compatibility, whilst also allowing new API/ABI entrypoints to be added without bumping the SONAME, which would require everything to be recompiled.

This relies on the dynamic loader being able to express this information and enforce it, which is the case for glibc's ld.so.

It sounds like the windows dynamic linker either doesn't have these capabilities, or has different ones, and that I've assumed too much.

Does the windows dynamic linker have any such abilities, or do we simply need to express the ABI version in the dll name, or somesuch?

Hope this makes sense

2021-01-09 03:47 Updated by: eliz
評語

Reply To davidmalcolm

$ rpm -q --provides libgccjit libgccjit = 10.2.1-1.fc32 libgccjit(x86-64) = 10.2.1-1.fc32 libgccjit.so.0()(64bit) libgccjit.so.0(LIBGCCJIT_ABI_0)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_1)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_10)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_11)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_12)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_13)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_2)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_3)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_4)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_5)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_6)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_7)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_8)(64bit) libgccjit.so.0(LIBGCCJIT_ABI_9)(64bit) }}} where the versions get detected by rpmbuild and "baked into" the package metadata. Similarly, packages that use libgccjit can record exactly which ABI they were built again in their package metadata, ensuring that RPM can enforce compatibility, whilst also allowing new API/ABI entrypoints to be added without bumping the SONAME, which would require everything to be recompiled. This relies on the dynamic loader being able to express this information and enforce it, which is the case for glibc's ld.so. It sounds like the windows dynamic linker either doesn't have these capabilities, or has different ones, and that I've assumed too much. Does the windows dynamic linker have any such abilities, or do we simply need to express the ABI version in the dll name, or somesuch?

Like I said: the Windows program loader checks at load time that all the entry points the program imports are actually present in the DLL, and if not, it will refuse to run the program.

If all the entry points are present, but some have an incompatible ABI, then the loader will not be able to detect that. But this is largely unimportant in practice, since on Windows programs are linked against the versioned DLL names, and the name is recorded in the binary. And since symlinking a versioned DLL to an unversioned name is not practiced on Windows (because symlink support is problematic and requires privilege elevation), the problem of an incompatible ABI for the same entry points doesn't happen in practice, because the versioned DLL is being looked up explicitly.

2021-01-09 07:47 Updated by: keith
評語

Reply To eliz

Why is libgccjit need to depend on libintl anyway?

I guess, to provide internationalization of diagnostic messages.

gcc.exe and cc1.exe, for example, don't depend on it.

I'm puzzled, because that's not what I'm seeing here:

$ mingw32-ldd dist/staged/bin/gcc.exe 
dist/staged/bin/gcc.exe
 +- libiconv-2.dll
 |   +- KERNEL32.dll
 |   +- msvcrt.dll
 |   +- msvcrt.dll
 +- libintl-8.dll
 |   +- ADVAPI32.DLL
 |   +- KERNEL32.dll
 |   +- msvcrt.dll
 |   +- msvcrt.dll
 |   +- libgcc_s_dw2-1.dll
 |   |   +- KERNEL32.dll
 |   |   +- msvcrt.dll
 |   +- libiconv-2.dll
 |       +- KERNEL32.dll
 |       +- msvcrt.dll
 |       +- msvcrt.dll
 +- KERNEL32.dll
 +- msvcrt.dll
 +- msvcrt.dll
 +- USER32.dll
and:
$ mingw32-ldd `find dist/staged -name cc1.exe`
dist/staged/libexec/gcc/mingw32/9.2.0/cc1.exe
 +- libgmp-10.dll
 |   +- KERNEL32.dll
 |   +- msvcrt.dll
 |   +- msvcrt.dll
 |   +- libgcc_s_dw2-1.dll
 |       +- KERNEL32.dll
 |       +- msvcrt.dll
 +- libiconv-2.dll
 |   +- KERNEL32.dll
 |   +- msvcrt.dll
 |   +- msvcrt.dll
 +- libintl-8.dll
 |   +- ADVAPI32.DLL
 |   +- KERNEL32.dll
 |   +- msvcrt.dll
 |   +- msvcrt.dll
 |   +- libgcc_s_dw2-1.dll
 |   |   +- KERNEL32.dll
 |   |   +- msvcrt.dll
 |   +- libiconv-2.dll
 |       +- KERNEL32.dll
 |       +- msvcrt.dll
 |       +- msvcrt.dll
 +- libisl-21.dll
 |   +- KERNEL32.dll
 |   +- msvcrt.dll
 |   +- msvcrt.dll
 |   +- libgmp-10.dll
 |       +- KERNEL32.dll
 |       +- msvcrt.dll
 |       +- msvcrt.dll
 |       +- libgcc_s_dw2-1.dll
 |           +- KERNEL32.dll
 |           +- msvcrt.dll
 +- KERNEL32.dll
 +- msvcrt.dll
 +- libmpc-3.dll
 |   +- KERNEL32.dll
 |   +- msvcrt.dll
 |   +- libgmp-10.dll
 |   |   +- KERNEL32.dll
 |   |   +- msvcrt.dll
 |   |   +- msvcrt.dll
 |   |   +- libgcc_s_dw2-1.dll
 |   |       +- KERNEL32.dll
 |   |       +- msvcrt.dll
 |   +- libmpfr-6.dll
 |       +- libgmp-10.dll
 |       |   +- KERNEL32.dll
 |       |   +- msvcrt.dll
 |       |   +- msvcrt.dll
 |       |   +- libgcc_s_dw2-1.dll
 |       |       +- KERNEL32.dll
 |       |       +- msvcrt.dll
 |       +- KERNEL32.dll
 |       +- msvcrt.dll
 |       +- libgcc_s_dw2-1.dll
 |           +- KERNEL32.dll
 |           +- msvcrt.dll
 +- libmpfr-6.dll
 |   +- libgmp-10.dll
 |   |   +- KERNEL32.dll
 |   |   +- msvcrt.dll
 |   |   +- msvcrt.dll
 |   |   +- libgcc_s_dw2-1.dll
 |   |       +- KERNEL32.dll
 |   |       +- msvcrt.dll
 |   +- KERNEL32.dll
 |   +- msvcrt.dll
 |   +- libgcc_s_dw2-1.dll
 |       +- KERNEL32.dll
 |       +- msvcrt.dll
 +- msvcrt.dll
 +- USER32.dll
However, I do see that the version of gcc.exe in FRS is double the size of my staged copy, and appears to be statically linked with both libiconv.a and libintl.a, rather than with libiconv.dll.a and libintl.dll.a. I think I know why, but I'd like to review the issue; the DLL dependencies were intended.

2021-01-09 09:02 Updated by: keith
評語

Reply To eliz

Btw, judging by the Info manual that comes with libgccjit, a more appropriate name for the DLL would be libgccjit-10.dll, since it implements LIBGCCJIT_ABI_10 (and the next versions of GCC advance the ABI by several more notches). MinGW64 also calls it libgccjit-0.dll, so we could adopt the same name, regardless. But then we will need to invent ABI numbers out of thin air for future releases, which might be sub-optimal.

Our DLL version numbering scheme, like that also used by Cygwin, is based on the libtool interface version numbering conventions; like Cygwin, we compute the number, included within the DLL file name, as the libtool current version number minus the libtool age. In this case, current would appear to be 10, but, as David has indicated, ABI 10 is backwardly compatible with ABI 0, and every version between. Thus, age would also be 10, and the appropriate current - age result is, indeed, 0.

This stratagem was originally promoted by Charles (Chuck) Wilson, who contributed to both MinGW and Cygwin; his rationale was to ensure that any dependency on a specific version of a DLL could always be satisfied, through backward compatibility, by the most recent release of that specific version, but there was never any expectation, much less guarantee, that older versions of any DLL would remain forward compatible with newer applications.

2021-01-09 17:22 Updated by: eliz
評語

Reply To keith

Reply To eliz

Btw, judging by the Info manual that comes with libgccjit, a more appropriate name for the DLL would be libgccjit-10.dll, since it implements LIBGCCJIT_ABI_10 (and the next versions of GCC advance the ABI by several more notches). MinGW64 also calls it libgccjit-0.dll, so we could adopt the same name, regardless. But then we will need to invent ABI numbers out of thin air for future releases, which might be sub-optimal.

Our DLL version numbering scheme, like that also used by Cygwin, is based on the libtool interface version numbering conventions; like Cygwin, we compute the number, included within the DLL file name, as the libtool current version number minus the libtool age. In this case, current would appear to be 10, but, as David has indicated, ABI 10 is backwardly compatible with ABI 0, and every version between. Thus, age would also be 10, and the appropriate current - age result is, indeed, 0.

I'm aware of the numbering scheme, I just wasn't sure that libtool was involved in the build and applied the scheme. From your description I deduced, perhaps incorrectly, that building libgccjit for MinGW was not yet supported, and that led me to the conclusion that perhaps the 0 part was some kind of default, not a number correctly calculated from the ABI data.

It is also a certain surprise for me to read that adding entry points is still considered to be a "compatible" ABI. I always thought that if the DLL name remains the same, then any program linked against any version with that name will be able to run with any other version of that DLL that has the same name. I guess there's something new to learn every day...

Is libgccjit indeed fully backward compatible, btw?

2021-01-09 18:28 Updated by: eliz
評語

Reply To keith

Reply To eliz

Why is libgccjit need to depend on libintl anyway?

However, I do see that the version of gcc.exe in FRS is double the size of my staged copy, and appears to be statically linked with both libiconv.a and libintl.a, rather than with libiconv.dll.a and libintl.dll.a. I think I know why, but I'd like to review the issue; the DLL dependencies were intended.

I think statically linking against libintl is a Good Thing, especially since, as we see, the newer libintl-8.dll is not 100% compatible, as it causes the program to use libintl's version of setlocale. If using that entry point could be avoided, then a dynamic dependency on libintl wouldn't be a problem, I think (or rather hope). Otherwise, it's a bit of DLL hell, given the same name of the DLL.

2021-01-10 21:40 Updated by: keith
評語

Reply To eliz

Reply To keith

Our DLL version numbering scheme, like that also used by Cygwin, is based on the libtool interface version numbering conventions; like Cygwin, we compute the number, included within the DLL file name, as the libtool current version number minus the libtool age. In this case, current would appear to be 10, but, as David has indicated, ABI 10 is backwardly compatible with ABI 0, and every version between. Thus, age would also be 10, and the appropriate current - age result is, indeed, 0.

I'm aware of the numbering scheme, I just wasn't sure that libtool was involved in the build and applied the scheme. From your description I deduced, perhaps incorrectly, that building libgccjit for MinGW was not yet supported, and that led me to the conclusion that perhaps the 0 part was some kind of default, not a number correctly calculated from the ABI data.

In the case of libgccjit, to get infrastructure appropriate to MinGW, I did have to patch the build system. Libtool isn't used, (and wasn't for the original ELF configuration either). However, that does not prevent us from adopting the libtool versioning convention; I simply deduced, from the original soname attributes, that the appropriate current - age value would be zero, and assigned it manually.

It is also a certain surprise for me to read that adding entry points is still considered to be a "compatible" ABI. I always thought that if the DLL name remains the same, then any program linked against any version with that name will be able to run with any other version of that DLL that has the same name. I guess there's something new to learn every day...

Depends on how you define "compatible". In the MinGW ecosystem, (as it is in Cygwin's), the version number encoded within the DLL name, (equivalent to current - age in libtool parlance), represents the oldest current version of the ABI, with which a particular release of the DLL remains backwardly compatible. However, there is absolutely no guarantee of forward compatibility; the most recent release of any particularly named version of a given DLL should be fully compatible with any application requiring (and possibly working with an older release of) a similarly named DLL, but older releases of that DLL may not support a newer application.

The original reference no longer appears to be accessible, but this Wayback Machine archived reference may help to explain it.

Is libgccjit indeed fully backward compatible, btw?

I don't know for sure, but on the basis of David Malcolm's statement, (and, AIUI, he is the developer and maintainer of libgccjit), I must assume that it is.

(Edited, 2021-01-11 07:06 Updated by: keith)
2021-01-10 22:03 Updated by: keith
評語

Reply To eliz

I think statically linking against libintl is a Good Thing,

You are fully entitled to that opinion, but please allow me to disagree...

especially since, as we see, the newer libintl-8.dll is not 100% compatible, as it causes the program to use libintl's version of setlocale. If using that entry point could be avoided, then a dynamic dependency on libintl wouldn't be a problem, I think (or rather hope). Otherwise, it's a bit of DLL hell, given the same name of the DLL.

Are you saying that you have applications which depend on your older release of libintl-8.dll, and which will not work with MinGW's latest libintl-8.dll release? If that is indeed the case, then either your libintl-8.dll, (which I assume you built yourself), may be incompatible with any MinGW release, or GNU gettext has introduced an incompatible change, upstream, without making the necessary adjustment in the libtool versioning attributes; in the latter case, that should be considered to be an upstream bug, and should be reported as such. (Do, please, remember that the MinGW/Cygwin DLL versioning conventions promise only backward compatibility for older applications; there is no guarantee that an outdated DLL release will remain forward compatible with newer applications).

2021-01-11 02:40 Updated by: eliz
評語

Reply To keith

Reply To eliz

I think statically linking against libintl is a Good Thing,

You are fully entitled to that opinion, but please allow me to disagree...

Can you tell why you disagree?

Are you saying that you have applications which depend on your older release of libintl-8.dll, and which will not work with MinGW's latest libintl-8.dll release?

No, that's not what I'm saying. I said "a bit of DLL hell" because I'll have to cope with 2 different DLLs with the same name.

2021-01-11 06:56 Updated by: keith
評語

Reply To eliz

Reply To keith

Reply To eliz

I think statically linking against libintl is a Good Thing,

You are fully entitled to that opinion, but please allow me to disagree...

Can you tell why you disagree?

We have umpteen different packages — each with a fairly significant build time — and each individually dependent on libintl. If a bug is identified, in libintl, and each of those umpteen dependent packages is statically linked with libintl.a, then when fixing the bug, I have to rebuild every one of those umpteen dependent packages. If they are linked with libintl-8.dll, then I have only to fix the bug, and deploy an updated libintl-8.dll, and all of the umpteen dependent packages inherit the fix, without further effort.

Additionally, although possibly a lesser advantage, but an advantage nonetheless: linking with shared libraries reduces package size, consequently reducing disk usage, and upload/download bandwidth.

Are you saying that you have applications which depend on your older release of libintl-8.dll, and which will not work with MinGW's latest libintl-8.dll release?

No, that's not what I'm saying. I said "a bit of DLL hell" because I'll have to cope with 2 different DLLs with the same name.

Why? You should need only the most recent release of libintl-8.dll, (whether yours or mine), and all of your dependent applications should be happy; no need for your own (older) version. Besides, if static linking is such a "good thing", why don't you just link your own applications statically? If you did this, why was your libintl-8.dll ever needed anyway?

Forgive me, but I simply don't understand your reasoning here.

2021-01-12 00:56 Updated by: eliz
評語

Reply To keith

Reply To eliz

Reply To keith

Reply To eliz

I think statically linking against libintl is a Good Thing,

Are you saying that you have applications which depend on your older release of libintl-8.dll, and which will not work with MinGW's latest libintl-8.dll release?

No, that's not what I'm saying. I said "a bit of DLL hell" because I'll have to cope with 2 different DLLs with the same name.

Why? You should need only the most recent release of libintl-8.dll, (whether yours or mine), and all of your dependent applications should be happy; no need for your own (older) version. Besides, if static linking is such a "good thing", why don't you just link your own applications statically? If you did this, why was your libintl-8.dll ever needed anyway? Forgive me, but I simply don't understand your reasoning here.

I've seen libintl-8.dll versions that caused crashes on program exit, so I'm trying to use only the ones I trust. Maybe the one we have now on the MinGW site can also be trusted, but I will need to collect that experience from scratch.

IOW, I simply value my extremely stable system, and would like it to remain stable. So I try to avoid unnecessary changes to the basics, and libintl is part of that.

I hope this explains my reasoning.

2021-01-21 00:37 Updated by: davidmalcolm
評語

Keith, thanks for all the porting work you've done.

I've now looked through the patches themselves, and suddenly realized that they conflict in places with

by Nicolas Bértolo which is on "master" for gcc 11, but not on the branches for older releases. It looks like this might do much of what your patches do. Is this patch in the code that you're working with? What GCC version are you using? Should this be backported; if so, which version?

I'm sorry for not noticing this earlier in this discussion (in my defence it was back in May, and 2020 has been a difficult year).

That said, your patches do have additional changes that appear to be necessary (in particular the mutex one). I wonder why Nicholas got it working, but you needed additional patches. Maybe he merely fixed the build, not the actually execution?

Do you have a copyright assignment in place with the FSF? According to

the FSF considers that for a patch or collection of patches that touch more than around 15 lines of code and/or text that is legally significant for copyright purposes, the FSF needs copyright papers for that contribution, which seems to be the case here (e.g. just the mutex one is > 15). Let me know if you want my help with this, or if this is an issue.

Looking at the patches themselves:

20-libiberty-mingw-host-shared.patch(971bytes)

Patch to avoid bogus libiberty/pic path references

As far as I can tell, Nicholas made similar changes as part of c83027f32d9cca84959c7d6a1e519a0129731501.

21-libgccjit-win32-fchmod.patch(1012bytes)

Patch to correct for gratuitous misuse of fchmod()

This patch conflicts with https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=c83027f32d9cca84959c7d6a1e519a0129731501 by Nicolas Bértolo which is on "master" for gcc 11, but not on the branches for older releases. Looking at it, I think he merely hacked out the call on Windows, which may lead to a run-time failure: on WIndows, is a DLL required to be executable for it to be linkable into the process? I also wondered if there's a possible security issue here with closing the file and changing its permissions by name, but the DLL is created within a tempdir that only the user should have access to, with a random name.

22-libgccjit-win32-dlfcn.patch(532bytes)

Patch to circumvent gratuitous references to undeclared <dlfcn.h> functions

Seems reasonable, though I wonder why Nicolas Bértolo didn't run into this.

23-libgccjit-win32-no-pthreads.patch(2KB)

Patch to eliminate -lpthread dependency
> @@ -2138,12 +2173,12 @@ void
>  playback::context::release_mutex ()
>  {
>    /* Release the big GCC mutex. */
>    JIT_LOG_SCOPE (get_logger ());
>    gcc_assert (active_playback_ctxt == this);
> +  JIT_CONTEXT_RELEASE_LOCK (jit_mutex);
>    active_playback_ctxt = NULL;
> -  pthread_mutex_unlock (&jit_mutex);
>  }

Usage of "active_playback_ctxt" must be guarded by the mutex, so the JIT_CONTEXT_RELEASE_LOCK needs to be after the

active_playback_ctxt = NULL;

i.e. where the pthread_mutex_unlock call is.

Looking at the jit_mutex_initializer and the docs for https://docs.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-initializecriticalsection I see that InitializeCriticalSection needs to be called exactly once on a CRITICAL_SECTION. I see that the code does this via "static jit_mutex;" Presumably this leads to jit_mutex's ctor being called once when the DLL is loaded. Does this work correctly? (I'm always a little nervous about global ctors; older toolchains don't always implement it reliably)

24-libgccjit-full-driver-name.patch(616bytes)

Correction for omitted $(exeext) in Makefile

Nicholas made the same change as part of c83027f32d9cca84959c7d6a1e519a0129731501.

25-libgccjit-mingw-link-options.patch(3KB)

Use linker options appropriate for Windows

As far as I can tell, Nicholas made similar changes as part of c83027f32d9cca84959c7d6a1e519a0129731501.

26-libgccjit-mingw-host-shared.patch(2KB)

Patch to override --enable-host-shared requirement

As far as I can tell, Nicholas made similar changes as part of c83027f32d9cca84959c7d6a1e519a0129731501.

27-libgccjit-invalid-assertion.patch(1KB)

Patch to correct invalid assumption in DIR_SEPARATOR verification assertion

Nicholas completely rewrote this code for Windows as part of c83027f32d9cca84959c7d6a1e519a0129731501. Sorry again for not realizing it earlier. I'd appreciate it if you could look over gcc/jit/jit-w32.c in his patch to see if you know of a better way to securely create a temporary directory on Windows (or, indeed, if there are other issues).

Thanks again for your work on this, and again, sorry for taking so long to spot the duplication.

2021-01-21 01:39 Updated by: eliz
  • Details Updated
評語

Reply To davidmalcolm

Keith, thanks for all the porting work you've done. I've now looked through the patches themselves, and suddenly realized that they conflict in places with https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=c83027f32d9cca84959c7d6a1e519a0129731501 by Nicolas Bértolo which is on "master" for gcc 11, but not on the branches for older releases. It looks like this might do much of what your patches do. Is this patch in the code that you're working with? What GCC version are you using? Should this be backported; if so, which version?

I'm not Keith, but let me say one thing that could answer several questions you ask: Nicolas was using MinGW64, a different flavor of GCC and runtime environment ported to native MS-Windows.

That said, your patches do have additional changes that appear to be necessary (in particular the mutex one). I wonder why Nicholas got it working, but you needed additional patches. Maybe he merely fixed the build, not the actually execution?

I believe MinGW64 uses pthreads as an integral part of their GCC port, so Nicolas didn't need this. IMNSHO, it is better to avoid gratuitous dependencies on pthreads, so I think what Keith did here is cleaner. However, if the MinGW64 port must use pthreads, then I guess both varieties will need to be supported.

22-libgccjit-win32-dlfcn.patch(532bytes) Patch to circumvent gratuitous references to undeclared <dlfcn.h> functions Seems reasonable, though I wonder why Nicolas Bértolo didn't run into this.

AFAIK, MinGW64 doesn't have the dlfcn.h header and the related emulations of Posix functions, while mingw.org's MinGW does have that.

2021-01-28 04:42 Updated by: keith
評語

Reply To davidmalcolm

I've now looked through the patches themselves, and suddenly realized that they conflict in places with https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=c83027f32d9cca84959c7d6a1e519a0129731501 by Nicolas Bértolo which is on "master" for gcc 11, but not on the branches for older releases. It looks like this might do much of what your patches do.

Quite possibly. As Eli has already pointed out, Nicolas is focussed on work for MINGW64 — actually the trademark-infringing mingw-w64 project — whereas my efforts are on behalf of the legitimate owner of the MinGW trademark. I guess it's inevitable that we would come up with some very similar changes. In any case, I'm not at all enthusiastic about the way in which Nicolas has bundled everything into just one patch; I much prefer a coherent series of granular patches, (which I can manage with Mercurial Queues), so that I can more readily adapt to variations between releases, (and to conflicts with other contributors' patches).

Is this patch in the code that you're working with?

No, I don't think so.

What GCC version are you using?

As I mentioned, in an earlier comment, the last GCC release I've published for MinGW is GCC-9.2.0; I've simply been tweaking my existing build tree for that, to explore the feasibility of incorporating libgccjit support, before I move on to GCC-10. (I tend to match my build cycles, as my available time permits, to the native compiler version on my Manjaro-Linux build machine; this is currently GCC-10.2).

Should this be backported; if so, which version?

I already have a significant collection of local patches — 28 currently, with 3 no longer required — which I apply for MinGW releases; I'm quite happy to continue with this publication strategy, so, while backporting may be useful, it's by no means imperative. I will not be looking at anything earlier than GCC-9.2.0, as a MinGW libgccjit candidate.

I'm sorry for not noticing this earlier in this discussion (in my defence it was back in May, and 2020 has been a difficult year).

Don't we all know it :-)

That said, your patches do have additional changes that appear to be necessary (in particular the mutex one). I wonder why Nicholas got it working, but you needed additional patches. Maybe he merely fixed the build, not the actually execution?

I can't speak for anything emanating from the mingw-w64 folks — they are a law on to themselves. Perhaps Nicolas was happy to introduce a dependency on a third-party (their own?) implementation of POSIX threads for Windows. I could have done likewise, but I much prefer the native solution.

Do you have a copyright assignment in place with the FSF?

I do, in respect of my contributions to the GNU troff (groff) project, but not specific to GCC.

Attachment File List

編輯

Please login to add comment to this ticket » 登入