autoPatchelfHook: Correctly detect PIE binaries

I originally thought it would just be enough to just check for an INTERP
section in isExecutable, however this would mean that we don't detect
statically linked ELF files, which would break our recent improvement to
gracefully handle those.

In theory, we are only interested in ELF files that have an INTERP
section, so checking for INTERP would be enough. Unfortunately the
isExecutable function is already used outside of autoPatchelfHook, so we
can't easily get rid of it now, so let's actually strive for more
correctness and make isExecutable actually match ELF files that are
executable.

So what we're doing instead now is to check whether either the ELF type
is EXEC *or* we have an INTERP section and if one of them is true we
should have an ELF executable, even if it's statically linked.

Along the way I also set LANG=C for the invocations of readelf, just to
be sure we don't get locale-dependent output.

Tested this with the following command (which contains almost[1] all the
packages using autoPatchelfHook), checking whether we run into any
library-related errors:

  nix-build -E 'with import ./. { config.allowUnfree = true; };
    runCommand "test-executables" {
      drvs = [
        anydesk cups-kyodialog3 elasticsearch franz gurobi
        masterpdfeditor oracle-instantclient powershell reaper
        sourcetrail teamviewer unixODBCDrivers.msodbcsql17 virtlyst
        vk-messenger wavebox zoom-us
      ];
    } ("for i in $drvs; do for b in $i/bin/*; do " +
       "[ -x \"$b\" ] && timeout 10 \"$b\" || :; done; done")
  '

Apart from testing against library-related errors I also compared the
resulting store paths against the ones prior to this commit. Only
anydesk and virtlyst had the same as they didn't have self-references,
everything else differed only because of self-references, except
elasticsearch, which had the following PIE binaries:

  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/autoconfig
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/autodetect
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/categorize
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/normalize

These binaries were now patched, which is what this commit is all about.

[1]: I didn't include the "maxx" package (MaXX Interactive Desktop)
     because the upstream URLs are no longer existing and I couldn't
     find them elsewhere on the web.

Signed-off-by: aszlig <aszlig@nix.build>
Fixes: https://github.com/NixOS/nixpkgs/issues/48330
Cc: @gnidorah (for MaXX Interactive Desktop)

1 files changed, 11 insertions, 2 deletions

diff --git a/pkgs/build-support/setup-hooks/auto-patchelf.sh b/pkgs/build-support/setup-hooks/auto-patchelf.sh
index 7c165627f72..d1ae317ff9a 100644
--- a/pkgs/build-support/setup-hooks/auto-patchelf.sh
+++ b/pkgs/build-support/setup-hooks/auto-patchelf.sh
@@ -7,7 +7,16 @@ gatherLibraries() {
 addEnvHooks "$targetOffset" gatherLibraries
 
 isExecutable() {
-    readelf -h "$1" 2> /dev/null | grep -q '^ *Type: *EXEC\>'
+    # For dynamically linked ELF files it would be enough to check just for the
+    # INTERP section. However, we won't catch statically linked executables as
+    # they only have an ELF type of EXEC but no INTERP.
+    #
+    # So what we do here is just check whether *either* the ELF type is EXEC
+    # *or* there is an INTERP section. This also catches position-independent
+    # executables, as they typically have an INTERP section but their ELF type
+    # is DYN.
+    LANG=C readelf -h -l "$1" 2> /dev/null \
+        | grep -q '^ *Type: *EXEC\>\|^ *INTERP\>'
 }
 
 # We cache dependencies so that we don't need to search through all of them on
@@ -157,7 +166,7 @@ autoPatchelf() {
       isELF "$file" || continue
       if isExecutable "$file"; then
           # Skip if the executable is statically linked.
-          readelf -l "$file" | grep -q "^ *INTERP\\>" || continue
+          LANG=C readelf -l "$file" | grep -q "^ *INTERP\\>" || continue
       fi
       autoPatchelfFile "$file"
     done < <(find "$prefix" -type f -print0)