AOSP build on ARM for ARM 3일차

jiwon

1년 ago

한 1주일 동안 VM을 꺼두고 있었는데, 이제 남은 돈은 아래와 같네요. 그래프를 보니, 빌드를 한번 시도 하고 이것저것 패키지들을 더 깔아서 인지, 하루 당 소모 비용 그래프가 좀 더 가팔라졌습니다. 아무래도 디스크 쓰는 게 거의 고정비용 형태로 나가는 것 같습니다.. 근데 1TB를 다 쓴 것도 아닌데, 1주일 정도에 몇 만원 수준이면(하루에 6천 원 조금 넘게 나가네요. 커피 한 잔씩 뿅~)… 일개 개인이 쓰기에는 확실히 비싸 보입니다. 그래서 오늘 남아 있는 금액은 160,913 크레딧… 이번 추석 연휴에는 프로젝트를 끝내야지 그러지 못하면, 그냥.. 좋은 경험(?)한 걸로 적당히 끝날 것 같네요..

이제좀 익숙해져서, VM을 가동시키도, apt upgrade도 해주고, aosp repo sync해주고(사실 안 하는 게 더 좋지만… 혹시라도 누군가 보고 해준 걸 넣어 줄 까봐… ^___^). 지난번은 lunch에서 ckati에러까지만 봤습니다. 오늘은 이걸 파서, 넘어가보도록 하겠습니다.

Pick from common choices above (e.g. 13) or specify your own (e.g. aosp_barbet-eng): aosp_arm64-eng
11:36:19 Build sandboxing disabled due to nsjail error.
11:36:19 Failed to run dumpvars: fork/exec prebuilts/build-tools/linux-x86/bin/ckati: exec format error

역시 구글에 관한 건 구글에 물어보면, 아래처럼 결과가 나옵니다. GNU Make의 대체품이라고 하네요. 빠르게 받아서 빌드해봅시다. 그냥 ARM에서 받아서 하려고 합니다. 굳이 X86에서 ARM으로 Cross Compile하지 않아도, 왠만한 복잡한 ARM명령들을 쓰지 않았다면 왠만해서는 될꺼에요.. 왠만하면… –;
https://github.com/google/kati

그리고 전에 prebuilts/build-tools를 자동으로 빌드하는 부분도 동시에 확인해서 진행해봅니다. 이쪽은 ninja가 타겟이 되겠습니다.
https://ninja-build.org/

prebuilts/build-tools# OUT_DIR=/mnt/bsp/out ./build-prebuilts.sh –armonarm
11:50:32 Build sandboxing disabled due to nsjail error.
11:50:32 Failed to run soong bootstrap: fork/exec prebuilts/build-tools/linux-x86/bin/ninja: exec format error

kati가 알아보기 조금 편하므로, 먼저 git clone해서 무작정 빌드를 해봅니다. 복잡하지는 않게 빌드는 끝나는데, 결과물을 확인해봅시다.

/prebuilts/build-tools/linux-arm64/src$ git clone https://github.com/google/kati.git
ID@AOSPonARM:/mnt/bsp/prebuilts/build-tools/linux-arm64/src/kati$ make ckati
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/affinity.o src/affinity.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/command.o src/command.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o
out/dep.o src/dep.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/eval.o src/eval.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/exec.o src/exec.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/expr.o src/expr.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/file.o src/file.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/file_cache.o src/file_cache.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/fileutil.o src/fileutil.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/find.o src/find.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/flags.o src/flags.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/func.o src/func.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/io.o src/io.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/log.o src/log.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/main.o src/main.cc
src/main.cc: In member function ‘void SegfaultHandler::dumpstr(const char) const’: src/main.cc:126:16: warning: ignoring return value of ‘ssize_t write(int, const void, size_t)’, declared with attribute warn_unused_result [-Wunused-result]
126 | (void)write(STDERR_FILENO, s, strlen(s));
| ~^~~~~~~~~
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/ninja.o src/ninja.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/parser.o src/parser.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/regen.o src/regen.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/rule.o src/rule.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/stats.o src/stats.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/stmt.o src/stmt.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/string_piece.o src/string_piece.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/stringprintf.o src/stringprintf.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/strutil.o src/strutil.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/symtab.o src/symtab.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/timeutil.o src/timeutil.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/var.o src/var.cc
echo ‘// +build ignore’ > out/version.cc
echo >> out/version.cc
echo ‘const char* kGitVersion = “09dfa26c4e59b15919aaad6986f9b47c883dc4f1”;’ >> out/version.cc
g++ -c -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o out/version.o out/version.cc
g++ -std=c++17 -g -W -Wall -MMD -MP -O -DNOLOG -march=native -o ckati out/affinity.o out/command.o out/dep.o out/eval.o out/exec.o out/expr.o out/file.o out/file_cache.o out/fileutil.o out/find.o out/flags.o out/func.o out/io.o out/log.o out/main.o out/ninja.o out/parser.o out/regen.o out/rule.o out/stats.o out/stmt.o out/string_piece.o out/stringprintf.o out/strutil.o out/symtab.o out/timeutil.o out/var.o out/version.o -lrt -lpthread
build completed successfully (36 seconds)

결과물 파일들도 일단 모두 ELF ARM aarch64로 빌드는 되었습니다.. 이제.. ckati가 어디 있나만 찾으면 되겠네요…

ID@AOSPonARM:/mnt/bsp/prebuilts/build-tools/linux-arm64/src/kati/out$ file *.o
affinity.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
command.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
dep.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (GNU/Linux), with debug_info, not stripped
eval.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (GNU/Linux), with debug_info, not stripped
exec.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
expr.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
file.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
file_cache.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
fileutil.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (GNU/Linux), with debug_info, not stripped
find.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
flags.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
func.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
io.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
log.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
main.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
ninja.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
parser.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
regen.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (GNU/Linux), with debug_info, not stripped
rule.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
stats.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
stmt.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
string_piece.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
stringprintf.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
strutil.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
symtab.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
timeutil.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
var.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
version.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped

그러던 와중에,, 그냥 ckati를 실행해보니. Debian Packag가 있네요.. 그냥 설치합시다. –;;

AOSPonARM:/mnt/bsp/prebuilts/build-tools/linux-arm64/src/kati/out$ sudo apt install ckati
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following NEW packages will be installed:
ckati
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 137 kB of archives.
After this operation, 399 kB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports focal/universe arm64 ckati arm64 9.0.0+r45-1build1 [137 kB]
Fetched 137 kB in 0s (1677 kB/s)
Selecting previously unselected package ckati.
(Reading database … 109331 files and directories currently installed.)
Preparing to unpack …/ckati_9.0.0+r45-1build1_arm64.deb …
Unpacking ckati (9.0.0+r45-1build1) …
Setting up ckati (9.0.0+r45-1build1) …
ID@AOSPonARM:/mnt/bsp/prebuilts/build-tools/linux-arm64/src/kati/out$

ninja도 있습니다. 설치 설치

AOSPonARM:/mnt/bsp/build$ sudo apt install ninja-build
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following NEW packages will be installed:
ninja-build
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 101 kB of archives.
After this operation, 322 kB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports focal/universe arm64 ninja-build arm64 1.10.0-1build1 [101 kB]
Fetched 101 kB in 0s (1580 kB/s)
Selecting previously unselected package ninja-build.
(Reading database … 109335 files and directories currently installed.)
Preparing to unpack …/ninja-build_1.10.0-1build1_arm64.deb …
Unpacking ninja-build (1.10.0-1build1) …
Setting up ninja-build (1.10.0-1build1) …
Processing triggers for man-db (2.9.1-1) …
ID@AOSPonARM:/mnt/bsp/build$

/usr/bin에 이제 필요한 ckati와 ninja는 ARM버전으로 존재하고 이제 AOSP빌드 환경에서 알아들을 수 있게 고쳐줘야겠네요.

ID@AOSPonARM:/mnt/bsp/build$ file /usr/bin/ninja
/usr/bin/ninja: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=e56feac78164a7baf11303dbd156ce6a7d545eb0, for GNU/Linux 3.7.0, stripped
ID@AOSPonARM:/mnt/bsp/build$ file /usr/bin/ckati
/usr/bin/ckati: ELF 64-bit LSB shared object, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=88a2e925c0c180176e82daa451820115e0534a6f, for GNU/Linux 3.7.0, stripped
ID@AOSPonARM:/mnt/bsp/build$

여전히 prebuilts/build-tools/linux-x86에서 툴 들을 찾으려 하여, /build아래의 여러가지를 살펴봅니다. linux-x86으로 검색 후 의심 가는 것들을 나열해보면, build/make/envsetup.sh, build/soong/soong_ui.bash 등이 보이고, envsetup.sh에서 PREBUILT_NAME으로 단순 linux만 확인해서 x86으로 가정하고 지정을 하는 부분을 일단 바꿔줘 봅니다.. 이제 linux-arm64에 다가 몽땅 몰아넣어야겠네요.

찾아볼게 기하급수로 늘어나네요.. 아래 정도가 아마 고정 path로 잡혀 있는 것 같으니. 수정..
build/soong/build_kzip.bash:43:declare -r go_extractor=$(realpath prebuilts/build-tools/linux-x86/bin/go_extractor) build/soong/scripts/reverse-deps.sh:96:ninja_bin=”${ANDROID_BUILD_TOP}/prebuilts/build-tools/linux-x86/bin/ninja” build/soong/scripts/setup-android-build.sh:65:”$source/prebuilts/build-tools/linux-x86/bin/nsjail”\ build/soong/scripts/transitive-deps.sh:176:readonly ninja_bin=”${ANDROID_BUILD_TOP}/prebuilts/build-tools/linux-x86/bin/ninja” build/soong/ui/build/sandbox_linux.go:51:const nsjailPath = “prebuilts/build-tools/linux-x86/bin/nsjail”

https://github.com/google/nsjail

이거 빌드하려면 protocol buffer를 설치해야 되겠네요.
No package ‘protobuf’ found g++ -O2 -c -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -fPIE -Wformat -Wformat-security -Wno-format-nonliteral -Wall -Wextra -Werror -Ikafel/include -std=c++14 -fno-exceptions -Wno-unused -Wno-unused-parameter cmdline.cc -o cmdline.o protoc –cpp_out=. config.proto make: protoc: Command not found make: *** [Makefile:86: config.pb.cc] Error 127

근데. WSL이라서인지 UBUNTU 20.04.5LTS 에서 바뀐건지.. snap으로 설치하고 옵션을 주라고 뜹니다. 근데 이건 꽝. 설치하지 맙시다.

————– 여기부터
AOSPonARM:/mnt/bsp/prebuilts/build-tools/linux-arm64/src/nsjail/nsjail$ sudo apt install protobuf
Reading package lists… Done
Building dependency tree
Reading state information… Done

No apt package “protobuf”, but there is a snap with that name.
Try “snap install protobuf”

E: Unable to locate package protobuf AOSPonARM:/mnt/bsp/prebuilts/build-tools/linux-arm64/src/nsjail/nsjail$ sudo snap install protobuf error: This revision of snap "protobuf" was published using classic confinement and thus may perform arbitrary system changes outside of the security sandbox that snaps are usually confined to, which may put your system at risk.

If you understand and want to proceed repeat the command including --classic.

AOSPonARM:/mnt/bsp/prebuilts/build-tools/linux-arm64/src/nsjail/nsjail$ sudo snap install protobuf –classic
protobuf 3.14.0 from Stuart Bishop (stub) installed
————— 여기까지… 일단 무시하셔도 좋습니다.

https://groups.google.com/g/open-lighting/c/gt-iuxiwWRU?pli=1 이 글을 보고 더 설치합니다. sudo apt install protobuf-compiler python-protobuf libprotobuf-dev

nsjail build 시에 에러 나는 상황이네요. 뭔가 할게 무지 많은데, 순서 정하기가 쉽지 않습니다. 이 글도 정리하면서 조각 순서 잡고 Step 별로 글을 만들어야 할 것 같은데. 일단 성공적인 step까지 한번 가본 뒤에 정리하겠습니다.

In file included from config.pb.cc:4:
config.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory
10 | #include
| ^~~~~~~~ compilation terminated. make: *** [Makefile:60: config.pb.o] Error 1 make: *** Waiting for unfinished jobs…. net.cc:54:10: fatal error: netlink/route/link.h: No such file or directory 54 | #include | ^~~~~~~~
compilation terminated.
make: *** [Makefile:60: net.o] Error 1
failed to build some targets (1 seconds)\

솔루션은 여기에 따라서. https://github.com/BVLC/caffe/issues/6774
make clean후에 해보기 –;;; 정말이네요.. 이제 뭔가 쉽게 잡힐만한 에러들이 보입니다.

net.cc:54:10: fatal error: netlink/route/link.h: No such file or directory
54 | #include
| ^~~~~~~~~~
compilation terminated.
make: *** [Makefile:60: net.o] Error 1
make: *** Waiting for unfinished jobs….

AOSPonARM:/mnt/bsp/prebuilts/build-tools/linux-arm64/src/nsjail/nsjail$ sudo apt install libnl-3-dev
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following additional packages will be installed:
libnl-3-200
The following NEW packages will be installed:
libnl-3-200 libnl-3-dev
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 145 kB of archives.
After this operation, 821 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 libnl-3-200 arm64 3.4.0-1 [51.5 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 libnl-3-dev arm64 3.4.0-1 [93.7 kB]
Fetched 145 kB in 0s (2068 kB/s)
Selecting previously unselected package libnl-3-200:arm64.
(Reading database … 110421 files and directories currently installed.)
Preparing to unpack …/libnl-3-200_3.4.0-1_arm64.deb …
Unpacking libnl-3-200:arm64 (3.4.0-1) …
Selecting previously unselected package libnl-3-dev:arm64.
Preparing to unpack …/libnl-3-dev_3.4.0-1_arm64.deb …
Unpacking libnl-3-dev:arm64 (3.4.0-1) …
Setting up libnl-3-200:arm64 (3.4.0-1) …
Setting up libnl-3-dev:arm64 (3.4.0-1) …
Processing triggers for libc-bin (2.31-0ubuntu9.9) …]

Makefile을 보니 아래가 맞는 모양입니다.

jiwon@AOSPonARM:/mnt/bsp/prebuilts/build-tools/linux-arm64/src/nsjail/nsjail$ sudo apt install libnl-route-3-dev
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following additional packages will be installed:
libnl-route-3-200
The following NEW packages will be installed:
libnl-route-3-200 libnl-route-3-dev
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 304 kB of archives.
After this operation, 1666 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 libnl-route-3-200 arm64 3.4.0-1 [138 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 libnl-route-3-dev arm64 3.4.0-1 [166 kB]
Fetched 304 kB in 0s (3732 kB/s)
Selecting previously unselected package libnl-route-3-200:arm64.
(Reading database … 110581 files and directories currently installed.)
Preparing to unpack …/libnl-route-3-200_3.4.0-1_arm64.deb …
Unpacking libnl-route-3-200:arm64 (3.4.0-1) …
Selecting previously unselected package libnl-route-3-dev:arm64.
Preparing to unpack …/libnl-route-3-dev_3.4.0-1_arm64.deb …
Unpacking libnl-route-3-dev:arm64 (3.4.0-1) …
Setting up libnl-route-3-200:arm64 (3.4.0-1) …
Setting up libnl-route-3-dev:arm64 (3.4.0-1) …
Processing triggers for libc-bin (2.31-0ubuntu9.9) …

이제 빌드 성공

AOSPonARM:/mnt/bsp/prebuilts/build-tools/linux-arm64/src/nsjail/nsjail$ make -j4
CFLAGS=-fPIE make -C kafel
make[1]: Entering directory ‘/mnt/bsp/prebuilts/build-tools/linux-arm64/src/nsjail/nsjail/kafel’
Makefile:27: warning: overriding recipe for target ‘test’
build/Makefile.mk:41: warning: ignoring old recipe for target ‘test’
make -C src PROJECT_ROOT=../
g++ -O2 -c -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -fPIE -Wformat -Wformat-security -Wno-format-nonliteral -Wall -Wextra -Werror -Ikafel/include -pthread -std=c++14 -fno-exceptions -Wno-unused -Wno-unused-parameter -I/usr/include/libnl3 config.cc -o config.o
make[2]: Entering directory ‘/mnt/bsp/prebuilts/build-tools/linux-arm64/src/nsjail/nsjail/kafel/src’
make[2]: Nothing to be done for ‘all’.
make[2]: Leaving directory ‘/mnt/bsp/prebuilts/build-tools/linux-arm64/src/nsjail/nsjail/kafel/src’
make[1]: Leaving directory ‘/mnt/bsp/prebuilts/build-tools/linux-arm64/src/nsjail/nsjail/kafel’
g++ -O2 -c -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -fPIE -Wformat -Wformat-security -Wno-format-nonliteral -Wall -Wextra -Werror -Ikafel/include -pthread -std=c++14 -fno-exceptions -Wno-unused -Wno-unused-parameter -I/usr/include/libnl3 net.cc -o net.o
g++ -O2 -c -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -fPIE -Wformat -Wformat-security -Wno-format-nonliteral -Wall -Wextra -Werror -Ikafel/include -pthread -std=c++14 -fno-exceptions -Wno-unused -Wno-unused-parameter -I/usr/include/libnl3 sandbox.cc -o sandbox.o
g++ -O2 -c -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -fPIE -Wformat -Wformat-security -Wno-format-nonliteral -Wall -Wextra -Werror -Ikafel/include -pthread -std=c++14 -fno-exceptions -Wno-unused -Wno-unused-parameter -I/usr/include/libnl3 subproc.cc -o subproc.o
g++ -O2 -c -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -fPIE -Wformat -Wformat-security -Wno-format-nonliteral -Wall -Wextra -Werror -Ikafel/include -pthread -std=c++14 -fno-exceptions -Wno-unused -Wno-unused-parameter -I/usr/include/libnl3 uts.cc -o uts.o
g++ -O2 -c -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -fPIE -Wformat -Wformat-security -Wno-format-nonliteral -Wall -Wextra -Werror -Ikafel/include -pthread -std=c++14 -fno-exceptions -Wno-unused -Wno-unused-parameter -I/usr/include/libnl3 user.cc -o user.o
g++ -O2 -c -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -fPIE -Wformat -Wformat-security -Wno-format-nonliteral -Wall -Wextra -Werror -Ikafel/include -pthread -std=c++14 -fno-exceptions -Wno-unused -Wno-unused-parameter -I/usr/include/libnl3 util.cc -o util.o
g++ -o nsjail caps.o cgroup.o cgroup2.o cmdline.o config.o contain.o cpu.o logs.o mnt.o net.o nsjail.o pid.o sandbox.o subproc.o uts.o user.o util.o config.pb.o kafel/libkafel.a -pie -Wl,-z,noexecstack -lpthread -lprotobuf -pthread -lnl-route-3 -lnl-3
build completed successfully (2 seconds)

AOSPonARM:/mnt/bsp/prebuilts/build-tools/linux-arm64/src/nsjail/nsjail$ file nsjail
nsjail: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=b2d42a87e718c1b0be7054e47c376d87d2844ed1, for GNU/Linux 3.7.0, not stripped