can't use traceconv to symbolize collected callstacks

[dshulyak]

there are no issues if i am running same binary on the host where i run perfetto.
but i cannot restore symbols if i am running it in container and trying to symbolize with traceconv symbolize.

PERFETTO_SYMBOLIZER_MODE=index PERFETTO_BINARY_PATH=./data/ traceconv symbolize docker.pftrace symbols.pftrace
[720.884] local_symbolizer.cc:382 Indexed: ./data/libc.so.6 (cd410b710f0f094c6832edd95931006d883af48e)
[720.885] local_symbolizer.cc:382 Indexed: ./data/profiler_exp (395e3956a5b66c6a29a6e17e6b955e5589d165bb)
[720.885] local_symbolizer.cc:382 Indexed: ./data/libm.so.6 (7d8778fca8ea4621b268cc03662855d0cd983439)

i assume that symbols.pftrace is correctly generated so i

cat docker.pftrace symbols.pftrace > symbolized.pftrace

but the callstacks are empty (only kernel is there, that was collected before symbolization)

[LalitMaganti]

Do you have llvm-symbolizer available (installed) inside your container? We need the ability to run it as a subprocess to be able to symbolize correctly.

[dshulyak]

i added llvm-symbolizer to docker container and it is available on the PATH. the output is the same.
just to clarify if if mattters, i was running traceconv symbolize on the host

[dshulyak]

and also traced and traced_perf are running on the host. not in the container

[rsavitski]

My guess: you're running the recording process without sufficient capabilities/permissions to access /proc/$pid/maps, and/or the loaded ELF libraries of the profiled processes.

ERROR_INVALID_MAP & ERROR_INVALID_ELF indicate there are unwinding errors at the time of recording, so there are no callframes to symbolise (and I would guess the ELF build IDs are therefore missing in the trace, leaving the symboliser with nothing to match, but I didn't double-check whether we log any errors in this situation).

The "simplest" approach would be to try running the recording process as root. Otherwise you'll need to make sure that the recording process has access to procfs (could be a matter of "hidepid" mounting option), has matching UID as the target process or another way to satisfy PTRACE_MODE_READ kernel permission checks, and can access the ELF files loaded into the target process (to recover the build id). I can't provide the full steps for the non-root setup since different linux distributions have different setups for procfs and file permissions.

[dshulyak]

i don't see any obvious problem. 12264 is a process running in docker that i am trying to profile

openat(AT_FDCWD, "/proc/12264/cmdline", O_RDONLY|O_CLOEXEC) = 31
read(31, "/app/profiler_exp\0", 511)    = 18
close(31)                               = 0
openat(AT_FDCWD, "/proc/12264", O_RDONLY|O_CLOEXEC|O_DIRECTORY) = 31
newfstatat(31, "", {st_dev=makedev(0, 0x18), st_ino=49108, st_mode=S_IFDIR|0555, st_nlink=9, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=0, st_size=0, st_atime=1747328024 /* 2025-05-15T18:53:44.738011964+0200 */, st_atime_nsec=738011964, st_mtime=1747328024 /* 2025-05-15T18:53:44.738011964+0200 */, st_mtime_nsec=738011964, st_ctime=1747328024 /* 2025-05-15T18:53:44.738011964+0200 */, st_ctime_nsec=738011964}, AT_EMPTY_PATH) = 0
openat(31, "maps", O_RDONLY|O_CLOEXEC)  = 32
openat(31, "mem", O_RDONLY|O_CLOEXEC)   = 33

it has access to maps, not sure what to expect next. should it read/mmap from maps and then from map_files? also it looks pretty much the same when symbolization works (e.g when i run target process on host instead of docker)

full trace from a session
docker_traced_perf.tar.gz

[dshulyak]

i missed that there was another thread. so apparently traced_perf tries to read an executable and fails in expected way (traced_perf is running on host)

openat(AT_FDCWD, "/app/profiler_exp", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

is it possible to read that from map_files instead?

[rsavitski]

EDIT(raced again): will post a separate reply below.

---

ok, so /proc/pid/maps access is probably fine if the open succeeds. That makes sense to me if the process is running as root - it'll operate on the root pid namespace and pass most capability/file permission checks.

So the next suspect is opening/reading referenced ELF files from disk for their DWARF info. We don't currently use /proc/pid/map_files, and instead let the unwinding library open ELFs by path (as obtained from /proc/pid/maps).

Sorry, I gave you an insufficient strace oneliner earlier, we want to capture both threads of traced_perf (the second thread is the one doing DWARF-based unwinding using libunwindstack). This invocation should do it:
strace -v -o /tmp/log -ffp $(pidof traced_perf)
(with per-thread output in /tmp/log.XXX files after detaching).

How do the paths being opened look like? Are they meaningful/accessible outside of the container? We're after lines like this (assuming the /maps parsing went fine):
openat(AT_FDCWD, "/usr/lib/clangd/bin/clangd", O_RDONLY|O_CLOEXEC) = 17

A snapshot of the /proc/pid/maps of the target process might be helpful as well. If you don't want to share the attachments widely but have a google drive account, feel free to share the files directly with me ([email protected]).

[rsavitski]

Right, so it's the mount namespace filepaths. We don't currently support /map_files, though it's been prototyped before. And I can't immediately think of any alternatives without local patching (or trying to run a second traced_perf inside of the container, if the mounts aren't process-private).

If your setup uses frame pointers everywhere, one option would be to try fp unwinding [1]. But you'd still need to manually guide the offline symbolisation as we'd still be missing the ELF build ids.

So at the moment I can only suggest trying to work around this limitation. But if you believe this setup is important to support, please create an issue in this github project.

[1] https://source.chromium.org/chromium/chromium/src/+/main:third_party/perfetto/protos/perfetto/config/profiling/perf_event_config.proto;l=158;drc=d196754d3c01f0053cf0cb76ca9053c16505d735

[dshulyak]

thanks for the support. i think can get around this problem, will open an issue if i will get stuck again

[rsavitski]

Btw, if you still see this: do you mind checking how perf was accessing the ELF files from the container? Was it via /proc/pid/fd/... reopening as root?

[dshulyak]

i think this is what it does. 172091 is the process that runs in container, and i put a binary at /tmp/profiler_exp.

openat(AT_FDCWD, "/proc/self/ns/mnt", O_RDONLY) = 1879
openat(AT_FDCWD, "/proc/172091/ns/mnt", O_RDONLY) = 1880
setns(1880, CLONE_NEWNS)                = 0
openat(AT_FDCWD, "/tmp/profiler_exp", O_RDONLY) = 1882
fcntl(1882, F_GETFD)                    = 0
newfstatat(1882, "", {st_mode=S_IFREG|0775, st_size=94585120, ...}, AT_EMPTY_PATH) = 0
pread64(1882, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20_\3\0\0\0\0\0"..., 64, 0) = 64
pread64(1882, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2688, 94582432) = 2688
pread64(1882, "\0.symtab\0.strtab\0.shstrtab\0.inte"..., 425, 94582000) = 425
pread64(1882, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\253\311\2005\210\30y\372\34W1\226\24\217\262\321"..., 36, 912) = 36
close(1882)                             = 0
setns(1879, CLONE_NEWNS)                = 0

attached full trace from sudo strace -o perf.log p^Cf record -p 172091 -g -F 99 -o perf.data

perf.tar.gz

[rsavitski]

Thank you, so perf changes between namespaces (setns()) using root-only capabilities. Something to consider for us when profiling outside of the android OS, where root might be an acceptable assumption.