shaka-packager/docs/linux_profiling.md

110 lines
3.8 KiB
Markdown
Raw Normal View History

# Linux Profiling
Profiling code is enabled when the `use_allocator` variable in gyp is set to
build: Stop using hermetic clang, libc++, etc This brings our default build config more in line with what is necessary for some platforms anyway: using the system-installed toolchain and sysroot to build everything. We will no longer fetch source or binaries for any specific build tools, such as libc++, clang, gold, binutils, or valgrind. The main part of this change is the changing of default gyp settings in gyp_packager.py. For this, a bug in gyp_packager.py had to be fixed, in which similar GYP_DEFINE key names (such as clang and host_clang) would conflict, causing some defaults not to be installed properly. In order to enable clang=0 by default, some changes had to be made in common.gypi: - compiler macros added to fix a compatibility issue between Chromium's base/mac/ folder and the actual OSX SDK - replaced clang_warning_flags variables with standard cflags settings, plus xcode_settings for OSX - turned off warnings-as-errors for non-shaka code, rather than allow-listing specific warning types, since we can't actually fix those warnings on any platform - disabled two specific warnings in shaka code, both of which are caused by headers from our non-shaka dependencies Also, one warning (missing "override" keyword) has been fixed in vod_media_info_dump_muxer_listener.h. Although these changes were done to make building simpler on a wider array of platforms (arm64, for example), it seems to make the build a bit faster, too. For me, at least, on my main Linux workstation: - "gclient sync" now runs 20-30% faster - "ninja -C out/Release" now runs 5-13% faster The following environment variables are no longer required: - DEPOT_TOOLS_WIN_TOOLCHAIN - MACOSX_DEPLOYMENT_TARGET Documentation, Dockerfiles, and GitHub Actions workflows have been updated to reflect this. The following GYP_DEFINES are no longer required for anyone: - clang=0 - host_clang=0 - clang_xcode=1 - use_allocator=none - use_experimental_allocator_shim=0 Documentation, Dockerfiles, and GitHub Actions workflows have been updated to reflect this. The following repos are no longer dependencies in gclient: - binutils - clang - gold - libc++ - libc++abi - valgrind The following gclient hooks have been removed: - clang - mac_toolchain - sysroot Change-Id: Ie94ccbeec722ab73c291cb7df897d20761a09a70
2021-07-28 20:01:19 +00:00
`tcmalloc` and `profiling` variable in gyp is set to `1`. That will build the
tcmalloc library, including the cpu profiling and heap profiling code into
shaka-packager, e.g.
GYP_DEFINES='profiling=1 use_allocator="tcmalloc"' gclient runhooks
If the stack traces in your profiles are incomplete, this may be due to missing
frame pointers in some of the libraries. A workaround is to use the
`linux_keep_shadow_stacks=1` gyp option. This will keep a shadow stack using the
`-finstrument-functions` option of gcc and consult the stack when unwinding.
## CPU Profiling
In order to enable cpu profiling, run shaka-packager with the environment
variable `CPUPROFILE` set to a filename. For example:
CPUPROFILE=/tmp/cpuprofile out/Release/packager
The cpu profile will be dumped periodically to the filename specified in the
CPUPROFILE environment variable. You can then analyze the dumps using the pprof
script (`packager/third_party/tcmalloc/chromium/src/pprof`). For example,
pprof --gv out/Release/packager /tmp/cpuprofile
This will generate a visual representation of the cpu profile as a postscript
file and load it up using `gv`. For more powerful commands, please refer to the
pprof help output and the google-perftools documentation.
For further information, please refer to
http://google-perftools.googlecode.com/svn/trunk/doc/cpuprofile.html.
## Heap Profiling
To turn on the heap profiler on shaka-packager, use the `HEAPPROFILE`
environment variable to specify a filename for the heap profile. For example:
HEAPPROFILE=/tmp/heapprofile out/Release/packager
The heap profile will be dumped periodically to the filename specified in the
`HEAPPROFILE` environment variable. The dumps can be analyzed using the same
command as cpu profiling above.
For further information, please refer to
http://google-perftools.googlecode.com/svn/trunk/doc/heapprofile.html.
Some tests fork short-living processes which have a small memory footprint. To
catch those, use the `HEAP_PROFILE_ALLOCATION_INTERVAL` environment variable.
#### Dumping a profile of a running process
To programmatically generate a heap profile before exit, use code like:
#include <packager/third_party/tcmalloc/chromium/src/gperftools/heap-profiler.h>
// "foobar" will be included in the message printed to the console
HeapProfilerDump("foobar");
Then add allocator.gyp dependency to the target with the above change:
'conditions': [
['profiling==1', {
'dependencies': [
'base/allocator/allocator.gyp:allocator',
],
}],
],
Or you can use gdb to attach at any point:
1. Attach gdb to the process: `$ gdb -p 12345`
2. Cause it to dump a profile: `(gdb) p HeapProfilerDump("foobar")`
3. The filename will be printed on the console, e.g.
"`Dumping heap profile to heap.0001.heap (foobar)`"
build: Stop using hermetic clang, libc++, etc This brings our default build config more in line with what is necessary for some platforms anyway: using the system-installed toolchain and sysroot to build everything. We will no longer fetch source or binaries for any specific build tools, such as libc++, clang, gold, binutils, or valgrind. The main part of this change is the changing of default gyp settings in gyp_packager.py. For this, a bug in gyp_packager.py had to be fixed, in which similar GYP_DEFINE key names (such as clang and host_clang) would conflict, causing some defaults not to be installed properly. In order to enable clang=0 by default, some changes had to be made in common.gypi: - compiler macros added to fix a compatibility issue between Chromium's base/mac/ folder and the actual OSX SDK - replaced clang_warning_flags variables with standard cflags settings, plus xcode_settings for OSX - turned off warnings-as-errors for non-shaka code, rather than allow-listing specific warning types, since we can't actually fix those warnings on any platform - disabled two specific warnings in shaka code, both of which are caused by headers from our non-shaka dependencies Also, one warning (missing "override" keyword) has been fixed in vod_media_info_dump_muxer_listener.h. Although these changes were done to make building simpler on a wider array of platforms (arm64, for example), it seems to make the build a bit faster, too. For me, at least, on my main Linux workstation: - "gclient sync" now runs 20-30% faster - "ninja -C out/Release" now runs 5-13% faster The following environment variables are no longer required: - DEPOT_TOOLS_WIN_TOOLCHAIN - MACOSX_DEPLOYMENT_TARGET Documentation, Dockerfiles, and GitHub Actions workflows have been updated to reflect this. The following GYP_DEFINES are no longer required for anyone: - clang=0 - host_clang=0 - clang_xcode=1 - use_allocator=none - use_experimental_allocator_shim=0 Documentation, Dockerfiles, and GitHub Actions workflows have been updated to reflect this. The following repos are no longer dependencies in gclient: - binutils - clang - gold - libc++ - libc++abi - valgrind The following gclient hooks have been removed: - clang - mac_toolchain - sysroot Change-Id: Ie94ccbeec722ab73c291cb7df897d20761a09a70
2021-07-28 20:01:19 +00:00
## Thread sanitizer (tsan)
To compile with the thread sanitizer library (tsan), you must set clang as your
compiler and set the `tsan=1` and `tsan_blacklist` configs:
CC=clang CXX=clang++ GYP_DEFINES="tsan=1 tsan_blacklist=/path/to/src/packager/tools/memory/tsan_v2/ignores.txt" gclient runhooks
NOTE: tsan and asan cannot be used at the same time.
## Adddress sanitizer (asan)
To compile with the address sanitizer library (asan), you must set clang as your
compiler and set the `asan=1` config:
CC=clang CXX=clang++ GYP_DEFINES="asan=1" gclient runhooks
NOTE: tsan and asan cannot be used at the same time.
## Leak sanitizer (lsan)
To compile with the leak sanitizer library (lsan), you must set clang as your
compiler and set the `lsan=1` config:
CC=clang CXX=clang++ GYP_DEFINES="lsan=1" gclient runhooks
## Reference
[Linux Profiling in Chromium](https://chromium.googlesource.com/chromium/src/+/master/docs/linux_profiling.md)