CC-BY-SA 3.0 - Attribution requirements and misc., PLEASE READ:

• This slide must remain as-is in this specific location (slide #1), everything else you are free to change; including the logo :-)
• Use of figures in other documents must feature the below "Originals at" URL immediately under that figure and the below copyright notice where appropriate.
• You are FORBIDDEN from using the default "About" slide as-is or any of its contents.

Copyright (C) 2013-2021, Opersys inc.

These slides created by: Karim Yaghmour

Originals at: http://www.opersys.com/training/

About this specific class

Android debugging is dirty business

Default AOSP builds have issues

... ergo ...

Runtime adjustments will be made

Goals - High Level

• Understand the set of debugging and performance monitoring tools and capabilities available in Android
• Understand the internals and limitations of each
• Get hands-on experience with the tools and capabilities
• Determine which ones are most appropriate, useful and/or important for a given task

Goals - Specifics

• Understand the Android stack's debugging mechanisms and their internals
• Debug from the app level all the way down to kernel drivers
• Using Linux debugging tools with Android
• Learning about Android-specific tools
• Monitor performance and latencies
• Quantify and analyze memory usage
• Breakpoint and step through the stack
• Apply commonly-used techniques for framework debugging
• Familiarize with lesser-known tools and capabilities built into Android

Prerequisites

• C/C++
• Java
• Linux command line
• Android internals
• Linux kernel internals
• Linux device drivers
• ARM architecture

Topics

1. Classifying and Analyzing Tools
2. Internals Architecture Quick Recap
3. AOSP Sources and Symbolic Debugging
4. Kernel Tools and Capabilities
5. Android-Agnostic User-Space Tools
6. Android-Specific User-Space Tools
7. Java Tools
8. System Services Interfacing
9. Other Tools and Techniques
10. glibc User-Space

Courseware

• These slides
• Exercises
• Online documentation

Hands-On Environment

Host

• Ubuntu-based system
• 150GB / AOSP

Target

• Hikey LeMaker 96boards
• Kirin 620 SoC
• ARM® Cortex™-A53 Octa-core 64-bit up to 1.2GHz (ARM v8 instruction set)
• 2GB LPDDR3 SDRAM @ 800MHz
• 8GB eMMC on board storage
• 96boards form-factor w/ expansion connectors

Classifying and Analyzing Tools

• Families of tools available in Linux/Android
• Use of sampling
• Use of software breakpoints
• Use of interrupts
• Ability to understand machine code
• Ability to read into internal data structures
• Use of statistics
• Use of buffering
• Time measurement
• Limitations
• Documentation vs. capabilities

Internals Architecture Quick Recap

• Hardware used to run Android
• AOSP
• Binder
• System Services
• HAL
• Call walkthrough
• System startup
• Debug setup
• Network boot
• Symbolic debugging

1. Hardware used to run Android

2. AOSP

3. Binder

4. System Services

5. HAL

6. Call walkthrough

7. System startup

8. Debug setup

9. Network boot

10. Symbolic debugging

AOSP Sources and Symbolic Debugging

1. Basics
2. Preparing for Studio import
3. Importing into Studio
4. Browsing the sources
5. Java debugging
6. C/C++ debugging

1. Basics

• repo
• build/envsetup.sh
  • godir
  • croot
  • mm
  • m
  • jgrep
  • cgrep
  • resgrep
• hmm
• lunch
• make -j8

2. Preparing for Studio import

AOSP:

• Get AOSP ... from Google or otherwise
• Extract if needed
• Configure, build, etc.

Android Studio:

• Get Android Studio from developer.android.com
• Extract
• Start and update and if needed

Create AOSP project files for Studio:

[aosp]$ make idegen && development/tools/idegen/idegen.sh

3. Importing into Studio

• Start Studio:
  • Choose "Open an Existing Android Studio Project"
  • Select android.ipr from AOSP
  • Let it finish indexing
  • If indexing keeps restarting:
    • May need to fix android.iml file by hand
    • See https://bit.ly/2QrN99g
    • Must add:

      <option name="ALLOW_USER_CONFIGURATION" value="false" />

4. Browsing the Sources

• Right-click object type to be taken to declaration
• Browse classes through “Structure”
• Right-click "Find Usages"
• Toggle open files (Alt-left, Alt-right)
• Many other shortcuts, see:
  https://developer.android.com/sdk/installing/studio-tips.html
• Issues:
  • Can't compile with Studio ... still need “make”
  • For Java only

5. Java Debugging

• Use Android Studio to attach to running process
• All other debugging operations are typical Studio operations

6. C/C++ Debugging

• Use of gdb and gdbserver
• Classic user-space symbolic debugging
• For better or worse:
  • It's practically the only thing out there
• Relies on ptrace() for all operations
• Tightly integrated with GNU toolchain
• Use -g or -ggdb flags when building
• Several front-ends: DDD, Eclipse CDT, ...
• man gdb
• gdb also has online help in its shell
• gdbserver binary part of AOSP
• Everything works the same on Android

6.1. Target side

AOSP already takes care of debug:

• "-g" flag added to all native binaries
• Unstripped binaries in out/target/product/[PRODUCT_NAME]/symbols/...

Two ways to operate gdbserver:

• Start app for debugging with gdbserver prepended

    # gdbserver localhost:2345 service list

• Attaching to running process

    # gdbserver --attach localhost:2345 30

Either way, need to forward the port on the host:

    $ adb forward tcp:2345 tcp:2345

6.2. Host side

Load file FIRST and then attach on host side

    $ arm-linux-androideabi-gdb
    ...
    (gdb) file out/target/product/hikey/symbols/system/bin/service
    (gdb) target remote localhost:2345
    (gdb) b main
    Cannot access memory at address 0x0
    Breakpoint 1 at 0x2a00146c: file frameworks/native/cmds/service/service.cpp, line 59.
    (gdb) cont
    Continuing.
    warning: Could not load shared library symbols for 11 libraries, e.g. /system/bin/linker.
    ...

    Breakpoint 1, main (argc=2, argv=0xbe882b74) at frameworks/native/cmds/service/service.cpp:59
    59    {
    (gdb) n
    60        sp<IServiceManager> sm = defaultServiceManager();
    (gdb) n
    59    {
    (gdb) n
    60        sp<IServiceManager> sm = defaultServiceManager();
    (gdb) n
    61        fflush(stdout);
    ...

Can also attach to system services' JNI code -- attach FIRST

$ arm-linux-androideabi-gdb
...
(gdb) target remote localhost:2345
(gdb) file out/target/product/hikey/symbols/system/bin/app_process 
(gdb) set solib-absolute-prefix out/target/product/hikey/symbols/
(gdb) set solib-search-path out/target/product/hikey/symbols/system/lib/
(gdb) b com_android_server_OpersysService.cpp:70
(gdb) cont
Continuing.
-----------------------------
root@android:/ # service call opersys 2 s16 adfasd
-----------------------------
[New Thread 576]
[Switching to Thread 576]

Breakpoint 1, write_native (env=0x5c94ad40, clazz=, 
    ptr=, buffer=0xa4f00005)
    at frameworks/base/services/jni/com_android_server_OpersysService.cpp:72
72        if (dev == NULL) {
(gdb) 

6.3. Using 'gdbclient'

$ gdbclient system_server
[1] 13531
Attached; pid = 1653
Listening on port 5039
GNU gdb (GDB) 7.6
...
Reading symbols from /media/6921e18a-5b32-4fe8-97cc-62a85a6f351f/home/karim/opersys-dev/android/aosp-5.0.2_r1-pristine/out/target/product/flo/symbols/system/bin/app_process32...done.
Remote debugging from host 127.0.0.1
warning: while parsing target library list: not well-formed (invalid token)
warning: Could not load shared library symbols for 29 libraries, e.g. 'U.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
readv () at bionic/libc/arch-arm/syscalls/readv.S:12
12        neg     r0, r0
Function "art_sigsegv_fault" not defined.
Breakpoint 1 (art_sigsegv_fault) pending.
ART debugging mode is enabled.
If you are debugging a native only process, you need to
re-enable normal SIGSEGV handling using this command:
  handle SIGSEGV print stop

(gdb) b com_android_server_power_PowerManagerService.cpp:126
Breakpoint 2 at 0xb404f404: file frameworks/base/services/core/jni/com_android_server_power_PowerManagerService.cpp, line 126.
(gdb) cont
Continuing.
[New Thread 30562]
[Switching to Thread 30562]

Breakpoint 2, android::nativeSetInteractive (env=0xaf6c30f0, clazz=0x9eff29ac, 
    enable=1 '\001')
    at frameworks/base/services/core/jni/com_android_server_power_PowerManagerService.cpp:126
126        if (gPowerModule) {
(gdb) n

Kernel Tools and Capabilities

1. Basic interfacing
2. Instrumentation
3. Analysis tools
4. Debugging

1. Basic Interfacing

• procfs
• sysfs
• configs
• debugfs
• dmesg/printk

1.1. procfs

• Mounted as /proc in Android
• Virtual filesystem maintained by kernel
• Traditionally the main way to expose internal info
• Since 2.6 it's meant for process info only
• All hardware-specific info is in sysfs instead
• Documentation/filesystems/proc.txt
• Used by a lot of tools: ps, top, uptime, etc.

1/              43/             60/             cpuinfo         net/
10/             45/             63/             crypto          pagetypeinfo 
1007/           46/             64/             devices         partitions
1061/           47/             7/              diskstats       sched_debug
11/             48/             726/            dma-mappings    schedstat
12/             488/            741/            driver/         self/
13/             49/             756/            execdomains     slabinfo
14/             499/            773/            fb              softirqs
2/              5/              8/              filesystems     stat
25/             51/             835/            fs/             swaps
26/             512/            9/              interrupts      sys/
27/             52/             909/            iomem           sysrq-trigger
28/             523/            920/            ioports         sysvipc/
29/             53/             935/            irq/            timer_list
3/              54/             950/            kallsyms        tty/
30/             547/            971/            kmsg            uptime
31/             55/             988/            kpagecount      version
33/             56/             buddyinfo       kpageflags      vmallocinfo
34/             57/             bus/            loadavg         vmstat
362/            572/            cgroups         locks           yaffs
386/            58/             cmdline         meminfo         zoneinfo
39/             59/             config.gz       misc
40/             593/            consoles        mounts
413/            6/              cpu/            mtd
4. Kernel Tools and Capabilities

What's in here?

• Kernel-specific info:
  • memory information (meminfo)
  • verion/build (version)
  • CPU info (cpuinfo)
  • interrupt info (irq/ and interrupts)
• One directory per PID:
  • memory maps (maps)
  • command line (cmdline)
  • mem file to access memory -- ptrace
  • sched stats (sched)
  • detailed process info (status)

A lot more stuff ...

1.2. sysfs

• Mounted as /sys in Android
• Virtual filesystem maintained by kernel
• Main way for kernel to publish its view of HW
• Tightly tied to kernel's device object model
• Enables hotplug functionality -- used by udev
• Allows user-space to write values to kernel-exposed properties
• Documentation/filesystems/sysfs.txt

block/       class/       devices/     fs/          module/
bus/         dev/         firmware/    kernel/      power/

1.3. configfs

• Would be mounted as /config if needed
• Not as often used as its counterparts
• Contrary to sysfs:
  • Enables user-space to create objects
• Used for configuring complex kernel-side subsystems:
  • USB composite devices
  • SCSI
• Documentation/filesystems/configfs/configfs.txt

1.4. debugfs

• Mount as /sys/kernel/debug
• Free "scratch area" for all things debugging
• No fixed rules of what can or has to be done
• Used by ftrace
• If you need to debug a driver, use this FS
• Documentation/filesystems/debugfs.txt

asoc          dri                ion      opp       sleep_time       
bdi           extfrag            kprobes  pinctrl   split_huge_pages 
binder        f72c0000.usb       kvm      pm_genpd  suspend_stats    
bluetooth     fault_around_bytes mali     pm_qos    tracing          
btrfs         gpio               memblock pstore    usb              
clk           hid                mmc0     ras       virtio-ports     
debug_enabled ieee80211          mmc1     regmap    wakeup_sources   
dma_buf       iio                mmc2     regulator 

1.5. dmesg/printk

• Meet the kernel's printf: printk()

• Defined: include/linux/printk.h

    int printk(const char *fmt, ...);

• Implemented: kernel/printk.c

• Can loose data in cases of large output
• Widely-used throughout kernel sources
• Don't call while holding lock:
  • Has lock contention of its own

2. Instrumentation

• mcount
• tracepoints
• user-space events
• USDT
• kprobes
• uprobes
• perf events
• HW counters
• HW breakpoints

2.1. mcount

• gcc-based mechanism
• Trigger on -pg flag
• Originally-designed for gprof

• Kernel-side implemented in assembly:

    arch/arm/kernel/entry-common.S

• Conditional to CONFIG_FUNCTION_TRACER

• Two possible behaviors -- CONFIG_DYNAMIC_FTRACE:
  • Hardcoded call
  • Dynamically-patched nop

2.2. Tracepoints

• Instrument your own code, for fun and profit

• In kernel:

  • Use built-in mechanism to define/use custom tracepoints

  • See

    kernel/tracepoint.c
    include/linux/tracepoint.h
    include/trace/*
    include/trace/events/* -- definition of all global static tracepoints
    Documentation/trace/tracepoints.txt

  • Example -- track context switches:

    • include/trace/sched.h uses this macro:

      TRACE_EVENT(sched_switch,...

    • This results in trace_sched_switch() to be created

    • kernel/sched/core.c uses this function

• Kernel instrumentation mechanism:
  • Conditional to CONFIG_JUMP_LABEL
  • If enabled, uses dynamically-patched nops
  • If disabled, uses classic if()
  • Beware of Qcom CONFIG_STRICT_MEMORY_RWX
• Probe using register_trace_subsys_eventname()

2.3. User-space events

• Write to ftrace's buffer
• That's what Android's systrace/atrace functionality does
• /sys/kernel/tracing/trace_marker
• It's just a file
• open(), write(), write(), write(), ...
• Read your events as part of ftrace's output

2.4. USDT

• Inspired by Dtrace user-space tracing
• Supported added in perf starting in kernel 4.10 http://www.brendangregg.com/perf.html#StaticUserTracing
• Not presently used in Android
• Has been review by Android trace tool developer: http://www.joelfernandes.org/linuxinternals/2018/02/10/usdt-notes.html
• Relies on sdt.h and sdt-config.h -- can be retrieved by Ubuntu systemtap-sdt-dev package.

#include <sys/sdt.h>
...
DTRACE_PROBE(provider, name)
...
DTRACE_PROBE4(provider, name, arg1, arg2, arg3, arg4)
...

• Adds a .notes.stapsdt section to ELF file
• Can tell perf tool to update cache:

  perf buildid-cache --add path_to_my_instrumented_binary
  

• Ref doc: https://sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps

2.5. kprobes

• Formal mechanism for dynamically adding probe points
• In mainline kernel since 2005:
  • Stems from IBM's previous work on DProbes
  • Trimmed-down version of DProbes functionality
• Requires module insertion
• Module must know insertion address/symbol
• 3 types of probes:
  • Kprobe => register_kprobe()
  • Jprobe => register_jprobe()
  • Kretprobe => register_kretprobe()
• Typically:
  • module_init() registers + provides handlers
  • module_exit() unregisters
• Documentation/kprobes.txt

What's a kprobe?

• Acts like a typical breakpoint
• Original instruction at destination is copied
• Breakpoint is inserted
• On hit, kprobe-registered pre_handler callback notified
• Copied instruction is single-stepped
• Then, kprobe-registered post_handler callback notified
• Execution continues at the next instruction
• Example: samples/kprobes/kprobe_example.c

What's a jprobe?

• DEPRECATED since October 2017
• It's a kprobe inserted at function entry-point
• Allows handler to inspect function's arguments
• Called function's stack is copied for inspection:
  • Only MAX_STACK_SIZE is copied -- 64 bytes on ARM
• Registered handler is called
• Copied stack is recopied over the original
  • gccism, see doc
• Example: samples/kprobes/jprobe_example.c

What's a kretprobe?

• Allows you to monitor function entry and exit
• kprobe inserted at function entry-point
• Return address is saved and replaced with handler
• Then entry_handler is called
• Function continues
• When function returns, return handler (handler) is called
• Example: samples/kprobes/kretprobe_example.c

Android support

• It's orthogonal to Android
• Kernel mechanism
• No user-space component
• No need for explicit Android support

Resources

https://lwn.net/Articles/132196/

http://www.linuxforu.com/2011/04/kernel-debugging-using-kprobe-and-jprobe/

https://sourceware.org/systemtap/kprobes/

2.6. uprobes

• User-space equivalent to kprobes

• Currently:

  • x86
  • PowerPC
  • ARM (32 and 64)
  • MIPS
  • S390

  • See:

    kernel/events/uprobes.c
    kernel/trace/trace_uprobe.c
    Documentation/trace/uprobetracer.txt

2.7. perf events

• Yet another source of kernel events ;)
• Partly required for PMU counter house-keeping
• Feeds into perf infrastructure
• Some of these are called "software" events, such as context switches, pages faults, etc.
• See include\linux\perf_event.h
• Example:

 static inline void perf_event_task_sched_in(struct task_struct *prev,
                         struct task_struct *task)

2.8. HW counters

• Count key HW events without SW support
• Very HW-specific:
  • Arch-specific
  • CPU-specific
• Handled by perf, for better or worse
• perf designed to measure on overflow

2.9. HW breakpoints

• Create breakpoints on memory access
• Core is also handled by perf
• kernel/events/hw_breakpoint.c
• samples/hw_breakpoint/data_breakpoint.c

3. Analysis Tools

• ftrace
• systrace
• perf
• SystemTap
• BCC/eBPF/...
• Other tools
  • LTTng
  • sysdig
  • ktap
  • dtrace
  • oprofile

3.1. ftrace

• Kernel function and event tracer
• Relies on:
  • gcc's "-pg" flag (i.e. mcount())
  • Tracepoints
• /sys/kernel/tracing/

  README                    options/                  trace_options
  available_events          per_cpu/                  trace_pipe
  available_tracers         printk_formats            tracing_cpumask
  buffer_size_kb            saved_cmdlines            tracing_enabled
  buffer_total_size_kb      set_event                 tracing_max_latency
  current_tracer            trace                     tracing_on
  events/                   trace_clock               tracing_thresh
  free_buffer               trace_marker
  

• Documentation/tracing/ftrace.txt

Kernel configuration options to watch for:

• CONFIG_FTRACE
• CONFIG_FUNCTION_TRACER
• CONFIG_FUNCTION_GRAPH_TRACER
• CONFIG_STACK_TRACER
• CONFIG_DYNAMIC_FTRACE

Implementation - kernel/trace/

blktrace.c               trace_events.c              trace_output.h
ftrace.c                 trace_events_filter.c       trace_printk.c
Kconfig                  trace_events_filter_test.h  trace_probe.c
Makefile                 trace_export.c              trace_probe.h
power-traces.c           trace_functions.c           trace_sched_switch.c
ring_buffer_benchmark.c  trace_functions_graph.c     trace_sched_wakeup.c
ring_buffer.c            trace.h                     trace_selftest.c
rpm-traces.c             trace_irqsoff.c             trace_selftest_dynamic.c
trace_branch.c           trace_kdb.c                 trace_stack.c
trace.c                  trace_kprobe.c              trace_stat.c
trace_clock.c            trace_mmiotrace.c           trace_stat.h
trace_entries.h          trace_nop.c                 trace_syscalls.c
trace_event_perf.c       trace_output.c              trace_uprobe.c

Check if tracing is on:

# cat tracing_on 
0

Check which tracers are available:

# cat available_tracers 
blk function_graph wakeup_rt wakeup function nop

Check the current tracer:

# cat current_tracer
nop

Set the current tracer:

# echo function > current_tracer

Enable tracing -- beware the space with ">":

# echo 1 > tracing_on

The raw events:

# ls events/*

Check the content of a trace:

# cat trace
# tracer: function
#
# entries-in-buffer/entries-written: 60028/3128571   #P:1
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
            adbd-55    [000] ....  1075.680000: __schedule <-schedule
            adbd-55    [000] ....  1075.680000: rcu_sched_qs <-__schedule
            adbd-55    [000] d...  1075.680000: deactivate_task <-__schedule
            adbd-55    [000] d...  1075.680000: dequeue_task <-deactivate_task
            adbd-55    [000] d...  1075.680000: update_rq_clock <-dequeue_task
            adbd-55    [000] d...  1075.680000: dequeue_task_fair <-dequeue_task
            adbd-55    [000] d...  1075.680000: update_curr <-dequeue_task_fair
            adbd-55    [000] d...  1075.680000: clear_buddies <-dequeue_task_fair
            adbd-55    [000] d...  1075.680000: account_entity_dequeue <-dequeue_task_fair
            ...

Stop tracing

# echo 0 > tracing_on

Clear a trace:

# echo > trace

Check buffer size:

# cat buffer_size_kb 
1408

Set buffer size:

# echo 2048 > buffer_size_kb

Use function graph tracer:

# echo function_graph > current_tracer

Restart tracing:

# echo 1 > tracing_on

Check graph tracer output:

# cat trace
# tracer: function_graph
#
# CPU  DURATION                  FUNCTION CALLS
# |     |   |                     |   |   |   |
 0)   0.000 us    |              } /* __sync_icache_dcache */
 0)   0.000 us    |              __sync_icache_dcache();
 0)   0.000 us    |              vm_normal_page();
 0)   0.000 us    |              __sync_icache_dcache();
 0)   0.000 us    |              _cond_resched();
 0)   0.000 us    |            } /* copy_pte_range */
 0)   0.000 us    |          } /* copy_page_range */
 0)               |          cap_vm_enough_memory() {
 0)   0.000 us    |            cap_capable();
 0)   0.000 us    |            __vm_enough_memory();
 0)   0.000 us    |          }
 0)   0.000 us    |          kmem_cache_alloc();
 0)               |          anon_vma_fork() {
 0)               |            anon_vma_clone() {
 0)   0.000 us    |              kmem_cache_alloc();
 0)   0.000 us    |              mutex_lock();
 0)   0.000 us    |              anon_vma_chain_link();
 0)   0.000 us    |              mutex_unlock();
 0)   0.000 us    |            }
...

Linux toolset

• trace-cmd (uses splice())-- not avail. in Android
  http://git.kernel.org/cgit/linux/kernel/git/rostedt/trace-cmd.git
• KernelShark -- not avail. in Android
  http://people.redhat.com/srostedt/kernelshark/HTML/
• Neither of these are included in the AOSP

Use in drivers

• In the long-term:
  • Create your own events with TRACE_EVENT() macro

• For short-term debugging/instrumentation:

  • Use trace_printk()
  • trace_printk() is EXPORT_SYMBOL_GPL()'ed

  • Make sure your module is GPL-licensed:

    MODULE_LICENSE("GPL");

  • Otherwise, symbol will be unresolvable at load time

Resources:

https://lwn.net/Articles/365835/

https://lwn.net/Articles/366796/

https://lwn.net/Articles/370423/

http://elinux.org/Ftrace

3.2. Systrace

• Introduced in 4.1, heavy updates in 4.2
• ... finicky ...
• Android stack feeds events into ftrace
• Same entries in /sys/kernel/tracing
• Neither trace-cmd nor KernelShark
• Android tools:
  • On the device: atrace -- native binary
  • On the host: systrace -- Python script
• systrace calls atrace over adb
• systrace-generated traces viewable w/ Chrome
  • And nothing but Chrome ... NIH?

# atrace --help
usage: atrace [options] [categories...]
options include:
  -a appname      enable app-level tracing for a comma separated list of cmdlines
  -b N            use a trace buffer size of N KB
  -c              trace into a circular buffer
  -k fname,...    trace the listed kernel functions
  -n              ignore signals
  -s N            sleep for N seconds before tracing [default 0]
  -t N            trace for N seconds [defualt 5]
  -z              compress the trace dump
  --async_start   start circular trace and return immediatly
  --async_dump    dump the current contents of circular trace buffer
  --async_stop    stop tracing and dump the current contents of circular
                    trace buffer
  --list_categories
                  list the available tracing categories

root@flo:/ # atrace --list_categories
         gfx - Graphics
       input - Input
        view - View System
     webview - WebView
          wm - Window Manager
          am - Activity Manager
          sm - Sync Manager
       audio - Audio
       video - Video
      camera - Camera
         hal - Hardware Modules
         app - Application
         res - Resource Loading
      dalvik - Dalvik VM
          rs - RenderScript
      bionic - Bionic C Library
       power - Power Management
       sched - CPU Scheduling
         irq - IRQ Events
        freq - CPU Frequency
        idle - CPU Idle
        disk - Disk I/O
        load - CPU Load
        sync - Synchronization
       workq - Kernel Workqueues
  memreclaim - Kernel Memory Reclaim
  regulators - Voltage and Current Regulators

Google's doc:
https://developer.android.com/studio/command-line/systrace

Also have a look at these:

/external/chromium-trace/systrace.py
/frameworks/native/cmds/atrace
/frameworks/base/core/java/android/os/Trace.java
/frameworks/base/core/jni/android_os_Trace.cpp
/system/core/include/cutils/trace.h
/system/core/libcutils/trace-dev.c
/system/core/libcutils/trace-host.c
/system/core/libutils/include/utils/Trace.h
/system/core/libutils/Trace.cpp

Look for:

• ATRACE* in c/cpp files
• Trace.traceBegin()/Trace.traceEnd() in Java files

• Use in C files in 4.4:

    #define ATRACE_TAG ATRACE_TAG_ALWAYS
    ...
    #include <cutils/trace.h>
    ...
    fct() {
          ...
          ATRACE_BEGIN("fct");
          ...
          ATRACE_END();
          ...
    }

• Use in C++ files -- you can also use ATRACE_CALL():

    #define ATRACE_TAG ATRACE_TAG_ALWAYS
    ...
    #include <utils/Trace.h>
    ...
    fct() {
          ATRACE_CALL();
          ...
    }

Gotchas:

• Enabling on the command line:

    # setprop debug.atrace.tags.enableflags ...

• Make sure the trace marker file is writeable (/sys/kernel/tracing/trace_marker):

  • Either mount debugfs at startup

  • Or:

    # chmod 222 /sys/kernel/tracing/trace_marker

3.3. perf

• Initial goal: formal interface for performance counters
  • oprofile used its own custom/external module for those
• Now spans more events than just PMU-based
• Has become a fundamental part of kernel toolset
• Poorly documented
• Steep learning curve
• Great for statistical analysis, not for detailed tracing
• Counters saved on context switch, if per-process
• Works great on x86, not so much on many ARM SoCs
• Fixes/support for ARM SoCs ... with some elbow grease
• Patch necessary for Hikey620
• Implemented in kernel/events/ and tools/perf/

• Documentation:
  • tools/perf/design.txt
  • tools/perf/Documentation/
  • https://perf.wiki.kernel.org/index.php/Tutorial
• MUST READ: "multiplexing and scaling events" in tutorial:
  • Trying to monitor more events than there PMU counters will result in multiplexing and scaling of data collection

• There's a perf system call:

    int sys_perf_event_open(struct perf_event_attr *hw_event_uptr,
                            pid_t pid, int cpu, int group_fd,
                            unsigned long flags);

• Requires CONFIG_PERF_EVENTS

• Unlike ftrace, really can't be used without perf command

    # perf

     usage: perf [--version] [--help] COMMAND [ARGS]

     The most commonly used perf commands are:
       annotate        Read perf.data (created by perf record) and display annotated code
       archive         Create archive with object files with build-ids found in perf.data file
       bench           General framework for benchmark suites
       buildid-cache   Manage build-id cache.
       buildid-list    List the buildids in a perf.data file
       diff            Read two perf.data files and display the differential profile
       evlist          List the event names in a perf.data file
       inject          Filter to augment the events stream with additional information
       kmem            Tool to trace/measure kernel memory(slab) properties
       kvm             Tool to trace/measure kvm guest os
       list            List all symbolic event types
       lock            Analyze lock events
       probe           Define new dynamic tracepoints
       record          Run a command and record its profile into perf.data
       report          Read perf.data (created by perf record) and display the profile
       sched           Tool to trace/measure scheduler properties (latencies)
       script          Read perf.data (created by perf record) and display trace output
       stat            Run a command and gather performance counter statistics
       test            Runs sanity tests.
       timechart       Tool to visualize total system behavior during a workload
       top             System profiling tool.

     See 'perf help COMMAND' for more information on a specific command.

Get basic stats:

    # perf stat -a sleep 5

     Performance counter stats for 'sleep 5':

           5014.375095 task-clock                #    1.000 CPUs utilized
                   371 context-switches          #    0.000 M/sec
                     0 CPU-migrations            #    0.000 M/sec
                   270 page-faults               #    0.000 M/sec
              49315140 cycles                    #    0.010 GHz                     [80.45%]
                     0 stalled-cycles-frontend   #    0.00% frontend cycles idle    [78.46%]
                     0 stalled-cycles-backend    #    0.00% backend  cycles idle    [96.02%]
              16766094 instructions              #    0.34  insns per cycle         [72.67%]
               1826454 branches                  #    0.364 M/sec                   [76.27%]
                158411 branch-misses             #    8.67% of all branches         [76.58%]

           5.013001679 seconds time elapsed

Monitor what functions are using the CPU:

       PerfTop:     935 irqs/sec  kernel:91.6%  exact:  0.0% [1000Hz cycles],  (all, 1 CPU)
    -------------------------------------------------------------------------------

                 samples  pcnt function               DSO
                 _______ _____ ______________________ _____________________

                   13.00 34.2% dvmAsmInstructionStart /system/lib/libdvm.so
                    6.00 15.8% strcmp                 /system/lib/libc.so
                    6.00 15.8% __vfprintf             /system/lib/libc.so
                    6.00 15.8% dlmalloc               /system/lib/libc.so
                    5.00 13.2% dvmJitToInterpNoChain  /system/lib/libdvm.so
    ...

See the events it can monitor:

     # perf list

    List of pre-defined events (to be used in -e):
      cpu-cycles OR cycles                               [Hardware event]
      stalled-cycles-frontend OR idle-cycles-frontend    [Hardware event]
      stalled-cycles-backend OR idle-cycles-backend      [Hardware event]
      instructions                                       [Hardware event]
      cache-references                                   [Hardware event]
      cache-misses                                       [Hardware event]
      branch-instructions OR branches                    [Hardware event]
      branch-misses                                      [Hardware event]
      bus-cycles                                         [Hardware event]

      cpu-clock                                          [Software event]
      task-clock                                         [Software event]
      page-faults OR faults                              [Software event]
      minor-faults                                       [Software event]
      major-faults                                       [Software event]
      context-switches OR cs                             [Software event]
      cpu-migrations OR migrations                       [Software event]
      alignment-faults                                   [Software event]
      emulation-faults                                   [Software event]

continued:

      L1-dcache-loads                                    [Hardware cache event]
      L1-dcache-load-misses                              [Hardware cache event]
      L1-dcache-stores                                   [Hardware cache event]
      L1-dcache-store-misses                             [Hardware cache event]
      L1-dcache-prefetches                               [Hardware cache event]
      L1-dcache-prefetch-misses                          [Hardware cache event]
      L1-icache-loads                                    [Hardware cache event]
      L1-icache-load-misses                              [Hardware cache event]
      L1-icache-prefetches                               [Hardware cache event]
      L1-icache-prefetch-misses                          [Hardware cache event]
      LLC-loads                                          [Hardware cache event]
      LLC-load-misses                                    [Hardware cache event]
      LLC-stores                                         [Hardware cache event]
      LLC-store-misses                                   [Hardware cache event]
      LLC-prefetches                                     [Hardware cache event]
      LLC-prefetch-misses                                [Hardware cache event]
      dTLB-loads                                         [Hardware cache event]
      dTLB-load-misses                                   [Hardware cache event]
      dTLB-stores                                        [Hardware cache event]
      dTLB-store-misses                                  [Hardware cache event]
      dTLB-prefetches                                    [Hardware cache event]
      dTLB-prefetch-misses                               [Hardware cache event]

continued:

      iTLB-loads                                         [Hardware cache event]
      iTLB-load-misses                                   [Hardware cache event]
      branch-loads                                       [Hardware cache event]
      branch-load-misses                                 [Hardware cache event]

      rNNN (see 'perf list --help' on how to encode it)  [Raw hardware event descriptor]

      mem:<addr>[:access]                                [Hardware breakpoint]

      sunrpc:rpc_call_status                             [Tracepoint event]
      sunrpc:rpc_bind_status                             [Tracepoint event]
      sunrpc:rpc_connect_status                          [Tracepoint event]
      ...
      sched:sched_wakeup_new                             [Tracepoint event]
      sched:sched_switch                                 [Tracepoint event]
      sched:sched_migrate_task                           [Tracepoint event]
      sched:sched_process_free                           [Tracepoint event]
      ...
      irq:irq_handler_entry                              [Tracepoint event]
      irq:irq_handler_exit                               [Tracepoint event]
      ...

All tracepoint events can be monitored by perf

In-depth profiling of a single application

• Commands
  • perf record -- generate
  • perf report -- analyze
  • perf annotate -- analyze
• Samples based on PMU counter overflow (2^64)
• They all operate on perf.data files
• Annotation requires compile with -ggdb
  • Otherwise you just get disassembly
• Annotation also requires rebuild w/ proper path to appropriate objdump
  • Recent versions have a --objdump= option
• Can record system-wide or one single process

Recording system-wide:

    # perf record -a sleep 30

Reading the report:

    # perf report
    no symbols found in /system/bin/mpdecision, maybe install a debug package?
    Failed to open /init, continuing without symbols
    no symbols found in /system/bin/mksh, maybe install a debug package?
    Failed to open /sbin/adbd, continuing without symbols
    ...
    Kernel address maps (/proc/{kallsyms,modules}) were restricted.

    Check /proc/sys/kernel/kptr_restrict before running 'perf record'.

    If some relocation was applied (e.g. kexec) symbols may be misresolved.

    Samples in kernel modules can't be resolved as well.
    ...

    # Events: 15K cycles
    #
    # Overhead          Command                    Shared Object               ...              Symbol
    # ........  ...............  ...............................  ....................................
    #
        61.17%             adbd  [unknown]                        [k] 0xc07c5cd4
         5.19%             adbd  dumpsys                          [.] 0x1b8f8 
         4.52%             perf  [unknown]                        [k] 0xc07c3fe0
         3.46%          swapper  [unknown]                        [k] 0xc07c5d0c
         2.71%           logcat  [unknown]                        [k] 0xc029b0d0
         2.57%      kworker/0:0  [unknown]                        [k] 0xc07c5cd4
         1.76%       mpdecision  [unknown]                        [k] 0xc029a77c
         1.53%    system_server  dumpsys                          [.] 0x3c18c 
         1.39%    system_server  [unknown]                        [k] 0xc0087710
         0.63%               ls  [unknown]                        [k] 0xc0008578
         0.51%             perf  dumpsys                          [.] 0x25fc8 
         0.41%  ndroid.launcher  dumpsys                          [.] _Z17dvmHeapBitmapWalkPK10HeapBitmapPFvP6ObjectPvES4_
         0.39%  d.process.media  dumpsys                          [.] 0x39c18 
         0.39%    system_server  dumpsys                          [.] 0x81740 
         0.37%    system_server  dumpsys                          [.] 0x5226  
         0.36%           logcat  dumpsys                          [.] 0x18f4  
         0.36%    system_server  dumpsys                          [.] dvmAsmInstructionStart
         0.32%               ps  [unknown]                        [k] 0xc07c7940
         0.28%             perf  dumpsys                          [.] dlfree
         0.27%  ndroid.launcher  [unknown]                        [k] 0xc07c58d4
         0.27%             perf  dumpsys                          [.] memcpy
     ...

Support in Android

• simpleperf tool in system/extras/simpleperf
• Won't build on user (production) builds
• Reduced perf toolset command surface
• Relies on Python scripts for some functionality
  • See scripts/ subdirectory
• Very detailed README.md at top level
• Can record multipled PIDs and TIDs simultaneously

3.4. SystemTap

• Problem:
  kprobes requires hand-crafted modules, for each probe point
• Need:
  Higher-level mechanism for defining and handling probe points
• Solution:
  SystemTap
• Built on kprobe mechanism
• External project from the kernel (IBM, RedHat, Intel)
• Effectively deprecates DProbes
• Full-fledged scripting language for creating/handling probes
• HUGE number of canned scripts for all sorts of uses
• https://sourceware.org/systemtap/

Android support

• None officially -- not in AOSP
• Maybe?:
  https://github.com/flipreverse/systemtap-android
• Also: requires a compiler to build the modules ...
• See here for a good discussion of the issues ... and a diagram:
  http://omappedia.org/wiki/Systemtap#Systemtap_and_Cross_Compilation

Resources

https://sourceware.org/systemtap/wiki

https://sourceware.org/systemtap/tutorial/

https://sourceware.org/systemtap/tapsets/

3.5. eBPF

• Problem:
  • SystemTap requires a compiler
  • SystemTap requires loading modules
• Need:
  • Something similar to SystemTap, minus its issues
• Based on "Berkeley Packet Filter":
  • Bytecode for packet filtering -- register-based VM
  • Originally designed in the early '90s
  • Introduced to Linux in the 2.5 series
  • See "The BSD Packet Filter: A New Architecture for User-level Packet Capture"

• BPF ISA out of date w/ modern-day, 64-bit architectures
• "extended BPF" (eBPF) introduced to:
  • Solve BPF shortcomings
  • Be useful as a general-purpose in-kernel VM
  • Added instruction to call on kernel functions
• In-kernel JIT, available for most major archs, incl. ARM
• Enables attaching a "program" to a kernel codepath
• Programs verified by "verifier" before loading
• Programs can be attached using:
  • Tracepoints
  • kprobes/uprobes
  • perf events
  • USDT probes
• Generally, programs unloaded at process exit

3.5.1. Verifier

• Checks that programs:
  • Cannot crash the kernel
  • Don't contain loops:
    • Only Direct Acyclical Graph (DAG) supported
  • Don't contain unused instructions
  • Terminate
  • Don't contain out-of-bounds jumps
  • Don't access out-of-range data
  • Can do pointer arithmetic only if they're allowed
  • Don't use uninitialized registers
  • Have limited kernel function access based on type
• Details in kernel/bpf/verifier.c
• Doc in Documentation/networking/filter.txt

3.5.2. bpf() system call

• System call for all eBPF functionality:
  • Load/control eBPF programs
  • Create/manage eBPF maps
• eBPF map is primary construct for user-space data exchange
• Program type dictates:
  1. Attachment location
  2. In-kernel helper functions available
  3. Direct access to network packet data
  4. First argument object type
• Program type is effectively API surface
• Several network-related types
• Tracing-related types:
  • BPF_PROG_TYPE_KPROBE
  • BPF_PROG_TYPE_TRACEPOINT
  • BPF_PROG_TYPE_EVENT

• Commands that can be provided to bpf() syscall:
  • BPF_MAP_CREATE
  • BPF_MAP_LOOKUP_ELEM
  • BPF_MAP_UPDATE_ELEM
  • BPF_MAP_DELETE_ELEM
  • BPF_MAP_GET_NEXT_KEY
  • BPF_PROG_LOAD
• There's a map of loaded eBPF programs
• Can jump to other loaded eBPF programs
• eBPF objects can be transfered over Unix domain sockets

3.5.3. Maps

• Data exchange between kernel and user-space
• Key-based access
• File descriptor returned on successful creation
• Generally deleted on file close, though can be kept around
• For each map:
  • Type
  • Max nbr elements
  • Value size (bytes)
  • Key size (bytes)
• Type examples:
  • BPF_TABLE
  • BPF_HASH
  • BPF_ARRAY
  • BPF_HISTOGRAM

3.5.4. Writing eBPF programs

• Kernel contains "raw" samples in samples/bpf/
• Relies on Clang/LLVM for building low-level eBPF portions
  • Clang has a -march=bpf parameter
• Humans use the "BPF Compiler Collection"
• Python-centric environment for writing eBPF programs
• Contains quite a few built-in tools:
  • See tools/
  • Look at *_example.txt
• Each program is combination of:
  • C-style eBPF program to be loaded into kernel
  • Python user-space code to process map data
• User-space and kernel space exchange data using maps
• BCC ensures that programs generates will pass verifier

3.5.5. Hello World!

#!/usr/bin/env python
# Copyright (c) PLUMgrid, Inc.
# Licensed under the Apache License, Version 2.0 (the "License")

# run in project examples directory with:
# sudo ./hello_world.py"
# see trace_fields.py for a longer example

from bcc import BPF

# This may not work for 4.17 on x64, you need replace kprobe__sys_clone with kprobe____x64_sys_clone
BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()

#!/usr/bin/env python

from bcc import BPF

program='''
int kprobe__sys_clone(void *ctx)
{
bpf_trace_printk("Hello, World!\\n");
return 0;
}
'''

BPF(text=program).trace_print()

3.5.6. Kernel Config

• Must haves:

CONFIG_KPROBES=y
CONFIG_KPROBE_EVENT=y
CONFIG_BPF_SYSCALL=y

• Optional:

CONFIG_UPROBES=y
CONFIG_UPROBE_EVENT=y

3.5.7. Android Integration

• BCC assumes a full Linux environment
• No AOSP integration of BCC
• Initial attempt: "BPFd"
• BPFd now deprecated
• Use of "Androdeb" project to load BCC
• Debian chroot() jail under Android
• Installs under /data/androdeb
• Occupies about 300MB
• Includes several key utilities: gcc, make, vi, nano, etc.
• Can get more packages using apt install

• Installation -- device must be connected to host:

  $ cd [aosp]/
  $ . build/envsetup.sh
  $ lunch hikey-eng
  $ cd external/
  $ git clone https://github.com/joelagnel/adeb.git
  $ cd adeb/
  $ ./adeb prepare --full

• Use:

  $ ./adeb shell
  root@localhost:/# 

• Trying BCC commands:

  root@localhost:/# execsnoop 
  PCOMM            PID    PPID   RET ARGS
  sh               2964   1944     0  
  getprop          2966   2964     0 /system/bin/getprop ro.product.device
  ps               2968   2964     0  
   ...      

References

• man 2 bpf
• "BPF Compiler Collection (BCC)"
• "bcc Reference Guide"
• "BPF: the universal in-kernel virtual machine"
• "A thorough introduction to eBPF"
• "Extending extended BPF"
• "Some advanced BCC topics"
• "Using user-space tracepoints with BPF"
• "BPF tracing filters"
• "adeb"
• "BCC (BPF compiler collection) for Android"

3.6. LTTng

• Complete rewrite of the Linux Trace Toolkit
• Extremely scalable, low-overhead
• Very effective user-space tracing (UST)
• Mostly maintained out of tree:
  • Loadable module
  • Relies on existing kernel tracepoint functionality
• http://lttng.org/
• Very powerful visualization tools
• No "official" support for or in Android:
  • Scattered patches for Android support
  • User-space tracing requires SHM
• trace_marker-like functionality upstreamed

Resources:

https://lwn.net/Articles/491510/

https://lwn.net/Articles/492296/

3.7. sysdig

• Yet another tracing tool
• Now mostly centered around containers
• See: http://sysdig.com

3.8. ktap

• Before eBPF took off, there was ktap
• Compiles scripts into bytecode
• Bytecode is interpreted by lua-based VM in kernel:
  • Seriously, it sounds scarier than it actually is
• Released in May 2013
• Initially positive feedback from key kernel developers
• Nack'ed by Ingo Molnar
• Aimed to be the "DTrace" of Linux
• References:

http://events.linuxfoundation.org/sites/events/files/lcjpcojp13_zhangwei.pdf

https://github.com/ktap/ktap

https://lwn.net/Articles/531059/

3.9. dtrace

• Legacy Sun tool
• Got a lot of marketing from Sun
• Offered in binary-only form in Linux for "ever"
• Released in source form after eBPF took off
• Mostly a historical footnote at this point

3.10. oprofile

• System profiler: both kernel and user-space
• Originally based on system timer
• Relies on performance counters
• Effectively obsoleted by perf
• http://oprofile.sourceforge.net

4. Debugging

• kgdb/kdb
• Other kernel debugging mechanisms
• JTAG

4.1. kgdb/kdb

• Built-in kernel debugger
• Two modes of operation:
  • kdb -> live analysis / peaking
    • Console/keyboard/serial
    • Magic Sysrq
  • kgdb -> source-level debugging
    • Remote gdb debugging
    • target remote ...
• "x86-centric" concept
• There's only so much you can do with this
• Documentation/DocBook/kgdb.tmpl

Internals

• Core: kernel/debug/debug_core.c
• Arch-specific code: arch/*/kernel/kgdb.c
• gdb stub: kernel/debug/gdbstub.c
• kdb front-end: kernel/debug/kdb
• kgdb I/O driver:
  • drivers/tty/serial/kgdboc.c
  • drivers/usb/early/ehci-dbgp.c
• Test suite: drivers/misc/kgdbts.c

4.2 Other kernel debugging mechanisms

• Crash dumps
  • kexec new kernel to dump-capture failed kernel
  • x86- and big-iron-centric
  • Documentation/kdump/kdump.txt
• Oops messages
  • Kernel errors/exceptions reported to dmesg
  • Documentation/oops-tracing.txt
• Dynamic debug
  • Dynamically-enable in-kernel debug info
  • Documentation/dynamic-debug-howto.txt

4.3. JTAG

• True geeks use JTAG debuggers
• See what the SoC is really doing
• Several vendors out there: Lauterbach, ARM, ...
• Typically N*$1,000, where N > 2
• Open source: Flyswater 2 (HW) + OpenOCD (SW)
• Use/operation JTAG-vendor specific
• Typical:
  • gdb-based
  • Setup file to prep processor for debug
  • Need vmlinux file
  • Module debugging requires relocation info

Android-Agnostic User-Space Tools

• strace
• ltrace
• top
• Core dumps
• LTTng UST
• apitrace
• GAPID

5.1. strace

• Classic Unix system call tracer
• Trace system calls and signals
• Relies on ptrace(), PTRACE_SYSCALL
• Pros:
  • Detailed info
  • Very simple to use
• Cons:
  • Modifies application behavior
• Included by default on Android
• external/strace/
• man strace on Linux host

• Several modes of operation:
  • Track existing PID
  • Start and track command
  • Save output in separate file
• Recommended
  • Use the -o flag to provide output file
  • Read output file separate from command output
• Beware:
  • Rumor has it that AOSP-packaged one sometimes has the wrong syscall table

        # strace -o data/logcat.strace logcat
        ...
        # cat data/logcat.strace 
        execve("/system/bin/logcat", ["logcat"], [/* 14 vars */]) = 0
        mprotect(0x4000f000, 4096, PROT_READ)   = 0
        open("/dev/null", O_RDWR)               = 3
        fcntl64(0, F_GETFL)                     = 0x2 (flags O_RDWR)
        fcntl64(1, F_GETFL)                     = 0x2 (flags O_RDWR)
        fcntl64(2, F_GETFL)                     = 0x2 (flags O_RDWR)
        close(3)                                = 0
        gettid()                                = 798
        set_tls(0x40010efc, 0x40010efc, 0, 0xffffffbc, 0x40010ffc) = 0
        sigaction(SIGILL, {0x40000a41, [], SA_RESTART|SA_SIGINFO}, NULL, 0x2a04b038) = 0
        sigaction(SIGABRT, {0x40000a41, [], SA_RESTART|SA_SIGINFO}, NULL, 0x2a04b038) = 0
        sigaction(SIGBUS, {0x40000a41, [], SA_RESTART|SA_SIGINFO}, NULL, 0x2a04b038) = 0
        sigaction(SIGFPE, {0x40000a41, [], SA_RESTART|SA_SIGINFO}, NULL, 0x2a04b038) = 0
        sigaction(SIGSEGV, {0x40000a41, [], SA_RESTART|SA_SIGINFO}, NULL, 0x2a04b038) = 0
        sigaction(SIGSTKFLT, {0x40000a41, [], SA_RESTART|SA_SIGINFO}, NULL, 0x2a04b038) = 0
        sigaction(SIGPIPE, {0x40000a41, [], SA_RESTART|SA_SIGINFO}, NULL, 0x2a04b038) = 0
        mmap2(NULL, 4092, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0) = 0x40012000
        open("/vendor/lib/liblog.so", O_RDONLY) = -1 ENOENT (No such file or directory)
        open("/system/lib/liblog.so", O_RDONLY) = 3
        ...

5.2. ltrace

• Problem:
  • Sometimes need to track library calls (.so)
  • strace can only trace system calls
• Solution:
  • ltrace
• Same concept as strace but for libraries
• Inserts breakpoints at symbol entry points
• Pros:
  • Detailed info
  • Very simple to use
• Cons:
  • Modifies application behavior
• Used to be included by default in Android

Resources

• man ltrace
• http://ltrace.org/ (link to git repo)
• http://www.opersys.com/blog/ltrace-internals-140120

5.3. top

• Works the same are regular Linux top

5.4. Core dumps

• Android doesn't generate core dumps by default
• Can, however, change that
• Check if core dumps enabled: ulimit -a
• Enable core dumps in a given shell: ulimit -c unlimited
• Need to copy core file from device to host
• Use cross-debugger to analyze content

5.5. LTTng UST

• User-space tracing mechanism for LTTng
• Provides integration with LTTng toolsuite
• Relies on SHM, which isn't exposed by Bionic
• http://lttng.org/ust
• Some Android efforts but no official support (yet)

5.6. apitrace

• "Generic" OpenGL tracing tool
• Relies on LD_PRELOAD
• No breakpoints = no behavior modification
• Records all calls to OpenGL
• Allows replay
• Has graphic analysis tools
• Some Android support upstream
• http://apitrace.github.io/
• https://github.com/apitrace/apitrace

5.7. GAPID

• Graphics API Debugger
• Recommended by Google to App developers
• Uses Android-custom mechanism to intercept calls
• See installer.cpp
• Like apitrace, doesn't rely on breakpoints
• Replay-capable
• Includes GUI tool (gapic)
• https://github.com/google/gapid

Android-Specific User-Space Tools

6.1. dumpstate / bugreport

• Get a complete dump of system state
• Reads from a lot of data sources
  • logcat
  • dumpsys
  • /proc
  • etc.
• Two versions:
  • dumpstate requires root
  • bugreport doesn't require root
• bureport invokes dumpstate over socket
• adb bugreport generates zip file
  • Works on any production device
• dumpstate HIDL/HAL enables custom data dumping

usage: dumpstate [-h] [-b soundfile] [-e soundfile] [-o file] [-d] [-p] [-z]] [-s] [-S] [-q] [-B] [-P] [-R] [-V version]
  -h: display this help message
  -b: play sound file instead of vibrate, at beginning of job
  -e: play sound file instead of vibrate, at end of job
    -o: write to file (instead of stdout)
    -d: append date to filename (requires -o)
    -p: capture screenshot to filename.png (requires -o)
    -z: generate zipped file (requires -o)
    -s: write output to control socket (for init)
    -S: write file location to control socket (for init; requires -o and -z)  -q: disable vibrate
    -B: send broadcast when finished (requires -o)
    -P: send broadcast when started and update system properties on progress (requires -o and -B)
    -R: take bugreport in remote mode (requires -o, -z, -d and -B, shouldn't be used with -P)
    -v: prints the dumpstate header and exit

6.2. Global properties

• init process maintains set of global properties
• Can:
  • List properties: getprop
  • Set properties: setprop
• Property files:
  • /default.prop
  • /system/build.prop
  • /system/default.prop
  • /data/local.prop
• Property triggers in init's .rc files

6.3. librank

Print library memory usage

    Usage: librank [ -P | -L ] [ -v | -r | -p | -u | -h ]

    Sort options:
        -v  Sort processes by VSS.
        -r  Sort processes by RSS.
        -p  Sort processes by PSS.
        -u  Sort processes by USS.
            (Default sort order is PSS.)
        -P /path  Limit libraries displayed to those in path.
        -R  Reverse sort order (default is descending).
        -h  Display this help screen.

     RSStot      VSS      RSS      PSS      USS  Name/PID
     55386K                                      /dev/ashmem/dalvik-heap
              29340K   29340K   23506K   23272K    com.android.systemui [645]
              13680K   13680K    7753K    7516K    com.android.launcher [765]
              11240K   11240K    5406K    5172K    system_server [565]
               7664K    7664K    1628K    1384K    com.android.phone [737]
               7552K    7552K    1521K    1280K    android.process.media [692]
               7392K    7392K    1326K    1076K    android.process.acore [818]
               7228K    7228K    1184K     940K    com.android.inputmethod.latin [710]
               7108K    7108K    1031K     784K    com.android.email [1091]
     ...

     40517K                                      anon_inode:dmabuf
              39972K   39972K   25758K   11544K    /system/bin/surfaceflinger [253]
              16172K   16172K    8142K     132K    system_server [565]
              11884K   11884K    5944K       4K    com.android.launcher [765]
                964K     964K     673K     408K    com.android.systemui [645]
                408K       0K       0K       0K    /system/bin/mediaserver [256]
                 40K       0K       0K       0K    /system/bin/qseecomd [341]

     19489K                                      /dev/ashmem/dalvik-aux-structure
               1480K    1456K    1194K    1184K    system_server [565]
               1812K    1740K    1116K    1088K    com.android.email [1091]
               1628K    1552K    1087K    1068K    com.android.phone [737]
               1824K    1740K    1076K    1044K    com.android.contacts [904]
               1656K    1572K    1050K    1028K    android.process.media [692]
               1760K    1684K     982K     944K    com.android.settings [801]
     ...

6.4. procmem

See PID's memory usage

    Usage: procmem [ -w | -W ] [ -p | -m ] [ -h ] pid
        -w  Displays statistics for the working set only.
        -W  Resets the working set of the process.
        -p  Sort by PSS.
        -m  Sort by mapping order (as read from /proc).
        -h  Hide maps with no RSS.

     # procmem 565
        Vss      Rss      Pss      Uss     ShCl     ShDi     PrCl     PrDi  Name
    -------  -------  -------  -------  -------  -------  -------  -------  
         4K       4K       4K       4K       0K       0K       4K       0K  �@
         4K       4K       0K       0K       4K       0K       0K       0K  /dev/ashmem/SurfaceFlinger
         4K       4K       2K       0K       4K       0K       0K       0K  /system/app/SettingsProvider.apk
         4K       4K       2K       0K       4K       0K       0K       0K  /system/app/SettingsProvider.apk
         0K       0K       0K       0K       0K       0K       0K       0K  /system/framework/framework_ext.jar
         8K       8K       0K       0K       8K       0K       0K       0K  /system/lib/libstagefright_yuv.so
         4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libstagefright_yuv.so
         4K       4K       4K       4K       0K       0K       4K       0K  /system/lib/libstagefright_yuv.so
         8K       8K       8K       8K       0K       0K       8K       0K  /system/app/SettingsProvider.apk
         0K       0K       0K       0K       0K       0K       0K       0K  /system/framework/core-junit.jar
         8K       8K       8K       8K       0K       0K       8K       0K  /system/framework/core-junit.jar
        32K      32K       8K       8K      24K       0K       8K       0K  /system/framework/core-junit.jar
         8K       8K       8K       8K       0K       0K       8K       0K  /dev/ashmem/dalvik-aux-structure
        64K      64K       7K       0K      64K       0K       0K       0K  /system/lib/libm.so
         0K       0K       0K       0K       0K       0K       0K       0K  /system/lib/libm.so
         4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libm.so
        16K      16K       8K       8K       8K       0K       8K       0K  /system/lib/libm.so
         0K       0K       0K       0K       0K       0K       0K       0K  /system/framework/android.policy.jar
         4K       4K       0K       0K       4K       0K       0K       0K  /system/framework/android.policy.jar
         8K       8K       8K       8K       0K       0K       8K       0K  /system/framework/android.policy.jar
     ...

6.5. procrank

See processes' memory usage, in order

    Usage: procrank [ -W ] [ -v | -r | -p | -u | -h ]
        -v  Sort by VSS.
        -r  Sort by RSS.
        -p  Sort by PSS.
        -u  Sort by USS.
            (Default sort order is PSS.)
        -R  Reverse sort order (default is descending).
        -w  Display statistics for working set only.
        -W  Reset working set of all processes.
        -h  Display this help screen.

    # procrank
      PID      Vss      Rss      Pss      Uss  cmdline
      565   77364K   77216K   36443K   24816K  system_server
      645   63492K   63172K   31496K   28356K  com.android.systemui
      253   64300K   51900K   31349K   15944K  /system/bin/surfaceflinger
      765   67408K   67116K   28784K   19532K  com.android.launcher
      818   35496K   35392K    7159K    5356K  android.process.acore
      737   35084K   34984K    6936K    5444K  com.android.phone
      254   37100K   36908K    6758K    4392K  zygote
      710   34420K   34340K    6347K    4916K  com.android.inputmethod.latin
      692   33404K   33236K    5879K    4644K  android.process.media
     1091   32892K   32736K    5436K    4232K  com.android.email
      256    9392K    8980K    5018K    4812K  /system/bin/mediaserver
      904   31524K   31356K    4505K    3336K  com.android.contacts
     1141   31468K   31316K    4336K    3160K  com.android.mms
     1052   31676K   31508K    4252K    3064K  com.android.providers.calendar
      801   31016K   30916K    4190K    2988K  com.android.settings
     1230   30896K   30728K    3955K    2784K  com.android.calendar
 ...

6.6. showmap

See objects mapped to process' address space

        
showmap [-t] [-v] [-c] [-q] <pid>
        -t = terse (show only items with private pages)
        -v = verbose (don't coalesce maps with the same name)
        -a = addresses (show virtual memory map)
        -q = quiet (don't show error if map could not be read)

    # showmap 565                                               
     virtual                     shared   shared  private  private
        size      RSS      PSS    clean    dirty    clean    dirty    # object
    -------- -------- -------- -------- -------- -------- -------- ---- ------------------------------
          68       60       60        0        0       60        0    1 /data/dalvik-cache/system@app@SettingsProvider.apk@classes.dex
         336      276      135      192        0       84        0    1 /data/dalvik-cache/system@framework@android.policy.jar@classes.dex
        1348       32        1       32        0        0        0    1 /data/dalvik-cache/system@framework@apache-xml.jar@classes.dex
         960       92        6       92        0        0        0    1 /data/dalvik-cache/system@framework@bouncycastle.jar@classes.dex
         124      112      112        0        0      112        0    1 /data/dalvik-cache/system@framework@com.quicinc.cne.jar@classes.dex
          28       12        0       12        0        0        0    1 /data/dalvik-cache/system@framework@core-junit.jar@classes.dex
        3320     1848      278     1780        0       68        0    1 /data/dalvik-cache/system@framework@core.jar@classes.dex
        1468       88        8       88        0        0        0    1 /data/dalvik-cache/system@framework@ext.jar@classes.dex
       11156     6216     1307     5680        0      536        0    1 /data/dalvik-cache/system@framework@framework.jar@classes.dex
         776       36        1       36        0        0        0    1 /data/dalvik-cache/system@framework@framework_ext.jar@classes.dex
        2384     1860     1593      440        0     1420        0    1 /data/dalvik-cache/system@framework@services.jar@classes.dex
          32       32       32        0        0       32        0    1 /data/data/com.android.providers.settings/databases/settings.db-shm
          32       32       32        0        0       32        0    1 /data/system/locksettings.db-shm
          48       32        0        0       32        0        0    1 /dev/__properties__ (deleted)
        8192       16       16        0        0        0       16    4 /dev/ashmem/CursorWindow: /data/data/com.android.providers.settings/databases/settings.db (deleted)
           4        4        0        4        0        0        0    1 /dev/ashmem/SurfaceFlinger read-only heap (deleted)
       ...

6.7. Logger / logcat

    Usage: logcat [options] [filterspecs]
    options include:
      -s              Set default filter to silent.
                      Like specifying filterspec '*:s'
      -f <filename>   Log to file. Default to stdout
      -r [<kbytes>]   Rotate log every kbytes. (16 if unspecified). Requires -f
      -n <count>      Sets max number of rotated logs to <count>, default 4
      -v <format>     Sets the log print format, where <format> is one of:

                      brief process tag thread raw time threadtime long

      -c              clear (flush) the entire log and exit
      -d              dump the log and then exit (don't block)
      -t <count>      print only the most recent <count> lines (implies -d)
      -g              get the size of the log's ring buffer and exit
      -b <buffer>     Request alternate ring buffer, 'main', 'system', 'radio'
                      or 'events'. Multiple -b parameters are allowed and the
                      results are interleaved. The default is -b main -b system.
      -B              output the log in binary
      ...

    --------- beginning of /dev/log/main
    I/BOOT    (  150): MSM target 'msm8960', SoC 'Surf', HwID '109', SoC ver '65536' 
    I/qcom-bluetooth(  289): /system/etc/init.qcom.bt.sh: init.qcom.bt.sh config = onboot 
    I/qrngd   (  275): qrngd has started:
    I/qrngd   (  275): Reading device:'/dev/hw_random' updating entropy for device:'/dev/random'
    I/DMM     (  305): DMM available. movable_start_bytes at 
    I/DEBUG   (  251): debuggerd: Jan 10 2014 20:38:46
    D/PPDaemon(  287): isHDMIPrimary: HDMI is not primary display
    D/PPDaemon(  287): CABL version 1.0.20120512
    I/qcom-bluetooth(  311): /system/etc/init.qcom.bt.sh: Bluetooth Address programmed successfully 
    --------- beginning of /dev/log/system
    I/Vold    (  246): Vold 2.1 (the revenge) firing up
    E/PPDaemon(  287): Failed to open the config file!
    D/Vold    (  246): Volume sdcard state changing -1 (Initializing) -> 0 (No-Media)
    D/QSEECOMD: (  293): qseecom listener services process entry PPID = 1
    D/QSEECOMD: (  293): Parent qseecom daemon process paused!! 
    D/QSEECOMD: (  341): QSEECOM DAEMON RUNNING 
    D/QSEECOMD: (  341): qseecom listener service threads starting!!! 
    D/QSEECOMD: (  341): Total listener services to start = 2
    D/QSEECOMD: (  341): Init dlopen(libdrmtime.so, RTLD_NOW) succeeds
    D/QSEECOMD: (  341): Init::Init dlsym(g_FSHandle atime_start) succeeds
    ...

    # log
    USAGE: log [-p priorityChar] [-t tag] message
        priorityChar should be one of:
            v,d,i,w,e

6.8. tombstones

Closest thing to "core dumps"

    # ls /data/tombstones/ -al
    drwxrwx--x system   system            1970-01-01 06:51 dsps
    drwxrwx--x system   system            1970-01-01 06:51 lpass
    drwxrwx--x system   system            1970-01-01 06:51 mdm
    drwxrwx--x system   system            1970-01-01 06:51 modem
    drwxrwx--x system   system            1970-01-01 06:51 wcnss

Usually actual files are called tombstone_XX where XX is a number.

    *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
    Build fingerprint: 'Android/aosp_arm/generic:4.4/KRT16M/eng.karim.20131112.142320:eng/test-keys'
    Revision: '0'
    pid: 1150, tid: 1150, name: vdc  >>> vdc <<<
    signal 13 (SIGPIPE), code -6 (SI_TKILL), fault addr --------
        r0 ffffffe0  r1 b7a5c028  r2 00000457  r3 00000888
        r4 b6ef01a4  r5 b7a5c028  r6 00000457  r7 00000004
        r8 00001000  r9 00000000  sl b6f00ee4  fp 0000000c
        ip b6efe2fc  sp bed41a30  lr b6ecb89f  pc b6ec7178  cpsr 20000010
        d0  a9c01b6937fe9a6b  d1  0000000000000000
        d2  0000000000000000  d3  0000000000000000
        d4  0000000000000000  d5  41cbff4d35800000
        d6  3f50624dd2f1a9fc  d7  c1d58ff925dc7ae1
        d8  0000000000000000  d9  0000000000000000
        d10 0000000000000000  d11 0000000000000000
        d12 0000000000000000  d13 0000000000000000
        d14 0000000000000000  d15 0000000000000000
        scr 00000010
    ...

    backtrace:
        #00  pc 00020178  /system/lib/libc.so (write+12)
        #01  pc 0002489d  /system/lib/libc.so (__sflush+54)
        #02  pc 00014393  /system/lib/libc.so (fclose+54)
        #03  pc 0000d939  /system/lib/libc.so
        #04  pc 0000eecc  /system/lib/libc.so (pthread_once+104)
        #05  pc 0000db93  /system/lib/libc.so
        #06  pc 00027ded  /system/lib/libc.so (__cxa_finalize+156)
        #07  pc 00027fe5  /system/lib/libc.so (exit+6)
        #08  pc 00000b03  /system/bin/vdc
        #09  pc 0000e23b  /system/lib/libc.so (__libc_init+50)
        #10  pc 000007f0  /system/bin/vdc

    stack:
             bed419f0  00000000  
             bed419f4  00000000  
             bed419f8  00000000
         ...

6.9. debuggerd

• Daemon running in background
• Allows catching crashing processes
• Set debug.db.uid to "greater than" UID to trigger

• Linker has code for latching to debuggerd

    I/DEBUG   (  365): ********************************************************
    I/DEBUG   (  365): * Process 984 has been suspended while crashing.  To
    I/DEBUG   (  365): * attach gdbserver for a gdb connection on port 5039:
    I/DEBUG   (  365): *
    I/DEBUG   (  365): *     adb shell gdbserver :5039 --attach 984 &
    I/DEBUG   (  365): *
    I/DEBUG   (  365): * Press HOME key to let the process continue crashing.
    I/DEBUG   (  365): ********************************************************

6.10. input

Send input to input layer

Usage: input [<source>] <command> [<arg>...]

The sources are: 
      dpad
      keyboard
      mouse
      touchpad
      gamepad
      touchnavigation
      joystick
      touchscreen
      stylus
      trackball

The commands and default sources are:
      text <string> (Default: touchscreen)
      keyevent [--longpress] <key code number or name> ... (Default: keyboard)
...

6.11. Control init services

• Stop service: stop servicename
• Start service: start servicename
• If no service name is specified:
  • zygote
  • surfaceflinger
• Can mark service as disabled in .rc files

6.12. run-as

Run a command under a given package's user ID

            run-as: usage: run-as <package-name> [--user <uid>] <command> [<args>]

6.13. schedtest

Test scheduler's ability to wake processes up after 1ms

    # schedtest
    max 3449    avg 1171
    max 3418    avg 1170
    max 3205    avg 1167
    max 2380    avg 1162
    max 3449    avg 1169
    max 9340    avg 1179
    max 3418    avg 1168
    max 3388    avg 1168
    max 3418    avg 1170
    max 3388    avg 1168
    max 3418    avg 1167
    ...

6.14. adb

Java Tools

• dalvikvm
• app_process
• dexdump
• jdb/jdwp
• junit
• Memory Profiler

7.1. dalvikvm

• Raw Dalvik VM
• Can't run Android code

• Seldom used

    # dalvikvm -help
  
    dalvikvm: [options] class [argument ...]
    dalvikvm: [options] -jar file.jar [argument ...]
  
    The following standard options are recognized:
      -classpath classpath
      -Dproperty=value
      -verbose:tag  ('gc', 'jni', or 'class')
      -ea[:<package name>... |:<class name>]
      -da[:<package name>... |:<class name>]
       (-enableassertions, -disableassertions)
      -esa
      -dsa
       (-enablesystemassertions, -disablesystemassertions)
      -showversion
      -help
      ...

7.2. app_process

• Magic command to start ART instance
• Entirely coded in C

• Used to start initial Zygote in init.rc

    service zygote /system/bin/app_process -Xzygote /system/bin --zygote --start-system-server

• Also used to start a variety of Java-coded framework commands:

  • am
  • pm
  • wm
  • svc
  • monkey
  • ...

7.3. dexdump

Dex file disassembler

    dexdump: [-c] [-d] [-f] [-h] [-i] [-l layout] [-m] [-t tempfile] dexfile...

     -c : verify checksum and exit
     -d : disassemble code sections
     -f : display summary information from file header
     -h : display file header details
     -i : ignore checksum failures
     -l : output layout, either 'plain' or 'xml'
     -m : dump register maps (and nothing else)
     -t : temp file name (defaults to /sdcard/dex-temp-*)

    # dexdump /system/app/Launcher2.apk
    Processing '/system/app/Launcher2.apk'...
    Opened '/system/app/Launcher2.apk', DEX version '035'
    Class #0            -
      Class descriptor  : 'Landroid/support/v13/app/FragmentCompat$FragmentCompatImpl;'
      Access flags      : 0x0600 (INTERFACE ABSTRACT)
      Superclass        : 'Ljava/lang/Object;'
      Interfaces        -
      Static fields     -
      Instance fields   -
      Direct methods    -
      Virtual methods   -
        #0              : (in Landroid/support/v13/app/FragmentCompat$FragmentCompatImpl;)
          name          : 'setMenuVisibility'
          type          : '(Landroid/app/Fragment;Z)V'
          access        : 0x0401 (PUBLIC ABSTRACT)
          code          : (none)
          ...

7.4. jdb/jdwp

• jdb = Java's gdb
• jdwp = Java Debug Wire Protocol
• In principle, can use jdb to debug Java processes
• In practice: use Android Studio
• References:
  http://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdwp-spec.html
  http://docs.oracle.com/javase/7/docs/technotes/guides/jpda/jdwpTransport.html
  

7.5. junit

• Java's unit testing framework
• Used extensively in Android
• References:
  https://developer.android.com/studio/test/
  http://junit.org/

7.6. Memory Profiler

• References:
  https://developer.android.com/studio/profile/memory-profiler

System Services Interfacing

• dumpsys
• service (espc. “service call” and aidl files)
• am
• pm
• wm
• svc
• monkey
• ANR dumps

8.1. dumpsys

• Allows you to poke system services
• Calls the system service's dump() function

• By default will dump all system services

    # dumpsys

• Can dump just one system service

    # dumpsys statusbar

• C-based tool

• Some system service dump functions can take parameters

• Of interest:
  • procstats
  • batterystats
  • usagestats
  • ... *stats
  • meminfo
  • gfxinfo
  • cpuinfo
  • ... *info

8.2. service

• Interact with system services

    Usage: service [-h|-?]
           service list
           service check SERVICE
           service call SERVICE CODE [i32 INT | s16 STR] ...
    Options:
       i32: Write the integer INT into the send parcel.
       s16: Write the UTF-16 string STR into the send parcel.
  
    Usage: service [-h|-?]
           service list
           service check SERVICE
                  service call SERVICE CODE [i32 N | i64 N | f N | d N | s16 STR ] ...
    Options:
       i32: Write the 32-bit integer N into the send parcel.
       i64: Write the 64-bit integer N into the send parcel.
       f:   Write the 32-bit single-precision number N into the send parcel.
       d:   Write the 64-bit double-precision number N into the send parcel.
       s16: Write the UTF-16 string STR into the send parcel.

• See system service's aidl file to get "CODE" and parameter list

• C-based tool

8.3. am

• Interact with the Activity Manager
• Allows you to send intents on the command line (very powerful)

Activity manager (activity) commands:
  help
      Print this help text.
  start-activity [-D] [-N] [-W] [-P <file>] [--start-profiler <file>]
          [--sampling INTERVAL] [--streaming] [-R COUNT] [-S]
          [--track-allocation] [--user <user_id> | current] <intent>
      Start an Activity.  Options are:
      -D: enable debugging
      -N: enable native debugging
      -W: wait for launch to complete
      --start-profiler <file>: start profiler and send results to <file>
      --sampling INTERVAL: use sample profiling with INTERVAL microseconds
          between samples (use with --start-profiler)
      --streaming: stream the profiling output to the specified file
          (use with --start-profiler)
      -P <file>: like above, but profiling stops when app goes idle
      --attach-agent <agent>: attach the given agent before binding
      -R: repeat the activity launch <count> times.  Prior to each repeat,
          the top activity will be finished.
      -S: force stop the target app before starting the activity
      --track-allocation: enable tracking of object allocations
      --user <user_id> | current: Specify which user to run as; if not
          specified then run as the current user.
      --stack <stack_id>: Specify into which stack should the activity be put.
  start-service [--user <user_id> | current] <intent>
      Start a Service.  Options are:
      --user <user_id> | current: Specify which user to run as; if not
...
</user_id></intent></user_id></stack_id></user_id></count></agent></file></file></file></intent></user_id></file></file>

8.4. pm

Interact with Package Manager

usage: pm path [--user USER_ID] PACKAGE
       pm dump PACKAGE
       pm install [-lrtsfd] [-i PACKAGE] [--user USER_ID] [PATH]
       pm install-create [-lrtsfdp] [-i PACKAGE] [-S BYTES]
               [--install-location 0/1/2]
               [--force-uuid internal|UUID]
       pm install-write [-S BYTES] SESSION_ID SPLIT_NAME [PATH]
       pm install-commit SESSION_ID
       pm install-abandon SESSION_ID
       pm uninstall [-k] [--user USER_ID] [--versionCode VERSION_CODE] PACKAGE
       pm set-installer PACKAGE INSTALLER
       pm move-package PACKAGE [internal|UUID]
       pm move-primary-storage [internal|UUID]
       pm clear [--user USER_ID] PACKAGE
       pm enable [--user USER_ID] PACKAGE_OR_COMPONENT
       pm disable [--user USER_ID] PACKAGE_OR_COMPONENT
       pm disable-user [--user USER_ID] PACKAGE_OR_COMPONENT
       pm disable-until-used [--user USER_ID] PACKAGE_OR_COMPONENT
       pm default-state [--user USER_ID] PACKAGE_OR_COMPONENT
       pm set-user-restriction [--user USER_ID] RESTRICTION VALUE
       pm hide [--user USER_ID] PACKAGE_OR_COMPONENT
       pm unhide [--user USER_ID] PACKAGE_OR_COMPONENT
       pm grant [--user USER_ID] PACKAGE PERMISSION
       pm revoke [--user USER_ID] PACKAGE PERMISSION
       pm reset-permissions
       pm set-app-link [--user USER_ID] PACKAGE {always|ask|never|undefined}
       pm get-app-link [--user USER_ID] PACKAGE
       pm set-install-location [0/auto] [1/internal] [2/external]
       pm get-install-location
       pm set-permission-enforced PERMISSION [true|false]
       pm trim-caches DESIRED_FREE_SPACE [internal|UUID]
       pm create-user [--profileOf USER_ID] [--managed] [--restricted] [--ephemeral] [--guest] USER_NAME
       pm remove-user USER_ID
       pm get-max-users<p></p>
...


8. System Services Interfacing

8.5. wm

Interact with Window Manager

usage: wm [subcommand] [options]
       wm size [reset|WxH|WdpxHdp]
       wm density [reset|DENSITY]
       wm overscan [reset|LEFT,TOP,RIGHT,BOTTOM]
       wm scaling [off|auto]
       wm screen-capture [userId] [true|false]

wm size: return or override display size.
         width and height in pixels unless suffixed with 'dp'.

wm density: override display density.

wm overscan: set overscan area for display.

wm scaling: set display scaling mode.

wm screen-capture: enable/disable screen capture.

wm dismiss-keyguard: dismiss the keyguard, prompting the user for auth if necessary.

wm surface-trace: log surface commands to stdout in a binary format.

8.6. svc

Interact with various system services

    Available commands:
        help         Show information about the subcommands
        power        Control the power manager
        data         Control mobile data connectivity
        wifi         Control the Wi-Fi manager
        usb          Control Usb state
        nfc          Control NFC functions
        bluetooth    Control Bluetooth service

8.7. monkey

• Interact with UI

• Can take scripts

        usage: monkey [-p ALLOWED_PACKAGE [-p ALLOWED_PACKAGE] ...]
                        [-c MAIN_CATEGORY [-c MAIN_CATEGORY] ...]
                        [--ignore-crashes] [--ignore-timeouts]
                        [--ignore-security-exceptions]
                        [--monitor-native-crashes] [--ignore-native-crashes]
                        [--kill-process-after-error] [--hprof]
                        [--match-description TEXT]
                        [--pct-touch PERCENT] [--pct-motion PERCENT]
                        [--pct-trackball PERCENT] [--pct-syskeys PERCENT]
                        [--pct-nav PERCENT] [--pct-majornav PERCENT]
                        [--pct-appswitch PERCENT] [--pct-flip PERCENT]
                        [--pct-anyevent PERCENT] [--pct-pinchzoom PERCENT]
                        [--pct-permission PERCENT]
                        [--pkg-blacklist-file PACKAGE_BLACKLIST_FILE]
                        [--pkg-whitelist-file PACKAGE_WHITELIST_FILE]
                        [--wait-dbg] [--dbg-no-events]
                        [--setup scriptfile] [-f scriptfile [-f scriptfile] ...]
                        [--port port]
                        [-s SEED] [-v [-v] ...]
                        [--throttle MILLISEC] [--randomize-throttle]
                        [--profile-wait MILLISEC]
                        [--device-sleep-time MILLISEC]
                        [--randomize-script]
                        [--script-log]
                        [--bugreport]
                        [--periodic-bugreport]
                        [--permission-target-system]
                        COUNT

8.8. ANR dumps

• If an app hangs, it'll generate an "Application Not Responding" event
• Info about those ANRs is dumped in files in /data/anr

Other Tools and Techniques

• Power management / DVFS
  • Documentation/cpu-freq/*
  • http://processors.wiki.ti.com/index.php/DVFS_User_Guide
• DS-5
  http://www.arm.com/products/tools/software-tools/ds-5/index.php
• sqlite3
• Using screen overlays (a-la CPU perf by Status Bar)

glibc User-Space

• Running "standard" glibc-based code with Android
• Integrating glibc in Android filesystem
• Building glibc-linked code to run with Android
• Interfacing between a glibc-based stack and the Android-stack

See courseware at
http://www.opersys.com/training/embedded-android