aboutsummaryrefslogtreecommitdiff
path: root/Documentation
diff options
context:
space:
mode:
Diffstat (limited to 'Documentation')
-rw-r--r--Documentation/ABI/testing/debugfs-kmemtrace71
-rw-r--r--Documentation/filesystems/pohmelfs/design_notes.txt70
-rw-r--r--Documentation/filesystems/pohmelfs/info.txt86
-rw-r--r--Documentation/filesystems/pohmelfs/network_protocol.txt227
-rw-r--r--Documentation/ftrace.txt1134
-rw-r--r--Documentation/kernel-parameters.txt33
-rw-r--r--Documentation/laptops/acer-wmi.txt10
-rw-r--r--Documentation/laptops/thinkpad-acpi.txt144
-rw-r--r--Documentation/sysrq.txt2
-rw-r--r--Documentation/tracepoints.txt21
-rw-r--r--Documentation/vm/kmemtrace.txt126
11 files changed, 1499 insertions, 425 deletions
diff --git a/Documentation/ABI/testing/debugfs-kmemtrace b/Documentation/ABI/testing/debugfs-kmemtrace
new file mode 100644
index 00000000000..5e6a92a02d8
--- /dev/null
+++ b/Documentation/ABI/testing/debugfs-kmemtrace
@@ -0,0 +1,71 @@
+What: /sys/kernel/debug/kmemtrace/
+Date: July 2008
+Contact: Eduard - Gabriel Munteanu <eduard.munteanu@linux360.ro>
+Description:
+
+In kmemtrace-enabled kernels, the following files are created:
+
+/sys/kernel/debug/kmemtrace/
+ cpu<n> (0400) Per-CPU tracing data, see below. (binary)
+ total_overruns (0400) Total number of bytes which were dropped from
+ cpu<n> files because of full buffer condition,
+ non-binary. (text)
+ abi_version (0400) Kernel's kmemtrace ABI version. (text)
+
+Each per-CPU file should be read according to the relay interface. That is,
+the reader should set affinity to that specific CPU and, as currently done by
+the userspace application (though there are other methods), use poll() with
+an infinite timeout before every read(). Otherwise, erroneous data may be
+read. The binary data has the following _core_ format:
+
+ Event ID (1 byte) Unsigned integer, one of:
+ 0 - represents an allocation (KMEMTRACE_EVENT_ALLOC)
+ 1 - represents a freeing of previously allocated memory
+ (KMEMTRACE_EVENT_FREE)
+ Type ID (1 byte) Unsigned integer, one of:
+ 0 - this is a kmalloc() / kfree()
+ 1 - this is a kmem_cache_alloc() / kmem_cache_free()
+ 2 - this is a __get_free_pages() et al.
+ Event size (2 bytes) Unsigned integer representing the
+ size of this event. Used to extend
+ kmemtrace. Discard the bytes you
+ don't know about.
+ Sequence number (4 bytes) Signed integer used to reorder data
+ logged on SMP machines. Wraparound
+ must be taken into account, although
+ it is unlikely.
+ Caller address (8 bytes) Return address to the caller.
+ Pointer to mem (8 bytes) Pointer to target memory area. Can be
+ NULL, but not all such calls might be
+ recorded.
+
+In case of KMEMTRACE_EVENT_ALLOC events, the next fields follow:
+
+ Requested bytes (8 bytes) Total number of requested bytes,
+ unsigned, must not be zero.
+ Allocated bytes (8 bytes) Total number of actually allocated
+ bytes, unsigned, must not be lower
+ than requested bytes.
+ Requested flags (4 bytes) GFP flags supplied by the caller.
+ Target CPU (4 bytes) Signed integer, valid for event id 1.
+ If equal to -1, target CPU is the same
+ as origin CPU, but the reverse might
+ not be true.
+
+The data is made available in the same endianness the machine has.
+
+Other event ids and type ids may be defined and added. Other fields may be
+added by increasing event size, but see below for details.
+Every modification to the ABI, including new id definitions, are followed
+by bumping the ABI version by one.
+
+Adding new data to the packet (features) is done at the end of the mandatory
+data:
+ Feature size (2 byte)
+ Feature ID (1 byte)
+ Feature data (Feature size - 3 bytes)
+
+
+Users:
+ kmemtrace-user - git://repo.or.cz/kmemtrace-user.git
+
diff --git a/Documentation/filesystems/pohmelfs/design_notes.txt b/Documentation/filesystems/pohmelfs/design_notes.txt
new file mode 100644
index 00000000000..6d6db60d567
--- /dev/null
+++ b/Documentation/filesystems/pohmelfs/design_notes.txt
@@ -0,0 +1,70 @@
+POHMELFS: Parallel Optimized Host Message Exchange Layered File System.
+
+ Evgeniy Polyakov <zbr@ioremap.net>
+
+Homepage: http://www.ioremap.net/projects/pohmelfs
+
+POHMELFS first began as a network filesystem with coherent local data and
+metadata caches but is now evolving into a parallel distributed filesystem.
+
+Main features of this FS include:
+ * Locally coherent cache for data and metadata with (potentially) byte-range locks.
+ Since all Linux filesystems lock the whole inode during writing, algorithm
+ is very simple and does not use byte-ranges, although they are sent in
+ locking messages.
+ * Completely async processing of all events except creation of hard and symbolic
+ links, and rename events.
+ Object creation and data reading and writing are processed asynchronously.
+ * Flexible object architecture optimized for network processing.
+ Ability to create long paths to objects and remove arbitrarily huge
+ directories with a single network command.
+ (like removing the whole kernel tree via a single network command).
+ * Very high performance.
+ * Fast and scalable multithreaded userspace server. Being in userspace it works
+ with any underlying filesystem and still is much faster than async in-kernel NFS one.
+ * Client is able to switch between different servers (if one goes down, client
+ automatically reconnects to second and so on).
+ * Transactions support. Full failover for all operations.
+ Resending transactions to different servers on timeout or error.
+ * Read request (data read, directory listing, lookup requests) balancing between multiple servers.
+ * Write requests are replicated to multiple servers and completed only when all of them are acked.
+ * Ability to add and/or remove servers from the working set at run-time.
+ * Strong authentification and possible data encryption in network channel.
+ * Extended attributes support.
+
+POHMELFS is based on transactions, which are potentially long-standing objects that live
+in the client's memory. Each transaction contains all the information needed to process a given
+command (or set of commands, which is frequently used during data writing: single transactions
+can contain creation and data writing commands). Transactions are committed by all the servers
+to which they are sent and, in case of failures, are eventually resent or dropped with an error.
+For example, reading will return an error if no servers are available.
+
+POHMELFS uses a asynchronous approach to data processing. Courtesy of transactions, it is
+possible to detach replies from requests and, if the command requires data to be received, the
+caller sleeps waiting for it. Thus, it is possible to issue multiple read commands to different
+servers and async threads will pick up replies in parallel, find appropriate transactions in the
+system and put the data where it belongs (like the page or inode cache).
+
+The main feature of POHMELFS is writeback data and the metadata cache.
+Only a few non-performance critical operations use the write-through cache and
+are synchronous: hard and symbolic link creation, and object rename. Creation,
+removal of objects and data writing are asynchronous and are sent to
+the server during system writeback. Only one writer at a time is allowed for any
+given inode, which is guarded by an appropriate locking protocol.
+Because of this feature, POHMELFS is extremely fast at metadata intensive
+workloads and can fully utilize the bandwidth to the servers when doing bulk
+data transfers.
+
+POHMELFS clients operate with a working set of servers and are capable of balancing read-only
+operations (like lookups or directory listings) between them.
+Administrators can add or remove servers from the set at run-time via special commands (described
+in Documentation/pohmelfs/info.txt file). Writes are replicated to all servers.
+
+POHMELFS is capable of full data channel encryption and/or strong crypto hashing.
+One can select any kernel supported cipher, encryption mode, hash type and operation mode
+(hmac or digest). It is also possible to use both or neither (default). Crypto configuration
+is checked during mount time and, if the server does not support it, appropriate capabilities
+will be disabled or mount will fail (if 'crypto_fail_unsupported' mount option is specified).
+Crypto performance heavily depends on the number of crypto threads, which asynchronously perform
+crypto operations and send the resulting data to server or submit it up the stack. This number
+can be controlled via a mount option.
diff --git a/Documentation/filesystems/pohmelfs/info.txt b/Documentation/filesystems/pohmelfs/info.txt
new file mode 100644
index 00000000000..4e3d5015708
--- /dev/null
+++ b/Documentation/filesystems/pohmelfs/info.txt
@@ -0,0 +1,86 @@
+POHMELFS usage information.
+
+Mount options:
+idx=%u
+ Each mountpoint is associated with a special index via this option.
+ Administrator can add or remove servers from the given index, so all mounts,
+ which were attached to it, are updated.
+ Default it is 0.
+
+trans_scan_timeout=%u
+ This timeout, expressed in milliseconds, specifies time to scan transaction
+ trees looking for stale requests, which have to be resent, or if number of
+ retries exceed specified limit, dropped with error.
+ Default is 5 seconds.
+
+drop_scan_timeout=%u
+ Internal timeout, expressed in milliseconds, which specifies how frequently
+ inodes marked to be dropped are freed. It also specifies how frequently
+ the system checks that servers have to be added or removed from current working set.
+ Default is 1 second.
+
+wait_on_page_timeout=%u
+ Number of milliseconds to wait for reply from remote server for data reading command.
+ If this timeout is exceeded, reading returns an error.
+ Default is 5 seconds.
+
+trans_retries=%u
+ This is the number of times that a transaction will be resent to a server that did
+ not answer for the last @trans_scan_timeout milliseconds.
+ When the number of resends exceeds this limit, the transaction is completed with error.
+ Default is 5 resends.
+
+crypto_thread_num=%u
+ Number of crypto processing threads. Threads are used both for RX and TX traffic.
+ Default is 2, or no threads if crypto operations are not supported.
+
+trans_max_pages=%u
+ Maximum number of pages in a single transaction. This parameter also controls
+ the number of pages, allocated for crypto processing (each crypto thread has
+ pool of pages, the number of which is equal to 'trans_max_pages'.
+ Default is 100 pages.
+
+crypto_fail_unsupported
+ If specified, mount will fail if the server does not support requested crypto operations.
+ By default mount will disable non-matching crypto operations.
+
+mcache_timeout=%u
+ Maximum number of milliseconds to wait for the mcache objects to be processed.
+ Mcache includes locks (given lock should be granted by server), attributes (they should be
+ fully received in the given timeframe).
+ Default is 5 seconds.
+
+Usage examples.
+
+Add (or remove if it already exists) server server1.net:1025 into the working set with index $idx
+with appropriate hash algorithm and key file and cipher algorithm, mode and key file:
+$cfg -a server1.net -p 1025 -i $idx -K $hash_key -k $cipher_key
+
+Mount filesystem with given index $idx to /mnt mountpoint.
+Client will connect to all servers specified in the working set via previous command:
+mount -t pohmel -o idx=$idx q /mnt
+
+One can add or remove servers from working set after mounting too.
+
+
+Server installation.
+
+Creating a server, which listens at port 1025 and 0.0.0.0 address.
+Working root directory (note, that server chroots there, so you have to have appropriate permissions)
+is set to /mnt, server will negotiate hash/cipher with client, in case client requested it, there
+are appropriate key files.
+Number of working threads is set to 10.
+
+# ./fserver -a 0.0.0.0 -p 1025 -r /mnt -w 10 -K hash_key -k cipher_key
+
+ -A 6 - listen on ipv6 address. Default: Disabled.
+ -r root - path to root directory. Default: /tmp.
+ -a addr - listen address. Default: 0.0.0.0.
+ -p port - listen port. Default: 1025.
+ -w workers - number of workers per connected client. Default: 1.
+ -K file - hash key size. Default: none.
+ -k file - cipher key size. Default: none.
+ -h - this help.
+
+Number of worker threads specifies how many workers will be created for each client.
+Bulk single-client transafers usually are better handled with smaller number (like 1-3).
diff --git a/Documentation/filesystems/pohmelfs/network_protocol.txt b/Documentation/filesystems/pohmelfs/network_protocol.txt
new file mode 100644
index 00000000000..40ea6c295af
--- /dev/null
+++ b/Documentation/filesystems/pohmelfs/network_protocol.txt
@@ -0,0 +1,227 @@
+POHMELFS network protocol.
+
+Basic structure used in network communication is following command:
+
+struct netfs_cmd
+{
+ __u16 cmd; /* Command number */
+ __u16 csize; /* Attached crypto information size */
+ __u16 cpad; /* Attached padding size */
+ __u16 ext; /* External flags */
+ __u32 size; /* Size of the attached data */
+ __u32 trans; /* Transaction id */
+ __u64 id; /* Object ID to operate on. Used for feedback.*/
+ __u64 start; /* Start of the object. */
+ __u64 iv; /* IV sequence */
+ __u8 data[0];
+};
+
+Commands can be embedded into transaction command (which in turn has own command),
+so one can extend protocol as needed without breaking backward compatibility as long
+as old commands are supported. All string lengths include tail 0 byte.
+
+All commans are transfered over the network in big-endian. CPU endianess is used at the end peers.
+
+@cmd - command number, which specifies command to be processed. Following
+ commands are used currently:
+
+ NETFS_READDIR = 1, /* Read directory for given inode number */
+ NETFS_READ_PAGE, /* Read data page from the server */
+ NETFS_WRITE_PAGE, /* Write data page to the server */
+ NETFS_CREATE, /* Create directory entry */
+ NETFS_REMOVE, /* Remove directory entry */
+ NETFS_LOOKUP, /* Lookup single object */
+ NETFS_LINK, /* Create a link */
+ NETFS_TRANS, /* Transaction */
+ NETFS_OPEN, /* Open intent */
+ NETFS_INODE_INFO, /* Metadata cache coherency synchronization message */
+ NETFS_PAGE_CACHE, /* Page cache invalidation message */
+ NETFS_READ_PAGES, /* Read multiple contiguous pages in one go */
+ NETFS_RENAME, /* Rename object */
+ NETFS_CAPABILITIES, /* Capabilities of the client, for example supported crypto */
+ NETFS_LOCK, /* Distributed lock message */
+ NETFS_XATTR_SET, /* Set extended attribute */
+ NETFS_XATTR_GET, /* Get extended attribute */
+
+@ext - external flags. Used by different commands to specify some extra arguments
+ like partial size of the embedded objects or creation flags.
+
+@size - size of the attached data. For NETFS_READ_PAGE and NETFS_READ_PAGES no data is attached,
+ but size of the requested data is incorporated here. It does not include size of the command
+ header (struct netfs_cmd) itself.
+
+@id - id of the object this command operates on. Each command can use it for own purpose.
+
+@start - start of the object this command operates on. Each command can use it for own purpose.
+
+@csize, @cpad - size and padding size of the (attached if needed) crypto information.
+
+Command specifications.
+
+@NETFS_READDIR
+This command is used to sync content of the remote dir to the client.
+
+@ext - length of the path to object.
+@size - the same.
+@id - local inode number of the directory to read.
+@start - zero.
+
+
+@NETFS_READ_PAGE
+This command is used to read data from remote server.
+Data size does not exceed local page cache size.
+
+@id - inode number.
+@start - first byte offset.
+@size - number of bytes to read plus length of the path to object.
+@ext - object path length.
+
+
+@NETFS_CREATE
+Used to create object.
+It does not require that all directories on top of the object were
+already created, it will create them automatically. Each object has
+associated @netfs_path_entry data structure, which contains creation
+mode (permissions and type) and length of the name as long as name itself.
+
+@start - 0
+@size - size of the all data structures needed to create a path
+@id - local inode number
+@ext - 0
+
+
+@NETFS_REMOVE
+Used to remove object.
+
+@ext - length of the path to object.
+@size - the same.
+@id - local inode number.
+@start - zero.
+
+
+@NETFS_LOOKUP
+Lookup information about object on server.
+
+@ext - length of the path to object.
+@size - the same.
+@id - local inode number of the directory to look object in.
+@start - local inode number of the object to look at.
+
+
+@NETFS_LINK
+Create hard of symlink.
+Command is sent as "object_path|target_path".
+
+@size - size of the above string.
+@id - parent local inode number.
+@start - 1 for symlink, 0 for hardlink.
+@ext - size of the "object_path" above.
+
+
+@NETFS_TRANS
+Transaction header.
+
+@size - incorporates all embedded command sizes including theirs header sizes.
+@start - transaction generation number - unique id used to find transaction.
+@ext - transaction flags. Unused at the moment.
+@id - 0.
+
+
+@NETFS_OPEN
+Open intent for given transaction.
+
+@id - local inode number.
+@start - 0.
+@size - path length to the object.
+@ext - open flags (O_RDWR and so on).
+
+
+@NETFS_INODE_INFO
+Metadata update command.
+It is sent to servers when attributes of the object are changed and received
+when data or metadata were updated. It operates with the following structure:
+
+struct netfs_inode_info
+{
+ unsigned int mode;
+ unsigned int nlink;
+ unsigned int uid;
+ unsigned int gid;
+ unsigned int blocksize;
+ unsigned int padding;
+ __u64 ino;
+ __u64 blocks;
+ __u64 rdev;
+ __u64 size;
+ __u64 version;
+};
+
+It effectively mirrors stat(2) returned data.
+
+
+@ext - path length to the object.
+@size - the same plus size of the netfs_inode_info structure.
+@id - local inode number.
+@start - 0.
+
+
+@NETFS_PAGE_CACHE
+Command is only received by clients. It contains information about
+page to be marked as not up-to-date.
+
+@id - client's inode number.
+@start - last byte of the page to be invalidated. If it is not equal to
+ current inode size, it will be vmtruncated().
+@size - 0
+@ext - 0
+
+
+@NETFS_READ_PAGES
+Used to read multiple contiguous pages in one go.
+
+@start - first byte of the contiguous region to read.
+@size - contains of two fields: lower 8 bits are used to represent page cache shift
+ used by client, another 3 bytes are used to get number of pages.
+@id - local inode number.
+@ext - path length to the object.
+
+
+@NETFS_RENAME
+Used to rename object.
+Attached data is formed into following string: "old_path|new_path".
+
+@id - local inode number.
+@start - parent inode number.
+@size - length of the above string.
+@ext - length of the old path part.
+
+
+@NETFS_CAPABILITIES
+Used to exchange crypto capabilities with server.
+If crypto capabilities are not supported by server, then client will disable it
+or fail (if 'crypto_fail_unsupported' mount options was specified).
+
+@id - superblock index. Used to specify crypto information for group of servers.
+@size - size of the attached capabilities structure.
+@start - 0.
+@size - 0.
+@scsize - 0.
+
+@NETFS_LOCK
+Used to send lock request/release messages. Although it sends byte range request
+and is capable of flushing pages based on that, it is not used, since all Linux
+filesystems lock the whole inode.
+
+@id - lock generation number.
+@start - start of the locked range.
+@size - size of the locked range.
+@ext - lock type: read/write. Not used actually. 15'th bit is used to determine,
+ if it is lock request (1) or release (0).
+
+@NETFS_XATTR_SET
+@NETFS_XATTR_GET
+Used to set/get extended attributes for given inode.
+@id - attribute generation number or xattr setting type
+@start - size of the attribute (request or attached)
+@size - name length, path len and data size for given attribute
+@ext - path length for given object
diff --git a/Documentation/ftrace.txt b/Documentation/ftrace.txt
index 803b1318b13..fd9a3e69381 100644
--- a/Documentation/ftrace.txt
+++ b/Documentation/ftrace.txt
@@ -15,31 +15,31 @@ Introduction
Ftrace is an internal tracer designed to help out developers and
designers of systems to find what is going on inside the kernel.
-It can be used for debugging or analyzing latencies and performance
-issues that take place outside of user-space.
+It can be used for debugging or analyzing latencies and
+performance issues that take place outside of user-space.
Although ftrace is the function tracer, it also includes an
-infrastructure that allows for other types of tracing. Some of the
-tracers that are currently in ftrace include a tracer to trace
-context switches, the time it takes for a high priority task to
-run after it was woken up, the time interrupts are disabled, and
-more (ftrace allows for tracer plugins, which means that the list of
-tracers can always grow).
+infrastructure that allows for other types of tracing. Some of
+the tracers that are currently in ftrace include a tracer to
+trace context switches, the time it takes for a high priority
+task to run after it was woken up, the time interrupts are
+disabled, and more (ftrace allows for tracer plugins, which
+means that the list of tracers can always grow).
The File System
---------------
-Ftrace uses the debugfs file system to hold the control files as well
-as the files to display output.
+Ftrace uses the debugfs file system to hold the control files as
+well as the files to display output.
To mount the debugfs system:
# mkdir /debug
# mount -t debugfs nodev /debug
-(Note: it is more common to mount at /sys/kernel/debug, but for simplicity
- this document will use /debug)
+( Note: it is more common to mount at /sys/kernel/debug, but for
+ simplicity this document will use /debug)
That's it! (assuming that you have ftrace configured into your kernel)
@@ -50,90 +50,124 @@ of ftrace. Here is a list of some of the key files:
Note: all time values are in microseconds.
- current_tracer: This is used to set or display the current tracer
- that is configured.
-
- available_tracers: This holds the different types of tracers that
- have been compiled into the kernel. The tracers
- listed here can be configured by echoing their name
- into current_tracer.
-
- tracing_enabled: This sets or displays whether the current_tracer
- is activated and tracing or not. Echo 0 into this
- file to disable the tracer or 1 to enable it.
-
- trace: This file holds the output of the trace in a human readable
- format (described below).
-
- latency_trace: This file shows the same trace but the information
- is organized more to display possible latencies
- in the system (described below).
-
- trace_pipe: The output is the same as the "trace" file but this
- file is meant to be streamed with live tracing.
- Reads from this file will block until new data
- is retrieved. Unlike the "trace" and "latency_trace"
- files, this file is a consumer. This means reading
- from this file causes sequential reads to display
- more current data. Once data is read from this
- file, it is consumed, and will not be read
- again with a sequential read. The "trace" and
- "latency_trace" files are static, and if the
- tracer is not adding more data, they will display
- the same information every time they are read.
-
- trace_options: This file lets the user control the amount of data
- that is displayed in one of the above output
- files.
-
- trace_max_latency: Some of the tracers record the max latency.
- For example, the time interrupts are disabled.
- This time is saved in this file. The max trace
- will also be stored, and displayed by either
- "trace" or "latency_trace". A new max trace will
- only be recorded if the latency is greater than
- the value in this file. (in microseconds)
-
- buffer_size_kb: This sets or displays the number of kilobytes each CPU
- buffer can hold. The tracer buffers are the same size
- for each CPU. The displayed number is the size of the
- CPU buffer and not total size of all buffers. The
- trace buffers are allocated in pages (blocks of memory
- that the kernel uses for allocation, usually 4 KB in size).
- If the last page allocated has room for more bytes
- than requested, the rest of the page will be used,
- making the actual allocation bigger than requested.
- (Note, the size may not be a multiple of the page size due
- to buffer managment overhead.)
-
- This can only be updated when the current_tracer
- is set to "nop".
-
- tracing_cpumask: This is a mask that lets the user only trace
- on specified CPUS. The format is a hex string
- representing the CPUS.
-
- set_ftrace_filter: When dynamic ftrace is configured in (see the
- section below "dynamic ftrace"), the code is dynamically
- modified (code text rewrite) to disable calling of the
- function profiler (mcount). This lets tracing be configured
- in with practically no overhead in performance. This also
- has a side effect of enabling or disabling specific functions
- to be traced. Echoing names of functions into this file
- will limit the trace to only those functions.
-
- set_ftrace_notrace: This has an effect opposite to that of
- set_ftrace_filter. Any function that is added here will not
- be traced. If a function exists in both set_ftrace_filter
- and set_ftrace_notrace, the function will _not_ be traced.
-
- set_ftrace_pid: Have the function tracer only trace a single thread.
-
- available_filter_functions: This lists the functions that ftrace
- has processed and can trace. These are the function
- names that you can pass to "set_ftrace_filter" or
- "set_ftrace_notrace". (See the section "dynamic ftrace"
- below for more details.)
+ current_tracer:
+
+ This is used to set or display the current tracer
+ that is configured.
+
+ available_tracers:
+
+ This holds the different types of tracers that
+ have been compiled into the kernel. The
+ tracers listed here can be configured by
+ echoing their name into current_tracer.
+
+ tracing_enabled:
+
+ This sets or displays whether the current_tracer
+ is activated and tracing or not. Echo 0 into this
+ file to disable the tracer or 1 to enable it.
+
+ trace:
+
+ This file holds the output of the trace in a human
+ readable format (described below).
+
+ latency_trace:
+
+ This file shows the same trace but the information
+ is organized more to display possible latencies
+ in the system (described below).
+
+ trace_pipe:
+
+ The output is the same as the "trace" file but this
+ file is meant to be streamed with live tracing.
+ Reads from this file will block until new data
+ is retrieved. Unlike the "trace" and "latency_trace"
+ files, this file is a consumer. This means reading
+ from this file causes sequential reads to display
+ more current data. Once data is read from this
+ file, it is consumed, and will not be read
+ again with a sequential read. The "trace" and
+ "latency_trace" files are static, and if the
+ tracer is not adding more data, they will display
+ the same information every time they are read.
+
+ trace_options:
+
+ This file lets the user control the amount of data
+ that is displayed in one of the above output
+ files.
+
+ tracing_max_latency:
+
+ Some of the tracers record the max latency.
+ For example, the time interrupts are disabled.
+ This time is saved in this file. The max trace
+ will also be stored, and displayed by either
+ "trace" or "latency_trace". A new max trace will
+ only be recorded if the latency is greater than
+ the value in this file. (in microseconds)
+
+ buffer_size_kb:
+
+ This sets or displays the number of kilobytes each CPU
+ buffer can hold. The tracer buffers are the same size
+ for each CPU. The displayed number is the size of the
+ CPU buffer and not total size of all buffers. The
+ trace buffers are allocated in pages (blocks of memory
+ that the kernel uses for allocation, usually 4 KB in size).
+ If the last page allocated has room for more bytes
+ than requested, the rest of the page will be used,
+ making the actual allocation bigger than requested.
+ ( Note, the size may not be a multiple of the page size
+ due to buffer managment overhead. )
+
+ This can only be updated when the current_tracer
+ is set to "nop".
+
+ tracing_cpumask:
+
+ This is a mask that lets the user only trace
+ on specified CPUS. The format is a hex string
+ representing the CPUS.
+
+ set_ftrace_filter:
+
+ When dynamic ftrace is configured in (see the
+ section below "dynamic ftrace"), the code is dynamically
+ modified (code text rewrite) to disable calling of the
+ function profiler (mcount). This lets tracing be configured
+ in with practically no overhead in performance. This also
+ has a side effect of enabling or disabling specific functions
+ to be traced. Echoing names of functions into this file
+ will limit the trace to only those functions.
+
+ set_ftrace_notrace:
+
+ This has an effect opposite to that of
+ set_ftrace_filter. Any function that is added here will not
+ be traced. If a function exists in both set_ftrace_filter
+ and set_ftrace_notrace, the function will _not_ be traced.
+
+ set_ftrace_pid:
+
+ Have the function tracer only trace a single thread.
+
+ set_graph_function:
+
+ Set a "trigger" function where tracing should start
+ with the function graph tracer (See the section
+ "dynamic ftrace" for more details).
+
+ available_filter_functions:
+
+ This lists the functions that ftrace
+ has processed and can trace. These are the function
+ names that you can pass to "set_ftrace_filter" or
+ "set_ftrace_notrace". (See the section "dynamic ftrace"
+ below for more details.)
The Tracers
@@ -141,36 +175,66 @@ The Tracers
Here is the list of current tracers that may be configured.
- function - function tracer that uses mcount to trace all functions.
+ "function"
+
+ Function call tracer to trace all kernel functions.
+
+ "function_graph_tracer"
+
+ Similar to the function tracer except that the
+ function tracer probes the functions on their entry
+ whereas the function graph tracer traces on both entry
+ and exit of the functions. It then provides the ability
+ to draw a graph of function calls similar to C code
+ source.
- sched_switch - traces the context switches between tasks.
+ "sched_switch"
- irqsoff - traces the areas that disable interrupts and saves
- the trace with the longest max latency.
- See tracing_max_latency. When a new max is recorded,
- it replaces the old trace. It is best to view this
- trace via the latency_trace file.
+ Traces the context switches and wakeups between tasks.
- preemptoff - Similar to irqsoff but traces and records the amount of
- time for which preemption is disabled.
+ "irqsoff"
- preemptirqsoff - Similar to irqsoff and preemptoff, but traces and
- records the largest time for which irqs and/or preemption
- is disabled.
+ Traces the areas that disable interrupts and saves
+ the trace with the longest max latency.
+ See tracing_max_latency. When a new max is recorded,
+ it replaces the old trace. It is best to view this
+ trace via the latency_trace file.
- wakeup - Traces and records the max latency that it takes for
- the highest priority task to get scheduled after
- it has been woken up.
+ "preemptoff"
- nop - This is not a tracer. To remove all tracers from tracing
- simply echo "nop" into current_tracer.
+ Similar to irqsoff but traces and records the amount of
+ time for which preemption is disabled.
+
+ "preemptirqsoff"
+
+ Similar to irqsoff and preemptoff, but traces and
+ records the largest time for which irqs and/or preemption
+ is disabled.
+
+ "wakeup"
+
+ Traces and records the max latency that it takes for
+ the highest priority task to get scheduled after
+ it has been woken up.
+
+ "hw-branch-tracer"
+
+ Uses the BTS CPU feature on x86 CPUs to traces all
+ branches executed.
+
+ "nop"
+
+ This is the "trace nothing" tracer. To remove all
+ tracers from tracing simply echo "nop" into
+ current_tracer.
Examples of using the tracer
----------------------------
-Here are typical examples of using the tracers when controlling them only
-with the debugfs interface (without using any user-land utilities).
+Here are typical examples of using the tracers when controlling
+them only with the debugfs interface (without using any
+user-land utilities).
Output format:
--------------
@@ -187,16 +251,16 @@ Here is an example of the output format of the file "trace"
bash-4251 [01] 10152.583855: _atomic_dec_and_lock <-dput
--------
-A header is printed with the tracer name that is represented by the trace.
-In this case the tracer is "function". Then a header showing the format. Task
-name "bash", the task PID "4251", the CPU that it was running on
-"01", the timestamp in <secs>.<usecs> format, the function name that was
-traced "path_put" and the parent function that called this function
-"path_walk". The timestamp is the time at which the function was
-entered.
+A header is printed with the tracer name that is represented by
+the trace. In this case the tracer is "function". Then a header
+showing the format. Task name "bash", the task PID "4251", the
+CPU that it was running on "01", the timestamp in <secs>.<usecs>
+format, the function name that was traced "path_put" and the
+parent function that called this function "path_walk". The
+timestamp is the time at which the function was entered.
-The sched_switch tracer also includes tracing of task wakeups and
-context switches.
+The sched_switch tracer also includes tracing of task wakeups
+and context switches.
ksoftirqd/1-7 [01] 1453.070013: 7:115:R + 2916:115:S
ksoftirqd/1-7 [01] 1453.070013: 7:115:R + 10:115:S
@@ -205,8 +269,8 @@ context switches.
kondemand/1-2916 [01] 1453.070013: 2916:115:S ==> 7:115:R
ksoftirqd/1-7 [01] 1453.070013: 7:115:S ==> 0:140:R
-Wake ups are represented by a "+" and the context switches are shown as
-"==>". The format is:
+Wake ups are represented by a "+" and the context switches are
+shown as "==>". The format is:
Context switches:
@@ -220,19 +284,20 @@ Wake ups are represented by a "+" and the context switches are shown as
<pid>:<prio>:<state> + <pid>:<prio>:<state>
-The prio is the internal kernel priority, which is the inverse of the
-priority that is usually displayed by user-space tools. Zero represents
-the highest priority (99). Prio 100 starts the "nice" priorities with
-100 being equal to nice -20 and 139 being nice 19. The prio "140" is
-reserved for the idle task which is the lowest priority thread (pid 0).
+The prio is the internal kernel priority, which is the inverse
+of the priority that is usually displayed by user-space tools.
+Zero represents the highest priority (99). Prio 100 starts the
+"nice" priorities with 100 being equal to nice -20 and 139 being
+nice 19. The prio "140" is reserved for the idle task which is
+the lowest priority thread (pid 0).
Latency trace format
--------------------
-For traces that display latency times, the latency_trace file gives
-somewhat more information to see why a latency happened. Here is a typical
-trace.
+For traces that display latency times, the latency_trace file
+gives somewhat more information to see why a latency happened.
+Here is a typical trace.
# tracer: irqsoff
#
@@ -259,20 +324,20 @@ irqsoff latency trace v1.1.5 on 2.6.26-rc8
<idle>-0 0d.s1 98us : trace_hardirqs_on (do_softirq)
+This shows that the current tracer is "irqsoff" tracing the time
+for which interrupts were disabled. It gives the trace version
+and the version of the kernel upon which this was executed on
+(2.6.26-rc8). Then it displays the max latency in microsecs (97
+us). The number of trace entries displayed and the total number
+recorded (both are three: #3/3). The type of preemption that was
+used (PREEMPT). VP, KP, SP, and HP are always zero and are
+reserved for later use. #P is the number of online CPUS (#P:2).
-This shows that the current tracer is "irqsoff" tracing the time for which
-interrupts were disabled. It gives the trace version and the version
-of the kernel upon which this was executed on (2.6.26-rc8). Then it displays
-the max latency in microsecs (97 us). The number of trace entries displayed
-and the total number recorded (both are three: #3/3). The type of
-preemption that was used (PREEMPT). VP, KP, SP, and HP are always zero
-and are reserved for later use. #P is the number of online CPUS (#P:2).
-
-The task is the process that was running when the latency occurred.
-(swapper pid: 0).
+The task is the process that was running when the latency
+occurred. (swapper pid: 0).
-The start and stop (the functions in which the interrupts were disabled and
-enabled respectively) that caused the latencies:
+The start and stop (the functions in which the interrupts were
+disabled and enabled respectively) that caused the latencies:
apic_timer_interrupt is where the interrupts were disabled.
do_softirq is where they were enabled again.
@@ -308,12 +373,12 @@ The above is mostly meaningful for kernel developers.
latency_trace file is relative to the start of the trace.
delay: This is just to help catch your eye a bit better. And
- needs to be fixed to be only relative to the same CPU.
- The marks are determined by the difference between this
- current trace and the next trace.
- '!' - greater than preempt_mark_thresh (default 100)
- '+' - greater than 1 microsecond
- ' ' - less than or equal to 1 microsecond.
+ needs to be fixed to be only relative to the same CPU.
+ The marks are determined by the difference between this
+ current trace and the next trace.
+ '!' - greater than preempt_mark_thresh (default 100)
+ '+' - greater than 1 microsecond
+ ' ' - less than or equal to 1 microsecond.
The rest is the same as the 'trace' file.
@@ -321,14 +386,15 @@ The above is mostly meaningful for kernel developers.
trace_options
-------------
-The trace_options file is used to control what gets printed in the trace
-output. To see what is available, simply cat the file:
+The trace_options file is used to control what gets printed in
+the trace output. To see what is available, simply cat the file:
cat /debug/tracing/trace_options
print-parent nosym-offset nosym-addr noverbose noraw nohex nobin \
- noblock nostacktrace nosched-tree nouserstacktrace nosym-userobj
+ noblock nostacktrace nosched-tree nouserstacktrace nosym-userobj
-To disable one of the options, echo in the option prepended with "no".
+To disable one of the options, echo in the option prepended with
+"no".
echo noprint-parent > /debug/tracing/trace_options
@@ -338,8 +404,8 @@ To enable an option, leave off the "no".
Here are the available options:
- print-parent - On function traces, display the calling function
- as well as the function being traced.
+ print-parent - On function traces, display the calling (parent)
+ function as well as the function being traced.
print-parent:
bash-4000 [01] 1477.606694: simple_strtoul <-strict_strtoul
@@ -348,15 +414,16 @@ Here are the available options:
bash-4000 [01] 1477.606694: simple_strtoul
- sym-offset - Display not only the function name, but also the offset
- in the function. For example, instead of seeing just
- "ktime_get", you will see "ktime_get+0xb/0x20".
+ sym-offset - Display not only the function name, but also the
+ offset in the function. For example, instead of
+ seeing just "ktime_get", you will see
+ "ktime_get+0xb/0x20".
sym-offset:
bash-4000 [01] 1477.606694: simple_strtoul+0x6/0xa0
- sym-addr - this will also display the function address as well as
- the function name.
+ sym-addr - this will also display the function address as well
+ as the function name.
sym-addr:
bash-4000 [01] 1477.606694: simple_strtoul <c0339346>
@@ -366,35 +433,41 @@ Here are the available options:
bash 4000 1 0 00000000 00010a95 [58127d26] 1720.415ms \
(+0.000ms): simple_strtoul (strict_strtoul)
- raw - This will display raw numbers. This option is best for use with
- user applications that can translate the raw numbers better than
- having it done in the kernel.
+ raw - This will display raw numbers. This option is best for
+ use with user applications that can translate the raw
+ numbers better than having it done in the kernel.
- hex - Similar to raw, but the numbers will be in a hexadecimal format.
+ hex - Similar to raw, but the numbers will be in a hexadecimal
+ format.
bin - This will print out the formats in raw binary.
block - TBD (needs update)
- stacktrace - This is one of the options that changes the trace itself.
- When a trace is recorded, so is the stack of functions.
- This allows for back traces of trace sites.
+ stacktrace - This is one of the options that changes the trace
+ itself. When a trace is recorded, so is the stack
+ of functions. This allows for back traces of
+ trace sites.
- userstacktrace - This option changes the trace.
- It records a stacktrace of the current userspace thread.
+ userstacktrace - This option changes the trace. It records a
+ stacktrace of the current userspace thread.
- sym-userobj - when user stacktrace are enabled, look up which object the
- address belongs to, and print a relative address
- This is especially useful when ASLR is on, otherwise you don't
- get a chance to resolve the address to object/file/line after the app is no
- longer running
+ sym-userobj - when user stacktrace are enabled, look up which
+ object the address belongs to, and print a
+ relative address. This is especially useful when
+ ASLR is on, otherwise you don't get a chance to
+ resolve the address to object/file/line after
+ the app is no longer running
- The lookup is performed when you read trace,trace_pipe,latency_trace. Example:
+ The lookup is performed when you read
+ trace,trace_pipe,latency_trace. Example:
a.out-1623 [000] 40874.465068: /root/a.out[+0x480] <-/root/a.out[+0
x494] <- /root/a.out[+0x4a8] <- /lib/libc-2.7.so[+0x1e1a6]
- sched-tree - TBD (any users??)
+ sched-tree - trace all tasks that are on the runqueue, at
+ every scheduling event. Will add overhead if
+ there's a lot of tasks running at once.
sched_switch
@@ -431,18 +504,19 @@ of how to use it.
[...]
-As we have discussed previously about this format, the header shows
-the name of the trace and points to the options. The "FUNCTION"
-is a misnomer since here it represents the wake ups and context
-switches.
+As we have discussed previously about this format, the header
+shows the name of the trace and points to the options. The
+"FUNCTION" is a misnomer since here it represents the wake ups
+and context switches.
-The sched_switch file only lists the wake ups (represented with '+')
-and context switches ('==>') with the previous task or current task
-first followed by the next task or task waking up. The format for both
-of these is PID:KERNEL-PRIO:TASK-STATE. Remember that the KERNEL-PRIO
-is the inverse of the actual priority with zero (0) being the highest
-priority and the nice values starting at 100 (nice -20). Below is
-a quick chart to map the kernel priority to user land priorities.
+The sched_switch file only lists the wake ups (represented with
+'+') and context switches ('==>') with the previous task or
+current task first followed by the next task or task waking up.
+The format for both of these is PID:KERNEL-PRIO:TASK-STATE.
+Remember that the KERNEL-PRIO is the inverse of the actual
+priority with zero (0) being the highest priority and the nice
+values starting at 100 (nice -20). Below is a quick chart to map
+the kernel priority to user land priorities.
Kernel priority: 0 to 99 ==> user RT priority 99 to 0
Kernel priority: 100 to 139 ==> user nice -20 to 19
@@ -463,10 +537,10 @@ The task states are:
ftrace_enabled
--------------
-The following tracers (listed below) give different output depending
-on whether or not the sysctl ftrace_enabled is set. To set ftrace_enabled,
-one can either use the sysctl function or set it via the proc
-file system interface.
+The following tracers (listed below) give different output
+depending on whether or not the sysctl ftrace_enabled is set. To
+set ftrace_enabled, one can either use the sysctl function or
+set it via the proc file system interface.
sysctl kernel.ftrace_enabled=1
@@ -474,12 +548,12 @@ file system interface.
echo 1 > /proc/sys/kernel/ftrace_enabled
-To disable ftrace_enabled simply replace the '1' with '0' in
-the above commands.
+To disable ftrace_enabled simply replace the '1' with '0' in the
+above commands.
-When ftrace_enabled is set the tracers will also record the functions
-that are within the trace. The descriptions of the tracers
-will also show an example with ftrace enabled.
+When ftrace_enabled is set the tracers will also record the
+functions that are within the trace. The descriptions of the
+tracers will also show an example with ftrace enabled.
irqsoff
@@ -487,17 +561,18 @@ irqsoff
When interrupts are disabled, the CPU can not react to any other
external event (besides NMIs and SMIs). This prevents the timer
-interrupt from triggering or the mouse interrupt from letting the
-kernel know of a new mouse event. The result is a latency with the
-reaction time.
+interrupt from triggering or the mouse interrupt from letting
+the kernel know of a new mouse event. The result is a latency
+with the reaction time.
-The irqsoff tracer tracks the time for which interrupts are disabled.
-When a new maximum latency is hit, the tracer saves the trace leading up
-to that latency point so that every time a new maximum is reached, the old
-saved trace is discarded and the new trace is saved.
+The irqsoff tracer tracks the time for which interrupts are
+disabled. When a new maximum latency is hit, the tracer saves
+the trace leading up to that latency point so that every time a
+new maximum is reached, the old saved trace is discarded and the
+new trace is saved.
-To reset the maximum, echo 0 into tracing_max_latency. Here is an
-example:
+To reset the maximum, echo 0 into tracing_max_latency. Here is
+an example:
# echo irqsoff > /debug/tracing/current_tracer
# echo 0 > /debug/tracing/tracing_max_latency
@@ -532,10 +607,11 @@ irqsoff latency trace v1.1.5 on 2.6.26
Here we see that that we had a latency of 12 microsecs (which is
-very good). The _write_lock_irq in sys_setpgid disabled interrupts.
-The difference between the 12 and the displayed timestamp 14us occurred
-because the clock was incremented between the time of recording the max
-latency and the time of recording the function that had that latency.
+very good). The _write_lock_irq in sys_setpgid disabled
+interrupts. The difference between the 12 and the displayed
+timestamp 14us occurred because the clock was incremented
+between the time of recording the max latency and the time of
+recording the function that had that latency.
Note the above example had ftrace_enabled not set. If we set the
ftrace_enabled, we get a much larger output:
@@ -586,24 +662,24 @@ irqsoff latency trace v1.1.5 on 2.6.26-rc8
Here we traced a 50 microsecond latency. But we also see all the
-functions that were called during that time. Note that by enabling
-function tracing, we incur an added overhead. This overhead may
-extend the latency times. But nevertheless, this trace has provided
-some very helpful debugging information.
+functions that were called during that time. Note that by
+enabling function tracing, we incur an added overhead. This
+overhead may extend the latency times. But nevertheless, this
+trace has provided some very helpful debugging information.
preemptoff
----------
-When preemption is disabled, we may be able to receive interrupts but
-the task cannot be preempted and a higher priority task must wait
-for preemption to be enabled again before it can preempt a lower
-priority task.
+When preemption is disabled, we may be able to receive
+interrupts but the task cannot be preempted and a higher
+priority task must wait for preemption to be enabled again
+before it can preempt a lower priority task.
The preemptoff tracer traces the places that disable preemption.
-Like the irqsoff tracer, it records the maximum latency for which preemption
-was disabled. The control of preemptoff tracer is much like the irqsoff
-tracer.
+Like the irqsoff tracer, it records the maximum latency for
+which preemption was disabled. The control of preemptoff tracer
+is much like the irqsoff tracer.
# echo preemptoff > /debug/tracing/current_tracer
# echo 0 > /debug/tracing/tracing_max_latency
@@ -637,11 +713,12 @@ preemptoff latency trace v1.1.5 on 2.6.26-rc8
sshd-4261 0d.s1 30us : trace_preempt_on (__do_softirq)
-This has some more changes. Preemption was disabled when an interrupt
-came in (notice the 'h'), and was enabled while doing a softirq.
-(notice the 's'). But we also see that interrupts have been disabled
-when entering the preempt off section and leaving it (the 'd').
-We do not know if interrupts were enabled in the mean time.
+This has some more changes. Preemption was disabled when an
+interrupt came in (notice the 'h'), and was enabled while doing
+a softirq. (notice the 's'). But we also see that interrupts
+have been disabled when entering the preempt off section and
+leaving it (the 'd'). We do not know if interrupts were enabled
+in the mean time.
# tracer: preemptoff
#
@@ -700,28 +777,30 @@ preemptoff latency trace v1.1.5 on 2.6.26-rc8
sshd-4261 0d.s1 64us : trace_preempt_on (__do_softirq)
-The above is an example of the preemptoff trace with ftrace_enabled
-set. Here we see that interrupts were disabled the entire time.
-The irq_enter code lets us know that we entered an interrupt 'h'.
-Before that, the functions being traced still show that it is not
-in an interrupt, but we can see from the functions themselves that
-this is not the case.
+The above is an example of the preemptoff trace with
+ftrace_enabled set. Here we see that interrupts were disabled
+the entire time. The irq_enter code lets us know that we entered
+an interrupt 'h'. Before that, the functions being traced still
+show that it is not in an interrupt, but we can see from the
+functions themselves that this is not the case.
-Notice that __do_softirq when called does not have a preempt_count.
-It may seem that we missed a preempt enabling. What really happened
-is that the preempt count is held on the thread's stack and we
-switched to the softirq stack (4K stacks in effect). The code
-does not copy the preempt count, but because interrupts are disabled,
-we do not need to worry about it. Having a tracer like this is good
-for letting people know what really happens inside the kernel.
+Notice that __do_softirq when called does not have a
+preempt_count. It may seem that we missed a preempt enabling.
+What really happened is that the preempt count is held on the
+thread's stack and we switched to the softirq stack (4K stacks
+in effect). The code does not copy the preempt count, but
+because interrupts are disabled, we do not need to worry about
+it. Having a tracer like this is good for letting people know
+what really happens inside the kernel.
preemptirqsoff
--------------
-Knowing the locations that have interrupts disabled or preemption
-disabled for the longest times is helpful. But sometimes we would
-like to know when either preemption and/or interrupts are disabled.
+Knowing the locations that have interrupts disabled or
+preemption disabled for the longest times is helpful. But
+sometimes we would like to know when either preemption and/or
+interrupts are disabled.
Consider the following code:
@@ -741,11 +820,13 @@ The preemptoff tracer will record the total length of
call_function_with_irqs_and_preemption_off() and
call_function_with_preemption_off().
-But neither will trace the time that interrupts and/or preemption
-is disabled. This total time is the time that we can not schedule.
-To record this time, use the preemptirqsoff tracer.
+But neither will trace the time that interrupts and/or
+preemption is disabled. This total time is the time that we can
+not schedule. To record this time, use the preemptirqsoff
+tracer.
-Again, using this trace is much like the irqsoff and preemptoff tracers.
+Again, using this trace is much like the irqsoff and preemptoff
+tracers.
# echo preemptirqsoff > /debug/tracing/current_tracer
# echo 0 > /debug/tracing/tracing_max_latency
@@ -781,9 +862,10 @@ preemptirqsoff latency trace v1.1.5 on 2.6.26-rc8
The trace_hardirqs_off_thunk is called from assembly on x86 when
-interrupts are disabled in the assembly code. Without the function
-tracing, we do not know if interrupts were enabled within the preemption
-points. We do see that it started with preemption enabled.
+interrupts are disabled in the assembly code. Without the
+function tracing, we do not know if interrupts were enabled
+within the preemption points. We do see that it started with
+preemption enabled.
Here is a trace with ftrace_enabled set:
@@ -871,40 +953,42 @@ preemptirqsoff latency trace v1.1.5 on 2.6.26-rc8
sshd-4261 0d.s1 105us : trace_preempt_on (__do_softirq)
-This is a very interesting trace. It started with the preemption of
-the ls task. We see that the task had the "need_resched" bit set
-via the 'N' in the trace. Interrupts were disabled before the spin_lock
-at the beginning of the trace. We see that a schedule took place to run
-sshd. When the interrupts were enabled, we took an interrupt.
-On return from the interrupt handler, the softirq ran. We took another
-interrupt while running the softirq as we see from the capital 'H'.
+This is a very interesting trace. It started with the preemption
+of the ls task. We see that the task had the "need_resched" bit
+set via the 'N' in the trace. Interrupts were disabled before
+the spin_lock at the beginning of the trace. We see that a
+schedule took place to run sshd. When the interrupts were
+enabled, we took an interrupt. On return from the interrupt
+handler, the softirq ran. We took another interrupt while
+running the softirq as we see from the capital 'H'.
wakeup
------
-In a Real-Time environment it is very important to know the wakeup
-time it takes for the highest priority task that is woken up to the
-time that it executes. This is also known as "schedule latency".
-I stress the point that this is about RT tasks. It is also important
-to know the scheduling latency of non-RT tasks, but the average
-schedule latency is better for non-RT tasks. Tools like
-LatencyTop are more appropriate for such measurements.
+In a Real-Time environment it is very important to know the
+wakeup time it takes for the highest priority task that is woken
+up to the time that it executes. This is also known as "schedule
+latency". I stress the point that this is about RT tasks. It is
+also important to know the scheduling latency of non-RT tasks,
+but the average schedule latency is better for non-RT tasks.
+Tools like LatencyTop are more appropriate for such
+measurements.
Real-Time environments are interested in the worst case latency.
-That is the longest latency it takes for something to happen, and
-not the average. We can have a very fast scheduler that may only
-have a large latency once in a while, but that would not work well
-with Real-Time tasks. The wakeup tracer was designed to record
-the worst case wakeups of RT tasks. Non-RT tasks are not recorded
-because the tracer only records one worst case and tracing non-RT
-tasks that are unpredictable will overwrite the worst case latency
-of RT tasks.
-
-Since this tracer only deals with RT tasks, we will run this slightly
-differently than we did with the previous tracers. Instead of performing
-an 'ls', we will run 'sleep 1' under 'chrt' which changes the
-priority of the task.
+That is the longest latency it takes for something to happen,
+and not the average. We can have a very fast scheduler that may
+only have a large latency once in a while, but that would not
+work well with Real-Time tasks. The wakeup tracer was designed
+to record the worst case wakeups of RT tasks. Non-RT tasks are
+not recorded because the tracer only records one worst case and
+tracing non-RT tasks that are unpredictable will overwrite the
+worst case latency of RT tasks.
+
+Since this tracer only deals with RT tasks, we will run this
+slightly differently than we did with the previous tracers.
+Instead of performing an 'ls', we will run 'sleep 1' under
+'chrt' which changes the priority of the task.
# echo wakeup > /debug/tracing/current_tracer
# echo 0 > /debug/tracing/tracing_max_latency
@@ -934,17 +1018,16 @@ wakeup latency trace v1.1.5 on 2.6.26-rc8
<idle>-0 1d..4 4us : schedule (cpu_idle)
+Running this on an idle system, we see that it only took 4
+microseconds to perform the task switch. Note, since the trace
+marker in the schedule is before the actual "switch", we stop
+the tracing when the recorded task is about to schedule in. This
+may change if we add a new marker at the end of the scheduler.
-Running this on an idle system, we see that it only took 4 microseconds
-to perform the task switch. Note, since the trace marker in the
-schedule is before the actual "switch", we stop the tracing when
-the recorded task is about to schedule in. This may change if
-we add a new marker at the end of the scheduler.
-
-Notice that the recorded task is 'sleep' with the PID of 4901 and it
-has an rt_prio of 5. This priority is user-space priority and not
-the internal kernel priority. The policy is 1 for SCHED_FIFO and 2
-for SCHED_RR.
+Notice that the recorded task is 'sleep' with the PID of 4901
+and it has an rt_prio of 5. This priority is user-space priority
+and not the internal kernel priority. The policy is 1 for
+SCHED_FIFO and 2 for SCHED_RR.
Doing the same with chrt -r 5 and ftrace_enabled set.
@@ -1001,24 +1084,25 @@ ksoftirq-7 1d..6 49us : _spin_unlock (tracing_record_cmdline)
ksoftirq-7 1d..6 49us : sub_preempt_count (_spin_unlock)
ksoftirq-7 1d..4 50us : schedule (__cond_resched)
-The interrupt went off while running ksoftirqd. This task runs at
-SCHED_OTHER. Why did not we see the 'N' set early? This may be
-a harmless bug with x86_32 and 4K stacks. On x86_32 with 4K stacks
-configured, the interrupt and softirq run with their own stack.
-Some information is held on the top of the task's stack (need_resched
-and preempt_count are both stored there). The setting of the NEED_RESCHED
-bit is done directly to the task's stack, but the reading of the
-NEED_RESCHED is done by looking at the current stack, which in this case
-is the stack for the hard interrupt. This hides the fact that NEED_RESCHED
-has been set. We do not see the 'N' until we switch back to the task's
+The interrupt went off while running ksoftirqd. This task runs
+at SCHED_OTHER. Why did not we see the 'N' set early? This may
+be a harmless bug with x86_32 and 4K stacks. On x86_32 with 4K
+stacks configured, the interrupt and softirq run with their own
+stack. Some information is held on the top of the task's stack
+(need_resched and preempt_count are both stored there). The
+setting of the NEED_RESCHED bit is done directly to the task's
+stack, but the reading of the NEED_RESCHED is done by looking at
+the current stack, which in this case is the stack for the hard
+interrupt. This hides the fact that NEED_RESCHED has been set.
+We do not see the 'N' until we switch back to the task's
assigned stack.
function
--------
This tracer is the function tracer. Enabling the function tracer
-can be done from the debug file system. Make sure the ftrace_enabled is
-set; otherwise this tracer is a nop.
+can be done from the debug file system. Make sure the
+ftrace_enabled is set; otherwise this tracer is a nop.
# sysctl kernel.ftrace_enabled=1
# echo function > /debug/tracing/current_tracer
@@ -1048,14 +1132,15 @@ set; otherwise this tracer is a nop.
[...]
-Note: function tracer uses ring buffers to store the above entries.
-The newest data may overwrite the oldest data. Sometimes using echo to
-stop the trace is not sufficient because the tracing could have overwritten
-the data that you wanted to record. For this reason, it is sometimes better to
-disable tracing directly from a program. This allows you to stop the
-tracing at the point that you hit the part that you are interested in.
-To disable the tracing directly from a C program, something like following
-code snippet can be used:
+Note: function tracer uses ring buffers to store the above
+entries. The newest data may overwrite the oldest data.
+Sometimes using echo to stop the trace is not sufficient because
+the tracing could have overwritten the data that you wanted to
+record. For this reason, it is sometimes better to disable
+tracing directly from a program. This allows you to stop the
+tracing at the point that you hit the part that you are
+interested in. To disable the tracing directly from a C program,
+something like following code snippet can be used:
int trace_fd;
[...]
@@ -1070,10 +1155,10 @@ int main(int argc, char *argv[]) {
}
Note: Here we hard coded the path name. The debugfs mount is not
-guaranteed to be at /debug (and is more commonly at /sys/kernel/debug).
-For simple one time traces, the above is sufficent. For anything else,
-a search through /proc/mounts may be needed to find where the debugfs
-file-system is mounted.
+guaranteed to be at /debug (and is more commonly at
+/sys/kernel/debug). For simple one time traces, the above is
+sufficent. For anything else, a search through /proc/mounts may
+be needed to find where the debugfs file-system is mounted.
Single thread tracing
@@ -1152,49 +1237,297 @@ int main (int argc, char **argv)
return 0;
}
+
+hw-branch-tracer (x86 only)
+---------------------------
+
+This tracer uses the x86 last branch tracing hardware feature to
+collect a branch trace on all cpus with relatively low overhead.
+
+The tracer uses a fixed-size circular buffer per cpu and only
+traces ring 0 branches. The trace file dumps that buffer in the
+following format:
+
+# tracer: hw-branch-tracer
+#
+# CPU# TO <- FROM
+ 0 scheduler_tick+0xb5/0x1bf <- task_tick_idle+0x5/0x6
+ 2 run_posix_cpu_timers+0x2b/0x72a <- run_posix_cpu_timers+0x25/0x72a
+ 0 scheduler_tick+0x139/0x1bf <- scheduler_tick+0xed/0x1bf
+ 0 scheduler_tick+0x17c/0x1bf <- scheduler_tick+0x148/0x1bf
+ 2 run_posix_cpu_timers+0x9e/0x72a <- run_posix_cpu_timers+0x5e/0x72a
+ 0 scheduler_tick+0x1b6/0x1bf <- scheduler_tick+0x1aa/0x1bf
+
+
+The tracer may be used to dump the trace for the oops'ing cpu on
+a kernel oops into the system log. To enable this,
+ftrace_dump_on_oops must be set. To set ftrace_dump_on_oops, one
+can either use the sysctl function or set it via the proc system
+interface.
+
+ sysctl kernel.ftrace_dump_on_oops=1
+
+or
+
+ echo 1 > /proc/sys/kernel/ftrace_dump_on_oops
+
+
+Here's an example of such a dump after a null pointer
+dereference in a kernel module:
+
+[57848.105921] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
+[57848.106019] IP: [<ffffffffa0000006>] open+0x6/0x14 [oops]
+[57848.106019] PGD 2354e9067 PUD 2375e7067 PMD 0
+[57848.106019] Oops: 0002 [#1] SMP
+[57848.106019] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:20:05.0/local_cpus
+[57848.106019] Dumping ftrace buffer:
+[57848.106019] ---------------------------------
+[...]
+[57848.106019] 0 chrdev_open+0xe6/0x165 <- cdev_put+0x23/0x24
+[57848.106019] 0 chrdev_open+0x117/0x165 <- chrdev_open+0xfa/0x165
+[57848.106019] 0 chrdev_open+0x120/0x165 <- chrdev_open+0x11c/0x165
+[57848.106019] 0 chrdev_open+0x134/0x165 <- chrdev_open+0x12b/0x165
+[57848.106019] 0 open+0x0/0x14 [oops] <- chrdev_open+0x144/0x165
+[57848.106019] 0 page_fault+0x0/0x30 <- open+0x6/0x14 [oops]
+[57848.106019] 0 error_entry+0x0/0x5b <- page_fault+0x4/0x30
+[57848.106019] 0 error_kernelspace+0x0/0x31 <- error_entry+0x59/0x5b
+[57848.106019] 0 error_sti+0x0/0x1 <- error_kernelspace+0x2d/0x31
+[57848.106019] 0 page_fault+0x9/0x30 <- error_sti+0x0/0x1
+[57848.106019] 0 do_page_fault+0x0/0x881 <- page_fault+0x1a/0x30
+[...]
+[57848.106019] 0 do_page_fault+0x66b/0x881 <- is_prefetch+0x1ee/0x1f2
+[57848.106019] 0 do_page_fault+0x6e0/0x881 <- do_page_fault+0x67a/0x881
+[57848.106019] 0 oops_begin+0x0/0x96 <- do_page_fault+0x6e0/0x881
+[57848.106019] 0 trace_hw_branch_oops+0x0/0x2d <- oops_begin+0x9/0x96
+[...]
+[57848.106019] 0 ds_suspend_bts+0x2a/0xe3 <- ds_suspend_bts+0x1a/0xe3
+[57848.106019] ---------------------------------
+[57848.106019] CPU 0
+[57848.106019] Modules linked in: oops
+[57848.106019] Pid: 5542, comm: cat Tainted: G W 2.6.28 #23
+[57848.106019] RIP: 0010:[<ffffffffa0000006>] [<ffffffffa0000006>] open+0x6/0x14 [oops]
+[57848.106019] RSP: 0018:ffff880235457d48 EFLAGS: 00010246
+[...]
+
+
+function graph tracer
+---------------------------
+
+This tracer is similar to the function tracer except that it
+probes a function on its entry and its exit. This is done by
+using a dynamically allocated stack of return addresses in each
+task_struct. On function entry the tracer overwrites the return
+address of each function traced to set a custom probe. Thus the
+original return address is stored on the stack of return address
+in the task_struct.
+
+Probing on both ends of a function leads to special features
+such as:
+
+- measure of a function's time execution
+- having a reliable call stack to draw function calls graph
+
+This tracer is useful in several situations:
+
+- you want to find the reason of a strange kernel behavior and
+ need to see what happens in detail on any areas (or specific
+ ones).
+
+- you are experiencing weird latencies but it's difficult to
+ find its origin.
+
+- you want to find quickly which path is taken by a specific
+ function
+
+- you just want to peek inside a working kernel and want to see
+ what happens there.
+
+# tracer: function_graph
+#
+# CPU DURATION FUNCTION CALLS
+# | | | | | | |
+
+ 0) | sys_open() {
+ 0) | do_sys_open() {
+ 0) | getname() {
+ 0) | kmem_cache_alloc() {
+ 0) 1.382 us | __might_sleep();
+ 0) 2.478 us | }
+ 0) | strncpy_from_user() {
+ 0) | might_fault() {
+ 0) 1.389 us | __might_sleep();
+ 0) 2.553 us | }
+ 0) 3.807 us | }
+ 0) 7.876 us | }
+ 0) | alloc_fd() {
+ 0) 0.668 us | _spin_lock();
+ 0) 0.570 us | expand_files();
+ 0) 0.586 us | _spin_unlock();
+
+
+There are several columns that can be dynamically
+enabled/disabled. You can use every combination of options you
+want, depending on your needs.
+
+- The cpu number on which the function executed is default
+ enabled. It is sometimes better to only trace one cpu (see
+ tracing_cpu_mask file) or you might sometimes see unordered
+ function calls while cpu tracing switch.
+
+ hide: echo nofuncgraph-cpu > /debug/tracing/trace_options
+ show: echo funcgraph-cpu > /debug/tracing/trace_options
+
+- The duration (function's time of execution) is displayed on
+ the closing bracket line of a function or on the same line
+ than the current function in case of a leaf one. It is default
+ enabled.
+
+ hide: echo nofuncgraph-duration > /debug/tracing/trace_options
+ show: echo funcgraph-duration > /debug/tracing/trace_options
+
+- The overhead field precedes the duration field in case of
+ reached duration thresholds.
+
+ hide: echo nofuncgraph-overhead > /debug/tracing/trace_options
+ show: echo funcgraph-overhead > /debug/tracing/trace_options
+ depends on: funcgraph-duration
+
+ ie:
+
+ 0) | up_write() {
+ 0) 0.646 us | _spin_lock_irqsave();
+ 0) 0.684 us | _spin_unlock_irqrestore();
+ 0) 3.123 us | }
+ 0) 0.548 us | fput();
+ 0) + 58.628 us | }
+
+ [...]
+
+ 0) | putname() {
+ 0) | kmem_cache_free() {
+ 0) 0.518 us | __phys_addr();
+ 0) 1.757 us | }
+ 0) 2.861 us | }
+ 0) ! 115.305 us | }
+ 0) ! 116.402 us | }
+
+ + means that the function exceeded 10 usecs.
+ ! means that the function exceeded 100 usecs.
+
+
+- The task/pid field displays the thread cmdline and pid which
+ executed the function. It is default disabled.
+
+ hide: echo nofuncgraph-proc > /debug/tracing/trace_options
+ show: echo funcgraph-proc > /debug/tracing/trace_options
+
+ ie:
+
+ # tracer: function_graph
+ #
+ # CPU TASK/PID DURATION FUNCTION CALLS
+ # | | | | | | | | |
+ 0) sh-4802 | | d_free() {
+ 0) sh-4802 | | call_rcu() {
+ 0) sh-4802 | | __call_rcu() {
+ 0) sh-4802 | 0.616 us | rcu_process_gp_end();
+ 0) sh-4802 | 0.586 us | check_for_new_grace_period();
+ 0) sh-4802 | 2.899 us | }
+ 0) sh-4802 | 4.040 us | }
+ 0) sh-4802 | 5.151 us | }
+ 0) sh-4802 | + 49.370 us | }
+
+
+- The absolute time field is an absolute timestamp given by the
+ system clock since it started. A snapshot of this time is
+ given on each entry/exit of functions
+
+ hide: echo nofuncgraph-abstime > /debug/tracing/trace_options
+ show: echo funcgraph-abstime > /debug/tracing/trace_options
+
+ ie:
+
+ #
+ # TIME CPU DURATION FUNCTION CALLS
+ # | | | | | | | |
+ 360.774522 | 1) 0.541 us | }
+ 360.774522 | 1) 4.663 us | }
+ 360.774523 | 1) 0.541 us | __wake_up_bit();
+ 360.774524 | 1) 6.796 us | }
+ 360.774524 | 1) 7.952 us | }
+ 360.774525 | 1) 9.063 us | }
+ 360.774525 | 1) 0.615 us | journal_mark_dirty();
+ 360.774527 | 1) 0.578 us | __brelse();
+ 360.774528 | 1) | reiserfs_prepare_for_journal() {
+ 360.774528 | 1) | unlock_buffer() {
+ 360.774529 | 1) | wake_up_bit() {
+ 360.774529 | 1) | bit_waitqueue() {
+ 360.774530 | 1) 0.594 us | __phys_addr();
+
+
+You can put some comments on specific functions by using
+trace_printk() For example, if you want to put a comment inside
+the __might_sleep() function, you just have to include
+<linux/ftrace.h> and call trace_printk() inside __might_sleep()
+
+trace_printk("I'm a comment!\n")
+
+will produce:
+
+ 1) | __might_sleep() {
+ 1) | /* I'm a comment! */
+ 1) 1.449 us | }
+
+
+You might find other useful features for this tracer in the
+following "dynamic ftrace" section such as tracing only specific
+functions or tasks.
+
dynamic ftrace
--------------
If CONFIG_DYNAMIC_FTRACE is set, the system will run with
virtually no overhead when function tracing is disabled. The way
this works is the mcount function call (placed at the start of
-every kernel function, produced by the -pg switch in gcc), starts
-of pointing to a simple return. (Enabling FTRACE will include the
--pg switch in the compiling of the kernel.)
+every kernel function, produced by the -pg switch in gcc),
+starts of pointing to a simple return. (Enabling FTRACE will
+include the -pg switch in the compiling of the kernel.)
At compile time every C file object is run through the
recordmcount.pl script (located in the scripts directory). This
script will process the C object using objdump to find all the
-locations in the .text section that call mcount. (Note, only
-the .text section is processed, since processing other sections
-like .init.text may cause races due to those sections being freed).
+locations in the .text section that call mcount. (Note, only the
+.text section is processed, since processing other sections like
+.init.text may cause races due to those sections being freed).
-A new section called "__mcount_loc" is created that holds references
-to all the mcount call sites in the .text section. This section is
-compiled back into the original object. The final linker will add
-all these references into a single table.
+A new section called "__mcount_loc" is created that holds
+references to all the mcount call sites in the .text section.
+This section is compiled back into the original object. The
+final linker will add all these references into a single table.
On boot up, before SMP is initialized, the dynamic ftrace code
-scans this table and updates all the locations into nops. It also
-records the locations, which are added to the available_filter_functions
-list. Modules are processed as they are loaded and before they are
-executed. When a module is unloaded, it also removes its functions from
-the ftrace function list. This is automatic in the module unload
-code, and the module author does not need to worry about it.
-
-When tracing is enabled, kstop_machine is called to prevent races
-with the CPUS executing code being modified (which can cause the
-CPU to do undesireable things), and the nops are patched back
-to calls. But this time, they do not call mcount (which is just
-a function stub). They now call into the ftrace infrastructure.
+scans this table and updates all the locations into nops. It
+also records the locations, which are added to the
+available_filter_functions list. Modules are processed as they
+are loaded and before they are executed. When a module is
+unloaded, it also removes its functions from the ftrace function
+list. This is automatic in the module unload code, and the
+module author does not need to worry about it.
+
+When tracing is enabled, kstop_machine is called to prevent
+races with the CPUS executing code being modified (which can
+cause the CPU to do undesireable things), and the nops are
+patched back to calls. But this time, they do not call mcount
+(which is just a function stub). They now call into the ftrace
+infrastructure.
One special side-effect to the recording of the functions being
traced is that we can now selectively choose which functions we
-wish to trace and which ones we want the mcount calls to remain as
-nops.
+wish to trace and which ones we want the mcount calls to remain
+as nops.
-Two files are used, one for enabling and one for disabling the tracing
-of specified functions. They are:
+Two files are used, one for enabling and one for disabling the
+tracing of specified functions. They are:
set_ftrace_filter
@@ -1202,8 +1535,8 @@ and
set_ftrace_notrace
-A list of available functions that you can add to these files is listed
-in:
+A list of available functions that you can add to these files is
+listed in:
available_filter_functions
@@ -1240,8 +1573,8 @@ hrtimer_interrupt
sys_nanosleep
-Perhaps this is not enough. The filters also allow simple wild cards.
-Only the following are currently available
+Perhaps this is not enough. The filters also allow simple wild
+cards. Only the following are currently available
<match>* - will match functions that begin with <match>
*<match> - will match functions that end with <match>
@@ -1251,9 +1584,9 @@ These are the only wild cards which are supported.
<match>*<match> will not work.
-Note: It is better to use quotes to enclose the wild cards, otherwise
- the shell may expand the parameters into names of files in the local
- directory.
+Note: It is better to use quotes to enclose the wild cards,
+ otherwise the shell may expand the parameters into names
+ of files in the local directory.
# echo 'hrtimer_*' > /debug/tracing/set_ftrace_filter
@@ -1299,7 +1632,8 @@ This is because the '>' and '>>' act just like they do in bash.
To rewrite the filters, use '>'
To append to the filters, use '>>'
-To clear out a filter so that all functions will be recorded again:
+To clear out a filter so that all functions will be recorded
+again:
# echo > /debug/tracing/set_ftrace_filter
# cat /debug/tracing/set_ftrace_filter
@@ -1331,7 +1665,8 @@ hrtimer_get_res
hrtimer_init_sleeper
-The set_ftrace_notrace prevents those functions from being traced.
+The set_ftrace_notrace prevents those functions from being
+traced.
# echo '*preempt*' '*lock*' > /debug/tracing/set_ftrace_notrace
@@ -1353,13 +1688,75 @@ Produces:
We can see that there's no more lock or preempt tracing.
+
+Dynamic ftrace with the function graph tracer
+---------------------------------------------
+
+Although what has been explained above concerns both the
+function tracer and the function-graph-tracer, there are some
+special features only available in the function-graph tracer.
+
+If you want to trace only one function and all of its children,
+you just have to echo its name into set_graph_function:
+
+ echo __do_fault > set_graph_function
+
+will produce the following "expanded" trace of the __do_fault()
+function:
+
+ 0) | __do_fault() {
+ 0) | filemap_fault() {
+ 0) | find_lock_page() {
+ 0) 0.804 us | find_get_page();
+ 0) | __might_sleep() {
+ 0) 1.329 us | }
+ 0) 3.904 us | }
+ 0) 4.979 us | }
+ 0) 0.653 us | _spin_lock();
+ 0) 0.578 us | page_add_file_rmap();
+ 0) 0.525 us | native_set_pte_at();
+ 0) 0.585 us | _spin_unlock();
+ 0) | unlock_page() {
+ 0) 0.541 us | page_waitqueue();
+ 0) 0.639 us | __wake_up_bit();
+ 0) 2.786 us | }
+ 0) + 14.237 us | }
+ 0) | __do_fault() {
+ 0) | filemap_fault() {
+ 0) | find_lock_page() {
+ 0) 0.698 us | find_get_page();
+ 0) | __might_sleep() {
+ 0) 1.412 us | }
+ 0) 3.950 us | }
+ 0) 5.098 us | }
+ 0) 0.631 us | _spin_lock();
+ 0) 0.571 us | page_add_file_rmap();
+ 0) 0.526 us | native_set_pte_at();
+ 0) 0.586 us | _spin_unlock();
+ 0) | unlock_page() {
+ 0) 0.533 us | page_waitqueue();
+ 0) 0.638 us | __wake_up_bit();
+ 0) 2.793 us | }
+ 0) + 14.012 us | }
+
+You can also expand several functions at once:
+
+ echo sys_open > set_graph_function
+ echo sys_close >> set_graph_function
+
+Now if you want to go back to trace all functions you can clear
+this special filter via:
+
+ echo > set_graph_function
+
+
trace_pipe
----------
-The trace_pipe outputs the same content as the trace file, but the effect
-on the tracing is different. Every read from trace_pipe is consumed.
-This means that subsequent reads will be different. The trace
-is live.
+The trace_pipe outputs the same content as the trace file, but
+the effect on the tracing is different. Every read from
+trace_pipe is consumed. This means that subsequent reads will be
+different. The trace is live.
# echo function > /debug/tracing/current_tracer
# cat /debug/tracing/trace_pipe > /tmp/trace.out &
@@ -1387,38 +1784,45 @@ is live.
bash-4043 [00] 41.267111: select_task_rq_rt <-try_to_wake_up
-Note, reading the trace_pipe file will block until more input is added.
-By changing the tracer, trace_pipe will issue an EOF. We needed
-to set the function tracer _before_ we "cat" the trace_pipe file.
+Note, reading the trace_pipe file will block until more input is
+added. By changing the tracer, trace_pipe will issue an EOF. We
+needed to set the function tracer _before_ we "cat" the
+trace_pipe file.
trace entries
-------------
-Having too much or not enough data can be troublesome in diagnosing
-an issue in the kernel. The file buffer_size_kb is used to modify
-the size of the internal trace buffers. The number listed
-is the number of entries that can be recorded per CPU. To know
-the full size, multiply the number of possible CPUS with the
-number of entries.
+Having too much or not enough data can be troublesome in
+diagnosing an issue in the kernel. The file buffer_size_kb is
+used to modify the size of the internal trace buffers. The
+number listed is the number of entries that can be recorded per
+CPU. To know the full size, multiply the number of possible CPUS
+with the number of entries.
# cat /debug/tracing/buffer_size_kb
1408 (units kilobytes)
-Note, to modify this, you must have tracing completely disabled. To do that,
-echo "nop" into the current_tracer. If the current_tracer is not set
-to "nop", an EINVAL error will be returned.
+Note, to modify this, you must have tracing completely disabled.
+To do that, echo "nop" into the current_tracer. If the
+current_tracer is not set to "nop", an EINVAL error will be
+returned.
# echo nop > /debug/tracing/current_tracer
# echo 10000 > /debug/tracing/buffer_size_kb
# cat /debug/tracing/buffer_size_kb
10000 (units kilobytes)
-The number of pages which will be allocated is limited to a percentage
-of available memory. Allocating too much will produce an error.
+The number of pages which will be allocated is limited to a
+percentage of available memory. Allocating too much will produce
+an error.
# echo 1000000000000 > /debug/tracing/buffer_size_kb
-bash: echo: write error: Cannot allocate memory
# cat /debug/tracing/buffer_size_kb
85
+-----------
+
+More details can be found in the source code, in the
+kernel/tracing/*.c files.
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 421920897a3..2895ce29dea 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -50,6 +50,7 @@ parameter is applicable:
ISAPNP ISA PnP code is enabled.
ISDN Appropriate ISDN support is enabled.
JOY Appropriate joystick support is enabled.
+ KMEMTRACE kmemtrace is enabled.
LIBATA Libata driver is enabled
LP Printer support is enabled.
LOOP Loopback device support is enabled.
@@ -259,6 +260,22 @@ and is between 256 and 4096 characters. It is defined in the file
to assume that this machine's pmtimer latches its value
and always returns good values.
+ acpi_enforce_resources= [ACPI]
+ { strict | lax | no }
+ Check for resource conflicts between native drivers
+ and ACPI OperationRegions (SystemIO and SystemMemory
+ only). IO ports and memory declared in ACPI might be
+ used by the ACPI subsystem in arbitrary AML code and
+ can interfere with legacy drivers.
+ strict (default): access to resources claimed by ACPI
+ is denied; legacy drivers trying to access reserved
+ resources will fail to bind to device using them.
+ lax: access to resources claimed by ACPI is allowed;
+ legacy drivers trying to access reserved resources
+ will bind successfully but a warning message is logged.
+ no: ACPI OperationRegions are not marked as reserved,
+ no further checks are performed.
+
agp= [AGP]
{ off | try_unsupported }
off: disable AGP support
@@ -617,6 +634,9 @@ and is between 256 and 4096 characters. It is defined in the file
debug_objects [KNL] Enable object debugging
+ no_debug_objects
+ [KNL] Disable object debugging
+
debugpat [X86] Enable PAT debugging
decnet.addr= [HW,NET]
@@ -1078,6 +1098,15 @@ and is between 256 and 4096 characters. It is defined in the file
use the HighMem zone if it exists, and the Normal
zone if it does not.
+ kmemtrace.enable= [KNL,KMEMTRACE] Format: { yes | no }
+ Controls whether kmemtrace is enabled
+ at boot-time.
+
+ kmemtrace.subbufs=n [KNL,KMEMTRACE] Overrides the number of
+ subbufs kmemtrace's relay channel has. Set this
+ higher than default (KMEMTRACE_N_SUBBUFS in code) if
+ you experience buffer overruns.
+
movablecore=nn[KMG] [KNL,X86-32,IA-64,PPC,X86-64] This parameter
is similar to kernelcore except it specifies the
amount of memory used for migratable allocations.
@@ -1546,6 +1575,8 @@ and is between 256 and 4096 characters. It is defined in the file
Valid arguments: on, off
Default: on
+ noiotrap [SH] Disables trapped I/O port accesses.
+
noirqdebug [X86-32] Disables the code which attempts to detect and
disable unhandled interrupt sources.
@@ -2364,6 +2395,8 @@ and is between 256 and 4096 characters. It is defined in the file
tp720= [HW,PS2]
+ trace_buf_size=nn[KMG] [ftrace] will set tracing buffer size.
+
trix= [HW,OSS] MediaTrix AudioTrix Pro
Format:
<io>,<irq>,<dma>,<dma2>,<sb_io>,<sb_irq>,<sb_dma>,<mpu_io>,<mpu_irq>
diff --git a/Documentation/laptops/acer-wmi.txt b/Documentation/laptops/acer-wmi.txt
index 2b3a6b5260b..5ee2a02b3b4 100644
--- a/Documentation/laptops/acer-wmi.txt
+++ b/Documentation/laptops/acer-wmi.txt
@@ -1,9 +1,9 @@
Acer Laptop WMI Extras Driver
http://code.google.com/p/aceracpi
-Version 0.2
-18th August 2008
+Version 0.3
+4th April 2009
-Copyright 2007-2008 Carlos Corbacho <carlos@strangeworlds.co.uk>
+Copyright 2007-2009 Carlos Corbacho <carlos@strangeworlds.co.uk>
acer-wmi is a driver to allow you to control various parts of your Acer laptop
hardware under Linux which are exposed via ACPI-WMI.
@@ -36,6 +36,10 @@ not possible in kernel space from a 64 bit OS.
Supported Hardware
******************
+NOTE: The Acer Aspire One is not supported hardware. It cannot work with
+acer-wmi until Acer fix their ACPI-WMI implementation on them, so has been
+blacklisted until that happens.
+
Please see the website for the current list of known working hardare:
http://code.google.com/p/aceracpi/wiki/SupportedHardware
diff --git a/Documentation/laptops/thinkpad-acpi.txt b/Documentation/laptops/thinkpad-acpi.txt
index 41bc99fa188..3d7650768bb 100644
--- a/Documentation/laptops/thinkpad-acpi.txt
+++ b/Documentation/laptops/thinkpad-acpi.txt
@@ -20,7 +20,8 @@ moved to the drivers/misc tree and renamed to thinkpad-acpi for kernel
kernel 2.6.29 and release 0.22.
The driver is named "thinkpad-acpi". In some places, like module
-names, "thinkpad_acpi" is used because of userspace issues.
+names and log messages, "thinkpad_acpi" is used because of userspace
+issues.
"tpacpi" is used as a shorthand where "thinkpad-acpi" would be too
long due to length limitations on some Linux kernel versions.
@@ -37,7 +38,7 @@ detailed description):
- ThinkLight on and off
- limited docking and undocking
- UltraBay eject
- - CMOS control
+ - CMOS/UCMS control
- LED control
- ACPI sounds
- temperature sensors
@@ -46,6 +47,7 @@ detailed description):
- Volume control
- Fan control and monitoring: fan speed, fan enable/disable
- WAN enable and disable
+ - UWB enable and disable
A compatibility table by model and feature is maintained on the web
site, http://ibm-acpi.sf.net/. I appreciate any success or failure
@@ -53,7 +55,7 @@ reports, especially if they add to or correct the compatibility table.
Please include the following information in your report:
- ThinkPad model name
- - a copy of your DSDT, from /proc/acpi/dsdt
+ - a copy of your ACPI tables, using the "acpidump" utility
- a copy of the output of dmidecode, with serial numbers
and UUIDs masked off
- which driver features work and which don't
@@ -66,17 +68,18 @@ Installation
------------
If you are compiling this driver as included in the Linux kernel
-sources, simply enable the CONFIG_THINKPAD_ACPI option, and optionally
-enable the CONFIG_THINKPAD_ACPI_BAY option if you want the
-thinkpad-specific bay functionality.
+sources, look for the CONFIG_THINKPAD_ACPI Kconfig option.
+It is located on the menu path: "Device Drivers" -> "X86 Platform
+Specific Device Drivers" -> "ThinkPad ACPI Laptop Extras".
+
Features
--------
The driver exports two different interfaces to userspace, which can be
used to access the features it provides. One is a legacy procfs-based
-interface, which will be removed at some time in the distant future.
-The other is a new sysfs-based interface which is not complete yet.
+interface, which will be removed at some time in the future. The other
+is a new sysfs-based interface which is not complete yet.
The procfs interface creates the /proc/acpi/ibm directory. There is a
file under that directory for each feature it supports. The procfs
@@ -111,15 +114,17 @@ The version of thinkpad-acpi's sysfs interface is exported by the driver
as a driver attribute (see below).
Sysfs driver attributes are on the driver's sysfs attribute space,
-for 2.6.23 this is /sys/bus/platform/drivers/thinkpad_acpi/ and
+for 2.6.23+ this is /sys/bus/platform/drivers/thinkpad_acpi/ and
/sys/bus/platform/drivers/thinkpad_hwmon/
Sysfs device attributes are on the thinkpad_acpi device sysfs attribute
-space, for 2.6.23 this is /sys/devices/platform/thinkpad_acpi/.
+space, for 2.6.23+ this is /sys/devices/platform/thinkpad_acpi/.
Sysfs device attributes for the sensors and fan are on the
thinkpad_hwmon device's sysfs attribute space, but you should locate it
-looking for a hwmon device with the name attribute of "thinkpad".
+looking for a hwmon device with the name attribute of "thinkpad", or
+better yet, through libsensors.
+
Driver version
--------------
@@ -129,6 +134,7 @@ sysfs driver attribute: version
The driver name and version. No commands can be written to this file.
+
Sysfs interface version
-----------------------
@@ -160,6 +166,7 @@ expect that an attribute might not be there, and deal with it properly
(an attribute not being there *is* a valid way to make it clear that a
feature is not available in sysfs).
+
Hot keys
--------
@@ -172,17 +179,14 @@ system. Enabling the hotkey functionality of thinkpad-acpi signals the
firmware that such a driver is present, and modifies how the ThinkPad
firmware will behave in many situations.
-The driver enables the hot key feature automatically when loaded. The
-feature can later be disabled and enabled back at runtime. The driver
-will also restore the hot key feature to its previous state and mask
-when it is unloaded.
+The driver enables the HKEY ("hot key") event reporting automatically
+when loaded, and disables it when it is removed.
-When the hotkey feature is enabled and the hot key mask is set (see
-below), the driver will report HKEY events in the following format:
+The driver will report HKEY events in the following format:
ibm/hotkey HKEY 00000080 0000xxxx
-Some of these events refer to hot key presses, but not all.
+Some of these events refer to hot key presses, but not all of them.
The driver will generate events over the input layer for hot keys and
radio switches, and over the ACPI netlink layer for other events. The
@@ -214,13 +218,17 @@ procfs notes:
The following commands can be written to the /proc/acpi/ibm/hotkey file:
- echo enable > /proc/acpi/ibm/hotkey -- enable the hot keys feature
- echo disable > /proc/acpi/ibm/hotkey -- disable the hot keys feature
echo 0xffffffff > /proc/acpi/ibm/hotkey -- enable all hot keys
echo 0 > /proc/acpi/ibm/hotkey -- disable all possible hot keys
... any other 8-hex-digit mask ...
echo reset > /proc/acpi/ibm/hotkey -- restore the original mask
+The following commands have been deprecated and will cause the kernel
+to log a warning:
+
+ echo enable > /proc/acpi/ibm/hotkey -- does nothing
+ echo disable > /proc/acpi/ibm/hotkey -- returns an error
+
The procfs interface does not support NVRAM polling control. So as to
maintain maximum bug-to-bug compatibility, it does not report any masks,
nor does it allow one to manipulate the hot key mask when the firmware
@@ -229,12 +237,9 @@ does not support masks at all, even if NVRAM polling is in use.
sysfs notes:
hotkey_bios_enabled:
- Returns the status of the hot keys feature when
- thinkpad-acpi was loaded. Upon module unload, the hot
- key feature status will be restored to this value.
+ DEPRECATED, WILL BE REMOVED SOON.
- 0: hot keys were disabled
- 1: hot keys were enabled (unusual)
+ Returns 0.
hotkey_bios_mask:
Returns the hot keys mask when thinkpad-acpi was loaded.
@@ -242,13 +247,10 @@ sysfs notes:
to this value.
hotkey_enable:
- Enables/disables the hot keys feature in the ACPI
- firmware, and reports current status of the hot keys
- feature. Has no effect on the NVRAM hot key polling
- functionality.
+ DEPRECATED, WILL BE REMOVED SOON.
- 0: disables the hot keys feature / feature disabled
- 1: enables the hot keys feature / feature enabled
+ 0: returns -EPERM
+ 1: does nothing
hotkey_mask:
bit mask to enable driver-handling (and depending on
@@ -618,6 +620,7 @@ For Lenovo models *with* ACPI backlight control:
and map them to KEY_BRIGHTNESS_UP and KEY_BRIGHTNESS_DOWN. Process
these keys on userspace somehow (e.g. by calling xbacklight).
+
Bluetooth
---------
@@ -628,6 +631,9 @@ sysfs rfkill class: switch "tpacpi_bluetooth_sw"
This feature shows the presence and current state of a ThinkPad
Bluetooth device in the internal ThinkPad CDC slot.
+If the ThinkPad supports it, the Bluetooth state is stored in NVRAM,
+so it is kept across reboots and power-off.
+
Procfs notes:
If Bluetooth is installed, the following commands can be used:
@@ -652,6 +658,7 @@ Sysfs notes:
rfkill controller switch "tpacpi_bluetooth_sw": refer to
Documentation/rfkill.txt for details.
+
Video output control -- /proc/acpi/ibm/video
--------------------------------------------
@@ -693,11 +700,8 @@ Fn-F7 from working. This also disables the video output switching
features of this driver, as it uses the same ACPI methods as
Fn-F7. Video switching on the console should still work.
-UPDATE: There's now a patch for the X.org Radeon driver which
-addresses this issue. Some people are reporting success with the patch
-while others are still having problems. For more information:
+UPDATE: refer to https://bugs.freedesktop.org/show_bug.cgi?id=2000
-https://bugs.freedesktop.org/show_bug.cgi?id=2000
ThinkLight control
------------------
@@ -720,10 +724,11 @@ The ThinkLight sysfs interface is documented by the LED class
documentation, in Documentation/leds-class.txt. The ThinkLight LED name
is "tpacpi::thinklight".
-Due to limitations in the sysfs LED class, if the status of the thinklight
+Due to limitations in the sysfs LED class, if the status of the ThinkLight
cannot be read or if it is unknown, thinkpad-acpi will report it as "off".
It is impossible to know if the status returned through sysfs is valid.
+
Docking / undocking -- /proc/acpi/ibm/dock
------------------------------------------
@@ -784,6 +789,7 @@ the only docking stations currently supported are the X-series
UltraBase docks and "dumb" port replicators like the Mini Dock (the
latter don't need any ACPI support, actually).
+
UltraBay eject -- /proc/acpi/ibm/bay
------------------------------------
@@ -847,8 +853,9 @@ supported. Use "eject2" instead of "eject" for the second bay.
Note: the UltraBay eject support on the 600e/x, A22p and A3x is
EXPERIMENTAL and may not work as expected. USE WITH CAUTION!
-CMOS control
-------------
+
+CMOS/UCMS control
+-----------------
procfs: /proc/acpi/ibm/cmos
sysfs device attribute: cmos_command
@@ -882,6 +889,7 @@ The cmos command interface is prone to firmware split-brain problems, as
in newer ThinkPads it is just a compatibility layer. Do not use it, it is
exported just as a debug tool.
+
LED control
-----------
@@ -893,6 +901,17 @@ some older ThinkPad models, it is possible to query the status of the
LED indicators as well. Newer ThinkPads cannot query the real status
of the LED indicators.
+Because misuse of the LEDs could induce an unaware user to perform
+dangerous actions (like undocking or ejecting a bay device while the
+buses are still active), or mask an important alarm (such as a nearly
+empty battery, or a broken battery), access to most LEDs is
+restricted.
+
+Unrestricted access to all LEDs requires that thinkpad-acpi be
+compiled with the CONFIG_THINKPAD_ACPI_UNSAFE_LEDS option enabled.
+Distributions must never enable this option. Individual users that
+are aware of the consequences are welcome to enabling it.
+
procfs notes:
The available commands are:
@@ -939,6 +958,7 @@ ThinkPad indicator LED should blink in hardware accelerated mode, use the
"timer" trigger, and leave the delay_on and delay_off parameters set to
zero (to request hardware acceleration autodetection).
+
ACPI sounds -- /proc/acpi/ibm/beep
----------------------------------
@@ -968,6 +988,7 @@ X40:
16 - one medium-pitched beep repeating constantly, stop with 17
17 - stop 16
+
Temperature sensors
-------------------
@@ -1115,6 +1136,7 @@ registers contain the current battery capacity, etc. If you experiment
with this, do send me your results (including some complete dumps with
a description of the conditions when they were taken.)
+
LCD brightness control
----------------------
@@ -1124,10 +1146,9 @@ sysfs backlight device "thinkpad_screen"
This feature allows software control of the LCD brightness on ThinkPad
models which don't have a hardware brightness slider.
-It has some limitations: the LCD backlight cannot be actually turned on or
-off by this interface, and in many ThinkPad models, the "dim while on
-battery" functionality will be enabled by the BIOS when this interface is
-used, and cannot be controlled.
+It has some limitations: the LCD backlight cannot be actually turned
+on or off by this interface, it just controls the backlight brightness
+level.
On IBM (and some of the earlier Lenovo) ThinkPads, the backlight control
has eight brightness levels, ranging from 0 to 7. Some of the levels
@@ -1136,10 +1157,15 @@ display backlight brightness control methods have 16 levels, ranging
from 0 to 15.
There are two interfaces to the firmware for direct brightness control,
-EC and CMOS. To select which one should be used, use the
+EC and UCMS (or CMOS). To select which one should be used, use the
brightness_mode module parameter: brightness_mode=1 selects EC mode,
-brightness_mode=2 selects CMOS mode, brightness_mode=3 selects both EC
-and CMOS. The driver tries to auto-detect which interface to use.
+brightness_mode=2 selects UCMS mode, brightness_mode=3 selects EC
+mode with NVRAM backing (so that brightness changes are remembered
+across shutdown/reboot).
+
+The driver tries to select which interface to use from a table of
+defaults for each ThinkPad model. If it makes a wrong choice, please
+report this as a bug, so that we can fix it.
When display backlight brightness controls are available through the
standard ACPI interface, it is best to use it instead of this direct
@@ -1201,6 +1227,7 @@ WARNING:
and maybe reduce the life of the backlight lamps by needlessly kicking
its level up and down at every change.
+
Volume control -- /proc/acpi/ibm/volume
---------------------------------------
@@ -1217,6 +1244,11 @@ distinct. The unmute the volume after the mute command, use either the
up or down command (the level command will not unmute the volume).
The current volume level and mute state is shown in the file.
+The ALSA mixer interface to this feature is still missing, but patches
+to add it exist. That problem should be addressed in the not so
+distant future.
+
+
Fan control and monitoring: fan speed, fan enable/disable
---------------------------------------------------------
@@ -1383,8 +1415,11 @@ procfs: /proc/acpi/ibm/wan
sysfs device attribute: wwan_enable (deprecated)
sysfs rfkill class: switch "tpacpi_wwan_sw"
-This feature shows the presence and current state of a W-WAN (Sierra
-Wireless EV-DO) device.
+This feature shows the presence and current state of the built-in
+Wireless WAN device.
+
+If the ThinkPad supports it, the WWAN state is stored in NVRAM,
+so it is kept across reboots and power-off.
It was tested on a Lenovo ThinkPad X60. It should probably work on other
ThinkPad models which come with this module installed.
@@ -1413,6 +1448,7 @@ Sysfs notes:
rfkill controller switch "tpacpi_wwan_sw": refer to
Documentation/rfkill.txt for details.
+
EXPERIMENTAL: UWB
-----------------
@@ -1431,6 +1467,7 @@ Sysfs notes:
rfkill controller switch "tpacpi_uwb_sw": refer to
Documentation/rfkill.txt for details.
+
Multiple Commands, Module Parameters
------------------------------------
@@ -1445,6 +1482,7 @@ for example:
modprobe thinkpad_acpi hotkey=enable,0xffff video=auto_disable
+
Enabling debugging output
-------------------------
@@ -1457,8 +1495,15 @@ will enable all debugging output classes. It takes a bitmask, so
to enable more than one output class, just add their values.
Debug bitmask Description
+ 0x8000 Disclose PID of userspace programs
+ accessing some functions of the driver
0x0001 Initialization and probing
0x0002 Removal
+ 0x0004 RF Transmitter control (RFKILL)
+ (bluetooth, WWAN, UWB...)
+ 0x0008 HKEY event interface, hotkeys
+ 0x0010 Fan control
+ 0x0020 Backlight brightness
There is also a kernel build option to enable more debugging
information, which may be necessary to debug driver problems.
@@ -1467,6 +1512,7 @@ The level of debugging information output by the driver can be changed
at runtime through sysfs, using the driver attribute debug_level. The
attribute takes the same bitmask as the debug module parameter above.
+
Force loading of module
-----------------------
@@ -1505,3 +1551,7 @@ Sysfs interface changelog:
0x020200: Add poll()/select() support to the following attributes:
hotkey_radio_sw, wakeup_hotunplug_complete, wakeup_reason
+
+0x020300: hotkey enable/disable support removed, attributes
+ hotkey_bios_enabled and hotkey_enable deprecated and
+ marked for removal.
diff --git a/Documentation/sysrq.txt b/Documentation/sysrq.txt
index afa2946892d..cf42b820ff9 100644
--- a/Documentation/sysrq.txt
+++ b/Documentation/sysrq.txt
@@ -115,6 +115,8 @@ On all - write a character to /proc/sysrq-trigger. e.g.:
'x' - Used by xmon interface on ppc/powerpc platforms.
+'z' - Dump the ftrace buffer
+
'0'-'9' - Sets the console log level, controlling which kernel messages
will be printed to your console. ('0', for example would make
it so that only emergency messages like PANICs or OOPSes would
diff --git a/Documentation/tracepoints.txt b/Documentation/tracepoints.txt
index 6f0a044f5b5..c0e1ceed75a 100644
--- a/Documentation/tracepoints.txt
+++ b/Documentation/tracepoints.txt
@@ -45,8 +45,8 @@ In include/trace/subsys.h :
#include <linux/tracepoint.h>
DECLARE_TRACE(subsys_eventname,
- TPPROTO(int firstarg, struct task_struct *p),
- TPARGS(firstarg, p));
+ TP_PROTO(int firstarg, struct task_struct *p),
+ TP_ARGS(firstarg, p));
In subsys/file.c (where the tracing statement must be added) :
@@ -66,10 +66,10 @@ Where :
- subsys is the name of your subsystem.
- eventname is the name of the event to trace.
-- TPPROTO(int firstarg, struct task_struct *p) is the prototype of the
+- TP_PROTO(int firstarg, struct task_struct *p) is the prototype of the
function called by this tracepoint.
-- TPARGS(firstarg, p) are the parameters names, same as found in the
+- TP_ARGS(firstarg, p) are the parameters names, same as found in the
prototype.
Connecting a function (probe) to a tracepoint is done by providing a
@@ -103,13 +103,14 @@ used to export the defined tracepoints.
* Probe / tracepoint example
-See the example provided in samples/tracepoints/src
+See the example provided in samples/tracepoints
-Compile them with your kernel.
+Compile them with your kernel. They are built during 'make' (not
+'make modules') when CONFIG_SAMPLE_TRACEPOINTS=m.
Run, as root :
-modprobe tracepoint-example (insmod order is not important)
-modprobe tracepoint-probe-example
-cat /proc/tracepoint-example (returns an expected error)
-rmmod tracepoint-example tracepoint-probe-example
+modprobe tracepoint-sample (insmod order is not important)
+modprobe tracepoint-probe-sample
+cat /proc/tracepoint-sample (returns an expected error)
+rmmod tracepoint-sample tracepoint-probe-sample
dmesg
diff --git a/Documentation/vm/kmemtrace.txt b/Documentation/vm/kmemtrace.txt
new file mode 100644
index 00000000000..a956d9b7f94
--- /dev/null
+++ b/Documentation/vm/kmemtrace.txt
@@ -0,0 +1,126 @@
+ kmemtrace - Kernel Memory Tracer
+
+ by Eduard - Gabriel Munteanu
+ <eduard.munteanu@linux360.ro>
+
+I. Introduction
+===============
+
+kmemtrace helps kernel developers figure out two things:
+1) how different allocators (SLAB, SLUB etc.) perform
+2) how kernel code allocates memory and how much
+
+To do this, we trace every allocation and export information to the userspace
+through the relay interface. We export things such as the number of requested
+bytes, the number of bytes actually allocated (i.e. including internal
+fragmentation), whether this is a slab allocation or a plain kmalloc() and so
+on.
+
+The actual analysis is performed by a userspace tool (see section III for
+details on where to get it from). It logs the data exported by the kernel,
+processes it and (as of writing this) can provide the following information:
+- the total amount of memory allocated and fragmentation per call-site
+- the amount of memory allocated and fragmentation per allocation
+- total memory allocated and fragmentation in the collected dataset
+- number of cross-CPU allocation and frees (makes sense in NUMA environments)
+
+Moreover, it can potentially find inconsistent and erroneous behavior in
+kernel code, such as using slab free functions on kmalloc'ed memory or
+allocating less memory than requested (but not truly failed allocations).
+
+kmemtrace also makes provisions for tracing on some arch and analysing the
+data on another.
+
+II. Design and goals
+====================
+
+kmemtrace was designed to handle rather large amounts of data. Thus, it uses
+the relay interface to export whatever is logged to userspace, which then
+stores it. Analysis and reporting is done asynchronously, that is, after the
+data is collected and stored. By design, it allows one to log and analyse
+on different machines and different arches.
+
+As of writing this, the ABI is not considered stable, though it might not
+change much. However, no guarantees are made about compatibility yet. When
+deemed stable, the ABI should still allow easy extension while maintaining
+backward compatibility. This is described further in Documentation/ABI.
+
+Summary of design goals:
+ - allow logging and analysis to be done across different machines
+ - be fast and anticipate usage in high-load environments (*)
+ - be reasonably extensible
+ - make it possible for GNU/Linux distributions to have kmemtrace
+ included in their repositories
+
+(*) - one of the reasons Pekka Enberg's original userspace data analysis
+ tool's code was rewritten from Perl to C (although this is more than a
+ simple conversion)
+
+
+III. Quick usage guide
+======================
+
+1) Get a kernel that supports kmemtrace and build it accordingly (i.e. enable
+CONFIG_KMEMTRACE).
+
+2) Get the userspace tool and build it:
+$ git-clone git://repo.or.cz/kmemtrace-user.git # current repository
+$ cd kmemtrace-user/
+$ ./autogen.sh
+$ ./configure
+$ make
+
+3) Boot the kmemtrace-enabled kernel if you haven't, preferably in the
+'single' runlevel (so that relay buffers don't fill up easily), and run
+kmemtrace:
+# '$' does not mean user, but root here.
+$ mount -t debugfs none /sys/kernel/debug
+$ mount -t proc none /proc
+$ cd path/to/kmemtrace-user/
+$ ./kmemtraced
+Wait a bit, then stop it with CTRL+C.
+$ cat /sys/kernel/debug/kmemtrace/total_overruns # Check if we didn't
+ # overrun, should
+ # be zero.
+$ (Optionally) [Run kmemtrace_check separately on each cpu[0-9]*.out file to
+ check its correctness]
+$ ./kmemtrace-report
+
+Now you should have a nice and short summary of how the allocator performs.
+
+IV. FAQ and known issues
+========================
+
+Q: 'cat /sys/kernel/debug/kmemtrace/total_overruns' is non-zero, how do I fix
+this? Should I worry?
+A: If it's non-zero, this affects kmemtrace's accuracy, depending on how
+large the number is. You can fix it by supplying a higher
+'kmemtrace.subbufs=N' kernel parameter.
+---
+
+Q: kmemtrace_check reports errors, how do I fix this? Should I worry?
+A: This is a bug and should be reported. It can occur for a variety of
+reasons:
+ - possible bugs in relay code
+ - possible misuse of relay by kmemtrace
+ - timestamps being collected unorderly
+Or you may fix it yourself and send us a patch.
+---
+
+Q: kmemtrace_report shows many errors, how do I fix this? Should I worry?
+A: This is a known issue and I'm working on it. These might be true errors
+in kernel code, which may have inconsistent behavior (e.g. allocating memory
+with kmem_cache_alloc() and freeing it with kfree()). Pekka Enberg pointed
+out this behavior may work with SLAB, but may fail with other allocators.
+
+It may also be due to lack of tracing in some unusual allocator functions.
+
+We don't want bug reports regarding this issue yet.
+---
+
+V. See also
+===========
+
+Documentation/kernel-parameters.txt
+Documentation/ABI/testing/debugfs-kmemtrace
+