Log and trace configuration

Open Liberty has a unified logging component that handles messages that are written by applications and the runtime, and provides First Failure Data Capture (FFDC) capability. Logging data that is written by applications by using the System.out, System.err, or java.util.logging.Logger streams is combined into the server logs.

A server has the following three primary log files:

  • console.log - This file is created by the server start command. It contains the redirected standard output and standard error streams from the underlying JVM process. This console output is formatted for human readability and it lacks some information that is useful for automated log analysis. The console.log file is reset when the server is restarted.

  • messages.log - This file contains all messages that are written or captured by the logging component. All messages that are written to this file contain additional information such as the message timestamp and the ID of the thread that wrote the message. This file is suitable for automated log analysis. This file does not contain messages that are written directly by the JVM process.

  • trace.log - This file is created only if you enable trace. It contains all the content of the messages.log file plus any enabled trace. This file does not contain messages that are written directly by the JVM process.

The following sections provide more information about configuring your Open Liberty logs:

Logging configuration

The logging component can be controlled through the server configuration. The logging component can be fully configured in your server.xml file by the logging element. However, logging is initialized before the server.xml file is processed so configuring logging through the server.xml file can result in early log entries that use a different log configuration from later ones. To avoid this problem, you can provide much of the logging configuration in the boostrap.properties file and in some cases by using environment variables. Avoid specifying different values for the same configuration property in both the bootstrap.properties and the server.xml file. For more information about logging configuration settings, see the Configuration settings by source table.

Log file storage management

The console.log file is created by redirecting the process stdout and stderr streams to a file. As a result, Open Liberty is unable to offer the same level of management, like log rollover, as it offers for the messages.log file. If you are concerned about the size of the console.log file, you can set the console log to OFF and use the messages.log file instead. All messages that are written or captured by the logging component are written to the messages.log file, and you can configure file rollover in the messages log.

However, when you set the console log level to OFF, messages from the underlying JVM are still written to the console.log file. The same principle applies to any messages that are written to the native standard out and standard error streams. The Open Liberty logging component does not capture or control the destination of messages that are written directly by the JVM process, such as -verbose:gc output. These messages are routed to the file system according to the active JVM settings and they do not appear in the messages.log file.

To prevent the logging component from writing to the console log and configure the messages.log file to roll over three times at 100 Mb, use the following configuration:

com.ibm.ws.logging.max.file.size=100
com.ibm.ws.logging.max.files=3
com.ibm.ws.logging.console.log.level=OFF
com.ibm.ws.logging.copy.system.streams=false

Log formats

Each of the three Open Liberty primary log files has a default log format and several alternative formats that you can specify in your logging configuration. The format that you choose for each log depends on the needs of your application and your infrastructure. For example, you might choose to output your messages log in JSON format if it is consumed by a log analysis platform. Alternatively, you might choose the TBASIC format for all three logs if you need a consistent log format across all of your log files.

  • The default format for the console.log file is DEV.

  • The default format for the messages.log file is SIMPLE.

  • The default format for the trace.log file is ENHANCED.

The following examples show sample output in each of the available log formats and list the logs for which each format is available: DEV, SIMPLE, JSON, TBASIC, ENHANCED, and ADVANCED.

DEV (console log)
[AUDIT   ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 7.967 seconds.
SIMPLE (console and messages logs)
[18/10/21 14:49:26:246 EDT] 0000003e com.ibm.ws.kernel.feature.internal.FeatureManager            A CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 7.844 seconds
JSON (console and messages logs)
{"type":"liberty_message","host":"HOST","ibm_userDir":"/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 7.967 seconds.","ibm_threadId":"0000003e","ibm_datetime":"2021-10-18T14:50:58.159-0400","ibm_messageId":"CWWKF0011I","module":"com.ibm.ws.kernel.feature.internal.FeatureManager","loglevel":"AUDIT","ibm_sequence":"1634583058159_0000000000009"}
TBASIC (console, messages, and trace logs)
[18/10/21 14:47:19:718 EDT] 0000003e FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 7.718 seconds.
ENHANCED (trace log)
[26/10/21 10:42:42:300 EDT] 0000006d id=00000000 x.com.ibm.ws.collector.manager.buffer.BufferManagerImpl      3 Adding event to buffer GenericData
[type=com.ibm.ws.logging.source.trace,ibm_datetime=1635259362300,ibm_messageId=null,ibm_threadId=109,module=com.ibm.ws.event.internal.EventImpl,severity=
<,loglevel=EXIT,ibm_methodName=setProperty,ibm_className=null,levelValue=400,threadName=Framework stop - Equinox Container: 2005505a-a1d7-46dc-abdd-
7230d055f81e,correlationId=,org=,product=,component=,message=Exit ,objectId=-964551509]
ADVANCED (trace log)
[26/10/21 10:29:25:011 EDT] 00000073  > UOW= source=com.ibm.ws.event.internal.EventImpl method=setProperty id=5f05cd40 org= prod= component=
          Entry
          service.objectClass
          java.util.concurrent.ExecutorService
          com.ibm.wsspi.threading.WSExecutorService

TBASIC log format

The TBASIC logging format provides a consistent log format across your Open Liberty console, message, and trace log files. The BASIC logging format exists for the trace log. However, the output differs between the trace log and that of the console and messages logs. The TBASIC format provides a consistent format option for the trace, console, and messages logs that matches the BASIC option for the trace log. The TBASIC format acts as an alias for the BASIC option.

You can specify the TBASIC format in the bootstrap.properties file, as shown in the following example:

com.ibm.ws.logging.message.format=tbasic
com.ibm.ws.logging.console.format=tbasic
com.ibm.ws.logging.trace.format=tbasic

You can also specify the TBASIC log format for the messages and console logs as an environment variable in your server.env file, as shown in the following example:

WLP_LOGGING_MESSAGE_FORMAT=TBASIC
WLP_LOGGING_CONSOLE_FORMAT=TBASIC

JSON logging

You can simplify log parsing by producing your logs in JSON format. JSON is a self-describing format that many log analysis tools can consume without requiring format-specific parsing instructions. You can configure Open Liberty logs to produce logs in JSON format either by editing the bootstrap.properties file or by specifying an environment variable. The following two examples show the configuration for each of these options:

  • Configure JSON logging in the bootstrap.properties file:

    com.ibm.ws.logging.message.format=json
    com.ibm.ws.logging.message.source=message,trace,accessLog,ffdc,audit
  • Configure JSON logging with environment variables in the server.env file:

    WLP_LOGGING_MESSAGE_FORMAT=json
    WLP_LOGGING_MESSAGE_SOURCE=message,trace,accessLog,ffdc,audit

Configurable JSON field names

When logs are in JSON format, you can use the jsonFieldMappings attribute to replace default field names with new field names. Replacing the default field names might be necessary if other servers in the same logging configuration use different field names than the Open Liberty default names. For example, an Open Liberty message is referred to by the message field name, but the message in another container might be in a field called log. In this case, two different visualizations of the messages show in the logs on a dashboard. If you modify the Open Liberty output field name so that it matches the other log, you can view them in the same visualization. The following examples show sample configurations for renaming a JSON field.

  • To configure a new field name, you can include the following environmental variable in the server.env file:

    WLP_LOGGING_JSON_FIELD_MAPPINGS=loglevel:level

    In this example, the loglevel field name is replaced by the level field name.

  • To configure a field name for a specific source, you can include the following environmental variable in the server.env file:

    WLP_LOGGING_JSON_FIELD_MAPPINGS=message:message:log

    In this example, the message field name is replaced by the log field name in the message log.

To omit a field from the logs, specify the field name without a replacement, as shown in the following example:

WLP_LOGGING_JSON_FIELD_MAPPINGS=defaultFieldName:

To rename or omit multiple fields, specify a comma-separated list of field name mappings.

For a full list of the default JSON field names, see the JSON log events reference list.

For more information, see Configurable JSON log field names.

Custom message and trace JSON fields

You can add custom fields to your JSON-formatted message and trace output to gather information about a particular issue or incident. For example, if you want to check the requests from a specific user, you can add a custom field to filter application logs by that user’s ID. You can add another field for the session ID so that you can analyze and filter application logs for a specific session.

The Open Liberty LogRecordContext API can add custom fields to your log and trace records. This API adds a field by specifying a value for an extension. To use the LogRecordContext API, first import the com.ibm.websphere.logging.hpel.LogRecordContext class. The following examples show how to add different kinds of custom fields to your JSON logs.

To add a string-valued field to your application logs, you can include the following line in your application:

LogRecordContext.addExtension("userName","bob");

The newly specified field is added to log and trace entries that are created on the same thread that executes the addExtension method. In this example, a custom field that is called userName is added for the bob user ID.

To add fields with boolean, float, int, or long values, the extension name must include the suffixes _bool, _float, _int, or _long, as shown in the following examples:

LogRecordContext.addExtension("extensionName_bool","true");
LogRecordContext.addExtension("extensionName_int","112233");
LogRecordContext.addExtension("extensionName_float","1.2");
LogRecordContext.addExtension("extensionName_long","132");

When you specify these suffixes to add non-string values, the resulting JSON field values are not enclosed in quotes in the logs. Only string-valued JSON field values are enclosed in quotes in the logs.

To remove custom fields from the logs, use the following method:

LogRecordContext.removeExtension(extensionName);

After you remove an extension, JSON output for subsequent logs and trace that are made on the same thread do not include that field.

Selectable HTTP access log JSON fields

When logs are in JSON format and the accessLog source is specified, you can replace the default HTTP access log JSON fields with a different set of fields. You can use the jsonAccessLogFields attribute to specify whether your access logs use the default fields or a set of fields that is specified by the logFormat attribute. You specify the replacement fields in the logFormat attribute of the accessLogging element within the httpEndpoint element.

With this configuration, you can receive information that is otherwise not available in JSON logs, such as the remote user ID, request headers, and more. These logs can be used by log analysis tools, such as the Elastic stack, to monitor your server. For more information about HTTP access log format options, see HTTP access logging.

The following example shows a configuration in the server.xml file to replace the default HTTP access log fields with fields that are specified by the logFormat attribute. HTTP access logging must be enabled to receive JSON access logs.

<httpEndpoint httpPort="9080" httpsPort="9443" id="defaultHttpEndpoint">
    <accessLogging logFormat='%R{W} %u %{my_cookie}C %s'/>
</httpEndpoint>
<logging jsonAccessLogFields="logFormat"/>

Container image logging configuration

In containerized environments, you can disable the messages log and format the console output as JSON by using environment variables, as shown in the following example:

WLP_LOGGING_MESSAGE_FORMAT=json
WLP_LOGGING_MESSAGE_SOURCE=
WLP_LOGGING_CONSOLE_FORMAT=json
WLP_LOGGING_CONSOLE_LOGLEVEL=info
WLP_LOGGING_CONSOLE_SOURCE=message,trace,accessLog,ffdc,audit

You can specify this configuration when you run the docker run command by using the -e flag to set the environment variables:

docker run -e "WLP_LOGGING_CONSOLE_SOURCE=message,trace,accessLog,ffdc"
           -e "WLP_LOGGING_CONSOLE_FORMAT=json"
           -e "WLP_LOGGING_CONSOLE_LOGLEVEL=info"
           -e "WLP_LOGGING_MESSAGE_FORMAT=json"
           -e "WLP_LOGGING_MESSAGE_SOURCE=" open-liberty

If you use Podman to manage your containers, run the following command:

podman run -e "WLP_LOGGING_CONSOLE_SOURCE=message,trace,accessLog,ffdc"
           -e "WLP_LOGGING_CONSOLE_FORMAT=json"
           -e "WLP_LOGGING_CONSOLE_LOGLEVEL=info"
           -e "WLP_LOGGING_MESSAGE_FORMAT=json"
           -e "WLP_LOGGING_MESSAGE_SOURCE=" open-liberty

Binary logging

Liberty has a high-performance binary log format option that reduces the resources that are needed to write trace files. For best performance, when you configure binary logging, set the console log level to OFF and use the messages log instead. You can configure binary logging in the bootstrap.properties file, as shown in the following example:

websphere.log.provider=binaryLogging-1.0
com.ibm.ws.logging.console.log.level=OFF
com.ibm.ws.logging.copy.system.streams=false

However, when you set the console log level to OFF, messages from the underlying JVM other other messages that are written to the native standard out and standard error streams are still written to the console.log file.

You can use the binaryLog command to convert the binary log to a text file, as shown in the following example for a server that is called defaultServer:

binaryLog view defaultServer

Time-based log rollover

You can enable time-based periodic rollover of Liberty messages and trace logs by specifying a log rollover start time and a log rollover interval duration. The specified rollover start time is the time of day when logs first roll over. The rollover interval duration is the time interval in between consecutive log rollovers.

For example, a server with a rollover start time of midnight and a rollover interval of 1 day rolls over the messages and trace logs once every day, at midnight.

You can configure rollover for Open Liberty messages and trace logs by either editing the bootstrap.properties file, specifying environment variables, or by using server configuration attributes. The following examples show the configuration for each of these options, with a log rollover start time of midnight and a log rollover interval of 1 day.

  • The following example configures time-based log rollover in the bootstrap.properties file.

    com.ibm.ws.logging.rollover.start.time=00:00
    com.ibm.ws.logging.rollover.interval=1m
  • The following example configures time-based log rollover with environment variables in the server.env file.

    WLP_LOGGING_ROLLOVER_START_TIME=00:00
    WLP_LOGGING_ROLLOVER_INTERVAL=1m
  • The following example configures time-based log rollover with server configuration attributes in the server.xml file.

    <logging rolloverStartTime="00:00" rolloverInterval="1d"/>

Configuration settings by source

The following table shows the equivalent server.xml file, bootstrap.properties file, and environment variable configurations along with brief descriptions. Settings that apply to the console log appear first, followed by settings that apply to the messages log, and then the trace log. Settings that apply either globally or to more than one log type appear last in the table. For more information, see the logging element.

Logging configuration settings
Server XML Attributebootstrap propertyEnv varDescription

consoleFormat

com.ibm.ws.logging.console.format

WLP_LOGGING_CONSOLE_FORMAT

This setting specifies the required format for the console. Valid values are DEV, SIMPLE, JSON, TBASIC format. By default, consoleFormat is set to DEV. For more information, see Log formats.

consoleLogLevel

com.ibm.ws.logging.console.log.level

WLP_LOGGING_CONSOLE_LOGLEVEL

This setting controls the granularity of messages that go to the console. The valid values are INFO, AUDIT, WARNING, ERROR, and OFF. The default is AUDIT. If using with the Eclipse developer tools this must be set to the default.

consoleSource

com.ibm.ws.logging.console.source

WLP_LOGGING_CONSOLE_SOURCE

This setting specifies a comma-separated list of sources that route to the console. It applies only when the console format is set to json. The valid values are message, trace, accessLog, ffdc, and audit. By default, consoleSource is set to message. To use the audit source, enable the Liberty Audit feature. To use the accessLog source you need to configure HTTP access logging.

copySystemStreams

com.ibm.ws.logging.copy.system.streams

If this setting is set to true, messages that are written to the System.out and System.err streams are copied to process stdout and stderr streams and so appear in the console.log file. If this setting is set to false, those messages are written to configured logs such as the messages.log file or trace.log file, but they are not copied to stdout and stderr and do not appear in console.log. The default value is true.

messageFileName

com.ibm.ws.logging.message.file.name

This setting specifies the name of the message log file. The message log file has a default name of messages.log. This file always exists, and contains INFO and other (AUDIT, WARNING, ERROR, FAILURE) messages in addition to the System.out and System.err streams . This log also contains time stamps and the issuing thread ID. If the log file is rolled over, the names of earlier log files have the format messages_timestamp.log

messageFormat

com.ibm.ws.logging.message.format

WLP_LOGGING_MESSAGE_FORMAT

This setting specifies the required format for the messages.log file. Valid values are SIMPLE, JSON, or TBASIC format. By default, messageFormat is set to SIMPLE. For more information, see Log formats.

messageSource

com.ibm.ws.logging.message.source

WLP_LOGGING_MESSAGE_SOURCE

This setting specifies a list of comma-separated sources that route to the messages.log file. This setting applies only when the message format is set to json`. The valid values are message, trace, accessLog, ffdc, and audit. By default, messageSource is set to message. To use the audit source, enable the Liberty audit-1.0 feature. To use the accessLog source you need to have configured httpAccessLogging.

suppressSensitiveTrace

This attribute, when set to true, prevents potentially sensitive information from being exposed in log and trace files. The server trace can expose sensitive data when it traces untyped data, such as bytes received over a network connection. The default value is false.

traceFileName

com.ibm.ws.logging.trace.file.name

This setting specifies the name of the trace log file. The trace.log file is created only if additional or detailed trace is enabled. stdout is recognized as a special value, and causes trace to be directed to the original standard out stream.

traceFormat

com.ibm.ws.logging.trace.format

This setting controls the format of the trace log. The default format for Liberty is ENHANCED. You can also use BASIC, TBASIC, and ADVANCED formats. For more information, see Log formats.

traceSpecification

com.ibm.ws.logging.trace.specification

This setting is used to selectively enable trace. The log detail level specification is in the following format:

component = level

The component specifies what log sources the level is set for. A component can be a logger name, trace group, or class name. The level specifies what level of trace to output for that component by using one of the following levels:

off, fatal, severe, warning, audit, info, config, detail, fine, finer, finest, all. Setting a trace component to off, fatal, severe, warning, audit, or info level has the same effect as setting it to the info level as info and higher levels cannot be disabled.

You can provide multiple log detail level specifications that are separated by colons.

An asterisk * acts as a wildcard to match multiple components based on a prefix. For example:

- * Specifies all traceable code that is running in the application server, including the product system code and customer code.

- com.ibm.ws.* Specifies all classes with the package name beginning with com.ibm.ws.

- com.ibm.ws.classloading.AppClassLoader Specifies the AppClassLoader class only.

The trace.log file is created and logs are written to it only if one or more trace components are set to config level or lower. For more information on logging levels, see Valid log detail levels for the trace log.

com.ibm.ws.logging.newLogsOnStart

If this setting is set to true when Open Liberty starts, any existing messages.log or trace.log files are rolled over and logging writes to a new messages.log or trace.log file. If this setting is set to false, messages.log or trace.log files only refresh when they hit the size that is specified by the maxFileSize attribute. The default value is true. This setting cannot be provided using the logging element in the server.xml file because it is only processed during server bootstrap.

hideMessage

com.ibm.ws.logging.hideMessage

You can use this setting to configure the messages keys that you want to hide from the console.log and messages.log files. When the messages are hidden, they are redirected to the trace.log file.

isoDateFormat

com.ibm.ws.logging.isoDateFormat

This setting specifies whether to use ISO-8601 formatted dates in log files. The default value is false.

If this setting is set to true, the ISO-8601 format is used in the messages.log file, the trace.log file, and the FFDC logs. The format is yyyy-MM-dd’T’HH:mm:ss.SSSZ.

If you specify a value of false, the date and time are formatted according to the default locale set in the system. If the default locale is not found, the format is dd/MMM/yyyy HH:mm:ss:SSS z.

maxFfdcAge

com.ibm.ws.logging.max.ffdc.age

WLP_LOGGING_MAX_FFDC_AGE

Use this setting to enable FFDC file purging based on age. This setting specifies the maximum file age before an FFDC file is deleted. Purges occur everyday at midnight. Specify a positive integer followed by a unit of time, which can be days (d), hours (h), or minutes (m). For example, specify 2 days as 2d. You can include multiple values in a single entry. For example, 2d6h is equivalent to 2 days and 6 hours.

appsWriteJson

com.ibm.ws.logging.apps.write.json

WLP_LOGGING_APPS_WRITE_JSON

When the message log or console is in JSON format, this setting allows applications to write JSON-formatted messages to those destinations, without modification.

jsonAccessLogFields

com.ibm.ws.json.access.log.fields

WLP_LOGGING_JSON_ACCESS_LOG_FIELDS

When logs are in JSON format, you can use this setting to replace the default HTTP access log JSON fields with fields that are specified by the logFormat attribute of the accesLogging element.

jsonFieldMappings

com.ibm.ws.logging.json.field.mappings

WLP_LOGGING_JSON_FIELD_MAPPINGS

When logs are in JSON format, use this setting to replace default field names with new field names or to omit fields from the logs. For more information, see Configurable JSON field names

logDirectory

com.ibm.ws.logging.log.directory

LOG_DIR

You can use this setting to set a directory for all log files, excluding the console.log file, but including FFDC. The default is WLP_OUTPUT_DIR/serverName/logs. It is not recommended to set the logDirectory in the server.xml file since it can result in some log data being written to the default location prior to when the server.xml file is read.

maxFiles

com.ibm.ws.logging.max.files

This setting specifies how many of each of the logs files are kept. This setting also applies to the number of exception summary logs for FFDC. So if this number is 10, you might have 10 message logs, 10 trace logs, and 10 exception summaries in the ffdc/ directory. By default, the value is 2. The console log does not roll so this setting does not apply to the console.log file.

maxFileSize

com.ibm.ws.logging.max.file.size

This setting specifies the maximum size (in MB) that a log file can reach before it is rolled. Setting the value to 0 disables log rolling. The default value is 20. The console.log does not roll so this setting does not apply.

stackTraceSingleEntry

com.ibm.ws.logging.stackTraceSingleEntry

WLP_LOGGING_STACK_TRACE_SINGLE_ENTRY

When this attribute is set to true, stack traces that are emitted to standard-out and standard-error are merged into a single log event. By default, this value is set to false and each individual line of the stack trace is emitted as an individual log event.

rolloverStartTime

com.ibm.ws.logging.rollover.start.time

WLP_LOGGING_ROLLOVER_START_TIME

Use this setting alone or with the rolloverInterval attribute to enable time-based log rollover for messages.log and trace.log files. This setting specifies the scheduled time of day for logs to first roll over. The rolloverInterval setting duration begins at the time that is specified by the rolloverStartTime attribute. Valid values follow a 24-hour ISO-8601 date-time format of HH:MM, where 00:00 represents midnight. Padding zeros are required. If the rolloverInterval attribute is specified, the default value of the rolloverStartTime attribute is 00:00, midnight. The console.log does not roll so this setting does not apply.

rolloverInterval

com.ibm.ws.logging.rollover.interval

WLP_LOGGING_ROLLOVER_INTERVAL

Use this setting alone or with the rolloverStartTime attribute to enable time-based log rollover for messages.log and trace.log files. This setting specifies the time interval in between log rollovers, in minutes if a unit of time is not specified. Specify a positive integer followed by a unit of time, which can be days (d), hours (h), or minutes (m). For example, specify 5 hours as 5h. You can include multiple values in a single entry. For example, 1d5h is equivalent to 1 day and 5 hours. If the rolloverStartTime attribute is specified, the default value of the rolloverInterval attribute is 1 day. The console.log does not roll so this setting does not apply.

Trace log detail levels

You can specify the traceSpecification attribute for the logging element in your server.xml file to selectively enable trace and set the trace log detail level for specific components in your server configuration. The following example specifies a trace specification for two groups of components.

<logging traceSpecification="com.myco.mypackage1.*=fine:com.myco.mypackage2.*=finest"/>

In this example, the two detail level specifications are separated by a colon (:).

The com.myco.mypackage1.*=fine value applies to all classes with the package name that begins with com.myco.mypackage1.

The com.myco.mypackage2.*=finest value applies to all classes with the package name that begins with com.myco.mypackage2.

The following table lists the valid log detail levels that you can specify in the traceSpecification attribute. You can specify an empty value for any component to disable all trace for that component. Any component that is not specified is initialized to a default state of *=info.

Setting a trace component to off, fatal, severe, warning, audit, or info level has the same effect as setting it to the info level as info and higher levels cannot be disabled. The trace.log file is created and logs are written to it only if one or more trace components are set to config level or lower.

Valid log detail levels for the trace log
Logging levelContent or Significance

off

Logging is turned off.

fatal

Task cannot continue and component, application, and server cannot function.

severe

Task cannot continue but component, application, and server can still function. This level can also indicate an impending unrecoverable error.

warning

Potential error or impending error. This level can also indicate a progressive failure. For example, the potential leaking of resources.

audit

Significant event that affects server state or resources

info

General information that outlines overall task progress

config

Configuration change or status

detail

General information that details subtask progress

fine

General trace

finer

Detailed trace that includes method entry, exit, and return values

finest

A more detailed trace that includes all the detail that is needed to debug problems

all

All events are logged. If you create custom levels, all includes those levels, and can provide a more detailed trace than finest.