AppInternals 10.10.0 is GA

Overview of Changes

For full information, please check the release notes

Part 1Part 2

Version 10.10.0

AppInternals Version 10.10.0 updates both the analysis server and agent software. It is available both as a SaaS and on-premises release. Version 10.10.0 of the agent requires 10.10.0 or later of the analysis server.

SNMP Alert Notifications

Alerts for threshold violations can now be sent as SNMP traps as well as email messages. The Alert Configuration screen has a new option to enable SNMP alerts:

Use the new SNMP Settings configuration screen to specify an SNMP management station that will receive the traps. For example:

The traps contain details of the threshold violation:

Ability to Define Transactional Thresholds Based on Exception Counts

Transactional thresholds can now be based on exception counts. The Add Transaction Threshold dialog shows an additional metric choice, Exceptions:

Like any threshold, such thresholds can be specified in an alert definition to generate notifications when they are violated.

Application Map Changes

Application Maps have the following enhancements:




Edit options to display additional metric data. The new options are to display response time, transaction count, CPU usage, and total response time:




New options to color server elements with color gradients that show relative values of either the total response time or threshold violation percentage for each server. For example, Color by Total Response Time shows total response time (calculated as the response time times the transaction count) compared to other servers. The darker the color, the longer the total response time relative to other servers:




A new “Edit Mode” is available only after you filter on an application definition in the Applications tab. When you navigate to the Application Map tab after filtering by an application definition, a notification panel and pencil icon indicates edit-mode options are available:

In edit mode, additional right click options manually add and remove server nodes and relationships:




A new “collapse” operation combines duplicate or redundant map elements. When you collapse one element into another, the first element no longer appears in the map:

Application Definitions Bulk Edit With Export and Import

New Export and Import options in the Manage Applications configuration screen make it easier to create or modify many application definitions.

This feature is parallel with the Transaction Types Bulk Edit With Export and Import introduced in Version 10.8.0.

JMX Monitoring of Uninstrumented Processes

The JMX sub-agent can now report data on any Java process. Previously, it reported metrics only for Java processes that were instrumented by with the JIDA Sub-Agent.

The Agent Details page has a new option to enable this JMX monitoring. This option is at the bottom of the page and is ON by default:

Uninstrumented Java processes report data on the Instances tab. The JMX icon distinguishes them from instrumented processes:

In addition, the JMX sub-agent reports additional metrics, for CPUs and threads:

You must configure non-instrumented Java processes so that the JMX sub-agent can connect to them. Add the following system properties into the Java command line: \  (IBM JVM only)
-Djava.rmi.server.hostname=<Host IP> \<Exposed port> \<Exposed Port> \ \

Geo Location from Transaction Trace Data

In earlier releases, to determine the geographic location of users, the analysis server required that the Collect End-User Experience Data option be enabled. This option collects web-page performance data by injecting a JavaScript snippet in monitored web pages. This data includes the user’s IP address, which the analysis server mapped to a corresponding location.

Unless the Collect End-User Experience Data option was enabled, the analysis server could not identify users’ locations. Now, if end-user experience data is not available, the analysis server attempts to find the user IP addresses in transaction trace data.

This means that the Search screen and other places in the analysis server interface will display Country values in many more cases.

Enhancements to calls Analysis Operator

The calls analysis operator displays a table with performance data for method calls. New for this release, calls accepts the -group_by argument to add a column for the specified field. For example, group_by transactiontype:

Supply any valid search field as the value to the -group_by argument. Use the new group_by_field value to the -sort_by argument to actually group results by the group-by field.

SaaS Analysis Server Works as Data Source for SteelCentral Portal

Version 10.10 of the SaaS analysis server can be specified as a data source in SteelCentral Portal. (Previous SaaS releases did not work as SteelCentral Portal data sources.) This integration is supported by SteelCentral Portal Version 1.5.2 and later

Note that, when adding the data source in SteelCentral Portal, you must first determine if the AppInternals user has access to more than one company (also called account and realm in AppInternals). In AppInternals, users with access to multiple companies see them when they log in and in the Realm drop-down list:

If the AppInternals user has access to more than one company, you must specify the company ID along with the user name in the SteelCentral Portal Add Data Source dialog box. In AppInternals, find the company ID by clicking the ACCOUNT link in the top menu bar to open the Account Details page in a separate browser window. The company ID is underneath the company name:

In the Username field of SteelCentral Portal Add Data Source dialog box, supply both the company ID and AppInternals user name separated by a backwards slash:


Users with access to a single company only can omit the company ID.

Windows Agent Support for Process Data (SaaS Analysis Server Only)

Version 10.10 agents installed on Windows systems will now automatically report details of process activity to the SaaS analysis server. Previously, only agents on Linux systems reported this data. (This feature is not available for on-premises analysis servers.)

Process reporting is enabled by default. The Agent Details page has an option to disable it.

Windows Agent Support for Network Data (BETA, SaaS Analysis Server Only)

Version 10.10 agents installed on Windows systems can now be configured to report details of network activity to the SaaS analysis server. Previously, only agents on Linux systems reported this data. (This feature is not available for on-premises analysis servers.)

Note: Beta Feature
Windows support for network data is a beta feature in Version 10.10. It is subject to change and changes may not be backwards-compatible with this release. Questions and feedback on the feature should not go through technical support. Please provide feedback, ask questions, and share your experience in the AppInternals area of Riverbed’s Splash! community:

This feature requires that the WinPcap library be installed on the Windows system. The library is not included as part of the agent installation. You can download WinPcap at After installing WinPcap, restart the AppInternals agent.

The Network Data section of the Agent Details page has settings to control whether network data is reported and configure which traffic will be monitored. These are the same controls as for Linux systems.

“Monitor of Monitors”: CLI Support for Installing Agent on Analysis Server

The new monitor-system CLI command makes it easy to install the AppInternals agent on an analysis server. This is useful to monitor performance of that analysis server. Specifically, the agent monitors the webui service that controls the Tomcat server that hosts the analysis server web user interface.

Typically, the agent reports data to another analysis server dedicated to monitoring similarly-deployed agents on many analysis servers. The dedicated analysis server is the “monitor of monitors”.

The agent is not installed by default. If the agent software has not been already installed on the analysis server, monitor-system displays an option to install it:

appinternals # monitor-system
/bin/mkdir: cannot create directory `/mnt/data/agent/tmp': File exists


    Riverbed SteelCentral AppInternals Agent Setup Menu:


    1) Install agent
    2) Return to CLI


    Select one of the options above and press enter [1]:

Install the agent as on any Linux system, specifying the “monitor of monitors” as its analysis server.

Running the monitor-server command after installing the agent displays options to manage the agent, including removing it:

appinternals # monitor-system
    Riverbed SteelCentral AppInternals Agent Command Menu:


    1) Show agent status
    2) Start agent
    3) Stop agent
    4) Restart agent
    5) Uninstall agent
    6) Return to CLI


    Select one of the options above and press enter [1]:

Monitoring of Docker Containers by Agent installed on Docker Host

This release allows a single agent installation on a Docker host to monitor any number of containers running on the host. This is in contrast with previous releases that required that the agent be installed on each container. The new approach offers the following advantages:




Because there is no DSA process, containers start faster




Upgrading the agent on the Docker host takes effect in all containers




Fewer connections to the analysis server




Reduced exposure of the Docker host to the container means fewer security concerns

Note that the previous approach of installing the agent on containers no longer works in Version 10.10. It will be supported again in future release.

The agent installed on the Docker host appears in the AppInternals interface agent like any other agent. For instance, it appears as an entry in the Agent List screen, can be configured in the Agent Details screen, and appears in the Servers tab like other agents.

By comparison, instrumented containers are not full-fledged agents. They do not appear in the Agent List screen and do not have their own Agent Details screen for configuration. The AppInternals interface uses the Docker icon () to indicate containers and instrumented processes running in them.

For example, the Servers tab shows the container ID and image name of Docker containers in the Server column of its table, denoted by the Docker icon. The Server Tagscolumn shows special “container tags” created automatically by the agent that give additional information about the container.

New Controller Service/Process for Agent

The Controller is a new agent component that starts, stops, and restarts agent-related processes on both Windows and Unix-like operating systems.

On Windows, there is now a single Controller service that replaces the four services in previous releases:

The new agent.bat file in the <installdir>\Panorama\hedzup\mn\bin\ directory allows command-line management:

C:\Panorama\hedzup\mn\bin>agent.bat ?
Valid commands are help, status, start, start <process name>, restart, restart <process name>, stop, stop <process name>


C:\Panorama\hedzup\mn\bin>agent status
[started] PID: 3820 Name: dsa Command: [.\DsaServer.exe, -d]
[started] PID: 3520 Name: agentrt Command: [.\agentrt_agent.exe, -d]
[started] PID: 9744 Name: dotnetagent Command: [.\dotnetagent2.exe, -d]
[started] PID: 12296 Name: npm Command: [.\npm_agent.exe]
[started] PID: 9928 Name: osda Command: [.\os_agent.exe, -d]

On Unix-like operating systems, the agent script in the <installdir>/Panorama/hedzup/mn/bin directory replaces dsactl but has similar commands. dsactl remains, but is a link to agent:

[root@11A bin]# ls -al agent dsactl
-rwxr-xr-x 1 root root 7871 Jun 13 12:39 agent
lrwxrwxrwx 1 root root  33 Jun 15 15:24 dsactl -> /opt/Panorama/hedzup/mn/bin/agent
[root@11A bin]# ./agent ?
Valid commands are help, status, start, start <process name>, restart, restart <process name>, stop, stop <process name>
[root@11A bin]# ./agent status
[started] PID: 4717 Name: dsa Command: [./dsa, -d]
[SaaS Only] Name: npm Command: [npm_agent]
[started] PID: 4854 Name: agentrt Command: [./agentrt]
[started] PID: 4861 Name: osda Command: [./os_agent]

Agent Support for Proxy Server Authentication

The Agent Support for Proxy Servers introduced in Version 10.8.0 allowed AppInternals agents to connect to the analysis server through a proxy server. However, agents did not support proxy servers that required authentication.

In this release, agents can connect to proxy servers that use Basic and Digest authentication (NTLM authentication is not supported). The agent installation on Windows and Unix-like operating systems prompts for details. For example, on Windows:

After installation, use the script in <installdir>/Panorama/hedzup/mn/support directory (<installdir>\Panorama\hedzup\mn\support\proxy_cred_util.bat on Windows systems) to change or test proxy-server connections. Run the script without any arguments for usage details:

[root@11A support]# ./
Error detected parsing command arguments [userid and password are required if -x option not specified]


usage: proxy_cred_util -u userid:password [-r realm] [-p proxyHost:proxyPort -d url] -x


      -u userid:password    - user ID and password for proxy authentication. Mutually exclusive
                              with -x option.
      -r realm              - (optional) the authentication realm for the proxy.  If omitted
                              the authentication will take place for any realm
      -p proxyHost:proxyPort - (optional) if verifiying proxy settings, the proxy address
                              and port.  If proxy port is omitted it defaults to 8080.
      -d url                - (optional) if verifiying proxy settings, the URL
                              we will use to make a test connection.
      -x                    - No authentication is required. Mutually excusive with -u option.

RPID: Automatic Process Instrumentation on Windows

The Riverbed Process Injection Driver (RPID) enables instrumentation for Java and .NET processes on Windows systems. After several releases of beta testing, it is enabled and started on Version 10.10 agent installations and upgrades.

RPID automatically injects into all .NET and Java processes without the use of the COR_* environment variables, JAVA_TOOL_OPTIONS, command line, script modifications, environment variables, or any other user intervention. There is no need to set environment variables, change command lines, or use tools such as JidaRegister.exe to set JAVA_TOOL_OPTIONS.

Because RPID is used by Riverbed Aternity as well as AppInternals, the agent installation creates RPID and related files in the %ProgramFiles(x86)%\Riverbed\ProcessInjection\bin\ directory.

Control process injection on Windows systems with the rpictrl.exe utility. For example:

C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe status


Riverbed Process Injection Control
Driver Version 10.10.586.0
Copyright 2014-2017. Riverbed Technology. All rights reserved.


Start type:                      system start
Status:                          running

See Automatic Process Instrumentation on Windows in the agent configuration documentation for details on RPID and rpictrl.exe.