New Relic's telemetry begins just like AppDynamics with a JAR file tossed in the lib directory and a
javaagent parameter added to the start command. The big difference is what you don't install: a monitoring server for digesting the data. This is all handled remotely on New Relic's servers. It's software as a service.
This sounded very inviting because I've wrestled with setting up a dashboard and server for Wily in the past. Letting someone else run this infrastructure was as good as it sounds. All of your account information is stuffed in the newrelic.yml file, and the statistics from your machines start appearing on your web pages at newrelic.com.
This simplicity seemed less exciting, though, after I started up the AppDynamics stats daemon without any trouble or confusion. The only extra thought AppDynamics required was to type the command line starting up the daemon in the right sequence with the server, which I had to try two or three times. Everything else was pretty automated, alleviating most of the advantages of leaving the stats collection to someone else. Both of these tools are dramatically easier to start up than what everyone endured even several years ago.
I'm not sure there's much of a security hole with sending information to New Relic, most of it will be dreadfully boring and useless information about which URL responds within how many milliseconds. New Relic does encrypt the data during the trip, but when it gets there, it's still in the control of another company. And then most of this data is recordable by outside visitors to your site, and they already know full well whether your website is working.
But some of the holes can be more subtle, and code can reveal secrets by mistake. For example, a supposedly confidential password might be used in the URL of some internal web service call that would normally stay secure inside the firewall. Some JDBC URLs, for instance, include database passwords. Or the programmer might punt on adding more security in the name of time. Will this information find its way to New Relic's servers? It may be hard to know because the person implementing the monitoring may just pop in the agent JAR and never think about the sensitive data.
Compared to AppDynamics, the style of New Relic's performance data presented by the website seems a bit more traditional. The main page tracks the average response time, the throughput, and the Apdex score, a stat that roughly measures the percentage of users who get their information quickly. The speediness of each web page is also tallied and graphed on another table. The free version shows the last 30 minutes. As with AppDynamics, if you want more information, both more metrics and a longer history, you'll need to upgrade.
More profiling information can be found by starting up a separate session. This records the amount of time spent on each of the major routines and subroutines in your system. In the past, I've usually relied upon the profiler with Eclipse to gather this information, but that's not always a good indication of what part of the system is exercised most with real client data in a working environment.
Some advantages of a central server become more apparent when you dig into deeper features, such as web integration and the API. New Relic's central server will export your incident reports to other websites such as Campfire, Lighthouse, or Pivotal Tracker. The monitoring server can start up a discussion on its own. There's also a connection with Twitter, although why anyone would want to share deployment and incident notifications with the entire world is something I'll never understand. Kids today want to get their server information mixed up with invitations to drink and reminders from Mom.
New Relic's top-down view of the threads in the application shows how much time is devoted to each thread and the methods used inside. The Wall Clock Time is useful for debugging problems with response time for users. An alternative view shows CPU Burn Time, which is useful for diagnosing long computations.
You don't need to use any of these outside features if you want to roll your own dashboard. There's also a generous API that exports most of the basic metrics about your application so that you can integrate this with other tools. Most of the data comes in XML, but there's even a cute little HTML fragment made just for AJAX calls. It's very simple to add website stats to any internal page.
All of these touches will be welcome by some of the more far-flung development teams. If your application is hosted on some distant server and your team works out of a virtual office using Campfire and Twitter to synchronise your workload, then this may be very attractive. If you're a bank that's paranoid about letting anything but heavily scrutinised HTML through your firewall, well it's just another reason to worry.
The free version of New Relic is just as much a brochure as the free version of AppDynamics. New Relic offers more thinly sliced pricing plans (Bronze, Silver, Gold, and Enterprise). Upgrading brings better data and longer storage. The free version is good enough if you want to know how fast your machines have been for the last 30 minutes, but it's still a bit too hard to pinpoint the error without some guessing and constant monitoring. The commercial levels offer more and better tools for identifying the errant transactions.