Rollbar vs Raygun vs Sentry vs Airbrake vs Bugsnag vs OverOps
We all want our application to run as smoothly as possible, but that’s not always the case. Once the application is up and running, we need to know if and when errors or exceptions are thrown. That’s why there are numerous error tracking tools in the market.
What is an error tracking tool? What can you actually do with it? What are the differences between the tools? Answers to these questions, along with some other relevant questions are waiting for you in the following post. Get ready to squash some bugs.
New Post: The Complete Guide to Error Tracking Tools https://t.co/XGNJy9Gt7K pic.twitter.com/XZrSPi01Dc
— OverOps (@overopshq) September 20, 2016
Error tracking tools to the rescue
Deploying fast and frequently might introduce a errors, exceptions and bugs into our application, and it’s best to identify them before your users do.
We’ve brushed the dust off our older error tracking tools post to see how they evolved, upgraded and changed during the past few months. We’ve pointed out some key features, pricing and special additions, so you’ll know which one is right for you.
Mix in some new tools with the older ones, and we got the current error tracking tools you need to know:
1. Rollbar
2. Raygun
3. Sentry
4. OverOps
5. Airbrake
6. BugSnag
7. StackHunter
1. Rollbar
The error tracking and monitor tool offers full-stack error monitoring for “all apps in any language”. It captures stack traces for exceptions and messages for log files, that are sent into Rollbar’s dashboard.
The dashboard presents all new and reactivated errors in your application (or items, as Rollbar calls them), that can be viewed via the occurrence graph. The detailed view includes how many times each error happened, the amount of unique IPs that were affected by it and whether it was critical or not.
While Rollbar can capture any uncaught errors, if you’re interested in capturing caught exceptions as well, you’ll have to manually send them to the dashboard.
Supported Languages and Frameworks
Rollbar has a long list of supported languages, that include JavaScript, Angular, Node.js, Python, Ruby, Django, PHP, Clojure, .NET, Android, iOS, Haskell, Drupal, Rails and many many more.
On the Java side you’ll need to use java-rollbar, a set of Maven modules that allows reporting issues to Rollbar in anything that can use Java. It’s important to mention that Java support is still under development, and Rollbar states that this library was written by “someone who knows C# much better than Java”.
Features
One of the things that caught our eye is Rollbar Zero. Just like the concept of emptying your email, the company encourages you to go over and “clean” every error that occurred. You can mute or hide certain items that you know about, define the severity of each error and get a notification if an error reoccurs. Each error can be sent to a team member to be taken care of, leaving the dashboard clean and error free.
If you want to see an error through the eyes of your system, Replays creates a rerun of GET requests that include the same URL, headers, cookies and other information needed to reproduce it. In addition, Person Tracking lets you see the event history for each user you’re interested in.
Integrations
Datadog
Flowdock
Github
Help Scout
Logentries
Logstash
Mailgun
PagerDuty
Slack
Trello
Webhooks
Pricing
The free version offers up to 5,000 Rollbar events per month, which is enough for a small project. The next pricing steps are $49/month, $149/month, $299/month and $599/month, based on the amount of events you’re interested in.
Bottom line: Rollbar has an impressive offering for numerous languages and frameworks, which is nice, but it requires some code changes to work, which might be a hassle to you.
2. Raygun
Raygun offers 2 services: Real User Monitoring (aka Pulse) and Crash Reporting. Pulse monitors sessions, performance issues, and bottlenecks, each with an option to set up an alert when they occur inside the code. It has a separate dashboard, that also informs you about loading times from specific countries, browsers and devices.
Crash Reporting aims to give instant notifications on errors and issues inside the app, and in this dashboard you’ll be able to see diagnostic reports and relevant needed data.
Supported Languages and Frameworks
Here you’ll find support for .NET, Android, Drupal, Go, iOS, JavaScript, Node.js, PHP, Python, Ruby, Unity and more.
If you’re looking for Java support, you need to add the Raygun4Java library to your application. This library will let you send exceptions right to Raygun’s dashboard. It’s a bit of a hassle, but there’s an extra explanation inside Raygun’s documentation.
Features
In order to keep your dashboard clean, Raygun presents multiple occurrence of a bug or an error in a single group, for faster handling or ignoring. The dashboard offers Trend Graphs that show how errors in your application change over time, and you can mark errors or instances with tags to know where they came from.
Every error group lets each team member see the status updates, comments and other information, and the dashboard counts the number of users affected by each error.
In addition, each error can be marked with custom tags, that in turn will send some more information whenever an exception occurs. The full text search option lets you search through the collected data, for easy recovery later on. You can also use webhooks and send notifications to third-party APIs, or your very own system.
Integrations
Asana
Assembla
BitBucket
Campfire
HipChat
Intercom
JIRA
PagerDuty
Slack
Trello
Webhooks
Zendesk
Pricing
Here’s where it gets tricky: Raygun treats Pulse and Crash Reporting as 2 different services, each with its own pricing plan. And no, there’s no bundle offer.
Pulse pricing starts at $99/month for 100,000 sessions per month. On the bright side and unlike page loads, a user session is the time between when a user visits your application to when they leave. The next step is $198/month for 200,000 sessions and then a big jump to $495/month, for 500,000 session.
Crash Reporting starts at $49/month for 25,000 error events per month, and up to 5 applications. $149/month will give you 250,000 error events in up to 20 applications, and for $499/month you can collect 3,000,000 error events in up to 50 applications.
There’s also an enterprise step for both services that starts at $999/month for unlimited sessions/events.
Bottom line: The dashboard looks clean and informative, but two different services with 2 pricing plans are already a hassle, especially if it basically summarizes existing log data.
3. Sentry
What started as a tiny bit of open source code grew to became a full-blown error monitoring tool, that identifies and debugs errors in production. You can use the insights to know whether you should spend time fixing a certain bug, or if you should roll back to a previous version.
The dashboard lets you see stack traces, with support for source maps, along with detecting each error’s URL, parameters and session information. Each trace can be filtered with app, framework or raw error views.
Supported Languages and Frameworks
JavaScript, Node.js, Python, Go, PHP, Ruby, C#, Objective-C and even Swift.
Raven is the Java client for Sentry, and it relies on popular logging libraries in order to capture and convert log files. The library takes that information and sends the relevant details into a Sentry instance. The company highly recommends using one of the supported logging frameworks to capture and send messages to Sentry.
Features
You can catch errors in real-time as you deploy, and every error includes information about software, environment and the users themselves. If you believe your users are a key point at understanding errors, Sentry lets you prompt users for feedback whenever they encounter errors. That allows comparing their individual experience to the data you already have.
Integrations
HipChat
GitHub
Slack
Trello
Heroku
Campfire
JIRA
PagerDuty
GitLab
Pricing
Hobbyists can use Sentry for free, with up to 5,000 events/day. The next pricing step range between $29–$249/month for up to 288,000 events/day, and the last step offers unlimited events for $999/month.
When you sign up you get free access to the medium tier for 14 days, so you can run wild and see how many events you really need before you start paying.
Bottom line: Sentry has an open approach in which we can ask our users questions about the application, while the rest of the information is based on what you can get directly from your log files.
4. OverOps
OverOps was made to give developers a simple answer to a complex question: When, where and why production code breaks. It captures exceptions and errors, and it’s the ONLY solution showing the variable state that caused it, across methods and machines.
With OverOps you can gain insights to the actual code that ran at the moment the exception or error were thrown. So you have everything you need to find, analyze and fix the errors as soon as they happen.
Supported Languages and Frameworks
OverOps offers support for all JVM languages, that includes Java, Scala, Clojure and Groovy, along with support for web containers and frameworks.
OverOps was built for Java production environments, and offers a simple installation process. Since it runs as a Java agent, there’s no need for code or build changes, just attach a Java agent to your JVM.
Features
If you still prefer going through your log files each time an error occurs, OverOps adds a link to each error that takes you directly to a web based dashboard, showing the complete source code for every frame in its stack trace with the variable values that caused it. You can prioritize by how often errors happen across your cluster, if they involve new or modified code, and whether that rate is increasing.
On the dashboard itself, you can customize alerts based on frequency of errors, filter errors by application, code location, time, transaction and tier, and see error frequency and impact over time.
Integrations
JIRA
Slack
HipChat
PagerDuty
Graphite
Grafana
Nagios
Zabbix
New Relic
AppDynamics
DataDog
Pricing
You get a free 14 days Pro trial, followed by the “Free Forever” account that supports 1 user and limited diagnostics. The Pro account will cost you $99/month per JVM for a monthly plan, or $79/month per JVM for a yearly plan. This offer includes up to 50 users and complete root cause analysis.
Bottom line: OverOps is the only tool that gives you the complete source code running within the JVM at the moment of error, with variable and object state across the entire call stack. On the down side, it currently only offers support for JVM languages. Try it for free.
5. Airbrake
Rackspace’s Airbrake offers a web based dashboard to capture and track the exceptions and errors in your application. The dashboard gives a pretty straightforward view of resolved and unresolved errors, the information about each one of them and a nice overview graph. Each error includes the environment in which it happened, when it was first identified, error type and more.
Supported Languages and Frameworks
Ruby, PHP, JavaScript, .NET, Python, Django, Node.js, iOS, Swift, Android, Go, Angular, Flask and some other names.
You can use Airbrake with Java through Maven and log4j, or by adding the Airbrake jar to your classpath. If you want to send uncaught exceptions to Airbrake, you need to use setUncaughtExceptionHandler as presented in the example below:
Features
Airbrake lets you see the stack trace and metadata for each error, and they are grouped by type, users and environment variables. That way, it’s easier to identify where exactly each error was created within the code.
There’s also a view for when each error was fixed and deployed, so you’ll know if they are new to the current cycle, or if they’ve re-occurred after they’ve been fixed. There’s also an option to see which team member caused which error, or which one is responsible for fixing them.
Integrations
Assembla
Bitbucket
Campfire
Flowdock
GitHub
GitLab
HipChat
JIRA
Lighthouse
Pivotal Tracker
Slack
Trello
Webhooks
Zapier
Pricing
The good news are that every user gets a free 30-days trial at Startup level. The bad news are that you have to enter your credit card in order to start using Airbrake.
The lowest pricing plan is $49/month, and that include 25 errors/minute in up to 5 applications. For 75 errors/minute in up to 15 applications you’ll be charged with $99/month, and the last step costs $249/month for 200 errors/minute for an unlimited number of applications.
Bottom line: We don’t really like to enter our credit card before we get to try out a product, and with all these alternatives it might be a reason to pick other tools over Airbrake.
6. Bugsnag
Bugsnag lets you monitor and capture the various errors and exceptions inside your application, and try to distinguish itself by focusing on the tools you need in order to solve them. Or in other words, the company acknowledge that once you’ve identified an error, you’ll need tools and workflows in order to fix them.
The dashboard gives an overview of all errors, with an option to filter out the unresolved ones. It groups errors that has the similar origin in the code, and helps visualize and compare the different error trends over time with graphs.
Supported Languages and Frameworks
Android, Go, iOS, .NET, Node.js, PHP, Python, Ruby, Unity.
You can add Bugsnag to your Java application using Gradle, Maven or through manual jar installation. There’s also an option to add Java using Spring and Spring boot projects, and you can view the full process here.
Features
The dashboards has a few features that include an option to snooze errors until they meet custom conditions, or a mute option in case they’re not important/critical for you. Each error comes with data about the number of users affected by it, how many times it happened and their stage.
The error status menu gives some extra information by showing the progress of the errors, and the activity feed specifies the full history of each error over time.
Integrations
DataDog
HipChat
Slack
Twilio
Assembla
Bitbucket
JIRA
Trello
PagerDuty
Webhooks
Pricing
The hobbyist plan is offered for free and supports up to 250 error reports per day, for one user. The Team pricing plan starts at $29/month and can go up to $499/month for up to 100,000 error reports per day for 30 team members. And similar to other services, the unlimited enterprise plan starts at $999/month.
Bottom line: Bugsnag can save us time, since it goes over the log files for us and create a dashboard of the errors and exceptions, while offering a better workflow while doing so.
7. StackHunter
StackHunter alerts via email whenever failures happen inside your application, giving the relevant details for analyzing and understanding each error. The information includes breakdowns by day, hour, user, location and type, along with the number of times this error occured.
Each error includes the stack trace, HTTP request, JVM properties, JMX attributes and of course, when and where it first happened. StackHunter watches for uncaught exceptions, along with errors logged through Log4J, Logback, and their own API.
Supported Languages and Frameworks
On the Java side, it’s a tool made specifically for capturing Java exceptions. It supports JDK 6 or higher, and requires Servlet 2.5 or higher container. Installation is pretty easy, and once you unpack the install package all you need to do is deploy the war file from server/stackhunter.war to your webserver.
Features
StackHunter tracks exceptions by automatically hooking into Log4J and Logback, which allows capturing exceptions you’re already logging. And the tool offers a single, self hosted, web based interface in which you can view all exceptions.
If you want to get into specific information, you can see all of your application’s exceptions, filter out unique exceptions, or view which ones affected the users or sessions. Also, similar exceptions are grouped together, so you’ll know what you should focus on.
Integrations
Log4J
Logback
Pricing
After the free 30 days trial, pricing starts at a one time fee of $499 for 1 monitored app and 1 monitored JVM, with 1 year of free upgrades. For 10 monitored apps and 10 JVMs the price jumps up to $2,495, and for $5,970 you’ll get 25 monitored apps and 25 monitored JVMs, both with 1 year of free upgrades.
Bottom line: StackHunter is a small beta tool made specifically for Java and basic as it is, it gets the job done.
Final thoughts
Deployments come and go, but errors and exceptions are here to stay. At least until we can hunt them down and fix them, that is. Error tracking tools are a key point in the application life cycle, and each one can introduce a whole new way of using our app, that we didn’t think of before.
There are a lot of options out there, and while they all focus on helping us find errors, each has its own pros and cons that might be critical for our app. If possible, we’d recommend taking advantage of the free trials offered and testing out these tools in your staging environment. Change isn’t easy, but it might help you make a better application for your users.
Using other tools we didn’t write about? Have anything else to add about one of the tools we wrote about? We would love to hear you out in the comments below.