Exceptional Blog

One of the most glaring omissions from Exceptional to date was the ability to search through exceptions. We've been aware of this from the day we started developing the app, but we decided it wasn't quite important enough to make the first release: we wanted to get the fundamentals right.

We've been working hard over the past week to get search up and running. And now it is.

As a bonus, you can now "Close all..." from within a search. So if you want to close a lot of exceptions matching a search, just hit the "Close all..." button after you search.

Recently we launched an update to our plugin that lets you install it as a gem. This means you can use Exceptional outside of the typical Rails setups that the plugin is targeting.

Installation

gem sources -a http://gems.github.com
sudo gem install contrast-exceptional

Usage

In your script/daemon/whatever you would write something like the following;

require 'rubygems'
require 'exceptional'

Exceptional.api_key = "YOUR-API-KEY-GOES-HERE"

Exceptional.rescue do
  nil.boom!
end

The Exception raised within the Exceptional.rescue block will be caught, posted to Exceptional and then raised.

For further documentation I'd love to say check out the RDoc but that's still in the works, we're working through some issues with the main Exceptional class documentation not showing up (because the current implementation is using class << self to create a Singleton class).

If you want to go ahead and dive in, please go ahead and check out the exceptional.rb source code itself.

• Exceptional docs

We were recently testing out a new release of our plugin with the magic of forked projects on github and can finally say it's ready in our official repository for consuming in your apps. We added a couple of features;

• Support for Thin
• Support for Phusion Passenger (mod_rails). As Passenger doesn't really like you spawning background threads, every time an Exception occurs the data is posted off to our app directly as opposed to the asynchronous background queue used for all other environments.
• Exceptional gem, if you want to use Exceptional outside of rails. We just pushed our gem to github, usage instructions are here.

Upgrading

You can install the plugin with:

script/plugin install git://github.com/contrast/exceptional.git --force

You will need to update your configuration file (located at config/exceptional.yml). You can either manually copy the example located in vendor/plugins/exceptional/exceptional.yml to config/ and enter your API Key, or download a prepared version from within your App's dashboard on Exceptional

Taking DHH's advice ("Stick a price tag on it") to heart and very much Getting Real, we're launching Exceptional paid plans today. We have four price plans and one free plan.

Price plans

The price plans are dead simple. You can check them out right now. In summary, we have a free plan that allows one app and one user to use the basics of Exceptional, all the way up to a maxi plan that supports 100 apps and 50 users for $150.

Thanks

We can only say the heartiest of thanks to everyone who partook in our beta. Your feedback has shaped the app and is continuing to shape it. Now that we're over the hump, we'll be cracking into our list of most-requested features that will add a lot more value. We're super excited to continue to shape the app into a powerful exception-tracking tool.

Migration

As a thank you, if you signed up for Exceptional before today, we're extending your free trial all the way to December 1st. If you want to continue to use the app beyond then, you're more than welcome to revert to the free plan, or simply sign up for a subscription before the end of November. Just click on "Account Settings" in the app.

The future

We're so excited to take Exceptional to new levels in the coming weeks and months. We're proud of what we've already managed to achieve, but we have plans to beef up the app with new features and more powerful tools. We'd love for you to be a part of it.

We've had lots of requests for Passenger support. Well we've been listening.

We're currently testing a new release of our plugin that comes with Passenger support built right in

You can install the plugin with:

script/plugin install git://github.com/davidjrice/exceptional.git --force

You may also need to install the json (that's right, json!) gem

sudo gem install json

Big thanks to Scott Chacon for his mod_rails patch that tided us through until now.

We now have a company profile on CrunchBase. Pretty cool!

Today we’re deploying a small list of updates based on the great feedback we’ve been getting. We’ve lots of bigger ones on the way, but we’ll be pushing little changes like this so you’re not kept waiting.

Dashboard details

The goal of the dashboard is to allow you to have a quick (10-second) glance at the state of all your apps. It still has some way to go, and we have some fun improvements planned. But for now, we’ve added some handy stats on the number of open, closed and ignored exceptions in each app with a link to each state within the exceptions index.

Close all exceptions

In the future, users with a large number of different exceptions will need more powerful browsing tools. We’ve got some half-baked features that will be polished and tested over the next month to help these guys out. But for now, one much-requested feature is a way to zap all open exceptions and see which ones crop up again (closed exceptions that reoccur in your app will reopen and you’ll get notified about it).

Notifications digest

I’m especially excited about this one. One of Exceptional’s key benefits is that you no longer get spammed with exception notifications. But, as we’ve learned with some help from our larger users, if you’ve got a lot of exceptions going down in your app, threaded and throttled notifications (Exceptional only mails you once on the 1st day an exception occurs, then on the 2nd, 3rd, 5th, 8th, 13th, 21st, 34th, 55th, 89th, 144th, 233rd, 377th, 610th, 987th, 1597th, 2584th, 4181st, 6765th… (recognise that sequence?)) just aren’t enough. If you have 500 new exceptions in a day, you get 500 mails. So…

We now only notify you by mail / SMS / Twitter / Campfire a maximum of six times each day. The first five are just regular as-it-happens notifications. This covers 80% (at least) of our users. The next is a digest!

We’ll be listening carefully to your feedback about whether or not this is right for you. We’re still learning and the digest over time will become a lot more useful.

Performance tweaks and other stuff

We’ve made a long list of fixes and tweaks which I won’t bore you with, including performance enhancements at the app level right across the board. Hope you like them! Please keep the great feedback coming and stick with us as we roll-out some far more significant changes and features over the next few weeks and months.

Better update your Fluid icons to match our shiny new site.

While we've been in beta we've had a few scaling issues atypical of an app that is so young. Given the nature of how Exceptional works, if several large apps are busy all at once and they're all throwing exceptions, that's a lot of data through our API to process.

We've addressed these issues on several fronts, and while we're still not done, we're quite far along at providing a solid service that you can rely on.

First up, our hardware. We're using Amazon's EC2 service. It just works. It's fast, powerful and cost effective. We're able to add instances as and when we need them to cope with demand.

Next, our API. This is something that we've put a lot of work into. We accept and write requests to disk as soon as they come through so as not to lock out any web-facing processes. Next, all requests are queued up and processed in the background. We'll be able to scale this out as we need to. Sometimes exceptions occur in the data processing queue that are caused by malformed XML or bugs on our side. The queue rescues these, flagging any requests that cause exceptions so that we can follow them up, then moving on to the next request in the queue. We're currently servicing hundreds of thousands of API requests per day.

We're very concerned about keeping our web interface snappy. We've optimised it for speed on an app level and we'll be working in the coming weeks to add speed on the http level. There are a slew of enhancements that we'll be making to improve app speed, but I'm already quite proud of the speed of the app as it is without any http level caching. See Jeremy Kemper's RailsConf keynote for more info on this.

Our database lives on its own EC2 instance. Like most web-apps, our database sees most of the pain of our busy API. We've worked hard to keep it really efficient. All 404 data is checksummed and we use incrementors wherever we can. Every exception coming through is also checksummed. After an initial 100 or so occurrences of an exception, we start incrementing, but taking samples of new exceptions rather than full data for every one. This way, we can give you a good overview of the active exceptions in your app while keeping our database size manageable.

We're using Amazon's Elastic Block Store for data persistence and nightly backups. EBS is an awesome service that allows us to take backups quickly and efficiently and restore data quickly in the event of failure.

There are a great many improvements that we plan to make over the coming weeks and months. We've built the architecture to scale and to handle large apps as well as smaller ones. We're keeping a good eye on CouchDB, a new(ish) database that is built from the ground up to be highly reliable and distributed. We already have a working implementation of the app built around storing data in CouchDB which we'll be testing and improving in the coming months.

Along with new features and improvements to the app in general, we're commited to making sure that it lives on solid, stable architecture so that ours is a service that we're proud of, and that you can trust.

Update: We were able to recover most of the data from this outage. If you feel your app is missing anything important, please contact us and we'll get it sorted. Thanks for your patience.

We'll write a full review very soon on all the changes and improvements we've pushed today, as well as a report on what went wrong during the migration of the exception data. But here's a quick summary:

• We pushed a major infrastructure release today along with quite a few interface tweaks and improvements and changes to our account system.
• We ran into a few issues whilst migrating the current data. Database issues meant that we had to restore the app from a backup and performed the upgrade on that data.
• The backup data is 24 hours old, so exceptions caught during that period were lost. We're very disappointed in ourselves for letting this happen and realise that this will inconvenience a lot of our users.
• We still have the log files during the period for which we lost exception data, so we're going to restore that data by parsing those over the weekend.
• All systems are back up and collecting data as of two hours ago.
• We profusely apologise again for the inconvenience; this is the first time we've had such serious data problems and we promise to take firm measures to make sure this never happens again.

Thanks for your patience and for sticking with us.

We're firm believers in building software for ourselves —we know what we like: simple, clean software that focuses on the core problem, with an interface that partners aesthetic beauty with just getting out of the way to help you get your job done. What delights us should delight our customers.

That's why I want to highlight how I used Exceptional on itself today as part of my own workflow.

While chatting in Campfire, a notification from Exceptional popped up:

Here's a link to the actual exception.

I was able to diagnose this exception very quickly:

Someone had linked to a page on Exceptional that didn't exist. But rather than throwing a 404 based exception such as an ActionController::RoutingError or ActiveRecord::RecordNotFound that we would catch separately, the app is trying to call a method on nil. Never good. So I need to build a handler. This isn't very urgent. After my exclamation of shock in Campfire though, I wanted to reassure the team that it's not too big a deal. I can quickly let everyone know using comments:

Meanwhile, back in Campfire, everyone who's chatting is kept up to date on the conversation about the exception.

Finally, as I want to deal with this issue, but not straight away, I used the "Send to Lighthouse" feature to post a message to our bug-tracker, and made the exception public so that I could link it up in this post.

Using Exceptional like this helps give us a really good understanding of the way the software should work. I'm constantly questioning the ways that we solve our problems, and in my daily use both wondering how to improve the app, and already getting antsy looking forward to using the features that are already committed to our development version.

Eating our own dog-food. It helps us, it helps our customers and best of all —it's loads of fun.

Make sure to swap in your API key

curl -H "Content-Type: text/xml" -vvvvvvvvvvX \
POST "http://getexceptional.com/errors/?api_key=YOUR_API_KEY&protocol_version=2" \
-d "<?xml version=\"1.0\" encoding=\"UTF-8\"?><error><agent_id>000</agent_id>
<controller_name>Foo</controller_name>
<action_name>bar</action_name>
<error_class>DodgyException</error_class>
<message>this is awesome</message>
<backtrace>the craziness</backtrace>
<occurred_at>Thu Sep 11 16:05:53 -0400 2008</occurred_at>
<rails_root>/var/www/woo/woo/woo</rails_root>
<url>http://www.getexceptional.com</url>
<environment>
  <server_name>lovely</server_name>
  <foo>nice</foo>
</environment>
<session>
  <foo>bar</foo>
</session>
<parameters>POST DATA</parameters>
</error>"

Pretty sweet. Thanks to Jan Lehnardt

Oops! We're SO sorry, we meant to send out invites to everyone who hadn't received an invite yet, but ended up sending an email to our entire beta signup list.

This is spam in our book, and I'm SO sorry for that, I pushed button.

Apology.send

Update: For the record, this should have been: BetaSignup.all(:conditions => {:used => false}).each { |b| b.invite }. The infinite benefit of hindsight

Hard to believe that it's been a week since we strolled into the Maritim hotel near Friedrichstrasse to bring Exceptional to RailsConf Europe 2008. The conference was really great, we met loads of fantastic people.

Everyone was really interested and supportive of Exceptional. Thanks so much to Geoffrey Grosenbach for giving us much needed encouragement, advice and awesome style. HUGE thanks to Jan Lehnardt for showing us around, organising the CouchDB meetup and even giving me a master class in CouchDB. CouchDB a super interesting database system, and I've already got a working prototype of Exceptional using CouchDB as its data store.

I also just want to mention how much of a gentleman David Heinemeier Hansson is. Let's face it, DHH made this conference exist by inventing Rails, apart from the overall contribution he's made to help bring Ruby into the mainstream. An interesting faux-pas of his was mentioning how he sat around a table at a modestly attended RubConf '01. David A. Black was quick to mention that it wasn't 2001, it was 2004. Which goes to show just how quick Ruby and Rails have become huge.

David is approachable, modest and just friendly. He was really supportive of our product and he was generous with time and conversation. One of David's main messages this year was: 'more people should contribute to open source.' He's definitely the kind of guy would give you the motivation to put stuff out there. David, thanks so much for your time, for your positive attitude and for Rails.

It was also great to meet way too many people than I can remember, but off the top of my head: Lena from Berlin, Pat (from Australia, of Riddle and Thinking Sphinx fame), Jon from Oxford, Craig and Nic from the BBC, Jonas from Scotland, Johan from Norway, of Gitorious fame, CJ from Sweden and so many others who were friendly, approachable and enthusiastic. RailsConf is so big that it can be intimidating, but everyone I met was just super cool.

Three things really stood out to me:

• The international element Sure, there were plenty of Americans, but from the small sample above, it's easy to see just how many countries are represented. This is a truly global community.
• The university-like atmosphere There's such an atmosphere of learning, of enthusiasm, of passion, but so concentrated and distilled. It's super infectious.
• The open-source connection It's awesome to meet some of the people who write and contribute to software that we use every day. Nothing quite as satisfying as looking someone in the eye and saying 'thanks'

I'm looking forward to next year's RailsConf, and definitely hope to get to a few more conferences in between.

We’re all off to Berlin for our second (Dave’s third!) RailsConf. In Portland this year we found some amazing early beta testers. They stuck with us through dozens of updates and upgrades, right up to today when we settled in to a new architecture at Amazon (more on that soon). This week, we’ll be inviting people to jump right in to our public beta with an end to the invite queue after we flush out the last few hundred great folks that are waiting to try our app.

If you’re at Berlin this week, please say hi. We’ll be wearing these goofy shirts with “Exceptional” written on the front. How clever.