Exceptional Blog

Wow it's been some ride. Rails 2.3 sees a major maturity step in the framework so many awesome features there that really make development that bit sweeter, here's a great article by Ryan Daigle summarizing the new features in Rails 2.3 if you don't already know what to expect! One of the most interesting developments is the tight integration with Rack out of the box (we're looking at some cool stuff to increase the performance of our data collector with Rack!). Anyway, with Rack the entire boot loading order of Rails and the Application server changes.

• You define an application with Rack, the app is loaded
• Then you load the server handler (mongrel etc.)

If, for example you were starting a Rails app with mongrel the traditional way;

• Mongrel is started
• The Rails app is loaded

So what does this mean for Exceptional? Well it means we can no longer determine the deployment environment on plugin initialization. We depended on that do decide wether to use our advanced, delayed data submission mode or simply performing everything within the request/response. Well after lots of discussion we decided to remove the advanced exception handling we had introduced. Using Threads was the wrong idea from the start because of the issues in several deployment environments, but it did bring a lot of nice performance gains and we have plans to make even more additions that would for example decrease your outgoing http requests and data volume.

We also got to fix a couple of edge case issues with the plugin thanks to some of our wonderful customers excellent feedback! We're in the process of improving our support systems to make it easier to publicly discuss any issues about Exceptional.

So at this point we have a very simple plugin and will be iterating on it quite heavily over the coming weeks with plenty of refactorings and improvements. We have a lot of tasty plans for the development and are in talks with some really cool people to add to our team to make it happen! We're really excited about all of this and can't wait to let you know, so stay tuned right here to find out!

If you want to get the latest Exceptional plugin, a simple reinstall will have you on your way!

script/plugin install --force git://github.com/contrast/exceptional.git

We work hard to ensure that Exceptional is as awesome as we can make it. We've got lots of improvements round the corner and some exciting stuff that we're planning to have ready in time for RailsConf in May.

We're really proud of our customers and we want to celebrate this on our homepage at getexceptional.com. If you've been using Exceptional and you'd like your logo displayed on our homepage under a "Who uses Exceptional?" header, we'd love to hear from you.

If you want your logo to be included, please send the URL you'd like us to link to, and a jpg, png or gif to feedback+logo@getexceptional.com

We'd also welcome short testimonials, if you fancy waxing lyrical!

To our great dismay, we had some downtime last week due to full hard drives. Unfortunately, this wasn't the end of our worries. Load on our service has been increasing steadily over the past few weeks as increased coverage of the app has driven a lot of new signups.

We've commenced a series of upgrades to introduce increased performance and reliability to the app. Here's the steps we've taken:

Further to this, we're commencing work this week on a set of reporting systems to help us make intelligent decisions about what directions to take in both infrastructure and within the app itself. I'll post info on these decisions as they're implemented.

Right now though, we hope you enjoy the increase in speed and reliability that these upgrades have afforded.

Big thanks to Ladislav Martincik who's been helping us out with the implementation.

The activity feed for your app is now available as an RSS feed. Makes sense, right? This has been one of our biggest requested features for quite a while, it's great to be able to deploy something useful.

Bonus: We've moved deployment from Mongrel to Phusion Passenger. Delighted so far with the results and the ease of deployment. Kudos to the Passenger team.

Last Friday we had two hours of downtime that were caused by our database hard drive filling up. This sparked a flood of error logs, causing our logging hard drives to fill up quickly too. We managed to mitigate the issue on Friday, but stable performance wasn't returned until this morning.

Hard drives running out of space is an embarrasing problem to have in this day and age, and definitely not something we take lightly. Our current production database hard drive is now at 50% capacity and our logging disks at varying states between 20% and 50%.

We have installed notification processes that will notify us when drives are over 70% capacity. This will give us time to investigate the filling of the drives and act if necessary.

We have also increased the coverage of our log rotation to mitigate against disk space usage.

We will continue to investigate other systems that we can put in place to improve our internal reporting and monitoring over the coming weeks. We'll let you know what they are as they happen and we hope that the measures that we're actively taking will increase and maintain your faith in our service.

We experienced about an hour of downtime this morning. Our database hard drive filled up due to a spike in activity over the last few days. The drive is about twice the size of the current database size, but the binary logs filled up the rest of the space.

We actively monitor the state of the drive, but it filled up just a bit quicker than we expected this time. The issue was resolved within about ten minutes of it coming to our attention.

We take this issue very seriously. We’ve had an excellent uptime record so far. This is our first downtime since launching the service last March, and we plan to maintain this reliability. We’ll actively be upgrading our infrastructure over the coming months to increase and continue to provide a reliable service.

Unfortunately, any exceptions that were posted to us between 9.30am and 10.30am GMT ( 1.30am and 2.30am PST ) will have been lost, as our database had no space to store them. Any exceptions that were in our queue before we ran out of space will be processed in rotation.

Straight away, we plan to install a new hard drive that will be several times the size of the current database, and we will monitor the growth of the database more closely.

We apologise for this downtime and please let us know if you have any queries or concerns.

Paul

Zachary Voase Zvoase on Twitter has posted this gist which enables Exceptional for use in Django. Awesome.

December marks our first month as a fully fledged software service, and it's great to celebrate Christmas having released Exceptional this year and got our production version out the door.

The very best wishes from all of us.

I've been working on a project lately that has required Internationalisation and it offered me a great opportunity to get up to speed with the great work that the Rails I18n team have been doing to bring standardised I18n to Rails.

The process, for the most part was painless and fun, it's obvious that a lot of thought and great work has gone into making Rails I18n the most simple solution that works and no more.

I did come across one issue that I couldn't find a direct solution to, however: managing translation files. I was maintaining the en.yml translation file during development, but there was no out of the box way to update the French translation file with changes in the English one.

It's not a hugely complex problem, but it didn't involve a few tricky fixes. The out of the box Ruby YAML library doesn't really like unicode, so using YAML.dump results in a bunch of messy unicode character translations. Further, the Ruby Hash#merge method only goes as far as one level of keys, so it requires a bit of recursive heroism.

Two knights in shining rubies came to the rescue here: Ya2Yaml (Yet another to_yaml) is UTF8 safe and handles foreign characters with ease and grace. Further, Rails itself has a deep_merge method that was added this year, which helps to merge multi-dimensional hashes.

I whipped up a little plugin that uses a simple rake task for merging updates to your master translation file. You can check it out on github or install it directly:

./script/plugin install http://github.com/paulca/merge_translation/tree/master

Obviously, this only works with Rails 2.2. Here's hoping it's helpful to anyone else who comes across this problem.

Tom Lianza got in touch with us a few weeks ago to say he was interested in writing an Exceptional plugin for .NET. Well it's hot out of the presses and you can Get it here.

You can also check out Tom's blog post.

This kind of thing is really exciting for Exceptional. Jan Lehnardt released his exceptional-php client earlier this year with help from Till Klampaeckel.

We're looking forward to seeing a lot more of this kind of thing. If you're working in a non-Ruby/Rails environment, do get in touch.

We received some reports from users that exceptions appeared to be handled fine on the Exceptional plugin side yet they were never showing up in the interface! After some investigation we managed to find that some installations of the plugin had been sending invalid JSON data.

Turns out this was down to a "feature" of Rails 1.2.x ActiveSupport::JSON encoding, allowing the developer to turn on/off quoting of JSON attributes. The default being off, means a lot of the legacy Rails applications out there are sending invalid JSON. Thankfully, this feature was removed for Rails 2.0.

This only affected a small amount of users, but it's important to us to get it sorted out. On the upside, thanks to our "dumb" collector architecture. No data was lost, we are going to begin the back dated processing once we have all our regular expressions golden.

Applying the fix

Here's how to install our plugin using our SVN mirror.

 script/plugin install http://contrast.svn.beanstalkapp.com/exceptional/trunk --force

One of the most glaring omissions from Exceptional to date was the ability to search through exceptions. We've been aware of this from the day we started developing the app, but we decided it wasn't quite important enough to make the first release: we wanted to get the fundamentals right.

We've been working hard over the past week to get search up and running. And now it is.

As a bonus, you can now "Close all..." from within a search. So if you want to close a lot of exceptions matching a search, just hit the "Close all..." button after you search.

Recently we launched an update to our plugin that lets you install it as a gem. This means you can use Exceptional outside of the typical Rails setups that the plugin is targeting.

Installation

gem sources -a http://gems.github.com
sudo gem install contrast-exceptional

Usage

In your script/daemon/whatever you would write something like the following;

require 'rubygems'
require 'exceptional'

Exceptional.api_key = "YOUR-API-KEY-GOES-HERE"

Exceptional.rescue do
  nil.boom!
end

The Exception raised within the Exceptional.rescue block will be caught, posted to Exceptional and then raised.

For further documentation I'd love to say check out the RDoc but that's still in the works, we're working through some issues with the main Exceptional class documentation not showing up (because the current implementation is using class << self to create a Singleton class).

If you want to go ahead and dive in, please go ahead and check out the exceptional.rb source code itself.

• Exceptional docs

We were recently testing out a new release of our plugin with the magic of forked projects on github and can finally say it's ready in our official repository for consuming in your apps. We added a couple of features;

• Support for Thin
• Support for Phusion Passenger (mod_rails). As Passenger doesn't really like you spawning background threads, every time an Exception occurs the data is posted off to our app directly as opposed to the asynchronous background queue used for all other environments.
• Exceptional gem, if you want to use Exceptional outside of rails. We just pushed our gem to github, usage instructions are here.

Upgrading

You can install the plugin with:

script/plugin install git://github.com/contrast/exceptional.git --force

You will need to update your configuration file (located at config/exceptional.yml). You can either manually copy the example located in vendor/plugins/exceptional/exceptional.yml to config/ and enter your API Key, or download a prepared version from within your App's dashboard on Exceptional

Taking DHH's advice ("Stick a price tag on it") to heart and very much Getting Real, we're launching Exceptional paid plans today. We have four price plans and one free plan.

Price plans

The price plans are dead simple. You can check them out right now. In summary, we have a free plan that allows one app and one user to use the basics of Exceptional, all the way up to a maxi plan that supports 100 apps and 50 users for $150.

Thanks

We can only say the heartiest of thanks to everyone who partook in our beta. Your feedback has shaped the app and is continuing to shape it. Now that we're over the hump, we'll be cracking into our list of most-requested features that will add a lot more value. We're super excited to continue to shape the app into a powerful exception-tracking tool.

Migration

As a thank you, if you signed up for Exceptional before today, we're extending your free trial all the way to December 1st. If you want to continue to use the app beyond then, you're more than welcome to revert to the free plan, or simply sign up for a subscription before the end of November. Just click on "Account Settings" in the app.

The future

We're so excited to take Exceptional to new levels in the coming weeks and months. We're proud of what we've already managed to achieve, but we have plans to beef up the app with new features and more powerful tools. We'd love for you to be a part of it.