The Exceptional client is now packaged as a gem to take advantage of versioning, simple installation and easier use outside rails. We've lots of new features planned, installing as a gem will make it easy for you to upgrade.

Once you've created your app on Exceptional, installing is as easy as:

1. Adding this to your environment.rb file

   config.gem 'exceptional', :version => '2.0.0'

2. gem install exceptional

3. exceptional install YOUR-API-KEY

   You will find your api key in your apps dashboard on Exceptional.

4. (Delete the old plugin if you had it installed)

   rm -rf vendor/plugins/exceptional

What else has changed?

We're now using api.getexceptional.com to handle api traffic. Using a separate domain name makes it easier for us to load balance traffic and provide a solid service. We changed the data format a little so that we can capture even more useful information about your errors.

Feedback?

We're always keen to hear your thoughts/ideas to help us make exceptional better. And always looking for an excuse to use the blink tag ;)

Tip for updating your servers

When it came to moving to using the gem, a quick way to install it on all our servers was using cap shell:

cap> sudo gem install exceptional

One of the most frustrating things, for us and our customers, has been the lack of visibility into the health of Exceptional. If there were delays we had no idea how long they were. When there were errors we often didn't find out about them until a customer got in touch.

Our new Exceptional Status site shows you near realtime data about delays in processing errors. It has already helped us respond quicker to problems, and is helping us prioritize improvements to our architecture. We'll improve it as we capture and analyze more data.

Please keep the good suggestions coming in on how we can improve Exceptional for you.

We're delighted with the number of apps and people using Exceptional. We've been the victim of our success in many ways. More apps means more errors, and processing and storing so much information about each error is taking it's toll. We're spending more time dealing with scaling issues, than releasing valuable features.

With that in mind, we've decided to cut down on the amount of information we store for apps on free accounts:

• Only store detailed information for the first occurrence of an error.
• Only store the urls and counts for the 404 feature (no user-agents or referrers).

These changes will give us some breathing room to work on adding value for both paying and free users.

We’re in London this weekend for RubyFoo and to mark the occasion, we thought it might be fun to give everyone an early look at what we’ve been up to.

We’ve been re-working and re-building a lot on the back-end so we can continue to scale but we’ve also been re-designing our interface and simplifying and improving how we manage errors.

We’re calling the new interface, along with a few new upcoming features, Exceptional 2. Starting today, you can switch between Exceptional 1 and the Exceptional 2 preview by hitting the buttons at the top right of the screen. Eventually, when its feature-complete, we’ll let you switch permanently to the new version.

So watch us work on Exceptional 2 over the next few weeks and months and please feel free to send us your feedback and any suggestions you have for further improvements. We’re dedicated to making Exceptional the very best error tracker and manager around and we know we’ll get there with your continued amazing support.

Please say hi if you’re at RubyFoo this weekend!

We've changed how our blog is hosted. Mephisto is no more. It is now generated using jekyll, a tasty bit of technology, along with disqus for comments, and a lick of design for good measure.

While chatting to a friend recently who works within a large Ruby on Rails development team in Germany, I asked him what exception tracking tool they use. While his answer of "None" didn't surprise me (my hunch is that the vast majority of ROR apps do not use any) but what followed certainly stumped me, "We don't need one, our software is really well tested/spec'ed and so we don't have run time exceptions".

Can this be the case ? Can you sufficiently test your software to a point where run-time exception management is not necessary ?

My experience tells me that this cannot be true. While effective testing at development time is fundamental to precise software engineering, it concentrates principally on 'expected failures'.

Run-time error management for me has unfortunately all too often uncovered 'unexpected failures' despite high coverage and comprehensively layered (unit, functional, integration, system) test suites. The data from run time error management (specifically exceptions) is of such critical value as the error is caused by a 'real user' executing a real action in the wild. I've found many times that the tests written as a result of fixing a bug discovered at run-time by a real user have been among the most important tests in the project.

What do you think ? What is your experience ? Please leave a comment below, or drop us a note at feedback@getexceptional.com.

2 Responses to "Why bother tracking Exceptions?"

1. Alex Gusev says:

   August 21st, 2009 at 01:30 PM

   Of course it is possible to

2. Ray says:

   August 21st, 2009 at 01:43 PM

   Run-time error management is essential I feel in dynamically typed languages, at the most basic level you simply don't have the safety net of type checking at compile or run-time, when you introduce users and external data this can be asking for it.

   With a framework such as rails that has a plug-in based culture lots of unknowns via 3rd parties become a factor and in my opinion error management becomes even more essential, more so in a true production environment.

3. Matt Jankowski says:

   August 21st, 2009 at 02:34 PM

   Was the claim that they don't use a WEB BASED tool (like exceptional or hoptoad), or that they literally don't have any tracking in place, at all, to know about production environment exceptions?

   We have a policy of TDD, and "test first" regression bug fixing - but I still feel like we'd be missing a lot w/out a tool to catch errors from live and report back to the team.

Many rails applications have daemon processes running for background processing. Exceptions from these daemon processes are just as important as core application exceptions but frequently are not monitored. It is very straight forward to set up Exceptional monitoring of daemon processes.

Below I go through how to create a background daemon using the daemons gem and the excellent daemon_generator Rails plugin.

We will use the famous blog example, to create a blog with expiring posts so that after a set period of time a background daemon process will delete the expired blog posts.

Creating a rails app with a background daemon monitored by Exceptional

1. Create a new rails application

$ rails blog

2. Add some functionality to the new app

$ ./script/generate scaffold Post title:string body:text expires_at:datetime

3. Test the new application to make sure its working

$ rake db:migrate
  $ ./script/server

Open up http://localhost:3000/posts

4. Create a new application on getexceptional.com

If you do not already have an exceptional account, then you can sign up for a free one, if you like.

Otherwise, create new app within your Exceptional account and give it a name (e.g ‘My Exceptional Daemon Blog’). You of course can re-use an existing test application if you wish.

5. Install The Exceptional Plugin

$ script/plugin install git://github.com/contrast/exceptional.git

6. Configure the Exceptional Plugin with the Exceptional API Key

Edit the config/exceptional.yml file

1. Overwrite PASTE_YOUR_API_KEY_HERE with your exceptional API_Key (available from the ‘installation instructions’ screen within Exceptional)
2. By default Exceptional is disabled for the development rails environment, so change enabled: true within the development scope (we will need this later to run and test the application as we will run everything under the development rails environment)

7. Install the daemons gem

The Daemons gem is a great gem for wrapping ruby code as a daemon process.

$ sudo gem install daemons

8. Install the daemon_generator rails plugin

The Daemon Generator rails plug-in provides a generator and some nice templates for managing the life-cycle of a ruby daemon process

$ script/plugin install git://github.com/dougal/daemon_generator.git

9. Create/Generate your Blog Posts Expiry Daemon

$ script/generate daemon expire_posts

This will generate some template files for our daemon including

create lib/daemons
  create script/daemons
  create lib/daemons/expire_posts.rb
  create lib/daemons/expire_posts_ctl
  create config/daemons.yml

10. Implement the Expire Posts Daemon Logic

Edit lib/daemons/expire_posts.rb template, we will wrap our daemon logic with Exceptional.rescue so that Exceptional will handle any exceptions thrown during processing

#!/usr/bin/env ruby

  ENV["RAILS_ENV"] ||= "production"

  require File.dirname(__FILE__) + "/../../config/environment"

  $running = true
  Signal.trap("TERM") do
  $running = false
  end

  while($running) do

  Exceptional.rescue do

  @expired_posts = Post.find(:all, :conditions => ['expiires_at < ?', Time.now])
  @expired_posts.each do |post|
  post.destroy
  end
  end

  sleep 60 #A Minute
  end

11. Start up your blog posts expire daemon

$ RAILS_ENV=development ./lib/daemons/expire_posts_ctl start

12. Error!

There was a tiny intentional typo in Step #10, spot it ?. If everything is setup and running correctly, you should now see an exception within your Exceptional Account.

expire_posts.rb# (ActiveRecord::StatementInvalid) "SQLite3::SQLException: no
  such column: expiires_at: SELECT * FROM \"posts\" WHERE (expiires_at < ....

13. Fix up your Daemon, restart it and your done!

$ RAILS_ENV=development ./lib/daemons/expire_posts_ctl stop

Fix up lib/daemons/expire_posts.rb (change expiires_at to expires_at)

$ RAILS_ENV=development ./lib/daemons/expire_posts_ctl start

14. Play

Go to http://localhost:3000/posts and create and test some expiring posts!

15. Let us know how you got on !

We always love to hear your comments, feedback and ideas. Please either leave a comment below, or let us know at feedback@getexceptional.com

2 Responses to "Are your daemons exceptional?"

1. Pete Roome says:

   August 13th, 2009 at 12:58 PM

   I have been using this plugin recently for a project i have been working on. It was incredibly easy to set up and worked perfectly in development, however i have been having some issues since launch, in production and im unable to actually get my jobs running. A bit annoying as i havent been able to resolve the issue.

   I'm going on to try Whenever (http://github.com/javan/whenever/tree/master), to see if i have any better luck. Fingers crossed.

   Thanks for the post though, ill bare this in mind if i ever decide to have a punt at using daemon_generator again.

2. Ijonas Kisselbach says:

   August 13th, 2009 at 05:07 PM

   Excellent stuff guys... Are daemons are not exceptional, but they soon will be :-) !

We had the pleasure of meeting Morten Bagji from Heroku recently who told us all about their platform. We love how beautifully simple it is ...

"It's like a rails deployment vending machine"

Getting a brand new Rails application deployed on Heroku with Exceptional is really simple. The steps below should take you no more than 5 minutes in total.

Before you start you will need:

• an account with Heroku http://heroku.com/signup,
• git installed

Creating a Heroku & Exceptional rails application

1. Create a new rails application

$ rails blog

2. Add some functionality to the new app

$ ./script/generate scaffold Post title:string body:text

3. Test the new application to make sure its working

$ rake db:migrate
$ ./script/server

Open up http://localhost:3000/posts

4. Create a new application on getexceptional.com

If you do not already have an exceptional account, then you can sign up for free one, if you like.

Otherwise, create new app within your Exceptional account and give it a name (e.g "My Exceptional Heroku Blog")

5. Install The Exceptional Plugin

$ script/plugin install git://github.com/contrast/exceptional.git

6. Configure the Exceptional Plugin with the Exceptional API Key

Edit the config/exceptional.yml file and overwrite PASTE_YOUR_API_KEY_HERE with your exceptional API_Key (available from the 'installation instructions' screen within Exceptional)

7. Initialize this rails application as a git repository

$ git init
$ git add .
$ git commit -m "My New Exceptional Heroku Rails App"

8. Install the Heroku Gem

$ sudo gem install heroku

9. Create your application on Heroku

You will need to enter your Heroku name and password during this step

$ heroku create MyExceptionalHerokuBlog

10. Push your code to Heroku

$ git push heroku master

11. Test your new heroku app in your browser

$ heroku open

Navigate to /posts

12. Error!!

You will suddenly get an error form your Rails application. Go to Exceptional and you will quickly see the problem:

ERROR: relation \"posts\" does not exist\n: SELECT * FROM \"posts\" " 

13. Migrate the Heroku Database

$ heroku rake db:migrate

14. Test the heroku app again ... all should work beautifully

$ heroku open

Navigate to /posts.

Simple!

Two weeks ago, the Exceptional API suffered some serious growing pains. It hung on some requests and responded with 500 errors on others. It even went offline at one point, much to our great regret.

Most of this pain was caused by some cumstomers' over-active apps; we worked with those guys to resolve this problem.

But we know better than most that these sort of emergencies are not freak occurrences; they happen and we'll have to deal with an influx of exceptions from one app again. (Just last week, one app sent us over a million exceptions in one day!) So we need to be able to handle situations like this gracefully and keep the Exceptional service fast, rock-solid and reliable.

I'm happy to report that last week we rolled out three changes that mean requests to our API are answered quickly, our exception queue is practically empty and in most cases, exceptions show up in Exceptional in less than a second.

Here are the changes we've made:

1. We set a relatively short timeout on our API so that our customers' apps will never hang if we suffer high load on it again.
2. We added a new API instance which means all API requests are load balanced across two large servers, eating all the requests thrown at it and responding to your app quickly.
3. We increased the exception processing processes tenfold; we now have 30 daemons crunching through the queue.

This is just the start of a list of improvements that we're getting through slowly but surely. We hope you'll stick around long enough to see what's in the pipeline.

Thanks so much again for your support and patience.

3 Responses to "Twice as hungry, ten times as busy"

1. Wes Winham says:

   July 20th, 2009 at 09:53 PM

   Have you considered rate-limiting on a per-customer basis? It seems like you could set a very very high limit (like 50k per 24 hour period) and come out way ahead when one customer has a problem and hits you 1 million times. I doubt they'd find the 950k duplicate exceptions that they didn't get to be a big loss.

2. Ryan Tomayko says:

   July 22nd, 2009 at 03:12 AM

   Yeah. Check out this rate limiting with memcache technique:

   http://simonwillison.net/2009/Jan/7/ratelimitcache/

   We're planning on using it to check for a variety of DOS attacks but I imagine it could be useful in situations like this as well. You might even rate limit using the exception name instead of the user id or IP -- a sort of dupe limiter.

3. Jared Dobson says:

   July 23rd, 2009 at 04:06 PM

   Nice job! Thanks for the improvements!!!

Exceptional has been growing steadily since the start of this year and we've been improving lots of little bits and pieces behind the scenes to keep it running smoothly. But recently the growth has overtaken our efforts and we've experienced some of our worst performance issues yet.

Starting on Thursday of last week (2 July 2009), our API began to respond quite slowly. By Friday (3 July 2009) it became unresponsive in some cases. On Sunday (12 July 2009), the API service went down completely for up to 8 hours. During this time, to our great regret, all exceptions posted to us were lost.

Since Friday (3 July 2009), we've been making some good progress with our architecture redesign and have also been working on some short-term solutions to these problems. At present, the service is online and processing 80% of all exceptions posted to us. The remainder of the requests are set to time-out quickly so that Exceptoinal does not cause our customers' applications to hang.

Our next step is to work with a very small number of our users (one or two) who are using over 80% of our API processing resources, and work with them to reduce this load. This will be complete by 18:00 GMT today, after which we expect to to be processing 100% of the exceptions posted to us again. We'll update you on this when we have news.

Update: This is now complete and all exceptions are being processed, however there is a large backlog in the exception queue which may take a few hours to clear.

After today, we'll be adding more API server instances to our cluster. This is currently the bottleneck in our service and will allow us to scale further.

After that, we'll begin distributing a new plugin, which will communicate with the Exceptional service in a smarter fashion and also cope with a situation where the host app has an exception crisis.

We really, really appreciate every single one of our customers and we regret terribly any inconvenience we cause you guys. You've been very patient with us and that means a lot.

Please e-mail feedback@getexceptional.com if these recent problems have inconvenienced you and we'll credit your account for a month.

Many thanks again for your support.

4 Responses to "Recent growing problems"

1. Lukas Fittl says:

   July 7th, 2009 at 05:43 PM

   Thanks for the update & for your hard work!

   Looking forward to the new plugin, the constant startup error messages kept annoying me.

   Cheers, Lukas

2. Matt says:

   July 8th, 2009 at 05:30 AM

   Could you expand on what changes are being made to the architecture and the way in which the plugin "will communicate with the Exceptional service in a smarter fashion and also cope with a situation where the host app has an exception crisis."?

   We love Exceptional but have been unable to use it on production systems (at least not where we have any kind SLA) because a problem with exceptional can, under some circumstances, bring our own system to its knees.

   We're hoping that the silver lining of this recent crisis is that the product will evolve to the point where we can enable it for critical production applications (which is where it would be most useful).

   Good work and good luck guys.

3. Darragh Curran says:

   July 8th, 2009 at 09:49 AM

   Hey Matt,

   The main change we'll make to the plugin is that communication with the api will not be in the request processing loop - so that slowness in the api will not mean slowness in your 500 pages or at startup.

   Another thing that's been hurting us, is when a popular site starts throwing 100's errors per second, (perhaps their db is down). We need to be able to handle this without having a negative effect on our other customers. Part of our approach will probably be a slightly smarter plugin client, perhaps batching errors, or some way of throttling them.

   Thanks for your patience. Darragh

4. Rishad Omar says:

   July 9th, 2009 at 09:36 AM

   I will be very interested if you publish a guideline (ideas) that you collect with your discussions and investigations with your clients - i.e. a guide on efficient dealing with Exceptions.

   One idea that I have is to keep a count of the number of Exceptionals logged in one request. If that exceeds 5 (for example) then the software is probably in a loop and instead of logging 100's of Exceptionals - rather log 1 summary.

   Thanks for your very useful service, Rishad