What I learned at Surge 2011

Last week I had the opportunity to attend the Surge 2011 conference in Baltimore, MD.  I thought it was a great conference, and I’m already looking forward to next year.  I’m sure there’s already a plethora of great blog posts on Surge, but here’s just some thoughts based on my experience.

In no particular order:

EC2 has changed the world, everybody hates EC2

I don’t think I heard a presentation where somebody didn’t use EC2 and the other assorted AWS products.  Amazon (as far as I know) was not represented at the conference, and it seemed awkward for them to not be there (to me, at least).  This conference was full of folks rigging up serious production architectures in EC2, and they all seemed to have their own private horror story about something that EC2 did to them.  

I’m really puzzled by why some of EC2s competitors aren’t just as popular, or why with all the talk of HA and redundancy, people seemed to think it a weird idea to run a production stack on EC2 and another provider, as opposed to running Hot/Hot (for the few actually doing so) on multiple EC2 availability zones.  

The good part about how transient EC2 is?  Everybody cares about service resiliency now.

Technology goes through fads, but good Engineering practices are true regardless

We’re all partial to the technologies we know and love (or know and hate!).  I think the biggest reason is inertia:  once we get familiar with a tool, it’s natural to see how to apply it in various ways and get comfortable with its implementation.  

The diversity of technology being employed by presenters at Surge was impressive.  I fully expected the Etsy presentation to talk about how they moved away from the big bad RDBMS (Postgres) and into YourFavoriteNoSQLSolutionGoesHere.  However, they migrated from Python and Postgres to the ultra trendy: PHP and MySQL!  What is this, 2003?

Seriously, Ross Snyder gave a great presentation and really highlighted the struggles Etsy went through with the traditional silos built between Dev and Ops.  The underlying technology played second fiddle to those company dynamics: the end result seemed less tied to those particular technologies than getting the relationships right.

Postgres is a lot more popular than I thought

I have nothing against Postgres, I’ve just never had opportunity to use it.  Nor have I ever heard anyone talk about it, much less say they actually used it.  I guess I’ve just been living in a MySQL-only bubble.  

Folks are a lot more serious about Operations work than in the past

I haven’t been to Velocity yet, so maybe I’m missing out.  But it’s refreshing to see so much focus on Operability.  No longer are Operations folks the ogres in the closet, or maybe it’s just the case that the ogres are getting better organized. :)  In any case, I’m all for it.

MongoDB’s Replica set and sharding configurations is what I always wanted for MySQL

I can’t speak for Mongo’s scalability, reliability, client feature sets, or whatever.  But I can say that the talk given by Matthias Stearn about scaling Mongo turned out to be more about High Availability and it was impressive.  

Cross-WAN replica sets with client-configurable quorum write requirements, oh la-la!  Master election priorities, replica auto-cloning, SSL inter-replica communication, and auto shard re-balancing!  Response-time based client -> server and server -> server connection management, so stuff stays local including quorum reads/writes (up to the minimum quorum requirement) and slave recloning from the nearest up-to-date replica.  Beautiful.

The replica sets in Mongo seem to operate not unlike MySQL with Tungsten Replicator/Enterprise (with at least a non-empty subset of these features).  It’s super refreshing to see this kind of stuff baked in to the core of the datastore and I really want to see how it progresses to more feature-rich RDBMSs.