Honesty is the best policy. We are down.

In case you have not noticed. We are down.

First and foremost. There is no data loss. There is/was no security exposure. We are doing an upgrade :)

I’m not going to butter this up. The extent of this outage was not planned and we know it is frustrating for you. I contemplated doing this in video form but I have not slept in a bit and my fear is my current appearance may not be very CEO like. While I am not sure this post is going to be received as being very CEO like, I’m going to make it anyway.

In all seriousness we were unaware of a limitation in a software / hardware combination and because of that, we have unplanned downtime. We owe you an explanation and next steps. Ultimately this combination was my decision and in that respect, I am the one who owes you the explanation.

What happened?

  • Dwolla’s database is getting really big really fast.
  • We made a small change yesterday and we planned to be updating for about 30 minutes.
  • During a transfer we found a limitation in the operating system we run on.
  • The transfer limitation meant a big upgrade. The upgrade took a better part of all night. Big upgrades do not happen very fast.
  • Now the data needs to be re-attached to the upgraded and re-configured operating system, and run through the paces. With things of this complexity. It takes time.

What is currently happening?

  • Our resources have shifted to assisting our hosting partner with the resolution and next steps.

A few things

  • This is a OS/hardware issue.
  • There is no data loss. There is/was no security exposure.
  • This is something we are solving and know how to deal with in the future. In the mean time we simply have to work through it. While it is unexpected and it is stressful, it needs to be recognized and dealt with.
  • This is not as simple as just throwing a bunch of data on the Amazon cloud or something to solve. We have a highly customized hosting environment for a reason and unfortunately due to this design we can’t just plop it over onto another computer. It has to be done very carefully by the right people.
  • We can not speed it up right now. There is no faster or easy button for what we are doing.

What we are doing to remedy the situation in the future

  • Modifications to the existing environment that deal with an OS limitation.
  • Groundwork laid for another environment (which will take considerably more time)that is on the docket for first quarter.

When we will be back up

  • Tonight. What we are doing right now is transferring massive amounts of data and we are doing that as quickly as humanly possible but as safely as humanly possible. I’ve made a choice not to lean into people to make them frantic and nervous. When people are frantic and nervous they make mistakes and the data is too precious. This needs to be done the right way even if typing this stinks… I feel like this is the right way to handle it.

Lastly

If you want to scream at anyone or just generally go off on someone. I would ask that you do it to me. Ultimately all the choices that led us here I signed off on and I am not going to sit here and point the finger at other people. This is on me. Our staff is working hard and smart. I am frustrated and I am upset (like many of you) but my energy now needs to stay with assisting with the resolution.

Please feel free to reach out to me directly. My e-mail is bmilne@dwolla.org.

  • http://www.desmoinesisnotboring.com Pete Jones

    Keep your head up Ben, you’ll be laughing at all this in a week.

  • http://www.ninthdivision.com Hasani Hunter

    I’m glad that you guys are transparent and honest about your outage. I really believe your customers will/do appreciate the information. Keep up the good work!

  • http://www.groupdynamic.us Alan Feirer

    “I’ve made a choice not to lean into people to make them frantic and nervous. When people are frantic and nervous they make mistakes and the data is too precious. This needs to be done the right way even if typing this stinks… I feel like this is the right way to handle it.”

    Your gut is good, Ben. Keep trusting it.
    The people you’ve surrounded yourself with are also good. Keep trusting them.

    I’m willing to bet the rest of us will keep on trusting YOU, too. Your transparency on this rocks, and, frankly, could serve as a model for other CEOs of companies large and small.

  • http://www.ndrix.com Michael Hendricks

    I think you’ve handled this very well. An open statement like this blog post is exactly the right thing to do. As they say, “anything that’s worth doing is worth doing slowly” I’d much rather have Dwolla back in a couple days with all my data than back right now with my account missing.

  • http://www.jeanmbaker.com Jean Baker

    Wait – is that a breath of fresh air I detect? A sincere apology, the decision to NOT “lean” on your people, and a promise to fix it as quickly and safely as possible? Whoa – what a concept!

    Good job!

  • Dee Vandeventer

    Ben,
    You’re doing the absolutely right thing by being swift and honest with what’s going on. If only Bill Clinton had told the truth right away, it would have all blown over (no pun intended) in a short time.

  • http://www.hemdesigns.com Harold Martin

    Growing pains Ben, growing pains. I think everyone should be very understanding and receptive especially since you took the time to address this now and now after it was back up. That shows guts. Proud to be a Dwollaite. :)

  • http://www.facebook.com/josh.oshiro Josh Oshiro

    Didnt think my 2000 dollar transfer would create any problems. i guess the 2 bit memory adressing on your old system wasnt enough after all :/

    • CaityJones

      Hey there Josh – are you having a problem with a current transaction, or are you referencing a past transaction? If so, drop me a line at support@dwolla.com. This post was published over 2 years ago, so I’m not sure if this has anything to do with any issue that you may be experiencing at the moment.

  • Kurt Cantin

    Is Dwolla down again?
    Nothing seems to work

  • Jon

    What happens to transfers that were transferred between mtgox to dwolla during down time?

    • CaityJones

      Hi Jon – transfers shouldn’t be affected. But feel free to shoot an email to support@dwolla.com with the details of your transaction and we can check it out!

  • TheRealdeal

    I specialize in system high availability, pooled services, disaster recovery and security measures. And I am looking for work! Why not do what you love and invest it in an idea that you love?! Is Dwolla hiring?