Dwolla.com updates

Yesterday afternoon, Dwolla’s service providers became the victim of a distributed denial of service event, resulting in limited or no availability to the website, Dwolla.com.

This advanced event, still persists today, and is preventing people from viewing the website and consequently accessing its services. We apologize for this inconvenience and are working hard with our service providers to resolve the issue.

In the meantime, we will continue to update this post with more details.

March 28, 2013 UPDATE 1:50pm CT

Third-party developers have been formally notified of the service interruption. Our team continues to work closely with service providers.

March 28, 2013 UPDATE 4:44pm CT

There’s been some misunderstanding concerning our hosting strategies. Dwolla uses isolated dedicated servers to host the Dwolla application. It does not share a virtual presence with other organizations inside a physical server. Our dedicated servers are kept in data centers specializing in redundancy and housing other isolated dedicated servers.

March 28, 2013 UPDATE 5:44pm CT

We’ve made meaningful progress with our hosting providers, and are now beginning to test accessibility with the web app. Mobile and API testing to begin soon.

March 29, 2013 UPDATE 10:15am CT

Dwolla has begun testing the API internally and hope to have its services back up today.

March 29, 2013 UPDATE 1:05pm CT

Initial configuration testing was successful, the API and mobile apps are back up and running. Please contact @BaconSeason if you have further questions.

CLARIFICATION  3:05pm CT

As of 1:30pm CST, Dwolla operations have returned to normal. We will continue to monitor and assess the situation and post any relevant updates here.

April 9, 2013 UPDATE 4:05pm CT

Summary

Between March 26 and March 29, 2013, Dwolla’s service provider began experiencing a large volume of traffic to its website. This event, called a DDoS attack, resulted in intermittent or no access to Dwolla through its online site and API offerings for a total of 24 hours.

Is my data and money safe?

Yes. Dwolla has conducted a thorough investigation since the incident. During the investigation, Dwolla’s dedicated security team did not find evidence to suggest that users’ account information or balances were affected by the event.

Solution

Dwolla and our hosting providers implemented numerous steps aimed at protecting access to Dwolla.com in event of a future attack. Although no system can guarantee complete availability during such events, we feel confident in the measures taken and remain vigilant.

 

  • Evan

    Am I correct in assuming that this is merely an access issue with the website, and has absolutely no risk on actual money?

    • dirk

      Yes, a ddos attack is basically the same as a bajillion people trying to access the website at once (but done with a computer), and the website crashes for a while. It shouldn’t affect the funds.

    • CaityJones

      Correct. It is a service denial, but our fraud team is actively monitoring the situation as well.

  • JLoganJ

    Any estimates on when service will be restored?

    • CaityJones

      I’ll be sure to add timing updates to the post as soon as I get them!

    • CaityJones

      If you haven’t tried yet, you can head over to the site and log in. We are still working on the API, making great progress.

  • DS

    And the payment API is also down today it seems, correct?

    • bpmilne

      DS – You got it. The consumer facing API is unavailable until the connectivity issue is resolved. #lessthanidealmorning

  • Itchy

    My favorite medium by which to fund my Mt. Gox account being down is in no way helping to curb my insane desire to buy bitcoins. I think I may just very well have myself a panic attack.

    • http://www.facebook.com/xof711 Xof Michot

      I’m with you on that one…

      • Charlie

        Last time I bought bitcoins, mybitcoin.com was immediately hacked and they were stolen (yes. I shouldn’t have had them there..) and this time I, also, can’t get them from Dwolla. Price has gone from 65 to 90+ in the time since I initiated a bank transfer to Dwolla. Arg!

        • utuxia

          me too. its crazy. it will hit 100 before i can even fund mt.gox.

          • Charlie

            dropped to $85 just now actually…. (mtgox via bitcoinwatch.com)

          • http://www.facebook.com/xof711 Xof Michot

            A correction will most likely occur… The crisis in Cyprus is artificially inflating BTC value. Hopefully, by then I can either fund mtGox or buy on Coinbase!!

          • utuxia

            i just bought 5 more on coinbase. I’m in it for the long haul. years. not months.

    • utuxia

      i think this is why dwolla is down. Everyone wants to fund mTgox to buy BTC. Did they consider that?

      • http://www.facebook.com/troy.benjegerdes Troy Benjegerdes

        Someone’s playing a game of global bitcoin market manipulation via DDOS

  • http://twitter.com/jakewastaken jakewastaken

    If I had a transfer already waiting to process, will this affect it?

  • John

    Will this slow down the payment transfer I made on Tuesday?

  • http://www.mobileappaddict.com/ John Defahl

    Yep, waiting to validate my bank account.. I feel like every second that goes by with Dowalla down, I am losing money..

    • JLoganJ

      Dwolla is losing money, too…

    • http://twitter.com/Varuka Varuka Salt

      I feel the same, but I do realize that if I’m not a customer yet, they’re not making money off me yet, so they’re going to get everyone verified as soon as they possibly can. It’s in their own self interest.

    • CaityJones

      Hey, John – we’re working hard to get the site up and running. We’ll have updates as soon as they’re available. Denial of service makes the site unavailable so you can’t land on the website and use it, but know that we have our fraud team monitoring as well.

  • jeff

    Why is your website being attacked?

    • CaityJones

      Dwolla utilizes a shared hosting provider, so any information we could provide would only be speculation at this time. Will provide updates as they come in.

      • http://www.facebook.com/xof711 Xof Michot

        Maybe it’s time to get your own dedicated hosting solution…

        • panicky

          especially when dealing with people’s money

      • CaityJones

        **Poor choice of words.** Dwolla uses isolated dedicated servers to host the Dwolla application, not residing on shared hosting service. What I meant by shared hosting is that there are multiple clients within data center, not sharing servers.

        • http://www.facebook.com/xof711 Xof Michot

          Gotcha… makes more sense! Next step has to be redundancy with other providers!!

  • Iyaun

    Did this only impact Dwolla.com (i.e. Dwolla targeted)?

    • http://twitter.com/jakewastaken jakewastaken

      Hopefully it was just the rack they were on.

      • CaityJones

        @iyaun:disqus @twitter-16077794:disqus Dwolla.com is a victim of this, along with others.

        • http://twitter.com/jakewastaken jakewastaken

          That makes me feel a little better that dwolla isn’t the sole target of an attack, but it still probably means it is even harder to come back online :-/

  • johnfein

    If I can help in any way, please let me know!

    • http://dwolla.com Jordan Lampe

      Thanks, John!

  • patroclus1

    Does this have anything to do with the DDOS against SpamHaus currently going on?

    • CaityJones

      It may or may not.. because the attack is affecting our hosting provider, we don’t have access to the data to draw conclusions.

  • Bob Armstrong

    I received an email from dwolla at 10:43 AM on 3/27 telling me that the dwolla facilitated transfer of funds from my verified checking account to my verified mt gox account was complete. It is 2:55 PM on 3/28 and I am still not seeing the funds in my Mt gox account. Am wondering what’s up?

  • http://www.facebook.com/profile.php?id=1033955636 Jerry Thibeau

    You all should have a replicated server at another hosting company for times like these. They make software to do this. Has anyone developed a disaster recovery plan? Who is running the show there, this should have been factored in when you started your business. I have a lot of money sitting in my account that I need to pay my contract workers with. If you’re going to handle money, your system needs to be bullet proof! I am going to have to explore other options now.

    • This x1000. You can hurt my feelings if you like, but do not mess with my money.

    • http://www.facebook.com/robertgenito Robert Genito

      I’m in with Jerry on this. Next thing: Dwolla’s database goes down, they cannot access backups because their disaster plan wasn’t tested, and no replicated DB existed. Then it takes me numbers and numbers of days, possibly even weeks to a month, to access my money!

      Dwolla, your 25 cent receiving fees looks less attractive now =

      • Nathan Gibson

        @facebook-1252763986:disqus

        Thanks for you feedback. As I mentioned in my previous reply to Jerry, Dwolla does utilize geographically distributed data center to support our website *and* databases. We also have an updated and tested disaster recovery plan.

    • utuxia

      agreed. cloudflare won’t cut it.

    • Jonah Dahlquist

      The problem is that it’s a DoS attack. As long as the DoS attack is running, no matter where the servers are, or how many backups they have, the attack will hit whatever machines they expose. If it was something that happened real quick and just needed to be reset, they wouldn’t have been down for nearly as long.

      Large financial companies like Visa have billions of dollars to put into security and stability systems that far exceed the industry standards.

      It appears to be back up, for me.

    • Nathan Gibson

      @Jerry

      My name is Nathan Gibson and I am part of Dwolla’s dedicated risk team. I appreciate your feedback. Dwolla does utilize geographically distributed data centers to support our system. During a DDoS attack, if one data center gets overwhelmed and stops responding, the secondary site takes on the complete load of the DDoS attack, and eventually becomes unresponsive also. Dwolla does have a disaster recovery plan we keep updated and tested on a regular basis. We are continually evolving and enhancing our operational security posture as new threats emerge. We have already made adjustments from this experience and will continue as we grow.

  • http://twitter.com/ManFmNantucket Mark L Palmer

    damn I’ve lost almost $900 not made on bitcoin due to this outage. Whats the plan to compensate or make funds available via alternate method?

    • utuxia

      oh please.

  • This is aggravating. I will not be doing business with Dwolla after this..

    • CaityJones

      It is aggravating, and I’m sorry for the inconvenience. We’re continuing to work through this as quickly and diligently as possible.

  • panicky

    I feel like I live in Cyprus, and the bank has just put a hold on my money

    • CaityJones

      A denial of service blocks access to the website. Funds are fine, and we do have our fraud team actively monitoring the entire situation.

      • panicky

        fraud team? you mean college students picking their acne with cheeto fingers?

        • CaityJones

          No college students, no acne, no cheeto fingers on the fraud team here. Just hard workers.

    • utuxia

      DOS is pretty common on the internet. Its rare, but it does happen. Its nothing like Cyprus. But maybe for a day. Their life savings have been destroyed. Consider that.

  • TheRealdeal

    I specialize in external & internal system upkeep including high availability, pooled services, disaster recovery and security measures. I just happen am looking for work! Why not do what you love and invest it in an idea that you love?! Is Dwolla hiring?

  • TheRealdeal

    I specialize in internal & external system high availability, pooled services, disaster recovery and security measures. And I am looking for work! Why not do what you love and invest it in an idea that you love?! Is Dwolla hiring?

  • TheRealdeal

    Will you expedite wire transfer transactions in order to compensate for this delay to access our site. Also, how can I reset my pin? i tried to transfer last night and it did not work.

  • http://thedanosphere.com Dan

    I’m going to have to echo what many users have already mentioned- if you have millions of users and their financial data in shared hosting environment that’s a very large security risk to each and every Dwolla user.

    Time to grow up, scale up, and get some dedicated boxes like a real company working with people’s money.

    I would like to give props to whoever runs your dwolla_support twitter account- very responsive and friendly.

    Hope the team can fix the problem and then move the hell away from that glaring ITsec hole when it comes to finData.

    • stenopad

      Wow, you clearly have no clue about security.

      • Nathan Gibson

        @Dan Hi there, there was some confusion on the post made earlier. We want to clarify by stating Dwolla uses isolated dedicated servers to host the Dwolla application, not residing on shared hosting service. What we meant by shared hosting is that there are multiple clients within data center, not sharing servers.

        • http://thedanosphere.com Dan

          Thanks- that is a solid, legit answer. And yep, that’s the way it should be done, kudos! Sorry for the confusion!

      • http://thedanosphere.com Dan

        And you do?

        If Dwolla is on using a shared host (ie: the same physical box hosts multiple sites/assets, not solely Dwolla) I could gain access to Dwolla data by via root access to the shared box (possibly through a vulnerability in one of the shared sites- sql injection, XSS, etc.) and then dump Dwolla data. If the Dwolla DB is on a different box (as it should be), I could dig through the Dwolla source on the infiltrated shared box and probably find the DB credentials somewhere in the source code, as well as its address, and easily tap into the DB box from the hacked shared box and then dump the database that way. If the DB is encrypted in some way bonus points to Dwolla, I’m blocked there.

        It’s easy to make blasé comments like yours without any backing data. Where am I wrong in this assessment? Do you have some implicit knowledge of Dwolla systems that I do not? All I know is a site the size of Dwolla using “shared hosting” doesn’t seem right… unless its for something trivial like a quick cache or something. That clearly is NOT the case as both the main site, AND the API are completely down right now.

        Please, drop some knowledge on me.

        • CaityJones

          Hey @thedanosphere:disqus – did @google-a498154e106eb18f712762e3ef435dc4:disqus’s comment help answer some of your questions? http://blog.dwolla.com/information-updates/#comment-845390454

          • http://thedanosphere.com Dan

            Yep Nathan hit it right on the head, thanks for clearing that up that also makes me feel a lot more secure in regards to my data. Keep up the good work!

  • Franklin Perez

    Does anyone at Dwolla have some type of estimate as to when the Dwolla website will be back up? I’m using Dwolla to fund my MtGox account.

    • CaityJones

      I’m not able to give an exact time, but we are getting closer. I’m sorry for the delay and the inconvenience it’s caused.

    • CaityJones

      If you haven’t checked out the site already – head on over and you should be able to log in. We’re still working on the API, but we’re making some great progress.

  • herbert

    INSANE to hear you guys are on a shared host! WTF!!! ok, maybe it’s fine for your .com but why is that causing your consumer faceing API to be down too? this DDOS attack is exposing how f**ked your infrastructure is… you guys have millions of dollars in investment. why not get a dedicated host on a closed, secure system?! SCARY

    • Dear Lord, I sure hope you don’t use the vast majority of the internet that is hosted on shared hosts like Heroku, RackSpace, Azure, Amazon EC2, etc.

      • herbert

        i use them all the time and build on them. i also build with a failure and disaster recovery plan in place over multiple AZs and multiple servers. I also never use DNS and hard wire IP routes for important services so I dont need to worry about DNS screw ups.

        • herbert

          btw there’s a difference between cloud server hosts (those who offer virtualized slices of larger computing slabs) vs shared hosts which stick multiple users on a single shared host.

          • utuxia

            there’s no way they are on a shared host. That would be a huge security risk.

          • herbert

            you would think / hope, right?

          • utuxia

            well, they are using cloudflare. I only used them when I was on a shared host. God only knows.

          • CaityJones

            Hey @e648ac9eb94a0bad5d9566bd2269c014:disqus and @utuxia:disqus – did @google-a498154e106eb18f712762e3ef435dc4:disqus answer any of these questions that you had? “We want to clarify by stating Dwolla uses isolated dedicated servers to host the Dwolla application, not residing on shared hosting service. What we meant by shared hosting is that there are multiple clients within data center, not sharing servers.” http://blog.dwolla.com/information-updates/#comment-845389351

    • Nathan Gibson

      Hi there, there was some confusion on the reply made earlier. We want to clarify by stating Dwolla uses isolated dedicated servers to host the Dwolla application, not residing on shared hosting service. What we meant by shared hosting is that there are multiple clients within data center, not sharing servers.

  • herbert

    thanks for censoring my comment only because i was criticizing your IT infrastructure. will never use DWOLLA EVER.. the word is out

    • CaityJones

      Hi Herbert – your comment was never censored. It’s still up and live.

      • herbert

        oh ok.. sorry i lost it in the flurry of comments.. i am very concerned about the security of your services. sorry for being so paranoid but i lost $$ in other hack attacks

        • CaityJones

          I understand the concern. The denial of service is just affecting access to the site in general, not funds. But we do have a fraud team that is actively monitoring the situation.

          • utuxia

            cloudflare sucks. they should ditch that crap.

      • http://twitter.com/WhatDanEats Dan

        i sent a large transcation to mt. gox yesterday and it hasn’t cleared get since they probably can’t access your site either. you guys are screwing me.

        • CaityJones

          Hi Dan – doing everything we can to get everything back up. I apologize for the inconvenience, I know it’s frustrating.

  • peacedude

    can you at least tell us if it will be today? Are our funds secure with you? Can you answer those 2 questions?

    • CaityJones

      Funds are okay, and we do have our fraud team actively monitoring the entire situation. We’ve been making good progress on getting dwolla.com up and running.

    • CaityJones

      If you haven’t checked out the site – you should be able to go there and log in.

  • bitoinmike

    looks like it is back up!!!

  • brneese

    if it’s a problem with a service provider, why isn’t more of the internet down?

  • Lucifer Dadson

    Aw, what? It was up a second ago?

  • Andrew Meek

    Seems like I can use the website just fine, yay! I know it’s not your fault Dwolla, sucks to have somebody shooting at you with a crowd getting mad at you for not dodging the bullets…

  • N0tM3

    Who would launch a dns attack against Dwolla? Paypal, Visa, MasterCard, American Express, Chase…need I go on? And that is just the “good guys” in the western hemisphere. Then there’s our friends at acronym “intelligence and law enforcement” agencies that need to fund their joyrides.

  • tedgarcooper

    Can’t log in via IOS apps. Kiosk does not show account ID and displays “null” for name. No balance info. Regular app terminates when logging in.

    • CaityJones

      While we’ve made meaningful progress with our hosting providers – we are now beginning to test accessibility with the web app. Hope to get everything else online soon.

  • Adrian

    Thanks for the updates. Looking forward for the service to become available soon.

    • CaityJones

      If you haven’t tried yet – head on over to the site. You should be able to log in. We’re still working on the API to get everything running smoothly again.

  • Brian

    This was no coincidence. DDoS attack on same day as Cypress comes back on-line.

  • http://www.facebook.com/troy.benjegerdes Troy Benjegerdes

    This problem is only going to happen again, and the only solution is going to be dedicated data centers that commit to fully transparency of who’s using their services, and where the traffic is coming from.

    The problem is by telling people where your servers are, you might get a physical attack. But at least with a physical attack, you call the cops, and the news carries stories about “terrorists attacking Iowa payment processor data center” instead of vague hand-waving about ‘DDOS’.

    If you want a dedicated, secure, powered by local renewable energy data center in Iowa, I have a farm, and I’d like to have some servers as an alternative crop.

  • http://www.prolimehost.com/dedicated_server.html dedicated hosting service

    This is the most important thing to focus on which type of hosting services we are using, is it working properly or not? Hosting having great presence in this internet world.