Goddamn it.

It's not often that one messes up really bad. But today is my day apparently.

TL;DR: I accidentally wiped a github organization that had a few popular repos on it. But it's all fixed now.

How the heck did this happen?

I was trying to remove a private repository, called nelmio, which incidentally has the same name as the organization it was in, so nelmio/nelmio. Then this happened:

  • I wanted to check repo permissions so I went to https://github.com/nelmio/nelmio/settings/collaboration then followed a team link which led to https://github.com/orgs/nelmio/teams/foo
  • Then I was like OK let's go back in the settings tab to delete this repo, except at this point the settings tab points to https://github.com/organizations/nelmio/settings/profile (i.e. the org settings not the repo)
  • So at the end of the settings tab I see the familiar red delete button, hit it, it tells me to type the repo name (nelmio) as usual, but obviously I've done this many times so I don't read the fine print. It turns out in this case it wanted me to confirm the org name and not repo name.
  • As I click "Confirm Delete" I saw that something in the message wasn't quite familiar, but then it reloaded the page and I find myself on the github home. I'm like "That's odd!", then more or less 2 seconds later this horrible feeling in my guts is confirmed, the entire org was wiped.

Mitigation

I immediately emailed GitHub support, and am still waiting for an answer. I kinda wish there was a hotline for such cases, even if it was billed 10 bucks a minute :)

After doing so, I started re-pushing repos into a new organization (nelmiobackup). I then changed the packagist.org package URLs to point to this new org, so that at least package installs should continue working relatively normally for the time being.

I did not want to re-create a nelmio org as I thought this might hamper any recovery effort by the github support folks, but someone had the great idea to do that for me, so now that the potential damage is done and they added me as owner, I forked everything from nelmiobackup to nelmio so that it's present at the old URL for people doing installs using composer.lock files that point to the old URL.

I hope GitHub will be able to fix this, but if not apparently http://ghtorrent.org/ has a ton of github data. We'll see what can be done.

Updates

Update1: GH support answered, it seems they can restore. I had to rename nelmio to nelmio-old for now to make room for the restore, so clones with old URLs will temporarily fail again.

Update2: All restored, I only have to re-create teams within the org, no biggie :) Total time was a bit over 1h from deletion, which isn't too bad in the grand scale of things.

P.S: If you feel like comparing this to the left-pad incident, this is quite different because I fucked up accidentally while the guy in question did it intentionally. I guess I can't stop you though.

P.P.S: If you want to laugh or say anything mean, please go away. I don't want to hear it right now. It is stressful enough as it is, and I am doing what I can.

May 31, 2016 // News

Post a comment

Subscribe to this RSS Feed Comments

2016-05-31 14:06:21

Xavier

Good luck Jordi in recovering stuff.
Shit happens, and everyone is aware that you're doing your best.

2016-05-31 14:13:14

Mike

Ah well the saying is "To err is human" ;)

2016-05-31 14:14:51

Jan

Thanks for the update! I guess you're not a robot and a human after all ;-) Can happen to the best of us. Hopefully GH can restore things back to the way they were.

2016-05-31 14:16:38

Gregcop1

Wow. That's harsh.
Kind of "Do you really want to format this partition ? - Yes... No!!!!!!!" in worst
I wish you good luck in recovering.

2016-05-31 14:30:12

Pierstoval

"rm -rf /" style

You were very lucky to retrieve the whole thing (y).

Congrats :)

2016-05-31 15:13:39

dbu

yay for backups! and a good github support. no coffee needed for you for the rest of today, i guess ;-)

2016-05-31 16:52:17

Agares

Well, shit happens. But I'm glad that you handled it very professionally!
And at least you've saved a few dollars on coffee for today! :D

2016-05-31 17:04:53

Adam

I got that sinking feeling in my stomach all over again just reading your post. We've all done it. You just learn from it an move on. Thanks for all that you do for the PHP community.

2016-05-31 17:23:29

Matthew Krauss

Can I laugh in a friendly way? You'll chuckle ruefully too one day. We've all had that gut-wrenching moment in one way or another! Thank you for stepping up and admitting it - this will probably serve to help someone down the line! Also, this is a great example of the advantages of the distributed nature of Git. Even if you didn't have all the repos to re-push and Github didn't have a backup, they would have been out there somewhere.

2016-05-31 17:40:21

Mathias

... then more or less 2 seconds later this horrible feeling in my guts is confirmed, the entire org was wiped.


I know that feeling, bro. ;-) Good that you reacted the way you did which certainly reduced the damage a lot.

Maybe Github should adopt the Google logic of not asking for a confirmation but offering an undo for a limited time instead. ;-)

2016-06-01 04:08:32

Fusty

Dang, bad day. Tomorrow will be better, hang in there.

2016-06-01 08:50:43

Addvilz

Shit happens, Jordi! We still love you tho. :)

2016-06-01 11:07:46

So it was you ;)

My composer went dead for a while, and Iam using that repo.

I tought that packagist was down.

2016-06-01 12:23:39

Jonathan

Jordi,
All this does is inspire confidence in people that view you as a superhero (I'm talking about myself.)

I like to know that someone who I view with such respect is capable of making a mistake. It makes me feel that with hard work I'll one day be near to your playing field, because I sure as hell make mistakes.

2016-06-01 15:56:18

Samantha

I once brought down Visa's payment settlement system. We've all done it at one point or another. Thanks for the very clear and prompt communication! <3

2016-06-01 16:25:25

SoboLAN

It's great that you realised what was going on and rapidly reacted.

Don't beat yourself up about it, accidents happen. The great thing about reasonably up-to-date backups is that you don't have to be stressed about these things all the time ;) .