I'm the 19yo guy that past Monday night set up https://filippo.io/Heartbleed/, the site to check if servers are safe from Heartbleed.

The site performed more than 60 millions tests last week. AMA!

A bit of media coverage: Forbes, LATimes. For the InfoSec crowd, I'm the one linked to by Schneier.

My Proof: https://filippo.io/Heartbleed/ama.html

Comments: 998 • Responses: 71  • Date: 

trunkzee757 karma

Hey Filippo, I just wanted to thank you for your work. During last week I've been upgrading ~1000 Servers to fix the heartbleed bug. Thanks to your tool I was able to keep customers on track without bothering me.

What do you think was the worst affected part of the heartbleed bug? Maybe Tor?

Did you get any Job offerings due to your recent work?

FiloSottile611 karma

:D

I don't know, I'm afraid we will discover it in the coming days. IMHO the most impacted will be the ones that are or have been slowest to fix, so some .gov, embedded...

Yeah, I did get some amazing offers, and I'm considering them these days (damn US immigration law, by the way!)

IndoPr0462 karma

What should an average internet person do after the Heartbleed?

FiloSottile818 karma

  • Install the Chromebleed or Foxbleed browser extension and not login to the sites that trigger an alert;
  • Think hard about all the important accounts one have, and go changing the passwords there (always a good thing); REMINDER: using different passwords is more important than using complex ones, write them down on paper if you need!
  • Wait for statements by the affected websites about what might have been leaked.

breakingsystems264 karma

Hey Filippo,

I'm the author of the FoxBleed addon. From this perspective, I wanted to thank you for the easy-to-use and reliable API! :)

Congratulations on the job offers and the attention btw! It's not often the case that a rapidly built site is both fully functional and good looking. You also handled the sudden rush and the minor hiccups really well, so well deserved!

Btw, here the addon:

https://addons.mozilla.org/de/firefox/addon/foxbleed/

And Chromebleed:

https://chrome.google.com/webstore/detail/chromebleed/eeoekjnjgppnaegdjbcafdggilajhpic


http://breaking.systems/blog/2014/04/foxbleed-check-your-frequently-visited-websites-en-passant

katiesam776 karma

Thanks so much to /u/FiloSottile and /u/breakingsystems.

I've installed Foxbleed add-on but it there any way we can test if it's working? I've visited lots of sites but nothing noticable has happened!

Is there a known site which hasn't be 'fixed' yet so we can see what happens?

FiloSottile7 karma

The Cloudflare Challenge site is vulnerable by design. I use that.

lomerell8 karma

Yeah, any advice for people who can't remember 46 different passwords already, much less change them into 46 new ones and then remember them?

FiloSottile21 karma

Lastpass, or write them down on paper. It's fine. We all do. It's SO MUCH BETTER than having the same everywhere.

ScelPol218 karma

First of, thanks for providing a free service which has helped a lot of people. I see that it is possible to donate on your website. Just wondering, how much money did you get out of this?

FiloSottile565 karma

I got a couple of thousands dollars, mainly via PayPal.

Ad companies offered more, and the donation link is tiny, but thinking that every time my phone buzz with a PayPal notification someone went to the trouble of clicking it and decided to send me money is AWESOME.

chiguireitor66 karma

And how much BTC? :)

FiloSottile110 karma

Not as much, a bit less than 1.

s3rv4i151 karma

Hey Filippo. Just wanted to thank you for your work. I think it's important that users (yeah, regular users, not sysadmins) could discover what sites were vulnerable to put pressure on them. This is a great contribution in fixing the internet!

Why Go? (I love Go but wanted to know why you chose it)

FiloSottile134 karma

Thanks! :D Honestly, this started as a sysadmin tool, but I've been delighted by how useful it has been for the users!

I'm in a serious "Go period", and I try to write in Go whenever I can. Here I blogged about some things I love of Go. Also it was incredibly fit for the task:

  • it has a good TLS library that was so easy to patch
  • it's easy to write enough to go online in a few hours
  • it's fast enough to carry the load by itself, with its built-in web server
  • EDIT: cross-compiling and static linking are awesome, I build the server binaries from my Mac to upload with one command

moredunsmore150 karma

I have nothing technical to contribute , I can barely change my passwords in a timly manner. So my question, what did you have for breakfast?

FiloSottile274 karma

This week I've almost lived on US time, but being in Italy this means waking up at 12-13 and having lunch.

So, pasta al pesto.

robespierring28 karma

wait a minute.... are you italian? The kind of "pasta al pesto" Italian? So..... When are you joining us on /r/italy for an AMA in italian?. Ti prego, ti prego, ti prego :)

FiloSottile37 karma

Ok, datemi fino a domani ;)

chiguireitor12 karma

Pesto, Alfredo and Napolitana.... national pride pasta!

FiloSottile96 karma

Secret: noone in Italy know what a Alfredo is.

XarothBrook132 karma

Hi Filippo,

Perhaps not something most people noticed, but why is your go app writing down the results of every request made to a log?

edit: https://github.com/FiloSottile/Heartbleed/blob/master/bleed_serve.go#L66-L79

FiloSottile198 karma

I feel like this data will make for a great overview of the impact, and it helped a lot debugging the site.

I tweeted a few times about logs and I want to stress that I don't log anything about the clients. Only results, and on a different system HTTP Referrals. Also there is no analytics or ads on the page to protect user privacy.

See also https://filippo.io/Heartbleed/faq.html#logs

emxiaks40 karma

This seems like standard practice. You would want to know the results of your heartbleed testing for the date/time. Keep in mind, this is not only for his website, anyone can download the source code and compile it. Any good sysadmin would want to know the dates/times that sites were affected and the dates/times they were patched.

XarothBrook37 karma

While I do agree with your assumption; do keep in mind that it was the webserver hosting the tests that was also doing the logging; this means that the webserver has a complete log of all systems that failed the test.

Let's assume that /u/FiloSottile was of malicious intent; he would be able to use that list to steal hundreds, if not thousands of keys from systems as people continued to run the tests on his site.

Now this is an extreme case, but I'm sure you can imagine the discussions one can have (like we had in our office) about blind trust, hours after a debacle like heartbleed

FiloSottile51 karma

Correct worry, but please consider masscan and zmap. ;)

eXplicits68 karma

You saved us hours of work, thank you kind internet sir.

For the AMA: How quickly did you get the site up following the announcement? I'm in the UK and turned up to work at 9am (the morning after the announcement) and the site was online, so you must have been pretty quick.

FiloSottile125 karma

<3

I'm in Italy, so almost the same timezone.

I read about it during the evening (few hours after the announcement?), started working on it at 1am and got a first version up around maybe 4am? I remember going to sleep around 8am.

UtterlyInsane137 karma

Dude you are like the Batman of the Internet. Can I give you a slogan? Yes? Cool.

"Heartbleed is out there, and I won't sleep until it does"

FiloSottile62 karma

I love it! :D

CuddlyLiveWires63 karma

How do we know you're not collecting a list of vulnerable sites?

Serious side: Thank you very much for the site. We put it to great use!

mesid27 karma

Just noticed, what is pay.reddit.com?

FiloSottile48 karma

Ah, it's a SSL enabled domain they use for payments, but that you can use for all the site with HTTPS Everywhere.

dweezil2256 karma

Were you at all concerned about possible legal ramifications for yourself? As a programmer in the US, I would never write such (an awesome and useful) tool, as our incredibly loosely worded laws would surely leave me open to prosecution should some idiot law enforcement agency decide they wanted to mess with me.

FiloSottile130 karma

Heh, good question. Didn't think about it at first, then some journalists started asking questions like yours.

The answer is multiple:

  • I'm in Italy, maybe this helps
  • Amazon has dealt with abuse reports from me (some have come), they are awesome! This shielded me a bit
  • Honestly, fuck them. Really. If someone is so stupid to want to build a case against me, do it. It's clear that my site was not malicious, and helped people.

It's not by letting us being scared away from doing what we think is right and awesome that we will make them stop. I donate to the EFF, but it's not enough, one has to act accordingly.

PS: any good lawyer? :P

LeaViljanen38 karma

How hard is it to detect Heartbleed, really?

I'm asking this considering the information on http://www.hut3.net/blog/cns---networks-security/2014/04/14/bugs-in-heartbleed-detection-scripts-

FiloSottile65 karma

Thanks a lot for the link, I have some fixing to do apparently. It is hard because TLS has so many different configurations (some even quirky, you should look at a TLS implementation, so many "shouldn't do this, but big sites need it"), and while developing one has access to a limited number of test servers.

Actually the sheer complexity of the protocol is what worries most some people in the InfoSec community.

Anyway, I made it a point to show a error whenever I wasn't 100% sure.

EyeSawThat34 karma

Do you use OpenSSL?

FiloSottile104 karma

Yes, and you do too. It's just everywhere.

If your question was "do you use OpenSSL for the tests", then no, I used the Golang crypto/tls stack.

termiterdrol30 karma

Can we expect any further Heartbleed threats in the future?

FiloSottile51 karma

Probably, can't tell when, can't tell what, but programmers are humans! Hopefully they will become more rare and less severe while we make our infrastructures better.

frankivo29 karma

Is it programmers or humans you want less of?

FiloSottile133 karma

Programmers, fuck competition :P

SidKetchum27 karma

What was the first project in which you have contributed? How do you start contributing open source programming? I have seen a lot of project on Github but I don't understand 90% of the code.

FiloSottile40 karma

My first serious project was youtube-dl.

You have to choose a GitHub repo (for example), get familiar with the code, watch the Issues and try to be of help there at first, then when you feel confident enough submit a Pull Request to fix something yourself and from there it's done :) Good luck!

marssantoso17 karma

You created youtube-dl? I think I used it like 4 years ago. How old were you when you created that?

FiloSottile39 karma

Nonono, didn't create it! I'm just a core developer. Started like, 2 years ago.

Zy0n21 karma

You're only 19? Wow, that's an incredible achievement! Are you a self taught programmer?

FiloSottile54 karma

Yep, there is no good High School IT education in Italy... I learned on Internet, started making bots for Wikipedia.

starcptn20 karma

Pancakes or waffles, my opinion of you hangs in the balance

FiloSottile47 karma

P... pancakes?

Inzire15 karma

How did you become so well integrated in OpenSSL? What do you do for a living/study? Thank you for the website tool, it works like a charm.

FiloSottile23 karma

I have no relation to OpenSSL. I just read the fix commit and worked from there. Please note that I did NOT find the bug.

I'm a freelancer for now, I'm considering some job offers.

Unidan14 karma

What part of your site are you most proud of, be it a technical bit you wrote for the site, or general impact?

FiloSottile22 karma

Umh... good question. I'd say the load it has been able to sustain thanks to Go. And the fact that it turned out to be user-friendly after all.

jsq13 karma

Hey, Filo! Glad to see this is doing so well. No questions, just dropping in (it's Jamie).

Shameless plug: Go get Chromebleed, based on Filo's service: here

FiloSottile12 karma

Hey there :) I suggested Chromebleed somewhere in the top comments, weight in there, maybe

PerfectlySaneTailor13 karma

Hey-J. Sartori here, wondered whether you remember me. I have nothing relevant to add, just wanted to say hi:)

FiloSottile17 karma

Hey, ma ciao :D

dieselxindustry12 karma

Do you know if the bug was attributed to malicious behavior or was it just a mistake made by the programmers?

Because it's open source, do you think its the fault of the companies that actually use it since they are vested everything on a technology they don't put funding into?

FiloSottile42 karma

My bet is completely on a mistake. We do them all the time, really.

Companies should have thrown much more money at OpenSSL, they need way more staff. It's critical infrastructure nowadays!

emxiaks10 karma

Hey Filippo, as a non-dev I found it to be a bit tedious compiling your code using "go". Any reason why you chose this approach?

FiloSottile17 karma

Hey, sorry to hear that. Go helped me a lot in developing the tool and the site, some reasons are here https://pay.reddit.com/r/IAmA/comments/233161/i_am_the_author_of_the_heartbleed_test_site_ama/cgsws8f

I should probably have provided some pre-built binary.

khaosoffcthulhu8 karma

What does the pay.reddit do?

FiloSottile7 karma

Ah, it's a SSL enabled domain they use for payments, but that you can use for all the site with HTTPS Everywhere.

WhitGoodman7 karma

Is it possible that the only reason that heartbleed was discovered was due to large companies wondering how the NSA was able to penetrate sensitive and what they believe was 'encrypted' information - after which the flaw was discovered?

FiloSottile15 karma

I don't think this is a likely scenario. It was probably not exploited at all before last week.

justanotherreddituse6 karma

After your discovery of HeartBleed, how much time did you spend writing this tool? Were you busy writing and improving it non stop? Were you in a panic to secure servers you were responsible for after the HeartBleed discovery?

Thanks for the tool btw, it was a great help to me. As a sysadmin that mainly deals with Windows, it was a bit confusing to get the tool running. It would have been great if I could have used packages from the stable repository instead of the unstable Debian repository. Support for scanning subnets in your tool would be nice too, I had to hack together a script to scan my network with your tool.

FiloSottile17 karma

2-3 hours to get the first version running on Monday.

Yes, I've been working on this since then, to improve it, fix it and keep it online. The first 3 days I actually slept really a little.

I don't have important servers to maintain myself, thankfully.

DANNYonPC6 karma

How long did it take to develop the site?

Why did you do it? you just heard the news and tought, lets make a test site?

60M tests and high traffic, how did the servers handle that stuff?

FiloSottile20 karma

2-3 hours to get the first version running on Monday. Since then to keep it online and fixed.

Basically, yes. Adam Langley made one for the "goto fail;" bug, so I thought it might be a fun and useful project (didn't expect it to become so huge!)

The servers got smashed on the ground a couple times at first. Then I rewrote the service in Go, put it behind a Amazon ELB, and started 40 m3.medium servers. That made it, I sustained 20,000 tests per minute at some point without problems. (Ah, and the html web site is static, hosted on GitHub Pages, zero problems with that.)

idontalwaysupvote3 karma

How much did it cost to host the program no Amazon? Did the project end up costing you money?

nhoss25 karma

Hey, I remember seeing your comment on HN and then seeing you post it there as an article a bit later.

How did you first handle scaling? How did you feel in those first couple of hours seening your tool get so popular?

FiloSottile11 karma

Hair on fire style :) I scaled up manually up to maybe 5-8 servers (behind a ELB), then rewrote the server to be pure Go (web.py was the bottleneck) and then turned to Ansible. Slept almost nothing ;)

I was amazed by the first thousand visits, that was my peak before last week. Then the millions came :D

Anterai4 karma

Best Pizza topping combination?

FiloSottile7 karma

Wurtel Wurstel and prosciutto cotto.

whosthetroll4 karma

Hey. Thanks for creating such a great site. DO you by chance have a test site that I could visit that has the heartbleed bug on it? When I install detection software like the foxbleed and chromebleed, I like to test it to make sure that it is works and see what it looks like when it detects a bug on a site.

Thanks

FiloSottile5 karma

I use https://www.cloudflarechallenge.com/ that is vulnerable by design.

TAscendor4 karma

At what age did you start programming and how many hours per day do you usually spend programming? Also much time do you spend on computer doing something, that is not programming, and how much time do you spend on acitvities that don't involve computer? Thanks :)

FiloSottile11 karma

I started around 13/14.

It depends... Let's say an average of 10? But it really depends on the days, I've spent periods without doing it much, and I've pulled quite some all-nighters. I used to bring my laptop and program at school, too. Now it's also kind of my job.

If we consider research, reading Hacker News, etc. broadly "programming", then little. I always have a Facebook tab somewhere, but I usually don't spend time just surfing YouTube or playing games.

I love my KTM Duke 690, so I spend a lot of time riding it. Then there are running, swimming, friends, girls, conferences...

I_enjoyfuckingthings8 karma

girls?

FiloSottile18 karma

girls

jspenguin3 karma

When the bug was first announced, I tried to use your tester, but it was not working; apparently, there was too much traffic. That prompted me to write my own tester in Python, which I posted on Reddit, and it got spread out over the globe, and apparently someone even incorporated it into Metasploit.

I haven't looked at your code, but I have a basic question about it: Does your tester perform a full key exchange before sending the heartbeat request? My script just sends a pre-fabricated ClientHello, then an unencrypted heartbeat request, and I wasn't able to get it to return more than 16k at a time.

FiloSottile6 karma

I do a true handshake with the crypto/tls lib. This makes it compatible with way more systems.

ars_ex_machina3 karma

Have you read this on /.? http://it.slashdot.org/story/14/04/13/1553258/private-keys-stolen-within-hours-from-heartbleed-openssl-site

Many commenters note that this effectively destroys ssl for all time and the industry needs to rebuild from the ground up. Your thoughts?

Also, I've read in several places now, that using your test effectively violates cybercrime laws and could be considers hacking attacks. Your thoughts on that as well?

FiloSottile5 karma

Meh, critical bugs happen. We need to make certificate revocations better, and fix the CA design, but we already knew that,

Re cybercrime: https://pay.reddit.com/r/IAmA/comments/233161/i_am_the_author_of_the_heartbleed_test_site_ama/cgsz52d

diyorgasms3 karma

So, coming from a Ruby/Java/Scala background, what are the best resources for learning how to write idiomatic Go code?

FiloSottile3 karma

The Getting Started docs are good. Then read the official FAQ etc.

Then the official Go blog posts, they are awesome.

Yep, the official docs rock!

JamesMean3 karma

How did you find the bug? Where you looking for bugs? Did you stumbled upon it when looking at the code or when using openSSL?

FiloSottile21 karma

I totally DID NOT FIND the bug. I created the test site, but all the glory for spotting it go to Neel Mehta, a Google engineer, and Codenomicon.

mikayakatnt2 karma

What type of hats do you like to wear?

FiloSottile5 karma

I love my cheap black Fedora.

kopilatis2 karma

Hey, awesome work. Could we get some numbers for example total hosts checked and how many of them were actually affected? (If you store them).

Thanks!

FiloSottile5 karma

I totally have to run some stats, but didn't have time up to now.

I'll write a post-mortem soon. I'll also probably release anonymized raw data.

rospaya2 karma

What do you think about the media panic concerning the bug? I've been getting calls from people screaming on the phone about a cataclism and internet catastrophe because he heard so on the news.

FiloSottile4 karma

This bug got some unusually good marketing, but if it helps getting it fixed everywhere, I don't see the damage.

(I feel the pain of having crazy customers, however)

synchroni_city1 karma

What language did you build this in? Was it complex to make?

FiloSottile2 karma

Go, some thoughts here https://pay.reddit.com/r/IAmA/comments/233161/i_am_the_author_of_the_heartbleed_test_site_ama/cgsws8f

Not much to make at first. Way harder to keep online and working with the traffic.

_72_65_64_64_69_74_1 karma

There are reports that the NSA knew about Heartbleed and exploited it. How do you feel about this?

FiloSottile6 karma

I don't know how much to trust these claims.

Better people than me stated that it's unlikely, it's too noisy for them.

lazy_troll21 karma

Why did you hack me?

FiloSottile4 karma

Because you are cute.

mycall1 karma

Go America!

FiloSottile1 karma

... Go Italy?

recursiveparanoia1 karma

how many government IPs have visited your website. I am operating under the assumption the NSA knew about this, which would mean very few if any NSA ip addresses would need to test for heart bleed.

FiloSottile3 karma

I don't log accesses.

steve6261 karma

How can a 37yo learn how to program and/or create websites? I'll have 2 years where I can put 2-4 hours a day into learning something and I think that this is it.

Thanks.

FiloSottile2 karma

There are really good resources online. I hear good things about http://www.codecademy.com/ but never tried it.

Start fiddling with Open Source projects as soon as you can.

rude_ass-8 karma

awesome achievement by 19.

so do u feel proud like OH YES, did this in 19!! or do u feel like WHY cudnt i do this by 14???

FiloSottile7 karma

I never think that much about my age, honestly, and the community helps a lot in this. Online few people care about your DOB.