I am the author of the Heartbleed test site. AMA!

Hey Filippo, I just wanted to thank you for your work. During last week I've been upgrading ~1000 Servers to fix the heartbleed bug. Thanks to your tool I was able to keep customers on track without bothering me.

What do you think was the worst affected part of the heartbleed bug? Maybe Tor?

Did you get any Job offerings due to your recent work?

:D

I don't know, I'm afraid we will discover it in the coming days. IMHO the most impacted will be the ones that are or have been slowest to fix, so some .gov, embedded...

Yeah, I did get some amazing offers, and I'm considering them these days (damn US immigration law, by the way!)

What should an average internet person do after the Heartbleed?

• Install the Chromebleed or Foxbleed browser extension and not login to the sites that trigger an alert;
• Think hard about all the important accounts one have, and go changing the passwords there (always a good thing); REMINDER: using different passwords is more important than using complex ones, write them down on paper if you need!
• Wait for statements by the affected websites about what might have been leaked.

Hey Filippo,

I'm the author of the FoxBleed addon. From this perspective, I wanted to thank you for the easy-to-use and reliable API! :)

Congratulations on the job offers and the attention btw! It's not often the case that a rapidly built site is both fully functional and good looking. You also handled the sudden rush and the minor hiccups really well, so well deserved!

Btw, here the addon:

https://addons.mozilla.org/de/firefox/addon/foxbleed/

And Chromebleed:

https://chrome.google.com/webstore/detail/chromebleed/eeoekjnjgppnaegdjbcafdggilajhpic

---

http://breaking.systems/blog/2014/04/foxbleed-check-your-frequently-visited-websites-en-passant

Thanks so much to /u/FiloSottile and /u/breakingsystems.

I've installed Foxbleed add-on but it there any way we can test if it's working? I've visited lots of sites but nothing noticable has happened!

Is there a known site which hasn't be 'fixed' yet so we can see what happens?

The Cloudflare Challenge site is vulnerable by design. I use that.

Yeah, any advice for people who can't remember 46 different passwords already, much less change them into 46 new ones and then remember them?

Lastpass, or write them down on paper. It's fine. We all do. It's SO MUCH BETTER than having the same everywhere.

First of, thanks for providing a free service which has helped a lot of people. I see that it is possible to donate on your website. Just wondering, how much money did you get out of this?

I got a couple of thousands dollars, mainly via PayPal.

Ad companies offered more, and the donation link is tiny, but thinking that every time my phone buzz with a PayPal notification someone went to the trouble of clicking it and decided to send me money is AWESOME.

And how much BTC? :)

Not as much, a bit less than 1.

Hey Filippo. Just wanted to thank you for your work. I think it's important that users (yeah, regular users, not sysadmins) could discover what sites were vulnerable to put pressure on them. This is a great contribution in fixing the internet!

Why Go? (I love Go but wanted to know why you chose it)

Thanks! :D Honestly, this started as a sysadmin tool, but I've been delighted by how useful it has been for the users!

I'm in a serious "Go period", and I try to write in Go whenever I can. Here I blogged about some things I love of Go. Also it was incredibly fit for the task:

• it has a good TLS library that was so easy to patch
• it's easy to write enough to go online in a few hours
• it's fast enough to carry the load by itself, with its built-in web server
• EDIT: cross-compiling and static linking are awesome, I build the server binaries from my Mac to upload with one command

I have nothing technical to contribute , I can barely change my passwords in a timly manner. So my question, what did you have for breakfast?

This week I've almost lived on US time, but being in Italy this means waking up at 12-13 and having lunch.

So, pasta al pesto.

wait a minute.... are you italian? The kind of "pasta al pesto" Italian? So..... When are you joining us on /r/italy for an AMA in italian?. Ti prego, ti prego, ti prego :)

Ok, datemi fino a domani ;)

Pesto, Alfredo and Napolitana.... national pride pasta!

Secret: noone in Italy know what a Alfredo is.

Hi Filippo,

Perhaps not something most people noticed, but why is your go app writing down the results of every request made to a log?

edit: https://github.com/FiloSottile/Heartbleed/blob/master/bleed_serve.go#L66-L79

I feel like this data will make for a great overview of the impact, and it helped a lot debugging the site.

I tweeted a few times about logs and I want to stress that I don't log anything about the clients. Only results, and on a different system HTTP Referrals. Also there is no analytics or ads on the page to protect user privacy.

See also https://filippo.io/Heartbleed/faq.html#logs

This seems like standard practice. You would want to know the results of your heartbleed testing for the date/time. Keep in mind, this is not only for his website, anyone can download the source code and compile it. Any good sysadmin would want to know the dates/times that sites were affected and the dates/times they were patched.

While I do agree with your assumption; do keep in mind that it was the webserver hosting the tests that was also doing the logging; this means that the webserver has a complete log of all systems that failed the test.

Let's assume that /u/FiloSottile was of malicious intent; he would be able to use that list to steal hundreds, if not thousands of keys from systems as people continued to run the tests on his site.

Now this is an extreme case, but I'm sure you can imagine the discussions one can have (like we had in our office) about blind trust, hours after a debacle like heartbleed

Correct worry, but please consider masscan and zmap. ;)

You saved us hours of work, thank you kind internet sir.

For the AMA: How quickly did you get the site up following the announcement? I'm in the UK and turned up to work at 9am (the morning after the announcement) and the site was online, so you must have been pretty quick.

<3

I'm in Italy, so almost the same timezone.

I read about it during the evening (few hours after the announcement?), started working on it at 1am and got a first version up around maybe 4am? I remember going to sleep around 8am.

Dude you are like the Batman of the Internet. Can I give you a slogan? Yes? Cool.

"Heartbleed is out there, and I won't sleep until it does"

I love it! :D

How do we know you're not collecting a list of vulnerable sites?

Serious side: Thank you very much for the site. We put it to great use!

I am! ^^ https://pay.reddit.com/r/IAmA/comments/233161/i_am_the_author_of_the_heartbleed_test_site_ama/cgsy7lh

Happy that it helped :)

Just noticed, what is pay.reddit.com?

Ah, it's a SSL enabled domain they use for payments, but that you can use for all the site with HTTPS Everywhere.

Were you at all concerned about possible legal ramifications for yourself? As a programmer in the US, I would never write such (an awesome and useful) tool, as our incredibly loosely worded laws would surely leave me open to prosecution should some idiot law enforcement agency decide they wanted to mess with me.

Heh, good question. Didn't think about it at first, then some journalists started asking questions like yours.

The answer is multiple:

• I'm in Italy, maybe this helps
• Amazon has dealt with abuse reports from me (some have come), they are awesome! This shielded me a bit
• Honestly, fuck them. Really. If someone is so stupid to want to build a case against me, do it. It's clear that my site was not malicious, and helped people.

It's not by letting us being scared away from doing what we think is right and awesome that we will make them stop. I donate to the EFF, but it's not enough, one has to act accordingly.

PS: any good lawyer? :P

How hard is it to detect Heartbleed, really?

I'm asking this considering the information on http://www.hut3.net/blog/cns---networks-security/2014/04/14/bugs-in-heartbleed-detection-scripts-

Thanks a lot for the link, I have some fixing to do apparently. It is hard because TLS has so many different configurations (some even quirky, you should look at a TLS implementation, so many "shouldn't do this, but big sites need it"), and while developing one has access to a limited number of test servers.

Actually the sheer complexity of the protocol is what worries most some people in the InfoSec community.

Anyway, I made it a point to show a error whenever I wasn't 100% sure.

Do you use OpenSSL?

Yes, and you do too. It's just everywhere.

If your question was "do you use OpenSSL for the tests", then no, I used the Golang crypto/tls stack.

Can we expect any further Heartbleed threats in the future?

Probably, can't tell when, can't tell what, but programmers are humans! Hopefully they will become more rare and less severe while we make our infrastructures better.

Is it programmers or humans you want less of?

Programmers, fuck competition :P

What was the first project in which you have contributed? How do you start contributing open source programming? I have seen a lot of project on Github but I don't understand 90% of the code.

My first serious project was youtube-dl.

You have to choose a GitHub repo (for example), get familiar with the code, watch the Issues and try to be of help there at first, then when you feel confident enough submit a Pull Request to fix something yourself and from there it's done :) Good luck!

You created youtube-dl? I think I used it like 4 years ago. How old were you when you created that?

Nonono, didn't create it! I'm just a core developer. Started like, 2 years ago.

You're only 19? Wow, that's an incredible achievement! Are you a self taught programmer?

Yep, there is no good High School IT education in Italy... I learned on Internet, started making bots for Wikipedia.

Pancakes or waffles, my opinion of you hangs in the balance

P... pancakes?

How did you become so well integrated in OpenSSL? What do you do for a living/study? Thank you for the website tool, it works like a charm.

I have no relation to OpenSSL. I just read the fix commit and worked from there. Please note that I did NOT find the bug.

I'm a freelancer for now, I'm considering some job offers.

What part of your site are you most proud of, be it a technical bit you wrote for the site, or general impact?

Umh... good question. I'd say the load it has been able to sustain thanks to Go. And the fact that it turned out to be user-friendly after all.

Hey, Filo! Glad to see this is doing so well. No questions, just dropping in (it's Jamie).

Shameless plug: Go get Chromebleed, based on Filo's service: here

Hey there :) I suggested Chromebleed somewhere in the top comments, weight in there, maybe

Hey-J. Sartori here, wondered whether you remember me. I have nothing relevant to add, just wanted to say hi:)

Hey, ma ciao :D

Do you know if the bug was attributed to malicious behavior or was it just a mistake made by the programmers?

Because it's open source, do you think its the fault of the companies that actually use it since they are vested everything on a technology they don't put funding into?

My bet is completely on a mistake. We do them all the time, really.

Companies should have thrown much more money at OpenSSL, they need way more staff. It's critical infrastructure nowadays!

Hey Filippo, as a non-dev I found it to be a bit tedious compiling your code using "go". Any reason why you chose this approach?

Hey, sorry to hear that. Go helped me a lot in developing the tool and the site, some reasons are here https://pay.reddit.com/r/IAmA/comments/233161/i_am_the_author_of_the_heartbleed_test_site_ama/cgsws8f

I should probably have provided some pre-built binary.

What does the pay.reddit do?

Ah, it's a SSL enabled domain they use for payments, but that you can use for all the site with HTTPS Everywhere.

Is it possible that the only reason that heartbleed was discovered was due to large companies wondering how the NSA was able to penetrate sensitive and what they believe was 'encrypted' information - after which the flaw was discovered?

I don't think this is a likely scenario. It was probably not exploited at all before last week.

After your discovery of HeartBleed, how much time did you spend writing this tool? Were you busy writing and improving it non stop? Were you in a panic to secure servers you were responsible for after the HeartBleed discovery?

Thanks for the tool btw, it was a great help to me. As a sysadmin that mainly deals with Windows, it was a bit confusing to get the tool running. It would have been great if I could have used packages from the stable repository instead of the unstable Debian repository. Support for scanning subnets in your tool would be nice too, I had to hack together a script to scan my network with your tool.

2-3 hours to get the first version running on Monday.

Yes, I've been working on this since then, to improve it, fix it and keep it online. The first 3 days I actually slept really a little.

I don't have important servers to maintain myself, thankfully.

How long did it take to develop the site?

Why did you do it? you just heard the news and tought, lets make a test site?

60M tests and high traffic, how did the servers handle that stuff?

2-3 hours to get the first version running on Monday. Since then to keep it online and fixed.

Basically, yes. Adam Langley made one for the "goto fail;" bug, so I thought it might be a fun and useful project (didn't expect it to become so huge!)

The servers got smashed on the ground a couple times at first. Then I rewrote the service in Go, put it behind a Amazon ELB, and started 40 m3.medium servers. That made it, I sustained 20,000 tests per minute at some point without problems. (Ah, and the html web site is static, hosted on GitHub Pages, zero problems with that.)

How much did it cost to host the program no Amazon? Did the project end up costing you money?

It might end up costing me, I still don't know.

https://pay.reddit.com/r/IAmA/comments/233161/i_am_the_author_of_the_heartbleed_test_site_ama/cgt00ge

Hey, I remember seeing your comment on HN and then seeing you post it there as an article a bit later.

How did you first handle scaling? How did you feel in those first couple of hours seening your tool get so popular?

Hair on fire style :) I scaled up manually up to maybe 5-8 servers (behind a ELB), then rewrote the server to be pure Go (web.py was the bottleneck) and then turned to Ansible. Slept almost nothing ;)

I was amazed by the first thousand visits, that was my peak before last week. Then the millions came :D

Best Pizza topping combination?

Wurtel Wurstel and prosciutto cotto.

Hey. Thanks for creating such a great site. DO you by chance have a test site that I could visit that has the heartbleed bug on it? When I install detection software like the foxbleed and chromebleed, I like to test it to make sure that it is works and see what it looks like when it detects a bug on a site.

Thanks

I use https://www.cloudflarechallenge.com/ that is vulnerable by design.

At what age did you start programming and how many hours per day do you usually spend programming? Also much time do you spend on computer doing something, that is not programming, and how much time do you spend on acitvities that don't involve computer? Thanks :)

I started around 13/14.

It depends... Let's say an average of 10? But it really depends on the days, I've spent periods without doing it much, and I've pulled quite some all-nighters. I used to bring my laptop and program at school, too. Now it's also kind of my job.

If we consider research, reading Hacker News, etc. broadly "programming", then little. I always have a Facebook tab somewhere, but I usually don't spend time just surfing YouTube or playing games.

I love my KTM Duke 690, so I spend a lot of time riding it. Then there are running, swimming, friends, girls, conferences...

girls?

girls

When the bug was first announced, I tried to use your tester, but it was not working; apparently, there was too much traffic. That prompted me to write my own tester in Python, which I posted on Reddit, and it got spread out over the globe, and apparently someone even incorporated it into Metasploit.

I haven't looked at your code, but I have a basic question about it: Does your tester perform a full key exchange before sending the heartbeat request? My script just sends a pre-fabricated ClientHello, then an unencrypted heartbeat request, and I wasn't able to get it to return more than 16k at a time.

I do a true handshake with the crypto/tls lib. This makes it compatible with way more systems.

Have you read this on /.? http://it.slashdot.org/story/14/04/13/1553258/private-keys-stolen-within-hours-from-heartbleed-openssl-site

Many commenters note that this effectively destroys ssl for all time and the industry needs to rebuild from the ground up. Your thoughts?

Also, I've read in several places now, that using your test effectively violates cybercrime laws and could be considers hacking attacks. Your thoughts on that as well?

Meh, critical bugs happen. We need to make certificate revocations better, and fix the CA design, but we already knew that,

Re cybercrime: https://pay.reddit.com/r/IAmA/comments/233161/i_am_the_author_of_the_heartbleed_test_site_ama/cgsz52d

So, coming from a Ruby/Java/Scala background, what are the best resources for learning how to write idiomatic Go code?

The Getting Started docs are good. Then read the official FAQ etc.

Then the official Go blog posts, they are awesome.

Yep, the official docs rock!

How did you find the bug? Where you looking for bugs? Did you stumbled upon it when looking at the code or when using openSSL?

I totally DID NOT FIND the bug. I created the test site, but all the glory for spotting it go to Neel Mehta, a Google engineer, and Codenomicon.

What type of hats do you like to wear?

I love my cheap black Fedora.

Hey, awesome work. Could we get some numbers for example total hosts checked and how many of them were actually affected? (If you store them).

Thanks!

I totally have to run some stats, but didn't have time up to now.

I'll write a post-mortem soon. I'll also probably release anonymized raw data.

What do you think about the media panic concerning the bug? I've been getting calls from people screaming on the phone about a cataclism and internet catastrophe because he heard so on the news.

This bug got some unusually good marketing, but if it helps getting it fixed everywhere, I don't see the damage.

(I feel the pain of having crazy customers, however)

What language did you build this in? Was it complex to make?

Go, some thoughts here https://pay.reddit.com/r/IAmA/comments/233161/i_am_the_author_of_the_heartbleed_test_site_ama/cgsws8f

Not much to make at first. Way harder to keep online and working with the traffic.

There are reports that the NSA knew about Heartbleed and exploited it. How do you feel about this?

I don't know how much to trust these claims.

Better people than me stated that it's unlikely, it's too noisy for them.

Why did you hack me?

Because you are cute.

Go America!

... Go Italy?

how many government IPs have visited your website. I am operating under the assumption the NSA knew about this, which would mean very few if any NSA ip addresses would need to test for heart bleed.

I don't log accesses.

How can a 37yo learn how to program and/or create websites? I'll have 2 years where I can put 2-4 hours a day into learning something and I think that this is it.

Thanks.

There are really good resources online. I hear good things about http://www.codecademy.com/ but never tried it.

Start fiddling with Open Source projects as soon as you can.

awesome achievement by 19.

so do u feel proud like OH YES, did this in 19!! or do u feel like WHY cudnt i do this by 14???

I never think that much about my age, honestly, and the community helps a lot in this. Online few people care about your DOB.