SSRF protection in Perl

There’s a class of attacks that don’t get a lot of attention called Server-Side Request Forgeries. We recently started protecting against this universally at FastMail (rather than the ad-hoc arrangement we had before). Here’s how we did it.

First, lets talk about SSRF. Think of a webapp that lets you give it a URL that it will do something with. Examples are imgur’s “upload image from URL” function, or any feed aggregator (eg feedly) or podcast application (eg PlayerFM) that lets you give it a feed URL. (Just a few examples from services I like and use). WebHooks are another obvious function. FastMail has some stuff like this too: you can specify an image URL to add to your file storage, or a CalDAV URL to sync with, or a POP server to get mail or SMTP server to send mail through.

The attack comes from the knowledge that the servers that actually connect out to external resources probably also have access to the internal network. This network will likely have databases, internal APIs and other stuff easily available, probably without authentication. So if you can send a hostname that resolves to one of the internal hosts, you might be able to make unexpected things happen. A simple GET URL might be limited in the damage it can cause, but for something like a POP server hostname you could conceivably get access to someone else’s mail if you convince it to connect to a server on the internal network.

If you want more examples of what’s possible if you can convince an internal server to do something for you, check out this awesome slide deck from a couple of security researchers at ONsec. The socket stuff in particular is utterly terrifying.

At FastMail we didn’t used to have much of this. The three main things that users could instruct to connect to external sites were those mentioned above - add image by URL, connect to POP server and connect to SMTP server. All of them sanitised their input, so there was no problem. We did have some external HTTP calls for access to third-party APIs (eg payment providers, SMS providers) but these don’t typically include user-supplied hostnames and so don’t have a problem.

A month ago we released the calendar we’ve been working on for the last year, and an important part of its functionality is to be able to sync your FastMail calendar with other CalDAV providers. So that’s user-provided hostnames right there. But we’ve also been adding a lot of other supporting code that made external connections, like OAuth authorisation, other APIs (Dropbox, Twitter, etc) and so on. Lots of new HTTP client work all of a sudden.

At the same time, we determined that LWP::UserAgent, the “standard” HTTP client library for Perl, was ridiculously slow when receiving data and was seriously hurting our the speed of our calendar client (which makes CalDAV requests to our backend servers). We did a quick look around and after a little testing switched over to HTTP::Tiny, a more recent, smaller, faster HTTP client library. The client instantly became responsive, so we didn’t look much further. There’s really a lot to like about HTTP::Tiny, mostly in that it does one thing really really well. Use it.

So getting back to SSRF. We decided we want to protect against that style of attack, but rather than add another lot of sanitisation we decided it’d be better to just switch every single HTTP client object in the entire system to use a single class, built on top of HTTP::Tiny, with sanitisation by default. That’s an internal module called ME::UserAgent, which is mostly just a thin wrapper around HTTP::Tiny that gives you a “paranoid” UA object that will fail if you try to request internal stuff or, with an explicit switch if you really want to be able to access internal stuff (as we do for our own client). We’re most of the way through porting everything away from LWP::UserAgent to ME::UserAgent (slightly tricky because it has a different interface) but the remaining instances have been checked and deemed safe. It’ll be finished soon.

So lets look at how we implemented the “paranoid” client. You can follow along at home by looking at HTTP::Tiny::Paranoid, but its so simple I’ll just paste it in here:

use parent 'HTTP::Tiny';
use Net::DNS::Paranoid;
use Class::Method::Modifiers;
    
my $dns = Net::DNS::Paranoid->new;
    
sub blocked_hosts { shift; $dns->blocked_hosts(@_) }
sub whitelisted_hosts { shift; $dns->whitelisted_hosts(@_) }
    
around _open_handle => sub {
    my $next = shift;
    my $self = shift;
    
    my ($req, $scheme, $host, $port) = @_;
    my (undef, $error) = $dns->resolve($host);
    die "$host: $error\n" if defined $error;
    
    $self->$next(@_);
};

Even if you don’t do Perl it shouldn’t be too hard to see what’s happening. You globally set up a list of black and/or white listed hosts (including IP address ranges; anything Net::DNS::Paranoid accepts), then you just call it normally. We hook HTTP::Tiny’s internal “connect to server” method and check the incoming hostname. If it matches or resolves to something blocked, then the call fails outright, and that’s end of it.

The possible downside of this that you have two DNS lookups for every connection (because HTTP::Tiny internally will do one) but I don’t really care about that because I have a fast DNS cache on every machine. HTTP::Tiny will generally keep connections open for a little while. In practice its a negligible overhead.

Now I actually found the slide deck linked above after doing this work, and I was relieved that we’d already done some work here but more generally, had switched away from LWP::UserAgent. LWP is something like cURL in that that it can deal with multiple URL schemes, so the redirect attack was possible. HTTP::Tiny purposely does not support anything other than HTTP and HTTPS. So we happily dodged a bullet there.

That’s pretty much it. I’ve seen other solutions to the problem, involving sanitising proxy servers or worse. This one works for us. If you’re already using HTTP::Tiny in your app, then you can get a little protection for free. Try it, let me know how you go.