My need for agent forwarding has almost completely vanished since `ProxyJump` was added a few years ago.
Host *.example.com
User runlevel1
IdentityFile ~/.ssh/id_rsa-foo
ProxyJump foo@jump.example.com
So when I SSH to qux.example.com, it first SSHs to foo@jump.example.com then through it to runlevel1@qux.example.com and my agent is used on each individually.
So there's no ssh-agent socket on the intermediate jump box.
EDIT2: For folks stuck on older SSH clients, you can do this with `ProxyCommand ssh -W ...` instead (also shown in the previous link) which is actually what ProxyJump is doing under-the-hood anyhow. https://github.com/openssh/openssh-portable/blob/d9dbb5d/ssh...
Agreed. What worries me is this statement: “While it is generally better for users to avoid the use of a forwarded agent altogether (e.g. using the ProxyJump directive), the agent protocol itself has offered little defence against this sort of attack.”
Which I cynically read as “Hey, there is a better way, but we’re going to try to make the worse way slightly more secure, here’s how!”
Some of the described behaviours strike me as opening up entirely new and hard-to-detect attack vectors.
That's exactly what it says. Like X11 forwarding, there are cases where agent forwarding is secure and cases where it's not, and unless the alternatives completely eliminate the need for this feature, improving it is always good.
There’s completely valid use-cases for which ProxyJump makes no sense. For example forwarding your SSH agent to a remote dev box so that you can push and pull from git.
ProxyJump also does not support pam_ssh_agent_auth which can be used for e.g. implementing sudo auth with ssh keys. ProxyJump only solves a small subset of what agent forwarding can be used for.
Huh, what I usually do is generate a new SSH key on the dev box and enroll it. That means if the dev box gets compromised, first, it can't access anything other than my git server, and second, I have independent audit logs of actions from my dev box's key.
What's the advantage of using SSH agent forwarding over either generating a new key (if you don't quite trust the remote box) or scp'ing your private key (if you do trust the remote box just as much as your local one)?
If you don’t really trust the remote box, then storing a key on it – even one created solely for that box – might not be a good option. Doesn’t really matter whether the key is completely and permanently unencrypted or you only decrypt it during use.
To my understanding, by forwarding the SSH agent you can keep the private material on the local box only. I’d have to read up the details before I could be sure this is true, but that’s also what the linked article says:
> The [...] ssh-agent [...] supports [...] a way to forward access to private keys to remote hosts, without exposing the private keys themselves.
Creating a separate private key solely for this box – but storing it locally – is then a measure you can take additionally.
When you're forwarding an agent the material itself is kept on your local client, but if someone has root on the server that your agent is forwarded to, they can hijack your ssh-agent socket and ssh to any other server "as you", and your local ssh-agent will comply with the request and forward the key.
There's a "lesser of two evils" argument you could make: by using a separate key that only has rights to git, you're protecting your "main" ssh identity from being hijacked. So the only thing they can do if they compromise the server is impersonate you on git. ...Exactly, that's arguably as bad as it gets. If you don't trust the jump box, do not use it.
I had to use a "meh" jump box in a previous job. It was locked down to an extent, but a bunch of people could easily get root on it. Since I had to be active on that box for a bunch of work (not just "passing through" for the duration of a git pull) I set a very short lifetime on my ssh-add, so I got prompted for my passphrase whenever it expired and a key was requested. Not great, but better than leaving it active continually while I'm connected.
But they can break into the dev box and inject malware (where by "malware" I mean like 'echo alias ssh=/tmp/evil-ssh >> ~/.bashrc') while I'm on vacation for two weeks, though?
And if I wasn't going to notice them breaking in while I'm away on vacation, can't they just break in again as soon as I return from vacation?
A computer isn't like a house. You don't inherently notice people breaking in when you're using it: they don't make sounds or show up in your line of sight. And you don't inherently fail to notice people breaking in when you're not: if there are logs (that get sent off the machine), there will still be logs when you get back. Digital burglars don't have any need to "scope out" your machine and "wait" for you to be on vacation.
That's why I think the ssh-agent model provides a false reassurance. The fact that you're actively connected to the machine doesn't make attacks any less likely, but it sure feels that way, and so you're likely to forward an agent to a machine where you wouldn't be comfortable forwarding the raw key - but you shouldn't be comfortable forwarding the agent either.
Sshuttle is amazing. I usually loathe vpn clients and all that surrounds them, especially when set up by bigco IT that treats Linux as second class citizen. sshuttle is such a breeze, it’s literally a single command that just works instantly, together with hardware keys you get 2fa without some vendor specific vpn auth app on your phone.
When the working copy of your git repo resides on a remote box R1 rather than your local machine. You want to ssh into the R1, edit files and commits there and then push from there to some other remote host R2. Meanwhile your private key for accessing R2 is on your local machine. R1 can only access your forwarded agent, but not your private key.
ProxyJump doesn't help you here because R1 does much more than just proxy your SSH connection.
Another thing that would be extremely helpful is improved logging. If the ssh-agent logged the entity that was attempting to hijack the connection it would be pretty easy to write detection logic for weird shit.
We'll probably publish about this in the next few months, my colleague added some logging and we wrote ~10 detections on the process tree + IPC signatures. It makes the entire attack extremely dangerous for an attacker.
Port 22 shouldn’t be accessible from the internet, in addition. Some people say it shouldn’t be port 22, but we dip into security by obscurity, while making it annoying for you to use.
Choosing some other port than 22 is not a measure to make things more secure; as you say, it would be security by obscurity. But it massively unclogs audit logs if, for some reason, SSH is exposed to the internet.
Where I'd advise against moving the SSH port, and other ports for that matter, is in the cases where a larger number of people might need access on and off. We had to deal with a client who believe that fail2ban and moving the SSH port was the two single greatest inventions of all time.
As such there's nothing wrong with either of those to solution, except if you have a contractor handling alerts during the night. Even with pretty good documentation, people will get this wrong, and they will accidentally lock them selfs out and not they can't do anything to fix your broken system.
I'm much more in favor of keeping everything "default" and locking down the network. You don't need to worry about ports and log spamming, if the only hosts that are even able to reach your bastion host is the two IPs you explicitly allowed in the firewall.
Exactly this. As an additional layer of obfuscation, it's useful-- it just should never be your only defense. People take the "never do security through obscurity" thing a little too literally.
Good point. At the very least, it buys you time while attackers try to deobfuscate. How valuable that is is left for everyoneto judge; especially considering what a sibling says about locking yourself out - basically you and your team might also have to deobfuscate and thus lose time.
Just in case anyone is wondering about their interesting choice of hostnames, they are very Greek in origin – somewhat appropriate for Geeks.
> Scylla and Charybdis were mythical sea monsters noted by Homer. [...]Scylla was rationalized as a rock shoal (described as a six-headed sea monster) on the Calabrian side of the strait and Charybdis was a whirlpool off the coast of Sicily. They were regarded as maritime hazards located close enough to each other that they posed an inescapable threat to passing sailors; avoiding Charybdis meant passing too close to Scylla and vice versa. According to Homer's account, Odysseus was advised to pass by Scylla and lose only a few sailors, rather than risk the loss of his entire ship in the whirlpool. [0]
This is distinct from the Hydra, which was a land based multi-headed beast, and guarded an entry to the underworld. [1]
So, OpenSSH thinks that you have to brave the seas to fight your demons on land.
Seems apt, I think.
(As an aside -- ``To be swept from Scylla to Charybdis'' is a (predominantly somewhat upper-class) British expression beloved by lawyers [e.g. [2]] meaning "caught between a rock and a hard place", and sometimes found in the formal judgements of some of the highest courts in the world, e.g. the ECJ ruling about whether or maintenance failures for airlines count as 'extraordinary' -- and therefore outside of the scope of the compensation legislation -- or not. It was found that they are 'part and parcel of an airline's operation'. I found this out as a teenager, when suing EasyJet for compensation I was owed. I won.)
My only wishlist item for `ssh-agent` that would make all use of the agent safer, or nearly as safe as a hardware key, is that it had the option for an interactive confirmation dialog or mechanism for the user to confirm the source, destination, key, and client making the signing request. This is how most crypto wallets work, and given that the SSH private key never leaves the original host's agent's memory, a confirmation on use mechanism would alleviate most remaining forwarded agent hijacking concerns.
Some SSH Agents can do that but SSH isn't "aware" of this. From SSH's perspective it's simply taking long and you can timeout your SSH connection with that. For Example, KeepassXC offers this feature to require confirmation.
The new ed25519-sk and ecdsa-sk keys do also require user interaction because you need to press your key (and also make many concerns of SSH Agent Forwarding Null and Void).
Yeah, I've been using a Yubikey for a bit over a year and I'm already a huge fan of the way it works with SSH/Git. It's a bit more of a pain to set up with WSL2 since you need something like https://github.com/BlackReloaded/wsl2-ssh-pageant but it's absolutely worth the setup time.
When I was at Google, there was a modified version of ssh-agent, that would simply ask before signing any login request. It popped up a window (x11 or curses, I think regulated by an external program) that asked "Do you want to connect to HOST?"
It seemed such an obvious feature that I was surprised it wasn't part of the standard distribution.
Blacklisting all connections by default on my own devices and networks.
Native tooling setup guides are easy to find, and software such as PiHole, AdGuard, and Little Snitch (Leaky Snitch btw) make it very manageable... once you get past the first two weeks of manual connection whitelist popups and log crawling hell, that is :)
Adding the permissions to the `ssh-add` call is probably easiest for the authors of ssh-add, but seems to me the wrong user interface for me.
Putting such information in a config file (or maybe attaching them to the key somehow, by default $PATH_TO_KEY.permissions or so?) would seem better to me than either requiring the user to write wrapper shell aliases or fetching the previous ssh-add command out of the shell's history.
Right; but with agent forwarding, an 'evil' process on the target machine, that has privileges to connect to your agent socket (ie, root, or another pid as uid), can ask your agent to sign anything. Your key doesn't get exposed, but the agent will cheerfully sign requests no matter where they are going or where they came from.
So, say a virus, or a worm or something just went through your ~/.ssh/config file, your ~/.bash_history and/or your ~/.ssh/known_hosts (if you arn't using host name hashing) -- they could attempt to connect to all your previous hosts with your agent connection, and usually the agent won't offer any clue that this is ongoing ...
I have an Ubuntu laptop that I've used for work for several years. I have SSH keys for a number of different clients on my machine, including some personal projects.
One day, when connecting to a machine over a jump server that utilized SSH forwarding the sysadmin was trying to help me debug something and sent me a list of all of my own keys. It wasn't just a list of the key I was using to connect to that machine but every single key on my laptop, complete with names.
Until he showed me that, I had no idea that the SSH Agent was quite literally sending the house. I was always under the impression that the only thing sent was what was needed to connect to the single machine.
Seeing as OpenSSH 8.8 was released in 2021-09, I assume the year is a typo.
That aside, this is a great feature that I haven't seen mentioned before. Up until now, retaining any form of clear separation of personas while working across teams and platforms with SSH keys has been a fools errand. This seems to make it sane!
I just use separated ssh-agent with separated keys when I may need to forward the agent.
For example, to deploy things to a production host from a build host I login to the build host using the agent with the production key and push things from there.
With that, the jump host support and the ability to login with one agent but to forward another I just see no point for what ssh is doing. As they wrote, they want to keep things simple. But this complicates things.
Once the ssh agent has access to a forwarded key from the client isn’t it just a matter of reverse engineering the agent to extract that key regardless of what restrictions the client has specified?
Can't the server still pretend to be connecting to one of the specified hops? For example if -h 'allowed.com' is specified then in /etc/hosts/ wouldn't an entry to 127.0.0.1 allowed.com allow the local server to receive the key?
There’s no such thing as “receiving the key”. At best, you can coerce the forwarded agent to sign things w/ a key that it has. It will never give you the key itself; that isn’t part of the agent’s available behaviors.
Yup. And because SSH is a well-designed protocol, the signature needed to prove your identity is fresh (both sides pick large random values) each time, so even though SSH agent was over-used and securing it better is important, even today you cannot obtain enduring credentials from it. If I cut off your access to my SSH agent today, you can't authenticate to my servers tomorrow using what you learned.
This also has the benefit that the SSH agent can offer this capability on behalf of physical hardware that won't give up the keys either. My Yubico Security Key won't tell anybody (even me) my SSH private keys, but since SSH agent only offers to make signatures, it can proxy that work to the Security Key as necessary.
The Yubico product won't sign anything without a physical gesture (touching a glowing icon on the key) and so now if my laptop is sat unused on my desk while I eat lunch it's impossible for a remote system to use my credentials to sign in to another system, even if it's hostile, and it has somehow taken over control of my local machine's SSH client or I've unwisely authorised SSH agent forwarding, because it cannot cause the touch sensor to get touched.
It looks like the dns name is just a convenience; it actually looks up the associated host key and uses that, not the hostname. So the attacker would have the private keys for the destination to make use of it; I think?
There isn’t even a draft specification for SSHv3 so anyone claiming to have a server/client implementation (closed source or otherwise) is playing fast and loose with the term to try to sell something.
Human types "3" when they meant "2" versus shadowy US government agency has invented a completely new cryptographic protocol that everybody needs to use yet somehow escaped mention anywhere except this one document... Who knows right?
It is amusing reading the confused warblings of people who've been handed a mandate for something that doesn't exist. "How do I enable SSHv3 on this Cisco router?". We don't have many pranks in our industry, perhaps enabling SSHv3 can be our left-handed screwdriver or gallon of prop wash.
I’m somewhat certain that document is supposed to say “SSLv3 or TLS”. It’s a minor typo but it brings into question the validity of the rest of the document.
SSLv3 seems unlikely in 2020 document about security. TLS 1.0 and TLS 1.1 are already known to be a bad idea by then, and SSLv3 was formally deprecated in 2015.
Whereas a typo of SSHv2 makes sense because you probably do want SSHv2 on products which have it, even if they also speak TLS 1.2 or better, since the functionality is orthogonal.
But the "Use SSH or TLS" interpretation seems to make as much sense as saying "use a square or a yellow" regarding shapes. SSH is not (easily, anyway) a substitute for TLS where TLS is applicable.
This document isn't suggesting which services to use it's telling you what should be enabled, it's titled "Hardening Network Devices" and says "Research should be done to determine what services are running by default. The following guidance will serve to determine the services that should be enabled or disabled"
Among the services they recommend disabling are: Daytime, BootP, and telnet.
So there's no ssh-agent socket on the intermediate jump box.
EDIT: More examples here: https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_J...
EDIT2: For folks stuck on older SSH clients, you can do this with `ProxyCommand ssh -W ...` instead (also shown in the previous link) which is actually what ProxyJump is doing under-the-hood anyhow. https://github.com/openssh/openssh-portable/blob/d9dbb5d/ssh...