Skip to content

Tag: SSH

Get real time Telegram notification when SSH login on Linux

Traditionally, Linux system admins use commands such as last, lastlog or some log analysis service to monitor user logins and catch suspicious activities. This is a good security practice. However these methods have their drawbacks – as the usual approch is to do periodic scans there could be delay between a login event and the report of incident. The delay could be minutes to hours depends on how the system is configured.

In this post I’m going to introduce a way to get near real time notification when a user login event occurred. This method make use of the popular instant messaging cilent Telegram and its Bot API to send a notification on user login so the delay is usually within just a couple of seconds!

The basic idea

Telegram Bot is a special kind of account that operated by software instead of real human being. Telegram provides a HTTP API to control the bot, to do things like send a message, or receive a message and process it as command, etc. etc. I have made use of that feature to do following things:

  1. On user login, run a script.
  2. The script calls Telegram Bot API to send a message.
  3. The message is sent by Telegram server to my Telegram account.
  4. My Telegram client on phone and PC receive the message, I know a login event occured.

All this happends right after the user logs in. I usually got the notification from Telegram on PC within 2 secs. Keep reading to see how to set up this.

Setup a Telegram bot

First of all, you’ll need to have a Telegram account and create a Telegram bot. I won’t go into detail for this as it is easy and is already well documented on Telegram website. Check it out here.

Once the setup of bot is finished, a token will be allocated for the bot. Keep that token. It will be used by the script to control the bot via Telegram HTTP API.

Connect the bot with your Telegram account

For obvious reason regarding privacy, a Telegram bot cannot initiate a message to a Telegram user. If you need to received messsages from a bot, you need to first start a chat with it and send a hello message. This process is like talking to a readl human account, just talk to the bot and send any text is OK.

After you initiate the chat, the bot and you is now in a conversation which represents by an unique ID in Telegram API. We need to find out this conversation ID, as it will be the “destination” for our script to send notification message to.

Here is how to get that chat ID – first send a message to the bot, then access the Telegram API getUpdates.

The API URL is https://api.telegram.org/bot<your bot token>/getUpdates. HTTP method GET should be used to access it. <your bot token> is the token allocated to your bot, as mentioned above.

I use curl to do this.

$ curl "https://api.telegram.org/bot<bot token>/getUpdates"
{"ok":true,"result":[{"update_id":xxxxxxxxx,
"message":{"message_id":1954,"from":{"id":xxxxxxxxx,"first_name":"Your first name","last_name":"Your last name","username":"Your telegram user name"},"chat":{"id":xxxxxxxxx,"first_name":"Your first name","last_name":"Your last name","username":"username","type":"private"},"date":1494671101,"text":"test"}}]}

The return message is a JSON text. This is how it looks after beautification.

{
  "ok": true,
  "result": [{
    "update_id": xxxxxxxxx,
    "message": {
      "message_id": 1954,
      "from": {
        "id": xxxxxxxxx,
        "first_name": "Your first name",
        "last_name": "Your last name",
        "username": "username"
      },
      "chat": {
        "id": xxxxxxxxx,          <--- This is the chat ID you need.
        "first_name": "Your first name",
        "last_name": "Your last name",
        "username": "username",
        "type": "private"
      },
      "date": 1494671101,
      "text": "test"
    }
  }]
}

Note down the value of message.chat.id, we will need it in the script.

Create a script to send message via the bot

Now we need to create a script which sends a predefined message via Telelgram.

Create a executable script file named /usr/local/bin/tgbot.sh and put below content in.

#!/bin/bash

KEY="<your bot token>"

URL="https://api.telegram.org/bot$KEY/sendMessage"

TARGET="<your chat ID>" # Telegram ID of the conversation with the bot, get it from /getUpdates API

TEXT="User *$PAM_USER* logged in on *$HOSTNAME* at $(date '+%Y-%m-%d %H:%M:%S %Z')
Remote host: $PAM_RHOST
Remote user: $PAM_RUSER
Service: $PAM_SERVICE
TTY: $PAM_TTY"

PAYLOAD="chat_id=$TARGET&text=$TEXT&parse_mode=Markdown&disable_web_page_preview=true"

# Run in background so the script could return immediately without blocking PAM
curl -s --max-time 10 --retry 5 --retry-delay 2 --retry-max-time 10 -d "$PAYLOAD" $URL > /dev/null 2>&1 &

Remember to replace <your bot token> and <your chat ID> with the real data of your own.

Basically, this script use curl to access the Telegram HTTP API to send a text message to the designated user. Those $PAM_* variables are provided by PAM, they will be explained later.

You can now run this script to test. If everything goes right, your Telegram client will receieve a message like this:

User  logged in on myhost at 2017-05-13 18:45:20 CST
Remote host: 
Remote user: 
Service: 
TTY:

That means the script is working. The real useful information is still empty, they will be filled in when running by PAM.

Run the script when a user logs in on the server

Now the final part is to make the script running when a user logs in on the server.

There are some information available on the Internet regarding this topic, however most of them just put the command to run in the user’s shell init script, such as .bashrc, or /etc/profile or a script under /etc/profile.d/. This way works, but is not secure. Because a user have access to his own shell init scripts such as .bashrc or .zshrc, a hacker who steal the user’s identitiy and log in on the server can easily remove the command from the script or remove the script itself entirely. This makes it practically useless – the script will only be run just once and send out the notification once, then the hacker is able to mute it.

A better approach is to use the capability of PAM. I’ve introduced it in an earlier post, read it if you are not familiar with PAM yet.

Here we are going to use the pam_exec module to accomplish our goal. This module is part of the PAM and it can run a command specified by user. We can use it to run the Telegram message sending script on user login, then we will be able to accomplish our goal. Here is how.

Edit the file /etc/pam.d/system-auth, this is the PAM service that will be called by many system commands that tries to authenticate a user, SSH is one of them. Put below line at the last line.

session optional pam_exec.so type=open_session seteuid /usr/local/bin/tgbot.sh

This command tells PAM to run the script /usr/local/bin/tgbot.sh after a user successfully authenticated himself and logged into the system. The pam_exec module will export a few environment variables so the script could read them and send in the message. Below are the explaination of the variables.

PAM_USER: name of the user name who is logging in.
PAM_RHOST: the remote host that the user connects from.
PAM_RUSER: the user name on the remote host that the user connects from.
PAM_SERVICE: the PAM service, which in some way represents the system service that user trying to access.
PAM_TTY: if the user logs in locally, this will be the TTY name. For remote login via SSH, it will just be “ssh”.

These variables are send in the Telegram message, so we can now who from where on when has logged in on which server via which service, enough for us to know whether it normal or suspecious.

As this file is only writable by root, a non-root user could not bypass this step. This is much secure than the shell init script method. Of course, if the root user is hacked it still could be modified and the notification could be muted. Some further facility could be deployed to mitigate this problem, such as file integrity check, which is beyond the scope of this article.

Test it

Now login in to the server via SSH, and you will be able to recieve a message on Telegram from the bot, like in below screenshot. Congratulations!

More

In fact, this is just an demostration of the functionality of pam_exec and Telegram bot. You can actually do more. A few ideas is on my mind.

  • Trigger a system task such as mount a remote filesystem on user login via pam_exec.
  • Send a log to a remote server on user login via pam_exec.
  • Send notification via Telegram on failure of a critical system crontab.
  • Send notification via Telegram when system update is available.
  • And more….
7 Comments

SSH power unleashed – Part 2: use SSH agent as an authentication provider

In the first part of this serial posts I’ve introduced SSH key and agent and their applications. In this one I’m going to take it further, let’s see how we can make use of SSH agent authentication to make things easier in other situations while still keep it secure.

A pain point of sudo

Sudo, the so commonly used tool that probably don’t need any introduction here. I find myself in dilemma while tryhing to keep secure AND convenient at the same time. By default, when you setup sudo to allow some user to own the root power, you ask him to authenticate himself by enter this password when he fires up sudo. It is usually set up by putting a line as below in /etc/sudoers file.

username        ALL=(ALL)       ALL

People like me may set up password very long so every time I ran sudo I have to type in that in a finger dance. Yes I’ve mentioned that in the last post already and I solved that by using SSH agent. Hint: it is going to help us again this time.

Some people trying to solve this problem by setting up sudo like below instead.

username        ALL=(ALL)       NOPASSWD: ALL

The NOPASSWD part makes sudo skip asking password for username when he issues sudo command, and grant him the root power silently. So as you already find out, although it is convenient, it is a huge risk – if the user accidently got hacked, the hacker is able to get root power like a piece of cake.

Let’s take the SSH agent authentication method further beyond just SSH connection

So you may wonder as I had before – we already have a way to authenticate ourselves to SSH command via the SSH agent, and skip password input in a secure way. Can we make use of that same facility on sudo? The answer is yes!

First we need to install the great pam_ssh_agent_auth package. This package provides the capability to authenticate via SSH agent in the PAM framework(I will cover that detail later in this post). This package is included in the repository of many Linux distros so just install it with your favorate package manager. As what I did on a CentOS server:

yum install pam_ssh_agent_auth

Then put a line in the file /etc/pam.d/sudo

#%PAM-1.0
auth sufficient pam_ssh_agent_auth.so file=~/.ssh/authorized_keys          <--- This is the line to put in
auth       include      system-auth
account    include      system-auth
password   include      system-auth
session    optional     pam_keyinit.so revoke
session    required     pam_limits.so

Make sure the line is located above other “auth” lines, like what I did in above example. This line tells PAM that when sudo is trying to authenticate a user, first try to use the pam_ssh_agent_auth module. If it succeeded, the user is authenticated and get sudo power. If it failed, try the next authentication method, the global system-auth method, which in most case would be asking for password.

The file=~/.ssh/authorized_keys parameter in that line tells the pam_ssh_agent_auth module to verify the user SSH agent against the public keys stored in his own home directory. You can also change that to some other file path, say, if you would like sudoers to authentication against an admin managed dedicated SSH key for sudo.

Once it is done, follow the same instructions in the last post regarding SSH agent and agent forwarding. Then you can sudo on a remote server without needing to input the password, while if you accidentally get hacked, the hacker won’t be able to sudo as he don’t have your SSH key.

The magic behind

So what happens behind all this? The major parts in this magic are: SSH agent, PAM, and the pam_ssh_agent_auth module.

I have already talked about SSH agent so I won’t repeat it. The pam_ssh_agent_auth module connects PAM and SSH agent. The key here is PAM.

What is PAM? PAM means Pluggable Authentication Modules. Here is what PAM says about itself in it’s man page.

Linux-PAM is a system of libraries that handle the authentication tasks of applications (services) on the system. The library provides a stable general interface (Application Programming Interface - API) that privilege granting programs (such as login(1) and su(1)) defer to to perform standard authentication tasks.

The principal feature of the PAM approach is that the nature of the authentication is dynamically configurable. In other words, the system administrator is free to choose how individual service-providing applications will authenticate users. This dynamic configuration is set by the contents of the single Linux-PAM configuration file /etc/pam.conf. Alternatively, the configuration can be set by individual configuration files located in the /etc/pam.d/ directory. The presence of this directory will cause Linux-PAM to ignore/etc/pam.conf.

My own sumary for PAM:

  • PAM provides a system with plugin capability, which is easy to extend for both developers and system admins. A plugin is called a module.
  • PAM provides a universal way for applications to use different methods for authentication provided by different modules. The application don’t need to change if the system provides password authentication in the beginning while later adds fingerprint authentication.
  • Each authentication method provided by PAM can be enabled / disabled / configured individually without interfering each other.
  • Different modules could be linked to gether to provide parallel or serial path of authentication flow.

PAM is a fundamental part of Linux system for many years. And many modules have been developed in this framework. The pam_ssh_agent_auth module I mentioned here, is one of them.

Sudo command, like many other applications in Linux, make use of PAM for user authentication. The file /etc/pam.d/sudo controls how sudo will make use of PAM for user authentication. Before we put the line in, it defaults to system-auth, which on most system usually is password authentication(check /etc/pam.d/system-auth if you are interested in what it does on your system). After we put the line in, sudo will first try to execute what that specific line instructs.

What the configuration line I put in /etc/pam.d/sudo does are explained below:

auth - Tells PAM this line is about a method of how to authenticate a user.
sufficient - Tells PAM that the user will be considered successfully authenticated if he passed this one, no need to try othe "auth" line after this one.
pam_ssh_agent_auth.so - Tells PAM to load and excute this module.
file=~/.ssh/authorized_keys - The parameter to the module, the meanning is already explained above.

So when a user fires up sudo, it calls PAM for authentication, PAM then look into the file /etc/pam.d/sudo and decide to try pam_ssh_agent_auth module first. This module then interacts with the SSH agent and verifys the priviate key information provided by SSH agent against the public keys in ~/.ssh/authorized_keys. If the verification susccess, the module turns back to PAM system and tells it the authentication is successful. As PAM sees the sufficient options here, it considers the whole authentication process is successful and sudo can move on. If the user has no SSH agent running, or the agent is not able to provide the correct key, the authentication attempt of this module failed, and PAM moves on to the next line, which allows the user to still authenticate with his password.

More than that

This approach is not specific to sudo. It can be enabled for any program that makes use of PAM for user authentication. So just open your mind and find out what makes you keep typing passwords. If it uses PAM, congratulations, you may save your fingers!

1 Comment

SSH power unleashed – Part 1: use private keys

I’m going to talk about some advanced tips regarding SSH in this series. Many people use it on a daily basisi, yet still only use it’s very basic functions: log into a server, do some work, log out. But SSH actually is very feature rich and flexible. It can even do something many people don’t heard before. Let’s first start with the well known public and private key authentication function.

Tip 1: never login use password. Use public key authentication instead.

The first rule of running a production server is , disable root remote login and password login for normal users. Make sure “PermitRootLogin=no” and “PasswordAuthentication=no” are in your SSHd configuration file, usually /etc/ssh/sshd_config.

You may also want to lockout your root passwd. This adds extra protection to your root power. it can be done by below command:

passwd -l root

Once root password is locked, no one can login as root with password. The only way to become root is to either login use some other authentication method, such as SSH public key, or use su/sudo to elevate from a normal user.

Now it is time to get a pair of public and private key for yourself to login. On a Linux machine, this can be done with below command.

Note: these should be done on your local machine, not your server. NEVER put your private key on remtoe server!

ssh-keygen -t rsa -b 2048

Then follow the on screen message to provide file name to save the keys(the default is OK), and the passphrase.

If you want ultra-secure keys, just raise the key bits given for -b option. 2048 is sufficient for nowadays, 4096 may give you longer confidence.

Now look at your ~/.ssh/ directory, your new keys are there. The public key file is named id_rsa.pub, and the private key file is named id_rsa. The public key file can be disclosed to the world, while the private key file should be kept safe, as safe as how you keep the key of your home, may be even more 😉 .

Now upload the public key file id_rsa.pub to your remote server, and make it useable for SSH. It can be done as below.

cat id_rsa.pub >> ~/.ssh/authorized_keys

This puts the content of your newly generated public key into the authorized_keys file, which will be check by SSH upon your login.

Now login from your local machine to the server with key authentication, you need to provide the passphrase of the private key instead of your system account password. And you can safely disable password login in SSHd configuration now.

Tip 2: use ssh-agent to cache your keys

Imagine that you manage a few servers, and you need to frequently login to perform some tasks. It’s going to be a pain to enter the password of the keys every time. It is quite boring for myself – my password is quite long and made of non-sense characters, everytime I enter the password it feels like a finger dance!

The ssh-agent command is made for such purpose. It can be used to invoke a shell and cache yoru private key in memory. Next time when a SSH key is requested, it will provide the data directly without requiring your to enter the password again. Let’s see how to do that.

First start a shell by ssh-agent:

ssh-agent /bin/bash

It seems nothing happend, you are just dropped back to shell prompt. But actually you are in a newly invoked shell now. In this shell, ssh-agent caches the private key password for you.

Now load your private key(s).

ssh-add

It will prompt you the file name of the current private key it going to load, ask you to input the password. Once you are done, the priviate key and its password is cached in memory. Now try to log into your remote server which has the public key installed. You will notice that no password is required during SSH login!

Tips 3: start ssh-agent upon login

Above tip makes life easier, let’s move on to see how we can make it even more easier.

Instead of start ssh-agent every time after you logged in your local machine, it is possible to start it automatically. If you are using Bash as your login shell, put below at the end of your ~/.bashrc file.

eval $(/usr/bin/ssh-agent -s)

/usr/bin/ssh-add

And below at the end of your ~/.bash_logout file.

if [ -n "$SSH_AGENT_PID" ] ; then
    /usr/bin/ssh-agent -k
fi

During your login, you will notice the message of “Agent PID xxxxx”, which means your newly added code in ~/.bashrc just ran and started ssh-agent and setup the environment for you. Then the ssh-add command asks you to load your private keys. Type in your password and your ssh-agent is up running just like described in Tip 2. You can SSH login to remote servers without typing in password.

When you logout, the code in ~/.bash_logout will make sure the agent is killed so no key data is left in memory.

Tip 4: use SSH agent forwarding

Still remember that I mentioned above the private key should be kept safe and never uploaded to any remote servers? What if you need to log into another server(let’s call it server B) from your remote server(let’s call it server A)? You don’t have the private key on server A. To log into server B, you have to either use password authentication, or from your local machine that have the private key. But what if you do need to connect from server A to B and don’t want to use the less secure password authentication way?

This is how SSH agent forwarding comes to save you. With this technology, you an “forward” the encrypted information of your private key located on your local machine, via server A, to server B, without actually copying the key file to server A. Let’s see how we can do this.

When you start the connection to server A, first make sure you followed Tip 2 & 3 and in the ssh-agent shell with private key loaded, then use the -A option like below:


ssh -A -p <remote_ssh_port> [email protected]_A

The -A option tells SSH program to forward local ssh-agent to server A. When you use SSH there, it can acccess the private key via the secure SSH connection between your local machine and server A.

Now if on server A you need to connect to server B, just run SSH and you will notice that no password is required, you just log into server B with your private key on your local machine!

If you are using PuTTY on Windows as your SSH client, the agent forwarding option is under Connection -> SSH -> Auth -> Authentication Parameters -> Allow agent forwarding, as shown in below screenshot.

Caution: always make sure server A is trusted before you forward your agent to it.

Leave a Comment