How to configure rsyslog 7.4.9 with TLS

07-10-2019

Linux

How to configure rsyslog 7.4.9 with TLS

If you’re looking to encrypt syslog transmissions between client and server, you can do so via rsyslog with TLS. Please see the external link to rsyslog.com below, in partiuclar the Overview section of the page describing encrypting syslog traffic with TLS (SSL). Also, if you’re an RHN subscriber, check out the RHN solution article.

In looking at the links, you may notice both pertain to rsyslog versions < 5.8.10, and the syntax for loading modules has changed in 7.4.9 and therefore examples provided on TLS configuration aren’t necessarily clear cut. In this article, I will show you how I combined these two external articles, and some insight garnered from rsyslog knowledgebase site kb.monitorware.com to work out the proper syntax for the latest stable release, rsyslog 7.4.9 (as of this writing).

As of this writing, I actively have over 1800 clients actively connected to my rsyslog server:

[root@rslserver ~]# netstat -an | grep ":514 " | wc -l  
1865

Specifications

Below are the specifications I used:

rsyslog certificate authority (CA) server:
- Red Hat Enterprise Linux 6 x86_64
  - with gnutls-utils-2.8.5-10.el6_4.2 package installed
rsyslog server and rsyslog client:
- CentOS Linux 5.6 x86_64
  - with the following packages installed directly from the v7-stable rsyslog repository
    - rsyslog-7.4.9-2.el5.centos
    - rsyslog-gnutls-7.4.9-2.el5.centos

How to configure rsyslog with TLS

rsyslog CA server -> linux01
rsyslog server -> rslserver
rsyslog client -> rslclient

Setup the CA (Certificate Authority)

The first thing that needs to be done is to setup the certificate authority, meaning we’ll be working with our RHEL6 server, linux01 to genereate a private key for the CA certificate and create a self signed certificate:

Generate a private key for the CA certificate

[root@linux01 ~]# certtool --generate-privkey --outfile ca-key.pem  
Generating a 2048 bit RSA private key...

Generate a self-signed CA Certificate. I set a 10 year expiration period.

[root@linux01 ~]# certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem  
Generating a self signed certificate...  
Please enter the details of the certificate's distinguished name. Just press enter to ignore a field.  
Country name (2 chars): US  
Organization name: kristianreese.com  
Organizational unit name: KB  
Locality name: STL  
State or province name: MO  
Common name: cacert  
UID:  
This field should not be used in new certificates.  
E-mail:  
Enter the certificate's serial number in decimal (default: 1395159808):  

Activation/Expiration time.  
The certificate will expire in (days): 3650  

Extensions.  
Does the certificate belong to an authority? (y/N): y  
Path length constraint (decimal, -1 for no constraint):  
Is this a TLS web client certificate? (y/N):  
Is this also a TLS web server certificate? (y/N):  
Enter the e-mail of the subject of the certificate: kb@kristianreese.com  
Will the certificate be used to sign other certificates? (y/N): y  
Will the certificate be used to sign CRLs? (y/N):  
Will the certificate be used to sign code? (y/N):  
Will the certificate be used to sign OCSP requests? (y/N):  
Will the certificate be used for time stamping? (y/N):  
Enter the URI of the CRL distribution point:  
X.509 Certificate Information:  
[...]  

Is the above information ok? (Y/N): y  

Signing certificate...

[root@linux01 ~]# ls -l  
total 136  
-rw-------  1 root root  1675 Mar 18 11:12 ca-key.pem  
-rw-r--r--  1 root root  1318 Mar 18 12:24 ca.pem  
[root@linux01 ~]#

ca-key.pem is like the key to your car, or your house. You wouldn’t give those to a stranger, or leave them laying around in a public place where they may get stolen. Treat this file in the same regard. Nobody but the server on which it was generated needs to have it, so keep it permissioned as 600.

Generate certificates for the rsyslog client and rsyslog server

Generate a private key for the rsyslog client (rslclient) server:

[root@linux01 ~]# certtool --generate-privkey --outfile rslclient-key.pem --bits 2048  
Generating a 2048 bit RSA private key...

Generate a certificate request for the rsyslog client (rslclient) server. Please be sure to enter the name of your system in the Common name field. The rsyslog authmode will be X509/name, meaning the certificate issued by the CA (linux01) is binding the public key to the Common name provided, and thus is the part that must match the host being authenticated.

[root@linux01 ~]# certtool --generate-request --load-privkey rslclient-key.pem --outfile request.pem  
Generating a PKCS #10 certificate request...  
Country name (2 chars): US  
Organization name: kristianreese.com  
Organizational unit name: KB  
Locality name: STL  
State or province name: US  
Common name: webserver01.kristianreese.com  
UID:  
Enter a dnsName of the subject of the certificate:  
Enter the IP address of the subject of the certificate:  
Enter the e-mail of the subject of the certificate:  
Enter a challenge password:  
Does the certificate belong to an authority? (y/N): n  
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N):  
Will the certificate be used for encryption (RSA ciphersuites)? (y/N):  
Is this a TLS web client certificate? (y/N): y  
Is this also a TLS web server certificate? (y/N): y  
[root@linux01 ~]#

If you have multiple clients belonging to the same subdomain, you could use a wildcard in the Common name field to cover all of your clients with this one certificate. Simply input the Common name as *.example.com

Generate the machine certificate using the request from step 2, and sign it with the Certifiate Authorities private key

[root@linux01 ~]# certtool --generate-certificate --load-request request.pem --outfile rslclient-cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem  
Generating a signed certificate...  
Enter the certificate's serial number in decimal (default: 1395162401):  

Activation/Expiration time.  
The certificate will expire in (days): 3650  

Extensions.  
Do you want to honour the extensions from the request? (y/N):  
Does the certificate belong to an authority? (y/N): n  
Is this a TLS web client certificate? (y/N): y  
Is this also a TLS web server certificate? (y/N): y  
Enter a dnsName of the subject of the certificate: webserver01.kristianreese.com  
Enter a dnsName of the subject of the certificate:  
Enter the IP address of the subject of the certificate:  
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N):  
Will the certificate be used for encryption (RSA ciphersuites)? (y/N):  
X.509 Certificate Information:  
[...]  

Is the above information ok? (Y/N): y  

Signing certificate...  
[root@linux01 ~]#

If you have multiple clients belonging to the same subdomain, you could use a wildcard in the dnsName field to cover all of your clients with one certificate. Simply input the dnsName as *.example.com (use your own domain name)

The request.pem file is no longer needed and may be removed. Things should look as follows:

[root@linux01 ~]# rm -f request.pem  
    [root@linux01 ~]# ls -l  
    total 144  
    -rw-------  1 root root  1675 Mar 18 11:12 ca-key.pem  
    -rw-r--r--  1 root root  1318 Mar 18 12:24 ca.pem  
    -rw-r--r--  1 root root  1424 Mar 18 13:07 rslclient-cert.pem  
    -rw-------  1 root root  1675 Mar 18 12:42 rslclient-key.pem  
    [root@linux01 ~]#

Repeat steps 1 through 4, but this time for the rsyslog server. In summary:
1. certtool --generate-privkey --outfile rslserver-key.pem --bits 2048
2. certtool --generate-request --load-privkey rslserver-key.pem --outfile request.pem
  1. Common name = FQDN of your rsyslog server -> rslserver.kristianreese.com
3. certtool --generate-certificate --load-request request.pem --outfile rslserver-cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem
  1. dnsName = FQDN of your rsyslog server -> rslserver.kristianreese.com
4. rm -f request.pem

Your directory should contain something like:

[root@linux01 ~]# ls -l  
total 148  
-rw-------  1 root root  1675 Mar 18 11:12 ca-key.pem  
-rw-r--r--  1 root root  1318 Mar 18 12:24 ca.pem  
</span><span style="color: rgb(0, 255, 0);"><span style="color: rgb(0, 255, 0);">-rw-r--r--  1 root root  1424 Mar 18 13:07 rslclient-cert.pem  
-rw-------  1 root root  1675 Mar 18 12:42 rslclient-key.pem</span>  
-rw-r--r--  1 root root  1415 Mar 18 13:42 rslserver-cert.pem  
-rw-------  1 root root  1675 Mar 18 13:40 rslserver-key.pem  
[root@linux01 ~]#

This completes the setup of the certificates. With the exception of ca-key.pem, these certificates need to be copied to their respective systems before we can continue with configuring rsyslog to use them. Remember to keep ca-key.pem safe! Nobody should have a copy of this file except the CA. In addition, each client-key should only be copied to the machine for which it was generated, except in the case of wildcard certificates in which case only those within the identified subdomain should receive a copy.

Copy certificates from CA to rsyslog server and rsyslog client

Where do the files get copied?

When the rsyslog rpm is installed, it creates a directory /etc/pki/rsyslog/. This is a good place to copy the files to:

Simply scp a copy of ca.pem, rslclient-cert.pem, and rslclient-key.pem to your rsyslog client system(s) /etc/pki/rsyslog/.
Simple scp a copy of ca.pem, rslserver-cert.pem, and rslserver-key.pem to your rsyslog server (the collector) /etc/pki/rsyslog/.

It’s worth double checking that the certificate files ownerships are set to root:root and 0600.
~# chmod 0600 /etc/pki/rsyslog/*

Configure rsyslog server

After installing the two rsyslog packages listed in the specifications section above (ie):

[root@rslserver ~]# rpm -qa | grep rsyslog  
rsyslog-7.4.9-2.el5.centos  
rsyslog-gnutls-7.4.9-2.el5.centos

you are ready to configure rsyslog.

Begin by editing /etc/rsyslog.conf and adding the following just after the defined MODULES section:

# Increase the amount of open files rsyslog is allowed, which includes open tcp sockets  
# This is important if there are many clients.  
http://www.rsyslog.com/doc/rsconf1_maxopenfiles.html  
$MaxOpenFiles 2048  

# make gtls driver the default  
$DefaultNetstreamDriver gtls  

# certificate files generated on RHEL6 and stored in /root  
$DefaultNetstreamDriverCAFile /etc/pki/rsyslog/ca.pem  
$DefaultNetstreamDriverCertFile /etc/pki/rsyslog/rslserver-cert.pem  
$DefaultNetstreamDriverKeyFile /etc/pki/rsyslog/rslserver-key.pem  

# Provides TCP syslog reception  
# for parameters see http://www.rsyslog.com/doc/imtcp.html  
module(load="imtcp"  
       MaxSessions="2000"  
       StreamDriver.mode="1"  
       StreamDriver.authmode="x509/name"  
       PermittedPeer="*.kristianreese.com")  
input(type="imtcp" port="514" name="tcp-tls")

For any concerns over the use of wildcard SSL certificates, consider using the $AllowedSender directive option available to the rsyslog configuration.

restart rsyslog:

/etc/init.d/rsyslog restart

Verify /var/log/messages did not log an error and that rsyslog is running with ps -ef | grep rsyslog

Configure rsyslog client

Again, after installing the same rsyslog packages as the rsyslog server, you are ready to configure rsyslog. For each client, instead of modifying /etc/rsyslog.conf, I created an include file within the /etc/rsyslog.d directory:

[root@0rslclient ~]# cat /etc/rsyslog.d/tls.conf  
# make gtls driver the default  
$DefaultNetstreamDriver gtls  

# certificate files  
$DefaultNetstreamDriverCAFile /etc/pki/rsyslog/ca.pem  
$DefaultNetstreamDriverCertFile /etc/pki/rsyslog/rslclient-cert.pem  
$DefaultNetstreamDriverKeyFile /etc/pki/rsyslog/rslclient-key.pem  

#### GLOBAL DIRECTIVES ####  

# Use default timestamp format  
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat  

# gtls Network Stream Driver  
#  x509/name - certificate validation and subject name authentication  
#  http://www.rsyslog.com/doc/ns_gtls.html  
$ActionSendStreamDriverAuthMode x509/name  
$ActionSendStreamDriverPermittedPeer rslserver.kristianreese.com  
$ActionSendStreamDriverMode 1 # run driver in TLS-only mode  

:msg, contains, "Failed password"  
authpriv.*                                              @@rslserver.kristianreese.com:514  
[root@rslclient ~]#

In this example, I am using a property-based filter (:msg) to capture only syslog messages that contain “Failed password” as commonly found in /var/log/secure during invalid login attempts. The @@ is intentional and tells rsyslog we’re wanting to send this message to a remote host.

restart rsyslog:

/etc/init.d/rsyslog restart

Verify /var/log/messages did not log an error and that rsyslog is running with ps -ef | grep rsyslog

Validation

The configuratio is now ready for validation. Login to the rslserver, and begin tailing /var/log/secure with the command:

~# tail -f /var/log/secure

Using another session, test by ssh’ing into the rslclient, and enter invalid credentials (remember we’re only capturing messages with the string “Failed password”). After the failed login attempts, the messages should display from the tail output on the rslserver.

If you want to log remote messages into their own log file, you can use filters… something like:

...
# The authpriv file has restricted access.  
:hostname, isequal, "rslserver"  
authpriv.*                                              /var/log/secure  
:msg, contains, "Failed password"  
authpriv.*                                              /var/log/secure.remote  
...

This setup logs anything destined for authpriv from the rslserver to its normal log /var/log/secure, and to /var/log/secure.remote. If you want messages to stop processing after an action, then read up on rsyslog.com about the stop directive.

If the message received is from an allowed client, and contains the phrase “Failed password”, it is only logged to /var/log/secure.remote.

Verification:

From the client machine, capture packets for examination of the before TLS and after TLS config to see for youself that the traffic is being encrypted. This can be used for importation into wireshark, for example.

rslclient ~# tcpdump -nnvvvS -s 0 -U -w /tmp/sniff.rsyslog dst <ip of rslserver> and dst port 514

Otherwise, from command line, you can simply observe the traffic:

Before

To listen/watch/verify for encrypted traffic, run tcpdump on the rslserver:
~# tcpdump -nn -s2048 -X host rslclient.domain.com

Here are some screenshots of a test I ran before configuring TLS to show that the traffic is in fact being encrypted after following the above steps:

AFTER

after TLS has been configured, run the tcpdump command again with a different file name:

~# tcpdump -nnvvvS -s 0 -U -w /tmp/sniff.myVPS.rsyslog dst <rslserver IP> and dst port 514

END

new home of kb.kristianreese.com

uncategorized

How to configure rsyslog 7.4.9 with TLS

How to configure rsyslog 7.4.9 with TLS

Specifications

How to configure rsyslog with TLS

Setup the CA (Certificate Authority)

Generate certificates for the rsyslog client and rsyslog server

Copy certificates from CA to rsyslog server and rsyslog client

Configure rsyslog server

Configure rsyslog client

Validation

Verification:

Before

AFTER