Category Archives: cloud

TLS Certificates 101

Over the last couple years I’ve been involved in many projects that require TLS certificates and some other technologies to provide security and establish encryption in transit for network communications. These technologies involve different concepts, protocols and standards such as mTLS, X509, PKI, digital signatures, hmac, symmetric and asymmetric encryption, different cryptographic algorithms etc and can feel very overwhelming especially for people that are new in the topic. 

I decided to write a quick blog post and share some of the lessons I’ve learned over the years. It is my intention to provide some guidance to newcomers when it comes to debugging these kinds of issues.

I don’t consider myself an expert in the whole TLS topic (I’m not a Cryptographer), however I’ve spent considerable time on this to the point that I feel very confident when debugging most issues related to TLS.

I firmly believe an expert is a person who has made so many errors and mistakes in the past and because of that he has accumulated so much knowledge to the point that he knows almost all the answers. I still need to make more 😉

Having said that, let’s start our journey into the TLS world.

What is TLS?

TLS, short for Transport Layer Security, is a cryptographic protocol designed to provide communications security over a computer network.

The TLS protocol is a complex piece of engineering with many security mechanisms and moving parts used to achieve security of data in transit by combining symmetric and asymmetric encryption techniques, if you want to learn more details about this technology I highly recommend to look at the TLS Wikipedia page, also here is an awesome diagram by Wazen Shbair that shows the different steps that happen during the TLS handshake process such as choosing a cipher suite or exchanging a shared key.

There are many different versions for this protocol but for practical purposes the only thing you need to know is:

  • TLS 1.0 and TLS 1.1 are known to be very vulnerable so you should never use them
  • TLS 1.2 is considered to be much more secure and is recommended to use
  • TLS 1.3 is an improved version of TLS 1.2 in terms of performance

What are TLS Certificates?

Excellent, now that you know TLS is the technology behind TLS certificates you may be wondering, what are TLS certificates?.

A TLS certificate is an implementation of a X509 Certificate, X509 is a standard that defines the file format used to store information related to an entity (among other things). This is very important because the main purpose of a certificate is to provide an identity to an entity, an entity can be anything, such as a website domain, a server, a piece of software, a workstation, a laptop or even a person. Similar to real life, people (entities) have birth certificates and driver’s licenses that prove who they are, this documents are backed up by government institution that we all trust (well … kinda), if you ask someone to prove their identity that person will probably show you their ID and if the ID looks damaged or you think there’s something phishy you can ask for additional forms of identification until you are convinced that you can trust them.

A TLS certificate will looks like this:

The TLS certificate contains many different fields like:

  • Subject name: the entity name, person name, website domain, etc
  • Issuer name: the authority that issued the certificate
  • Period of validity: the certificate is not valid before or after this dates
  • Additional cryptographic information and digital signatures

When an application (like your web browser) connects to a website by typing the IP address or the domain, ie: www.alevsk.com, the server behind will return a TLS certificate, the browser will then look at it and decide what to do next (exchange keys and establish a secure connection) based on the fields above.

As you can see, it is the client’s responsibility to verify these TLS certificates, the server may offer a perfectly fine certificate and the client could still choose to throw it away. With that said, I decided to write this blog post to share my experience debugging server applications during countless hours just to find the issue was on the client side or in the certificate itself, customers will swear the server is broken and not working when in reality it was their client not trusting the certificate authority or the clock their was misconfigured.

But before doing that I want to show you a couple of examples of TLS certificates being used on some applications, for that I have prepared a couple of code snippets.

Suppose you are running a simple web server written in go like this.

package main

import (
  "fmt"
  "net/http"
)

func hello(w http.ResponseWriter, req *http.Request) {
  secret := req.URL.Query().Get("secret")
  fmt.Fprintf(w, fmt.Sprintf("pong %s\n", secret))
}

func main() {
  address := "0.0.0.0:8080"
  http.HandleFunc("/ping", hello)
  fmt.Println("Starting server on", address)
  if err := http.ListenAndServe(address, nil); err != nil {
     return
  }
}

Let’s query the server running on port 8080 via CURL.

curl http://localhost:8080/ping\?secret\=eaeae
pong eaeae

With curl I’m performing a GET request and the server is replying fine, notice how I’m passing a secret via the secret parameter in the URL, everything looks good so far but there’s a problem.

This is an insecure web server, hence all the traffic going from the client (curl command) to the server (Go application) can be captured and the secret can be retrieved in plain text.

Let’s fix this by adding a TLS certificate to this web server, but first, how do we get one? Well, there are two types of certificates: self-signed certificates and certificates issued/signed by a certificate authority.

A certificate authority is an entity that stores, signs, and issues digital certificates. Going back to the society metaphor, it will be the equivalent to a government institution that many people trust. 

  • Self-signed certificate: ID document that you crafted yourself and its not valid
  • Certificate signed by a certificate authority: birth certificate, drivers license or ID document that is backed up by the government institution
  • Certificate authority: The government institution that most people trust by default

Using a TLS certificate signed by a certificate authority has many advantages that I’m going to discuss in some other blog post, for now it’s enough to say that you have to pay in most cases to get one of those but it’s worth it. Now let’s generate some certificates.

Tools for playing with TLS certificates

I’ll focus on self-signed certificates for this example and I’m going to show you how to generate them using three different tools.

Mkcert

Mkcert is a tool created by Filippo Valsorda. Mkcert is a simple zero-config tool to make locally trusted development certificates with any names you’d like, you can download it and installed following the instructions directly from this Github repository https://github.com/FiloSottile/mkcert

mkcert "localhost"

#Created a new certificate valid for the following names 📜
# - "localhost"

#The certificate is at "./localhost.pem" and the key at #"./localhost-key.pem" ✅

#It will expire on 14 December 2024 🗓

Certgen

Certgen is a tool created by MinIO, the startup I currently work with. Certgen is a dead simple tool to generate self signed certificates, you can download it and installed following the instructions directly from this Github repository https://github.com/minio/certgen

certgen --host localhost
#2022/09/14 21:58:54 wrote public.crt
#2022/09/14 21:58:54 wrote private.key

OpenSSL 

OpenSSL is a tool that doesn’t require any introduction, it has been part of the TLS arsenal of system administrators and network engineers for decades, if you wish to use openssl to generate certificates the process is a little bit more manual than with the tools above but still is fairly simple to use.

openssl genrsa -out private.key 2048

cat >> certificate.cnf << 'END'
[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
O = "http-server"
C = US
CN  = "localhost"

[req_ext]
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
END

openssl req -new -config certificate.cnf -key private.key -out certificate.csr

openssl x509 -signkey private.key -in certificate.csr -req -days 365 -out public.crt

#Certificate request self-signature ok
#subject=O = http-server, C = US, CN = localhost

Doesn’t really matter which tools you use to get your certificates as long as they are not malformed. Going back to our Go code I made a couple of changes and the code looks like this now.

package main

import (
  "fmt"
  "net/http"
)

func helloTLS(w http.ResponseWriter, req *http.Request) {
  secret := req.URL.Query().Get("secret")
  fmt.Fprintf(w, fmt.Sprintf("pong %s\n", secret))
}

func main() {
  address := "0.0.0.0:8080"
  http.HandleFunc("/ping", helloTLS)
  fmt.Println("Starting server on", address)

  if err := http.ListenAndServeTLS(address, "public.crt", "private.key", nil); err != nil {
     return
  }
}

The most relevant change is that I’m using the function http.ListenAndServeTLS now which is pretty explanatory. This function allows me to pass two files: the TLS certificate (also called the public key) and the corresponding private key. As mentioned before the TLS certificate contains information about the expiration date, valid domains, digital signature, certificate authority, among other, that will be relevant to the client while the private key will remain secret in the server and be used by the Go application to decrypt encrypted messages sent by the client during the TLS handshake. If you want to learn more about Public Key Cryptography I highly recommend looking at the PKI Wikipedia page.

I’ll run the Go program again and this time I’ll use the https protocol in the curl command.

curl https://localhost:8080/ping\?secret\=eaeae -k
pong eaeae

Before continuing I have to mention, just for the sake of the example and to demonstrate how TLS is securing the communication, I’ve included the -k flag in the curl command, passing this flag will make curl disable all TLS verifications. If you inspect the traffic on wireshark this time you’ll only see the TLS handshake and encrypted data after that, no more parameters in plain text.

TLS certificates

If you are not able to inspect SSL/TLS traffic in wireshark try adding the custom server port under Edit -> Preferences -> Protocols -> HTTP -> SSL/TLS Ports. You can add your custom port, ie: 8080. Change it to: 443,8080.

Debugging TLS certificate issues

Now comes the fun part, and where I’ve “invested” countless hours trying to fix other people’s problems, debugging TLS certificates issues.

If I go back and remove the -k flag from my curl command I get the following output.

curl https://localhost:8080/ping\?secret\=eaeae                                     
#curl: (60) SSL certificate problem: self-signed certificate
#More details here: https://curl.se/docs/sslcerts.html

#curl failed to verify the legitimacy of the server and therefore could not
#establish a secure connection to it. To learn more about this situation and
#how to fix it, please visit the web page mentioned above.

The error message is pretty clear, curl is failing to verify the certificate because this is a self-signed certificate, meaning it cannot be trusted. How does curl know this is a self-signed certificate? 

Well it is very simple, remember when I said there were only two types of certificates? 

There are two fields in the certificate: Subject Name and Issuer Name, if both fields match then this is a self-signed certificate. If they don’t then this is a certificate issued/signed by a certificate authority, that may or may not be trusted by the client.

The solution for this is quite simple, in your client application use the same certificate itself to verify its authenticity, since the certificate is technically its own authority, with curl you need to include the –cacert flag and specify the path to the public.crt used by the server.

curl https://localhost:8080/ping\?secret\=eaeae --cacert public.crt
pong eaeae

The above is one of the most common TLS issues I’ve encountered in the past few years and I hope I did a good job explaining the root cause, how to approach the problem and finally how to find a solution.

Next is a list of the most common TLS issues I’ve seen and some advices on how to debug them:

TLS ErrorDebugging advice
SSL certificate problem: self-signed certificateAsk people to provide you with the public key of the server (or download it yourself with openssl, ie: echo | openssl s_client -servername www.alevsk.com -connect www.alevsk.com:443 | sed -n ‘/^—–BEGIN CERT/,/^—–END CERT/p’ > public.crt) and then pass it with the –cacert flag in the curl command
SSL certificate problem: unable to get local issuer certificateUpdate ca certificates in the client machine (sudo update-ca-certificates) or ask for the ca.crt (certificate authority certificate) and pass it with the –cacert flag in the curl command
X.509 Certificate Signed by Unknown AuthorityClient doesn’t trust the certificate authority that issued the certificate, if you have the ca.crt files you can use openssl to verify the chain of trust: openssl verify -verbose -CAfile root.pem -untrusted intermediate.pem server.pem 
SSL: no alternative certificate subject name matches target host name ‘XXXX’The certificate used by the server is not valid for any of the domains the client is trying to use to connect, are you using an IP address instead of a domain name? Check the whole list of domains inside the certificate and make sure they match to what you are using in your curl command
SSL certificate problem: certificate has expiredMake sure the client and server clocks and correctly configured and in sync
SSL certificate problem: certificate is not yet validMake sure the client and server clocks and correctly configured and in sync
SSL: certificate subject name ‘XXXX’ does not match target host name YYYYThe certificate used by the server is not valid for any of the domains the client is trying to use to connect, are you using an IP address instead of a domain name? Check the whole list of domains inside the certificate and make sure they match to what you are using in your curl command
Public key certificate and private key doesn’t matchPretty explanatory, I’ve seen this happening mostly when people copy a bunch of keypairs around and get confused, you can use openssl to verify this
Connection error:The SSL connection could not be established, see inner exception..Check which version of TLS is the client using, most probably is using an old vulnerable version like 1.0 or 1.1, you need to upgrade your client TLS version to TLS 1.2 or above.

Test this using:

curl --tls-max 1.0 https://<endpoint>

As I remember more errors and ways of how I approach the problem to find a solution I may add them to the list, most of the errors are very explanatory but for some reason users struggle with them, when it comes to TLS errors they may think it’s some kind of obscure or arcane technology but its not.

Takeaway

As I said before, I’ve spent countless hours debugging this kind of issues, my main advice will be: instead of jumping directly into pulling server/application logs first look as the client side, always use the curl command first for testing, if the customer provide you with some client certificates for mTLS authentication or a ca.crt file and you are not even make them work with curl then definitely the issue is in the client side and not in your application.

  • Pay attention to how the client is referencing the server application, what domain the client is using in the curl command?
  • Make sure the client and server clocks and correctly configured and in sync
  • If the certificate contains wildcards make sure those are valid for the domain the server is using and the client is referencing
  • Make sure the public key and the private key matches, you can use OpenSSL to verify this

Here are some useful TLS certificates debugging tools I use:

Happy hacking

Compilation of open-source security tools & platforms for your Startup

This compilation of open-source tools aim to provide resources you can use for some of the step of the secure development life cycle of your organization, ie:

  1. Security Training
  2. Security Architecture Review
  3. Security Requirements
  4. Threat Modeling
  5. Static Analysis
  6. OpenSource Analysis
  7. Dynamic Analysis
  8. Penetration Testing

If you think I should add a new tool to the list you can open a Github issue or send a PR directly.

User management

Secret management

IDS, IPS, Firewalls and Host/Network monitoring

Data visualization

Web Application Firewall

Object Storage

VPN

Security training platforms

Static analysis tools

Dynamic analysis tools

Misc

Stop passing secrets via environment variables to your application

docker inspect command showing container secrets

Environment variables are great to configure and change the behavior of your applications, however there’s a downside for that, if someone uses the `docker inspect` command your precious secrets will get revealed there, because of that you should never pass any sensitive data to your container using environment variables (the `-e` flag), I’ll show you an example.

Suppose you have a simple python application (Download the source code of the app here: https://github.com/Alevsk/docker-containers-env-vars-security ) that returns the hmac signature for a provided message using a configured secret, the code will look like this:

app_secret = os.environ.get('APP_SECRET')
if app_secret is None:
    app_secret = ""

# derive key based on configured APP_SECRET
salt = binascii.unhexlify('aaef2d3f4d77ac66e9c5a6c3d8f921d1')
secret = app_secret.encode("utf8")
key = pbkdf2_hmac("sha256", secret, salt, 4096, 32)

app = Flask(__name__)

@app.route("/")
def hello():
    message = request.args.get('message')
    if message is None:
        return "Give me a message and I'll sign it for you"
    else:
        h = hmac.new(key, message.encode(), hashlib.sha256)
        return "<b>Original message:</b> {}<br/><b>Signature:</b> {}".format(message, h.hexdigest())

if __name__ == "__main__":
    app.run()

Pretty straightforward, then you build the docker image with:

docker build -t alevsk/app-env-vars:latest .

And run the application on a container with:

docker run --rm --name hmac-signature-service -p 5000:5000 -e APP_SECRET=mysecret alevsk/app-env-vars:latest

Test the endpoints works fine running a `curl` command:

curl http://localhost:5000/?message=eaeae

<b>Original message:</b> eaeae<br/><b>Signature:</b> cce4625d3d586470bc84ac088b6e2ae24c012944832d54ab42a999de94252849%

Perfect, everything works as intended, however if you inspect the running container the content of `APP_SECRET` is leaked.

docker inspect hmac-signature-service
    ..
    ...
    "Env": [
        "APP_SECRET=mysecret",
        "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "LANG=C.UTF-8",
        "GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568",
        "PYTHON_VERSION=3.9.9",
        "PYTHON_PIP_VERSION=21.2.4",
        "PYTHON_SETUPTOOLS_VERSION=57.5.0",
        "PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/3cb8888cc2869620f57d5d2da64da38f516078c7/public/get-pip.py",
        "PYTHON_GET_PIP_SHA256=c518250e91a70d7b20cceb15272209a4ded2a0c263ae5776f129e0d9b5674309"
    ],
    ...
    ..

Additionally, you can get the `process id` of the app inside the container (`1869799` in this case) and then look at the content of the `/proc/[pid]/environ` file and your application secret will be there too.

docker inspect hmac-signature-service
       ...
       ..
       .
       "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 1869799,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-12-14T08:40:24.338541774Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },

pstree -sg 1869799

systemd(1)───containerd-shim(1869776)───gunicorn(1869799)─┬─gunicorn(1869799)
                                                          ├─gunicorn(1869799)
                                                          ├─gunicorn(1869799)
                                                          └─gunicorn(1869799)

sudo cat /proc/1869799/environ

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binHOSTNAME=6a9110ff90e1APP_SECRET=mysecretLANG=C.UTF-8GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568PYTHON_VERSION=3.9.9PYTHON_PIP_VERSION=21.2.4PYTHON_SETUPTOOLS_VERSION=57.5.0PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/3cb8888cc2869620f57d5d2da64da38f516078c7/public/get-pip.pyPYTHON_GET_PIP_SHA256=c518250e91a70d7b20cceb15272209a4ded2a0c263ae5776f129e0d9b5674309HOME=/root%

Mount secret file to the container and read from there instead

You can fix that by doing a small change in the application source code, mainly how the application reads the secret, this time instead of reading from the `APP_SECRET` environment variable the app will read from a file located at `/tmp/app/secret`.

secret_path = "/tmp/app/secret"
app_secret = open(secret_path).readline().rstrip() if os.path.exists(secret_path) else ""

Build the docker image and run the container mounting the secret file.

docker build -t alevsk/app-env-vars:latest .
docker run --rm --name hmac-signature-service -p 5000:5000 -v $(pwd)/secret:/tmp/app/secret alevsk/app-env-vars:latest

The secret file looks like this:

mysecret

Try `curl` again:

curl http://localhost:5000/?message=eaeae

<b>Original message:</b> eaeae<br/><b>Signature:</b> cce4625d3d586470bc84ac088b6e2ae24c012944832d54ab42a999de94252849%

The generated signature looks good, if you inspect the container you will not see the secret used by the application.

docker inspect hmac-signature-service
    ..
    ...
    "Env": [
        "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "LANG=C.UTF-8",
        "GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568",
        "PYTHON_VERSION=3.9.9",
        "PYTHON_PIP_VERSION=21.2.4",
        "PYTHON_SETUPTOOLS_VERSION=57.5.0",
        "PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/3cb8888cc2869620f57d5d2da64da38f516078c7/public/get-pip.py",
        "PYTHON_GET_PIP_SHA256=c518250e91a70d7b20cceb15272209a4ded2a0c263ae5776f129e0d9b5674309"
    ],
    ...
    ..

Also, if you exec into the container by running `docker exec -it hmac-signature-service sh` the `APP_SECRET` environment variable is not there, nor in the `/proc/1/environ` file or in the `printenv` command output.

cat /proc/1/environ

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binHOSTNAME=afe80d4cafcaLANG=C.UTF-8GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568PYTHON_VERSION=3.9.9PYTHON_PIP_VERSION=21.2.4PYTHON_SETUPTOOLS_VERSION=57.5.0PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/3cb8888cc2869620f57d5d2da64da38f516078c7/public/get-pip.pyPYTHON_GET_PIP_SHA256=c518250e91a70d7b20cceb15272209a4ded2a0c263ae5776f129e0d9b5674309HOME=/root

printenv

HOSTNAME=afe80d4cafca
PYTHON_PIP_VERSION=21.2.4
SHLVL=1
HOME=/root
GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/3cb8888cc2869620f57d5d2da64da38f516078c7/public/get-pip.py
TERM=xterm
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
PYTHON_VERSION=3.9.9
PYTHON_SETUPTOOLS_VERSION=57.5.0
PWD=/app
PYTHON_GET_PIP_SHA256=c518250e91a70d7b20cceb15272209a4ded2a0c263ae5776f129e0d9b5674309

What if I cannot change the source code of my application?

In case you don’t have access or cannot change the source code of the application not all is lost This time, instead of passing the `APP_SECRET` environment variable via docker `-e` flags, you will mount a `secret` file directly into the container and then modify the container entry point to source from that secret file.

The `secret` file will contain something like this:

export APP_SECRET="mysecret"

Launch the container like this:

docker run --rm --name hmac-signature-service -p 5000:5000 -v $(pwd)/secret:/tmp/app/secret --entrypoint="sh" alevsk/app-env-vars:latest -c "source /tmp/app/secret && gunicorn -w 4 -b 0.0.0.0:5000 main:app"

This will prevent the `APP_SECRET` environment variable to be displayed if someone runs the `docker inspect` command.

docker inspect hmac-signature-service
    ..
    ...
    "Env": [
        "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "LANG=C.UTF-8",
        "GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568",
        "PYTHON_VERSION=3.9.9",
        "PYTHON_PIP_VERSION=21.2.4",
        "PYTHON_SETUPTOOLS_VERSION=57.5.0",
        "PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/3cb8888cc2869620f57d5d2da64da38f516078c7/public/get-pip.py",
        "PYTHON_GET_PIP_SHA256=c518250e91a70d7b20cceb15272209a4ded2a0c263ae5776f129e0d9b5674309"
    ],
    ...
    ..

However `APP_SECRET` will still be visible inside `/proc/[container-parent-pid]/environ` (requires to be inside the container or root privileges on the machine running the container)

Takeaway

Environment variables are great but the risk of leaking secrets is just not worth it, if you give access to the docker command to somebody on your system that person can pretty much look what’s running inside the container by running the docker inspect command (having docker access is equivalent to have root access on the system anyways), because of this reason it’s preferable that applications read configurations and secrets directly from files and to leverage on the OS file system permissions.

Download the source code of the app here: https://github.com/Alevsk/docker-containers-env-vars-security

Happy hacking

10 things you should be doing if you care about security in your Tech Startup

I’ve been working in the startup world as a Software Engineer for a little bit more than two years now, as most of you already know, I’m very passionate about information security so I decided to create a list of things you can do to protect your technology Startup (most of them for free).

SPOILER ALERT: This publication is not going to be your typical article about which crypto cipher is better to use, IDS comparisons or talking about specific DLP products , instead, I would like to cover 10 actions (more like advices) you can take if you value your product, your data, your employees and if you want to protect your Startup in general.

So if you are the CEO, CTO, some high executive or a decision maker in your Startup this information is for you.

1: Enforce the use of password managers

Everything starts with a password, literally, sign-in into your computer is one of the first thing most of you do every morning. Whether email clients, social networks, instant messaging apps, or online banking all this requires the user to provide a password in order to access the service so it’s natural for common users to want to think in a password only 1 time and then reuse it across multiple services.

Reusing passwords (even with small variations) it’s a bad thing because once your password is guessed/stolen it can be used to compromise all your other accounts (facebook, twitter, instagram, gmail, outlook, etc), attackers can automate the process using hacking tools such as credmap: The Credential Mapper.

So how do we prevent employees passwords to be guessed (dictionary attack) while at the same time make sure they are using strong and unique passwords on each one of their accounts? The answer is Password Managers.

Password Managers allow you to have one master password (for choosing a strong master password please refer to my talk How to create secure passwords) and then generate all the others you need based on a secure configuration such as secret length, character types, etc.

So the next time you want to access your favorite social network you just need to copy and paste the password, that also prevents your password for being stolen in case of a keylogger attack. You don’t want your community manager accounts to be stolen right?

There are a lot of good solutions out there for managing your passwords, some of them are free and open-source and some others requires you to buy a license, I personally use KeePass which is free, here is a list of the most popular password managers, doesn’t matter which one you want to choose but go ahead and start using password managers if you are not doing it yet!

2: Use multi-factor authentication if possible

The key of security is to add multiple layers of protection so in case one of them fails the other ones handle the risk, in particular for protecting accounts we can suggest our employees to use 2 factor or multi-factor authentication every time they can, so if a data breach happen and the passwords are stolen and cracked, attackers are still unable to log into the accounts because they are missing the token generator.

Now a days most of the more popular services support multi-factor authentication using one time passwords, token generators (such as google authenticator) or even hardware authenticator devices.

Personally I use a Yubico authenticator key and I’m very happy with it 🙂 every time I need to access my accounts from a new IP address or an unrecognized browser, websites such as Facebook or Gmail will ask for my authentication key, that’s very helpful because even if my password is leaked/cracked or someone guess it, they still need the physical key to access the service. 

If you don’t have a budget or prefer not to spend money on this you still can enforce multi-factor authentication using these free apps (every employee can have a token generator right on his smartphone):

3: Choose a secure instant messaging application

Every organization use some kind of real time communication application (Slack, Microsoft Teams, etc) and sometimes employees need to share sensitive information between them, they do not realize the information is also being shared with the third-party service provider who can read it.

Fortunately, nowadays more and more services support security features such as end to end encryption which means all communications between devices are encrypted (each device has a public and a private key) and not even the service provider can read them because they don’t have the private keys.

Another cool feature is self-destruct messages, basically you can set a timer so messages only exists during a particular amount of time after being send and then are destroyed, very useful when you want to share sensitive data such as passwords.

Some free apps that include these features are:

4: Securing all Email communications

Email communications is an essential part in every organization, making it a very attractive vector for attackers, according to a new report from PhishMe, 91% of Cyberattacks start with a phishing email, so even if you have advanced network controls, deceiving your users is still easy.

Nowadays attackers have access to sophisticated phishing tools like SET (Social-Engineer Toolkit) or Gophish which they use to target your employees, they also have access to large repositories of open-source phishing tools they use to tune and adapt their attacks to specific people.

Most of this tools allow attackers to spoof corporate emails and trick your users into downloading malicious files into their systems and into your network, spotting spoofed email addresses is very difficult for common users however using security software like PGP (Pretty Good Privacy) & GPG can help you to mitigate the issue.

Enforcing the use of software such as GPG (GNU Privacy Guard) could help your startup in many ways, like verify the legitimacy of a received messages or encrypt an email content so only a specific user can read them.

You can verify if a message you receive is legit by using the public key of the sender (usually another employee in the organization), meaning: if the person that sent you the email also signed the message using his private key and that private key is associated with the public key you have, then you are guaranteed the message is coming from the right person.

I know this sounds a little bit confusing at the beginning, but the main idea is that every person in the company has a key pair, a public key and a private key, everybody exchanges their public keys while keep their private keys to themself, so when I want to send a message to Mr John Doe I write the message normally and then I proceed to sign it with my own secret key, optionally I can encrypt the message using the public key of John Doe, so the message can only be decrypted and read it by the private/secret key of John, finally John can use my public key to verify the signature I applied to the original message (the one I generated with my private key).

If you still don’t get it don’t worry about it, nowadays most email clients support PGP and the process for verifying and decrypting emails is automatically, there is also a chrome extension called FlowCrypt that I highly recommend!

This message was encrypted with my public key and then sent to me, not even google can read this.
The message decrypted on my browser via the FlowCrypt browser extension

5: Encrypting all your drives

Now we are introducing the concept endpoint protection and “data loss prevention“, in fact I think most of you already use some form of data encryption software, I’m not going to go deep into the details but encrypting your drives could protect your data in many cases, ie: someone steal a company hard drive and try to mount it in another computer to read the information.

If your employees are MacOS users, the operating system already come shipped with FileVault enabled by default, if they use Windows they can use BitLocker and if they use a modern Linux distribution (ie: Ubuntu) full disk encryption is also available.

Data encryption has pros and cons, but the benefits are superior from a privacy and security stand point so I highly recommend to use full disk encryption in all company devices if possible, also the solutions I mentioned above are all free, so you don’t need to spend any money on this one too in order to protect your employees.

6: Encourage secure coding best practices

Usually, when you start a new company then financial resources are limited and you need to be very careful with the people you hire, basically you want to have the best developers, people that are really good at whatever they do but also are wiling to learn and adapt to different situations, you want Rockstar developers.

Rockstars developers have the potential for learning anything, so feed them with the right content, Open Security Training contains great resources about different topics of security like Introduction to Secure Coding, the best part is, are you ready?, its all FREE! in fact this is how I have been learning about security all this years.

Besides Open Security Training there is also The Open Web Application Security Project (OWASP), which is also a good resource for starters so they can learn how to create secure web applications and also secure mobile apps.

Everybody can learn about Security these days, encourage your developers to do it (give them time and resources) and your team will become stronger!, here are some extra sources I had used in the past:

7: Consider hiring a security expert to join your team or an external security team

This advice is more for mature startups or executives who already have a budget to spend on cybersecurity, but it can also apply if you are a small startup and have some friends in the security community.

The idea is to have someone in your team that can give you advice and guidance on different security matters, ie: implementing a security plan for the software development process, do threat modeling in your organization, security infrastructure (IDS, IPS, firewalls, etc), security training, network protection or just make sure your employees are safe are just some examples of things your tech Startup needs from a security perspective.

Besides having your own security guy consider hiring an external security team too, having the security assessment of an external team allow you to simulate more realistic attacks to your organization so you can be more prepared when the real thing happen.

Here are some personal thoughts about security people:

  • Security people are different
  • We enjoy talking about security all the time
  • We want to get asked about how to protect X or Y technology
  • We enjoy challenges and puzzles
  • we enjoy to break stuff and tell you how to fix them.

8: Start a bug bounty program

Companies doesn’t like the idea of their product being hacked, personally I believe that way of thinking need to change because it’s a good thing to have a group of white hat hackers finding vulnerabilities in your software before the bad guys do it.

You can start a bug bounty program with a well defined scope so people can try to hack your product legally (you can even set some special environments for this), there are some guidelines regarding how much to pay depending on the type of vulnerability but if you are still a small startup you can also offer some “swag” like t-shirts or gadgets.

In return you get (most of the time) an army of high quality security researchers that will deliver good vulnerability reports, including how to fix your security issues, everybody wins 🙂

Some popular bug bounty platforms right now are:

9: Encourage a cybersecurity culture in the Startup

The success of the cybersecurity strategy in the organization depends pretty much on the people, you can not just spend a lot of money on security assets like Firewalls and Antivirus and expect everything to be magically safe, it’s not possible because people are always the weakest part in the chain. Security is like a game and everybody need to play including high executives like CEOs and CTOs.

In order to have a culture of cybersecurity organizations have tried different things through the years, even punishing their employees, which is not very effective because people end hating security policies. In general people tend to care about security only when affects them directly but they also like rewards so there is a “new” trend in the security community about using gamification in which basically you reward your employees when they have a responsible security behaviour.

Those action might include:

  • Employees getting rewards when reporting phishing emails
  • Escort people without badge outside the facilities
  • Report suspicious USB drives or hardware that should not be there to the IT/Security department.
  • Enforce people to lock their workstation when not using them by sending emails (using the unlocked account) about free donuts for the whole floor/department/team :p

The idea of all this is to be fun while at the same time the organization become more secure against external threats.

10: Be transparent about Security issues and data breaches

Your biggest fear became true, your Startup got hacked and your information is all over the Internet, If you followed all my advices chances are your sensitive information like passwords are encrypted, which is useless for the attackers, however you still have a moral (and in some places legal) duty, you need to notify your customers and employees about the data breach (according to GDPR you have 72 hours to report a personal data breach after it’s discovered) basically every minute you wait is a minute attackers can invest into cracking and recovering the information so it’s better to communicate the incident, so people can start acting accordingly (change passwords, cancel credit cards, etc).

If you decide to hide the breach and continue without doing anything eventually everybody is going to know about hack and your reputation will be irreversibly damaged (nobody will trust you anymore) so its better to be open with your customers an accept the failure, the shame will be momentary but you will do the right thing.

There is no such thing as a Silver bullet in Cybersecurity, It’s not a matter of if you are going to be hacked or not, it’s about when is going to happen and if your organization is going to be prepared, and this is true for all companies.

Some final thoughts

Security people are often seen as blockers in the organization but I assure you, they have good intentions so please listen to them. Security is hard to implement and even harder to maintain so if you are the CEO/CTO/[Person with authority] of the startup consider to join the security team so you can experience first hand the whole process 🙂

Finally, all these advices are based on my personal opinion (I’m just a security enthusiast) so if you think I should add something else please leave it in the comments.

Happy hacking 🙂

#Docker para #hackers y pentesters, ejecutando #metasploit desde un container

Se acabó el 2016 y como ultima publicación del año les traigo un tutorial exprés que involucra docker y seguridad informática. En publicaciones anteriores explicaba que durante estos últimos meses he estado trabajando bastante con docker, orchestration e infraestructura de cloud en general (parte habitual en un trabajo de full stack engineer).

Docker es una herramienta muy poderosa para desarrolladores pues nos ayuda a construir imágenes con todas sus dependencias y nos deja el paso libre para enfocarnos en lo que realmente importa: deployar rápidamente una aplicación (o varias) que sabemos que va a funcionar.

Bajo esa premisa no es de extrañarse que la comunidad de seguridad haya adoptado docker tan rápidamente, docker es una herramienta fantástica 🙂 y así como nos permite dockerizar una aplicación también podemos dockerizar herramientas de seguridad y en general cualquier cosa que tengamos en nuestro arsenal para pentest.

La gente que trabaja o ha trabajado en seguridad, específicamente en el área de penetration testing, estará de acuerdo en que uno de los recursos más importantes que tenemos son las ventanas de tiempo, por lo general cuando se realiza una prueba de penetración a alguna aplicación o sistema se hace durante un periodo de tiempo bien definido, el tiempo es valioso y no podemos desperdiciarlo en instalar y configurar herramientas, o peor aun ¿que pasa si la infraestructura que estamos auditando nos bloquea? ¿cuánto tiempo vamos a invertir en preparar un nuevo nodo desde donde podamos lanzar ataques y recibir shells?, para todo lo anterior llega docker al rescate 🙂

En este tutorial mostraré como ejecutar una de las herramientas de seguridad más populares utilizando docker: metasploit, específicamente utilizaremos el módulo exploit/multi/handler para recibir sesiones de meterpreter.

ojo: no voy a mostrar como dockerizar metasploit, eso lo dejamos para un siguiente tutorial donde veamos como dockerizar aplicaciones

Ejecutando metasploit desde un contenedor de docker

Para evitarte el problema de abrir los puertos en tu router y hacer un mapeo de puertos para exponer tu maquina a internet, puedes contratar un vps con algún proveedor de tu elección, hoy en día es muy sencillo contratar un vps y puedes tener uno en línea prácticamente en minutos, yo recomiendo digitalocean por qué sus vps son baratos y el soporte es muy bueno, con un nodo de 10 USD al mes es suficiente para correr una imagen de metasploit, puedes contratar el de 5 USD pero tendrás que habilitar el swap o si no quieres gastar dinero siempre puedes aprovechar la promoción que te ofrece Amazon Webservices (más o menos 1 año de uso gratuito de una instancia micro)

Sea cual sea el proveedor que hayas elegido el siguiente paso es instalar docker en tu instancia, para este tutorial lo haré sobre ubuntu / debian pero podrías instalarlo en el sistema operativo de tu elección, acá tienes una lista de sistemas operativos soportados

Desde la terminal de tu instancia y como root vamos a ejecutar algunos comandos para instalar herramientas necesarias como compiladores de gcc/g++, algunas librerías, utilidades, etc. al final vamos a instalar docker

[bash]
apt-get install build-essential
apt-get install libxslt-dev libxml2-dev zlib1g-dev –yes
apt-get install docker
apt-get install docker.io
[/bash]

Lo siguiente que vamos a hacer es crear un directorio en nuestra instancia, este directorio lo vamos a utilizar como un volumen persistente cuando ejecutemos metasploit desde el contenedor para poder almacenar ahí todo el loot, scripts y en general archivos que nos genere la herramienta.

[bash]
mkdir /root/.msf4
[/bash]

Llego el momento, en el docker registry oficial existe una imagen llamada remnux/metasploit que contiene todo lo necesario para ejecutar la herramienta, ejecutamos el siguiente comando y docker comenzara a descargar la imagen y posteriormente procederá a correr el contenedor.

[bash]
docker run –rm -it -p 443:443 -v ~/.msf4:/root/.msf4 -v /tmp/msf:/tmp/data remnux/metasploit
[/bash]

En el tutorial de docker anterior explicaba para que era cada parametro, en resumen -p nos permite mapear puertos y -v definir volúmenes persistentes (mapear una carpeta de nuestro sistema de archivos a una del sistema de archivos virtual del contenedor).

Una vez la imagen haya sido descargada el contenedor será creado y todas las dependencias necesarias comenzaran a ser instaladas, nos olvidamos de instalar todas las gemas y resolver conflictos y nos podemos ir por un café ya que es un proceso bastante automatizo 🙂

Después de unos minutos tenemos un nodo listo para recibir conexiones.

En el caso de necesitar más listeners no hay problema pues podemos ejecutar múltiples contenedores de metasploit en diferentes puertos y así tener nuestras shells organizadas, y si nuestro servidor es baneado rápidamente podemos desplegar otro ejecutando esos 6 comandos.

Como lo he comentado en varios artículos, docker es una herramienta muy poderosa que puede ser utilizada en varias situaciones además del desarrollo de software, como lo vimos en este tutorial. Puedes dockerizar casi cualquier cosa, yo en lo personal tengo una imagen con un set de herramientas que utilizo en mi día a día (fierce, dirb, sqlmap, nmap, enum4linux, hashcat, Responder, etc.)

Saludos y Happy Hacking.