Monday, 27 May 2013

GStreamer toy application template


This time I'd like to share some piece of code I find very useful when experimenting with GStreamer. This includes writing some small tests to understand some features and all sort of debugging of specific cases after distilling them from larger applications. The thing is that I don't want to start over and over again with the common stuff whenever I want to quickly hack a GStreamer application. This is mostly about writing the main() function and some bits and pieces:

  • parse command line options
  • initialise GStreamer
  • create the pipeline
  • optionally add a probe
  • optionally send EOS
At the same time I want to keep it simple and not turn it into some versatile utility which would obfuscate the main idea. Plain simple. I know there's gst-template project but as I understand it's more GStreamer plugin oriented and helps with some GLib/GObject boilerplate.

The template

The goal for this "template" is to be reusable but mostly for some throw-away prototypes or tests. But still I wanted to keep the quality as high as possible, include proper error checking etc. as in practice a lot of prototype code ends up in production.

The cool thing about it is that it reads the pipeline description from the command line which is very useful when experimenting/debugging. It also offers sending EOS event upon SIGINT (-e), adding probe on a pad (-p) and verbose output (-v) if e.g. one wants see the output of a identity element. There are few gaps to fill in, all marked with TODO, e.g. the type of probe and probe return value. As I said I didn't want to over-complicate this so I left it to edit in the code.

Building and running

The source code is available here. This time I'm using uninstalled version of GStreamer's master branch. On my Debian laptop I build it as follows:

libtool --mode=link \
  gcc -O2 \
  $(pkg-config --cflags --libs gstreamer-1.0) \
  gst-test-app.c \
  -o /tmp/test
And a couple of tests:
[kris@lenovo-x1 master]$ /tmp/test -e fakesrc ! fakesink
** Message: Running...
^C** Message: Handling interrupt:  forcing EOS on the pipeline

** Message: End of stream
** Message: Returned, stopping playback
[kris@lenovo-x1 master]$ /tmp/test -e -p "sink:sink" videotestsrc num-buffers=5 ! fakesink name=sink
** Message: Successfully installed probe on 'sink:sink'
** Message: Running...
Got event: stream-start

** (lt-test:27135): WARNING **: TODO: Implement me!!!
Got event: caps

** (lt-test:27135): WARNING **: TODO: Implement me!!!
Got event: segment

** (lt-test:27135): WARNING **: TODO: Implement me!!!
Got event: eos

** (lt-test:27135): WARNING **: TODO: Implement me!!!
** Message: End of stream
** Message: Returned, stopping playback
VoilĂ ! Enjoy!

GStreamer toy application template

Wednesday, 8 May 2013

C14N for signed XML


Here's another, somewhat exotic subject which is canonicalisation, quite often abbreviated C14N. No, this has nothing to do with the pope and saints. This is about getting the canonical form of an XML file. Why? Well, XML is pretty ambiguous format where there's a lot of white space characters (for poor human readability), no strict element ordering or implicit namespaces. So what? This means that for a dumb machine two identically looking XML files may be very different. An extreme example would be a terminating new line character which is very difficult to spot. So what? Sometimes you want to find out if two XML files are identical. Or even more—whether anyone has tampered with it and whether it's been sent by someone you expect if it's transferred over the internet. This all leads to a state of the art, 21st century invention—signed XML or XMLDSig.

XML files are not perfect but some people love them. This is not going to be an overview of this format as this is a very broad topic and I'm not eligible to write such overview. I'm just a lowly engineer. Let me just say that XML is useful sometimes. I'd even dare to say that it's inevitable to use sometimes (as a dead simple INI format would raise too many eye-browses... OK, I won't be sarcastic any more). XML is inevitable sometimes. So I convinced you and you think that signed XML is cool and you want to have one, aye? Your mates will be jealous. You may be even tempted to get your partner's name (or your pet's name if you don't have a partner or anyone or anything you're attached to) into XML form, sign it and tattoo on your thigh. I'll leave it to you and focus only on one mysterious step toward that goal which is canonicalisation.

Since you're still reading, you're either one of my committed readers or you're genuinely interested in XMLDSig. I won't recommend reading XMLDSig reference nor C14N unless you have a strong urge to wade through rather dry standards. I also won't recommend implementing anything from scratch. Check out xmlsec library whether it suits you. I confess I knew about xmlsec but decided to use bare libxml2 and get my hands dirty for at least two reasons—I was scared and I wasn't sure what I was doing. I wanted to reassure myself by having as much control as possible. I hope you're in a better position.

Not very discerning dissection

Briefly XMLDSig consists of three parts:

SignedInfo describes how the content is signed, KeyInfo allows to identify the signer and Object is the signed content1. Since signing and verification involve computing the hash (digest) and asymmetric cryptography, care need to be taken to sign and verify exactly the same content. Those beasts are ruthless if there's even a single byte changed. To get a canonical form of an XML content, some uniform rules need to be applied so that two logically equal XML files look exactly the same. Then the result can be either signed or verified.

Once you've got a glimpse of the whole process, you may want to (or someone forces you to) do it programmatically. Now you might be tempted to call it a day after finding xmlC14NDocSave() function in the libxml2 API documentation. But before you cross this bridge, you need to answer first what's your quest. And believe it or not, this is not easy. The catch is that the xmlC14NDocSave() takes a set of nodes you want it to operate on and in fact it doesn't do all the dirty job for you. You need to provide it a set of the right nodes in the right order. Here's the XPath incantation:

descendant-or-self::* | descendant-or-self::text()[normalize-space(.)] |
.//attribute::* | .//namespace::* | .//comment()
There's some uncertainty whether to use normalize-space(.) or not and what the relative order of attributes and namespaces should be. This unfortunately depends on who you talk to and where you received signed XML from or where you intend to send it. For example signed XML files in Adobe AIR packages require space normalization while xmlsec tool doesn't. This is mundane and brutal reality of format incompatibilities. Beware.

Getting hands dirty

Since I got your full and utter attention and you're so excited that you probably dropped some of your late at-the-desk lunch onto your smart looking office trousers (or onto your pants if you're home alone or in the United States of America), let me show you some sample code. Full source code along with test scripts is available here. Here's the essence with mundane error checking removed here for brevity:

  // Now this is old granny's secret recipe for a delicious cheesecake.
  const xmlChar C14N_CONTENT[] =
    "descendant-or-self::* | descendant-or-self::text()[normalize-space(.)]"
    "| .//attribute::* | .//namespace::* | .//comment()";

  // Get some cheese... all sub-document content we need for canonicalisation.
  const auto sinfo =
    make_scoped (xmlXPathEvalExpression (C14N_CONTENT, ctx.get ()),

  // And finally... bake it!
  xmlC14NDocSave (doc.get (), sinfo->nodesetval, 0, NULL, 0, file_out, 0) < 0);
The full implementation is available here. And here's how I compile it on my Fedora 18 laptop:
g++ -O3 -std=c++11 \
  $(pkg-config --cflags --libs libxml-2.0) \
  xmldsig-c14n.cpp \
  -o /tmp/c14n-exe
And here's a Bash one-liner that takes a signed canonicalised XML on its input and verifies it (provided that you have one—see below what to do if you don't). For convenience I split it into multiple lines but essentially it's a single line command.
/tmp/c14n-exe "/default:Signature/default:SignedInfo" /tmp/sample.xml |
openssl dgst -sha1 -binary -verify <(
  printf -- \
    "-----BEGIN CERTIFICATE-----\n%s\n-----END CERTIFICATE-----\n" \
    "$(xmllint \
        --xpath "//*[local-name()=\"X509Certificate\"][1]/text()" \
        /tmp/sample.xml)" | \
  openssl x509 -pubkey -noout) \
  -signature <(
    xmllint \
      --xpath "//*[local-name()='SignatureValue'][1]/text()" \
      /tmp/sample.xml | \
    openssl base64 -d)
A more convenient script is here. If you don't have a XMLDSig file at hand, you can generate one with the script I provided.
./ /tmp/c14n-exe > /tmp/sample.xml
./ /tmp/c14n-exe /tmp/sample.xml
That's it for now. Enjoy!

[1] Actually Object is digested and the digest is included in SignedInfo which itself is digested and signed.
C14N for signed XML

Saturday, 13 April 2013

Verifying signatures with OpenSSL API

A brief introduction

This is another article where I try to tame OpenSSL API using C++11. This time round I describe a small example showing how to verify signed data programatically. There are many message formats catering for different needs. In this example I show how to verify that the data is not tampered and is sent from a party identified by a PKI certificate. Please refer to my other article to learn how to verify a certificate.

Again I want to emphasize that you should not implement this functionality as you can use openssl tool:

openssl dgst -verify test-key-pub.pem \
  -signature /tmp/signature </tmp/data
This command checks that the data stored in /tmp/data is not tampered. The tool calculates a checksum (a digest) and verifies it with the signature stored in /tmp/signature. The signature has been signed with the private key paired with the public key stored in test-key-pub.pem. If there's a certificate associated with the public key available, it can also be verified to see wether the data hasn't been signed by an intruder in the middle.

As you can see, there's no need to invent the wheel if your requirements are simple enough. Depending on the circumstances this approach might not be sufficient or acceptable, and only then you should come to grips with your own implementation.

Producing or verifying a signature is rather expensive operation as it involves asymmetric cryptography. In practice a digest is produced first (e.g. using SHA1) and then the digest is signed with one of the asymmetric keys. The verification comprises applying the same digest function to the received data and checking whether the signature of that digest "matches" when using the other key of the asymmetric pair. You don't have to worry about these details though as they are hidden behind the OpenSSL API. Hopefully this also allays concerns about the use of the openss dgst command which stands for "digest". The signature is simply another step in the process of digesting data.


Code for this example is available here. There's also a very basic test script provided.

The main three functions we are going to use are EVP_VerifyInit_ex(), EVP_VerifyUpdate(), and EVP_VerifyFinal(). The first two of them are simply aliases (macros) of equivalent "digest" functions. Of course you shouldn't abuse them and better use the macros provided to be explicit about the intentions. Please also note that in general *_ex() versions of OpenSSL API functions are recommended if available as they are more general and allow you to use an engine. If you don't intend to use an engine simply use nullptr.

The "trinity" is a common pattern in the OpenSSL API.

First you initialise the algorithm, then there's a one or more updates that feed the algorithm with data, and in the end you finalise the algorithm. The update step allows to process "streamed" data, i.e. you feed the algorithm with data as it arrives. If all data is available at once, you can make only one update call. In many situation though you might want to process data in chunks, e.g. when you read a large file or from a network socket.

More details about the API used in this example are available in the manual so there's no point in duplicating them here. If you're off-line and have openssl-devel (or equivalent) package installed (which you should in order to compile this example), you can also use info or man pages. Don't forget to read about EVP_MD_CTX_create() and EVP_MD_CTX_destroy().

Build and test

This is how I build the example on my Fedora 18 laptop:

g++ -std=c++11 -O3 -DNDEBUG signature-verify.cpp \
  -lcrypto -o /tmp/my-verifier
I think that the most frustrating thing about keys, certificates and all this cryptographic stuff is testing. Creating test assets (key material, certificates etc.) can be truly onerous. But this is still not as hard as testing a full production system with real cryptographic material (very often hardware assited), so let's get on with it:
# generate test private key and associated certificate
openssl req -x509 -newkey rsa:2048 \
  -keyout test-key-priv.pem -subj "/CN=FakeSigner" \
  -passout pass:none -out test-cert.pem

# sign some test data
echo -n "test" | tee /tmp/data | \
openssl dgst -sha1 -sign test-key-priv.pem \
  -passin pass:none -out signature
And finally we can run our verifier:
# verify signed data
/tmp/my-verifier test-cert.pem /tmp/data signature
As the steps above are a bit tedious, you can use a test script I provide here. Simply give it the path to the verifier executable as an argument and that's it. It creates a temporary scratch directory where it generates the assets and runs rudimentary tests using the executable provided. As I wanted to keep it dead simple, it doesn't provide any additional options like preserving the scratch directory, setting verbosity level etc. It'll probably evolve in future incarnations once I've got examples in my repository a bit reorganised.
./ /tmp/my-verifier

Happy verifying!

Verifying signatures with OpenSSL

Tuesday, 26 March 2013

Verifying certificates with OpenSSL API

I'd like to allude to my previous article where I superficially described how certificates and PKI work, at least the way I understand it. This time I want it to be more tangible and hope to make it helpful to anyone grappling with a problem: how to verify certificates programatically with OpenSSL API.

Let me make it clear—don't use OpenSSL API, don't write any C/C++ or any other application. Check out ready to use tool, namely openssl verify command:

[kris@lenovo-x1 tmp]$ cat c1.pem c2.pem c3.pem > untrusted.pem
[kris@lenovo-x1 tmp]$ openssl verify \
  -CAfile ca.pem \
  -untrusted untrusted.pem \
leaf.pem: OK
This should cover most common cases. Sure, one may say this is not a production tool and despite been written by experts (the authors of the OpenSSL package) it's hard to audit or it doesn't provide required features from the command line. Someone may not be able to deploy the executable onto the target system due to space limitations or may not be able to execute it from their program. There are plenty of reasons why someone would want to use OpenSSL API directly. But if you're not constrained by any of those reasons, just don't, unless you want to learn something or do it for dubious fun. And the OpenSSL API is not the most beautiful one. See this article on how botched SSL APIs make it easy to introduce vulnerabilities into applications.

I doubt you can get the ball rolling after reading the rather sloppy and incomplete documentation. Maybe it's a good starting point but sooner or later you'll be discontented. The way I learned how to use the API—and I believe it's the best way—is to read OpenSSL source code and maybe run the harder bits under the debugger. Certificate verification in particular is implemented in apps/verify.c. A good code cross-reference tool might be handy as well as any grep-ish approach will be hindered by wodge of preprocessor generated functions.

It's also interesting to see how C++11 features can be leveraged to deliver robust and nice to read code. Some developers slog through C in their applications as they feel they have to because OpenSSL is written in C. Sure, I appreciate OpenSSL itself is written in C as it's portable, but writing robust code in C is really hard and unpleasant. I wouldn't bother writing an example for this article in C at my leisure. I wouldn't even do it for money if there was no good reason for it. With C++11 on the other hand it's fun!

This is a rudimentary example and there are no classes and all that fancy stuff. I wanted to keep it dead simple. You may wonder why I bother checking all error codes and throw everywhere. Well, the reason is simple—I never know when I want to copy it (ehem, re-factor) into some utility, class, library or anything else in production code. I strongly believe that it's very important to start with a prototype (the simpler the better) to explore new areas and spot problems as soon as possible. But the sad thing is that quite often prototype code ends up in a final release, most notably because "if it works, don't touch it". I see nothing wrong in reusing prototype code that proved to work really well. But the caveat is to make it robust from the outset. So all this "small" things like checking error codes, deallocating those tiny bits and pieces may not matter now, but they might do in the future. And you don't want to go through the code scouring it and looking for these little nuisances. It's much easier to deal with the details as you write the "essential" code.

Enough preaching so let's do it, shall we? The source code is available as usual on GitHub here.

I want to declare some function return types with my beloved std::unique_ptr. As this is a very lightweight smart pointer, the pointee deleter is part of its type. This is different when compared to std::shared_ptr which uses type erasure for the deleter. As I don't care too much about the performance here, I want to do the same for my std::unique_ptr:

template<class T>
using scoped_ptr = std::unique_ptr<T, std::function<void (T *const)>>;
And I also reuse a teensy make_scoped() utility from one of the previous articles. It's noteworthy to say that this might be considered as std::unique_ptr abuse as it just happens that objects returned by OpenSSL API are literary pointers. Had they been some sort of handles only, I wouldn't have used std::unique_ptr here. And I named these utilities scoped_ptr and make_scoped to make them different from proposed std::make_unique().

Before you can call X509_verify_cert(), you need to prepare certificate store context which comprises ultimately trusted root certificate, untrusted intermediate certificates and the leaf certificate. This is all done in the main() function and is fairly easy to follow. Quite challenging is providing the actual error message. In this example I limited it to loading some rather succinct message from the OpenSSL library and extracting certificate expiry dates as I've found it quite common verification failure reason (dealing with a lot of generated short-term test certificates). See the formidable print_verification_failure_msg(). Basically OpenSSL provides verification algorithm but you could provide your own. Believe me though—you don't want to do it. So here's a simplified sequence (please refer to the source code for detailed error handling):

// Let's create a certificate store for the root CA
const auto trusted =
  make_scoped (X509_STORE_new (), X509_STORE_free);

// The lookup method is owned by the store once created
// and added to the store
const auto lookup =
  X509_STORE_add_lookup (trusted.get (), X509_LOOKUP_file ());

// Load the root CA into the store
X509_LOOKUP_load_file (lookup, argv[1], X509_FILETYPE_PEM);

// Create a X509 store context required for the verification
const auto ctx =
  make_scoped (X509_STORE_CTX_new (), X509_STORE_CTX_free);

// Now our untrusted (intermediate) certificates (if any)
const auto untrusted =
  read_untrusted (argv + 2, argv + argc - 1);

// And our leaf certificate we want to verify
const auto cert =
  read_x509 (argv[argc - 1]);

// Initialize the context for the verifiacion
X509_STORE_CTX_init (
  ctx.get (), trusted.get (), cert.get (), untrusted.get ());

// Verify!
const int result = X509_verify_cert (ctx.get ());

To build and try the example I did the following on my Fedora 18 laptop:

[kris@lenovo-x1 kriscience]$ g++ -std=c++11 -O2 -DNDEBUG \
cert-verify.cpp -lssl -lcrypto -o /tmp/test
[kris@lenovo-x1 kriscience]$ !$ ca.pem c1.pem c2.pem c3.pem leaf.pem
Verification OK
This is very trivial example but it's a foothold for someone starting their adventure with OpenSSL API. A more representable application would use policy verification, time parameter and CRL checks. Maybe one day I'll present it here.

Saturday, 2 March 2013

Supporting trusted but untrusted certificates with OpenSSL

This time for something completely different I'll broach a bit intimidating area—PKI certificate chains that link back both to trusted and untrusted root certificates. That is, how to recognise different trees from quite a long way away.

For some of the readers that know at least a little bit about the matter this might be a quick and easy recipe to solve their problem if they understand it. For those who have no idea of what I'm talking about this still might be an interesting reading if they like exotic trips. Don't worry though, as I'll provide some basic introduction without going into much detail. There's already a smorgasbord of material to learn from if someone's interested.

I won't even bother providing links as a reference (except for salient ones) as I find it overwhelming and distracting when a very specific story is sprinkled with lots of links for everything. Sure, one may ignore them (so I'd waste my time providing them) but equally one may look up something on their own if they have an urge to do so (whatever you look for, if you need an authoritative information, always reach out for RFCs). This is what I find to be a pragmatic approach which I think is different from purely scientific one.

A bit of introduction

PKI stands for Public Key Infrastructure and is a monster which a lot of people find hard to get on with. For this article it's only important to know that it provides means of distributing some cryptographic material referred to as public key in a trusted manner.

What is a public key you may ask and why is it public? Some smart guys in the past have combined a bunch of mathematical operations with huge random numbers and have come up with something known as asymmetric keys. Keys in turn are specially crafted strings of digits used as an input for ciphers. So you have: data + key -> cipher => blob (a `cipher' is a function that eats a `data' and a `key' and spews out a `blob'). Asymmetric means that there's a pair of keys related to each other through some mathematical operations—you can use one to encipher data and the second one to decipher it and the other way around. If you keep one of them secret and give the other one to everyone else you have public-private key pair.

Publishing keys

When you give out your public key, someone may want to use it to encipher something that is only intended for you (some secret) as you are supposed to use your private key to decipher it. The problem is that by simply publishing your public key there's no guarantee that it's actually your key. No one can really trust it to encipher secrets they want to share only with you. Here is where certificates come into play.

Certificates leverage at least two properties of asymmetric ciphers: authenticity and integrity of data. Again, using some smart mathematical operations and having one of the asymmetric keys, one can tell that the data has not been tampered with and has been ciphered with the other key of the same pair. This process is known as signing and verification and the data exchanged between parties is called a signature.

Certificates are structured bits of information where the essential part is a signature of one party's (A) public key created with a more trusted party's (B) private key. Such signature in turn can be verified with a more trusted party's (B) public key which again can be signed by a more trusted party's (C) private key (another certificate). This forms a chain of trust (chain of certificates) which ends up at the top with a root certificate.

Root certificates

Root certificates are special as there's no one who would signed them. They are in fact self-signed. So how does it all make sense? At the very top is a human. It might be you or a system administrator (some people claim they are not humans). No matter how complex processes and policies are employed, it's always a human being that makes the ultimate decision: I trust something or not.

Root certificate bundles

A system that is supposed to use PKI needs a set of trusted certificates also known as a root CA certificate store (CA stands for Certificate Authority) or a root CA certificate bundle (CA bundle for short). Whatever chain of certificates it needs to verify, it expects that the top-level untrusted certificate in that chain is signed with one of the root certificates stored in the root CA bundle which it ultimately trusts. As a PC user you get a CA bundle onto your system usually distributed with a web browser so you can use HTTPS protocol. You might have been prompted by a browser with some sort of security exception pop-up when visiting a website using HTTPS protocol whose authenticity couldn't be verified by the browser based on the CA bundle installed (actually the certificate chain sent to your browser as part of the TLS protocol could not be verified or for example the website domain name did not match the one signed by the leaf certificate). The browser in such situation might leave the decision up to you whether to trust the website or not.


This short introduction just scratched the surface. It's not even a top of an iceberg. There's a lot more to talk about, like on what basis a human being can make a decision to trust an authority (or simply someone else's public key) or how to ensure key privacy, what different key usages are and how they are ensured and enforced, cipher suites properties etc. Firstly, I'm not an expert nor a scientist, and secondly, it's not directly related to the rest of this article.

What this article is actually about?

Given the introduction above or the knowledge you might have already had and a CA bundle, you might face a situation when you need to verify a certificate chain as follows: A signed by B signed by C signed by D but C is in your CA bundle (your system trusts it ultimately) and D is not (your system doesn't trust it). I'm pretty sure it happened at least once in everyone's life, even in my dog's life which I actually don't have.

You may wonder how it's possible to have certificate C in the bundle (self-signed) and at the same time have it signed with certificate D. Here's where I'd like to refer to some external resource where it's nicely depicted. Note that some people may burble here something about cross-signing but it's not specific only to this situation as cross-signing simply refers to anything else than self-signing. Don't get confused by the diagram on the website I referred to. The article there says that one certificate is signed by two other certificates. But it's not a certificate that is signed nor any certificate signs another one. It's keys what actually gets signed. So by saying that certificate B signs certificate A, one (hopefully) means that the public key represented by certificate A is signed by the private key corresponding to the public key represented by certificate B.

What's the catch?

The catch is that if use OpenSSL, you may not be able to verify a chain which contains a root certificate included in the system CA bundle but signed in that chain by some other non-trusted root certificate. This is not what is expected in many situations. Fear not my friend as you're not left alone. OpenSSL has a solution for this situation: X509_V_FLAG_TRUSTED_FIRST verification flag. It tells it to not follow the chain once a trusted certificate is found.

Now the problem is that it's not available in all OpenSSL versions. The way I understand OpenSSL releases is that at the time of this writing there are three "production" branches available: 0.9.8x, 1.0.0x and 1.0.1x. You're very likely using one of them. None of them supports X509_V_FLAG_TRUSTED_FIRST though. All you need to do is to apply the following patch from the OpenSSL mainline. Now given a naughty certificate chain in chain.pem and a CA bundle in ca-bundle.crt you may try:

[kris@lenovo-x1 kriscience]$ openssl verify \
-CAfile ca-bundle.crt -untrusted chain.pem chain.pem

chain.pem: C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) \
2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public \
Primary Certification Authority - G5
error 20 at 2 depth lookup:unable to get local issuer certificate
but if you use a new -trusted_first option, it should succeed:
[kris@lenovo-x1 kriscience]$ openssl verify \
-CAfile ca-bundle.crt -trusted_first -untrusted chain.pem chain.pem
cert.pem: OK
Now all you need to do is to convince your client application to use X509_V_FLAG_TRUSTED_FIRST option. For example if you are using libcurl, you may want to apply this patch:
--- a/lib/ssluse.c
+++ b/lib/ssluse.c
@@ -1651,6 +1651,26 @@ ossl_connect_step1(struct connectdata *conn,
           data->set.str[STRING_SSL_CRLFILE]: "none");
+  if (1) { // artificial scope to keep the patch local
+    X509_VERIFY_PARAM *x509_param = X509_VERIFY_PARAM_new();
+      if (!x509_param) {
+          failf(data,"failed to create X509 verification parameter");
+          return CURLE_OUT_OF_MEMORY;
+      }
+      if (!X509_VERIFY_PARAM_set_flags(x509_param, X509_V_FLAG_TRUSTED_FIRST)) {
+          failf(data,"failed to set X509 flag - trusted certificate first");
+      }
+      if (!SSL_CTX_set1_param(connssl->ctx, x509_param)) {
+          failf(data,"failed to set X509 trusted certificate first parameter");
+      }
+      X509_VERIFY_PARAM_free(x509_param);
+  }
   /* SSL always tries to verify the peer, this only says whether it should
    * fail to connect if the verification fails, or if it should continue
    * anyway. In the latter case the result of the verification is checked with

Saturday, 2 February 2013

Handling POSIX signals

When programming for *nix systems, like it or not, sooner or later your application will be bothered with POSIX signals. And even if signals are considered to be a broken design, you have to live with them, hammered with some trepidation. And even if you've tinkered with it and managed to install some handlers, you can't really do much in a signal handler context. You can merely set some flag and get out of there if you don't want to get into trouble. And if you have to whack some threads in, you feel completely screwed as there's no certainty whatsoever of which thread would receive signals. A bit intimidating, isn't it?

There's an attempt to remedy this creepy situation with sigwait(3) and there's a nice article about it. So fear no more, you're not stranded as there's even a better solution: signalfd(2). It gives you a file descriptor so you can choose how to handle signals—be it blocking read(2) in a separate thread, some sort of poll(2) which nicely integrates with polling loops (GMainLoop, DBusWatch etc.) or anything else you'd like to do with a file descriptor.

The problem

Before I show an example of one approach to signal handling, I'd like to elaborate a bit more on problems you may encounter with "traditional" signal handling. It'll be easier for me to use a multi-threaded application example although single-threaded applications are also negatively affected but in a more subtle manner. Feels a bit contrived but I've seen it in the field.

Let's suppose someone wants to use a thread synchronization mechanism (e. g. a mutex) in a signal handler. Signals are considered to be software interruptions and their handlers are executed uninterrupted. If someone attempts to protect some shared (global) data with thread-based locks, it will apparently lead to a deadlock:

int globalData;
std::mutex globalMutex;

void foo()
  // ... some stuff

    std::lock_guard<std::mutex> lock(globalMutex);
    globalData = 1; // deal with the global data

  // ... some other stuff

void handler()
  std::lock_guard<std::mutex> lock(globalMutex);
  if (globalData == 0) // deal with the global data
    // ... something miserable
  } else {
    // ... something funny
It's conceivable that foo() can grab the mutex right before the process receives a signal. The handler() function is called as the signal handler. It tries to get the mutex without a success and it ends up with a deadlock since it is not scheduled out and doesn't let foo() release the mutex. Completely botched.

On the other hand, we cannot leave global data unprotected since there's no guarantee that the data can be accessed atomically. We could use atomics here but some architectures still use gUSA to provide atomicity which might complicate things in the context of a signal handler and in general the resource you want to protect might not be as simple as a single atomic structure.

The solution

I chose to civilize signal handling by making it thread-friendly (I can use mutexes and what not). The signal handler runs in a dedicated thread but there are other ways of doing it as the signal delivery is serialized by signalfd(2). You may even use it in a single-threaded application where you will probably have some sort of a main loop and check for any signals delivered to the application process at your leisure.

The example application is available here. Below I present only the main idea.

// protect some shared resource, let it be the standard output
std::mutex g_mutex;

void signal_handler(const int fd) {
  struct ::signalfd_siginfo si;
  while (const int bytes = ::read(fd, &si, sizeof(si))) {
    std::lock_guard<std::mutex> lock(g_mutex);
    switch (si.ssi_signo) {
    // Handle signals here

int main() {
  ::sigset_t mask;
  // block all signals
  ::pthread_sigmask(SIG_BLOCK, &mask, NULL);

  // unblock all signals but deliver through a file descriptor
  const int fd = ::signalfd(-1, &mask, 0);
  std::thread t(signal_handler, fd);
  // do some other stuff
First it blocks all signals and then unblocks them in the signalfd(2) call so they all are delivered through a file descriptor. The signals do not affect the application (except for SIGKILL and SIGSTOP) until the application reads from the descriptor and decides what to do about them. Nice and elegant.

And here's an example session on a x86 Linux PC with GCC 4.7.2.

$ g++ -O3 -std=c++11 signal-handler.cpp -pthread -o /tmp/test
$ !$ &
[1] 4135
$ kill -SIGUSR1 %
Received signal 10
$ kill -SIGUSR2 %
Received signal 12
$ kill -SIGTERM %
Quitting application...

Bear in mind that any approach to signal handling assumes you have control over the main thread of the application. If you are writing a library that is called from the main application that you can't control, then you're thrown back at its way of signal handling and other libraries it uses. This is a difficult situation and frankly there's no robust solution to it. Every new thread inherits signal mask from its parent thread, so when it's started it potentially has all signals unblocked until it blocks them with pthread_sigmask(3) call. Until that point new threads may still receive unsolicited signals.

Having the caveat above in mind I wish you best luck with signal handling. It doesn't have to be difficult. Just try to push traditional way of doing it toward oblivion and embrace signalfd(2) as soon as you can.

Saturday, 19 January 2013

Sneaky resources

What's the fuss all about?

Every so often an application has to deal with some sort of resources. There's been plenty of books and articles written about it and they are all useful. They even attempt to go beyond managing memory as this is only one of many resources. The resources often mentioned are network connections, database connections, file system objects and the like. Not only the application has to release them when it's done with them but also has to ensure that they are not held when errors occur and a different execution path is taken.

In C++ error conditions are very often signalled by throwing an exception which automatically unwinds the call stack. This is very handy if one knows how to leverage it. For unwary this could be pernicious though. In some ways C is more convenient in this regard as there could not be any exception thrown when a C function is called. The real problem I often see in code is when both worlds have to coexist:

  • C++ -> C call
    The most likely scenario is when a C++ code calls some C functions that return some sort of handles (very often pointers) to some resources (or simply memory) which are expected to be released with another (complementary) C function call.
  • C -> C++ call
    The other scenario is when a C code calls some C++ function (quite often some sort of C++ callback function registered with a C API). Care need to be taken to not let any exceptions back into C as this causes undefined behaviour. If you're lucky the program aborts immediately, otherwise you have to gape at meaningless call stacks and waste some time to figure out what happened. In this situation (if you know that C calls C++) a good idea is to set a breakpoint on a library routine that is responsible for delivering exceptions (for example __cxa_throw in glibc) and laboriously debug the program to find which exception gets into C.
In this article I want to focus on the C++ calling C scenario.

Most of resources are released by the operating system if the application exits or terminates. So network connections, file descriptors and all sort of sockets are closed, memory is released etc. There are some less obvious resources though which result in leaving undeleted temporary files behind, unreleased IPC resources or (for paranoid) unwiped memory (RSA encryption/decryption). I admit I haven't ever attempted to deal with application recovery and I'm usually happy if the application at least reports some error message and simply exits. But in the face of a less obvious resources leakage and simply for diligence I like to have all toys tidied up nicely no matter what bad course the application has taken. This does not include an immediate abortion which is really horrible and there's not much one can do about it. Even if releasing resources on application exit doesn't matter now, some bits of code or the entire application may get reused somewhere where it matters.

Some unwary programmer may write something like this:

  char *const p = get_some_c_string();
  std::string s(p);
  // do some fancy stuff with the string
Now this is a typical scenario when talking about exceptions. In this situation a lot of operations with std::string (including its creation) can throw. This will inevitably leak memory returned by get_some_c_string(). I won't dwell on it as a lot has been said about it elsewhere and it usually ends up using some sort of smart pointers.

What to do? What to do?

Let me introduce some small utility function that creates a scoped pointer that draws upon std::unique_ptr from C++11.
template<class T, class D = std::default_delete<T>>
constexpr std::unique_ptr<T, D>
make_scoped(T *const ptr, D deleter = D())
  return std::unique_ptr<T, D>(ptr, deleter);
Now let's assume that one wants to verify a certificate using OpenSSL. This task itself could probably make a lot of people nervous not to mention they would have to manage related resources properly. Now let's focus on adding some policies to the verification process:
addPolicy(X509_VERIFY_PARAM *const params, const std::string& policy)
  auto policyObj =
    make_scoped(OBJ_txt2obj(policy.c_str(), 1), ASN1_OBJECT_free);
  if (!policyObj) {
    throw std::runtime_error("Cannot create policy object");

  if (!X509_VERIFY_PARAM_add0_policy(params, policyObj.get())) {
    throw std::runtime_error("Cannot add policy");

  // committed
An interesting use case here is a "transactional" approach. Note that OpenSSL functions that have 0 in their name take ownership of objects passed to them. This is explained in the notes section here but it may appear to you to be the opposite if you read it for the first time. The addPolicy() function creates a managed policy object and do not hesitate to throw if something goes wrong. But once we successfully passed the ownership of the policy object to another object, we can give up its ownership in our scope (remember that C functions do not throw) and the "transaction" is "committed". Nice and easy. Then some part of initialization of the verification process could look like this:
const auto params =
  make_scoped(X509_VERIFY_PARAM_new(), X509_VERIFY_PARAM_free);
  params.get(), X509_V_FLAG_INHIBIT_ANY | X509_V_FLAG_INHIBIT_MAP);
addPolicy(params.get(), "");

// set up other stuff and do the verification
// ...
This is even nicer. It's a bliss and harmony.

I mentioned that letting the program exit without releasing memory shouldn't make any harm as the operating system would reclaim it anyway. But if you're paranoid and deal with some secrets in memory, then this becomes an issue. To address this problem just do the following:

  auto rsa = make_scoped(RSA_new(), RSA_free);
  // do some stuff with the RSA key
  // ...
Note that RSA_free() function erases the memory.

It's also convenient to do some more complicated things with lambdas which in pre-C++11 would have to be wrapped into a functor:

const auto untrustedCerts =
    [](STACK_OF(X509)* const s) { sk_X509_pop_free(s, X509_free); });
// now add untrusted certificates to the stack
// ...

No worries

Using some C++11 goodies can help you to manage different sort of resources and focus on the problem rather worrying about releasing stuff especially in error paths. It also gives a confidence and clarity of what is released where and how. In some scenarios you can also use the transactional approach.

Source code for the examples above is available here. On my Fedora 17 I built it as follows:

g++ -std=c++11 scoped-ptr-example.cpp -lcrypto -o /tmp/scoped
For those who can't afford using C++11 but can use Boost libraries, the following could be a replacement for a scoped pointer with a custom deleter:
#include <boost/function.hpp>
#include <boost/interprocess/smart_ptr/unique_ptr.hpp>

namespace detail {

template<class T>
class unique_ptr_deleter
    template<class D>
    unique_ptr_deleter(const D& d) : deleter(d) {}

    void operator()(T* const p) throw() { deleter(p); }

    const boost::function<void (T* const)> deleter;

} // namespace detail

template<class T>
struct unique_ptr
    typedef boost::interprocess::unique_ptr<
      T, detail::unique_ptr_deleter<T> > type;
This can be used as follows:
const unique_ptr<RSA>::type rsa(RSA_new(), RSA_free);