Debugging VxWorks Semaphores – give them a name!

Debugging semaphore interactions can be a tricky problem. When there is a deadlock or weird delay because of excessive priority inheritance, it is never just one mutex semaphore, but often the unintended interaction of several semaphores in multiple tasks. Some of the semaphores are part of the VxWorks I/O services, and some are in your application.

Your tools tell you what the semaphore ID is, but often that is not enough to identify the semaphore. If it is someone else’s code, and they have created lots of semaphores, you can spend hours trying to identify which bits of code are involved.

To add to the complexity, some semaphores only exist for a few microseconds, so examining the ID in a log does not help you understand where it was created. During my tenure as a field engineer I have helped more than one Wind River customer debug tricky issues with semaphores, and I have the missing hair to prove it.

Wouldn’t it be nice if you could identify each semaphore with a unique name?  “Yes,” you say, “that would be nice, but I am not going to rewrite all the code to use a different API.”

Would you be willing to add another header in the existing code?

In recent versions of VxWorks, there is the concept of a named semaphore that you create with semOpen() rather than semXCreate(). The primary use of semOpen() is to share a semaphore between memory contexts, so the same semaphore can be used in the kernel and an RTP, or multiple RTPs. The semOpen can also create a private semaphore that has the same scope as a semaphore instantiated with semXCreate(); (just don’t put a backslash at the beginning of the name).

The astute reader will have guessed what comes next? Yes, some example code with fancy macros. First the trivial example of the modified code with a new header semCreateWithOpen.h.

And then of course the fancy headers contents:

The tools will show you the name if it is present, you do not need to use an alternate API or different configuration. Try out semShow() and see what happens? You should see something like this:

Now do you know exactly where that semaphore came from?

You will need to add the Extended object library (SEM_OBJ_OPEN) support in your kernel, if you have not been configured for RTP support. You will want to add similar functions to the header for counting, and read-writing semaphores if you use them.

Would your team of experienced VxWorks engineers like a few more debugging hints? I would like to recommend our VxWorks Application Debugging Use Cases as an excellent way of spending some hands on time upgrading their skills.