27.06.2013 - Securely storing passwords in databases

Whenever I hear about users credential data being stolen from servers I wonder how good (or bad) it was stored in regards of security. There are several layers of security I apply to passwords. The following article will explain these layers and my reasons for them. I’ll explain these starting with the worst example I’ve ever seen in production.

Starting of really bad

I was assinged to work on a web application once, that contained the following table:


This was a table storing the plain text password along with the username and the email address. This is the absolute worst case scenario, because users tend to reuse the same username, email and password combinations. It is even likely that the password for the e-mail address would be the same as stored in the database. If an attacker would get access to this database it would not only harm the application the password was stolen from, but possibly the whole digital identity. If the same credentials were used for an online store it is possible the attacker could get access to credit card numbers and similar sensitive information.

Another very critical point is that the password may contain personal information (like religious view or sexual orientation) you are not allowed to store without explicit permission of the user. I’m not aware of any lawsuite regarding this topic, but I wouldn’t want to be the first.

Hashing passwords

To improve this it is actually a wise decision never to store the users password, only a hashcode of this password. I rely on hashing algorithms natively supported the technology stack used when developing an application. In the past I commonly used MD5, but nowadays I tend to use SHA-1. Feel free to pick whatever hashing algorithm suits your needs or that you are familiar with. The only restriction is that you should never use an algorithm known to be weak or even one that was already broken.

Important note: Some might argue SHA-1 is already broken. This is formally correct, but right now there is no implementation for the strategy proposed in breaking it. Next to that it’s an algorithm supported by many layers of the technology stack (e.g. MariaDB and Java). I would no longer use it if I would be starting a new application, but rather SHA-256.


Now you only have stored the password with a little bit of security applied.

Protecting users with weak passwords

It is hard to determine the plain text password from the stored hashes, and it is really expensive. You can use brute force to calculate the hashcode for lots and lots of passwords until you find a match. But the greatest threat to breaking the passwords are rainbow tables. A rainbow table basically is a really large table storing the hashcode along with its input. The table is then used to lookup hashcodes to quickly get the input. Common passwords, like ‚123456‘, will be broken within a fraktion of seconds. To protect users with weak passwords you need to use ’salting‘. This basically means you append a constant random string to the password prior to calculating the hashcode. If the users passwords was 123456 and our salt would be atho3aifeiR2, the string sent to the hashcode calculation would be atho3aifeiR2123456. If you choose your random string wisely it’s unlikely rainbow tables will contain this. Next to that most of the rainbow tables were built using only short passwords. If the salt itself is lengthy (e.g. 12 chars and more) it provides and additional chance of it not being in the rainbow table. So never use short and easy salts, like ‚1‘ because this does not provide any security at all.

Double hashing passwords

Now your passwords are stored pretty secure. But if the attacker was able to obtain your salt with the hashed passwords he could still build a rainbow table for all passwords starting with your salt. For simple passwords this would still harm the security. I’m using a chain of hashing algorithms to work around this. Basically you salt the password and hash it as done before, but after that you go ahead and calculate the hashcode of the hashcode. You can repeat this step several times, like md5(md5(md5('atho3aifeiR2' + password))). This way a possible attacker can not use any rainbow table but only rely on brute force attacks. The great advantage of this approach is that the attacker needs to know your implementation to actually produce usable results. If he was able to break into your database server but not your application server, your users passwords are safe.

Protecting against weak algorithms

f you used a weak algorithm (or one that was broken after you introduced it) your users still could be at risk. If you use a chain of different algorithms this threat is actually minimized. So our pseudo-code looks like this SHA2(sha1(md5('atho3aifeiR2' + password)), 256). In case all of these were broken, you can still wrap the whole chain using a stronger algorithm, like SHA2(SHA2(sha1(md5('atho3aifeiR2' + password)), 256), 512), and apply the outer most hashfunction to your existing data, e.g. UPDATE user SET password = SHA2(password, 512).

18.03.2013 - Repair broken paging in Adobe LiveCycle ES4 after upgrading from ES2

My company maintains a software of one of our customers that is using Adobe LiveCycle to render HTML/PDF forms. Our customer got a support contract with us and with Adobe so that we will keep the software running on the new LiveCycle versions. This meant that we were updating our software recently to use ES4. The update went pretty smooth except for a single fact: Paging would no longer work to any page after the second page.

The paging itself was implemented using the simple server-side-script “xfa.host.pageDown()”. Paging from the first to the second page worked fine and paging back as well. Just paging from page 2 to page 3 would just bring us back to page 2. So we started to research that. We came up with a testcase that worked and simplified our production code and added complexity to the testcase until there were almost no differences in the requests sent to the LiveCycle webserivce. The only remaining difference was how the XDP file was sent to the server. In both cases we just provided an URL. In our testcase we were using a simple Apache httpd hosted web directory, in our production code we were hosting it using a custom Java Servlet from a database. We dumped the traffic between the LiveCycle server and the file server hosting the XDP and finally found a tiny but important difference. The webserver hosted file was using “Last-Modified” and “If-Modifed-Since”, which seems to be the default for Apache httpd. The traffic looked like this:

  • the initial GET request came in from the LiceCycle server, containing a “Last-Modified: 1970-01-01”, Apache httpd return HTTP status code 200 with data and a modification timestamp of the file
  • following GET requests from the LiceCycle server contained a “Last-Modified” with the timestamp of the file, and Apache httpd returned HTTP status code 304 Not modified

Our servlet wasn’t using the Last-Modified header and was returning every request with HTTP status 200 and the data. This tiny little difference caused LiveCycle not to jump to the third page. So what do we learn from this experience: Use HTTP status codes properly and evaluate Last-Modified header manually (if your framework doesn’t do this for you). And make sure that your application logic return status code 304 on consecutive requests by LiveCycle if the content didn’t change.

13.02.2013 - Testing different libGL implementations

I recently learned that the Intel i965 hardware is capable of handling GLSL 1.30, but nobody got around to add support for this in the DRI/mesa 3D driver. The developers at Intel are focusing on their latest hardware which means GLSL 1.30 support won’t be added soon. The core mesa supports GLSL 1.30 and later generations of the Intel hardware are supported as well. Luckily Intel provides all the https://01.org/linuxgraphics/documentation/driver-documentation-prms that allows anyone to implement it.

But how do we get started? First of all you need to test the driver and see where you are starting. The best way to do this is using http://cgit.freedesktop.org/piglit. Piglit is an awesome tool maintained by the open source mesa developers to perform reproducible tests. You can see it as a conformance suite or a regression test. So the first thing is figure out what tests from piglit can be used to check for GLSL 1.30 support. Piglit ships lots of tests for specification conformance, including tests/spec/glsl-1.30.

Piglit also ships some test profiles, including all.tests. This will execute all tests known to piglit. For our purpose it is sufficient only to run the GLSL 1.30 tests. I went ahead and commented the tests not relevant for now.

$ ./piglit-run.py tests/all.tests results/2013-02-13_base
$ ./piglit-run.py tests/all.tests results/2013-02-13_base_130
$ ./piglit-summary-html.py --overwrite out results/2013-02-13_base*

I started with the command listed above. It will run the piglit tests using your currently installed driver. The first run will be using GLSL 1.20 which is announced by the driver, the second run will use 1.30. After that you can generate a nice HTML form to show the differences. If everything went fine you will quickly see tests changing from skipped to failed or passed. These will be the tests you will need to look into.

For me the tests caused a GPU lockup somewhere in the tests. This is to be expected when overriding the GLSL version to something not supported by the driver. Only use this when you are developing the graphics driver! Piglit already wrote parts of the result file so I started to compare the incomplete differences and picked the first that failed. In my case this is varying-packing-simple float separate -auto -fbo which caused a segfault (it’s not the GPU lockup, but let’s start with an easy example). Before you start of installing debug packages for you distribution you should verify that a failure is still present in the latest git code. It is best to build mesa using the same configuration options as your distribution does (I’m not going into the details of how to get these as they vary from distribution to distribution).

Now on to the interesting part: How do you use a different version of libGL and the DRI drivers without messing with your installed drivers? It’s pretty easy once you know the trick. First of all you need to tell the linker where to look for shared objects.

$ ldd varying-packing-simple float separate
  libGL.so.1 => /usr/lib/libGL.so.1 (....)
$ export LD_LIBRARY_PATH=/home/cbrill/source/mesa/lib
$ ldd varying-packing-simple float separate
  libGL.so.1 => /home/cbrill/source/mesa/lib/libGL.so.1 (....)

Now you managed to load a different libGL.so when you are executing you tests. But be aware: libGL will dynamically load your installed hardware drivers! This might cause some crashes if the installed driver does not match the development libGL.

$ export LIBGL_DRIVERS_PATH=/home/cbrill/source/mesa/lib
$ export EGL_DRIVERS_PATH=/home/cbrill/source/mesa/lib/egl

Now you are using your a local copy of mesa to test your implementation. The benefit of user space 3D drivers is pretty huge here, because otherwise you’d have to load/unload a kernel module very often. And debugging crashes in kernel space is much more complicated than in user space.

In my case the segfaulting test no longer crashed when using mesa from git. I’m currently in the process of generating a piglit run but I get a GPU lockup and don’t see the results. A good recommendation would be to run tests from a different machine that is not affected by your test (e.g. serial console or ssh connection). To run applications on the display when connected by ssh you need to do the following on you test machine

$ xhost +

After that that you can run any command, e.g. glxgears, on that display by doing:

$ DISPLAY=:0 glxgear

Note: for a complete list of environment variables see the https://www.mesa3d.org/envvars.html.

06.12.2012 - How to create a Android file browser in 15 minutes

Android provides very useful components to create Apps very fast. That sounds like a marketing claim, does it? OK, let’s see if that acutally is true! Let’s create an Android file browser.

Let’s start off by create a default „Hello World“-App that will consist of a layout file and a matching activity. Knowning what we want to build we name the layout activity_list_files.xml and name the Activity ListFileActivity. If you are developing using eclipse the default layout editor will show up displaying your wonderful „Hello World“-App. Let’s start by replacing the layout using the following code:

<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
  android:layout_width="match_parent" android:layout_height="match_parent">
  <ListView android:id="@android:id/list" android:layout_width="match_parent"
    android:layout_height="wrap_content" />

Now you have a very simple layout which only consists of a single element, a ListView. The ListView will be the component displaying our files. That’s it? Yes it is! But how will it be populated with data? That’s where our activity comes into play.

public class ListFileActivity extends ListActivity {

  private String path;

  protected void onCreate(Bundle savedInstanceState) {

    // Use the current directory as title
    path = "/";
    if (getIntent().hasExtra("path")) {
      path = getIntent().getStringExtra("path");

    // Read all files sorted into the values-array
    List values = new ArrayList();
    File dir = new File(path);
    if (!dir.canRead()) {
      setTitle(getTitle() + " (inaccessible)");
    String[] list = dir.list();
    if (list != null) {
      for (String file : list) {
        if (!file.startsWith(".")) {

    // Put the data into the list
    ArrayAdapter adapter = new ArrayAdapter(this,
        android.R.layout.simple_list_item_2, android.R.id.text1, values);

  protected void onListItemClick(ListView l, View v, int position, long id) {
    String filename = (String) getListAdapter().getItem(position);
    if (path.endsWith(File.separator)) {
      filename = path + filename;
    } else {
      filename = path + File.separator + filename;
    if (new File(filename).isDirectory()) {
      Intent intent = new Intent(this, ListFileActivity.class);
      intent.putExtra("path", filename);
    } else {
      Toast.makeText(this, filename + " is not a directory", Toast.LENGTH_LONG).show();

Let’s take a closer look at the activity. First of all I’ll give a short overview of the thoughts behind this activity: The activity displays all files within a given path. If we switch the path, we switch to a new activity displaying all files within the given directory. Easy right? To achieve that we extend the ListActivity which provides us methods like getListAdapter(), setListAdapter() and onListItemClick() to interact with the list in our layout. So when our Activity is started by Android and onCreate gets called the first thing we do (after setting the layout) is: Set the path as our activitiy’s title. In the next step we are going to read all the files from the given path, but there’s a catch: Android locks several paths from being accessed by apps. This is intended and necessary to keep apps from spying on each other (the „/data“-folder for example contains all the databases from the apps, including stored credentials). For this reason we add a note to the title and we need to check if File.list() actually returns an array. The javadoc states that it will only return null if you called the method on a file, but it also is true for inaccessible directories.

The lower part of onCreate is also interesting. See how we create our data adapter? We use Android’s ArrayAdapter, which allows us to pass it a List and tell it how to render it. Really easy and actually suited for many needs. Finally the onListItemClick will launch a new instance of our ListFileActivity to display another folder. Using the back button on your device will get you back to the previous folder.

You can see that creating a very simple file browser can be done in a very short amount of time and using very few lines of code. Of course this could be improved a lot (e.g. splitting onCreate into several methods) but there are already really good file browsers available on the Google Play store (and it would take way more than 15 minutes to match their features). Anyway I hope you enjoyed this little introduction into Android development using ListView.

Thanks to Simeon M. and Laurent T. for pointing out minor typos!

28.11.2012 - Anatomie eines git Commit-Kommentars

Everyone using git as version control system will sooner or later figure out how a git-commit comment should look like. For all readers who didn’t yet figure it out, I’ll explain it in an example:

Short description, max. 80 chars

Optional longer text describing the commit. It is allowed to be multiline and
should explain the reason for the commit.

Optional signatures.

You might now ask: Why is it done like this? Let’s take a real world example to discover the reason:

mesa: remove '(void) k' lines

Serves no purpose as the k parameter is used later in the code.

This commit removed several lines of code. If the author would have omitted a comment, you would be wondering why he did that. Lucky for us git enforces commit messages. In many cases the first line of the comment isn’t really precise, but rather follows the pattern of “stating the obvious”. It helps to find a commit, but it does not necessarly explain the reason. This only becomes obvious from the following lines.


Git allows the user to add signatures to a commit message, e.g.:

Signed-off-by: Christoph Brill <webmaster@egore911.de>

These signatures carry additional information about the people involved with the commit. There is no real standard for these signatures, but the following are proven to be useful in several projects:

  • Signed-off-by: The commit was added by the signer to the repository. It is meant (more or less) as specifying an additional author
  • Reviewed-by: The commit was read by the signer and got positive feedback
  • Tested-by: The signer applied the commit locally and tested it

Everyday usage

In a perfect world everyone would be writing detailed comments (within the sourcecode as well as in commit messages). But reality show, that this is not necessarly the case. Even in very disciplined projects some commits will fail this ideal criteria. Git (and many other version control systems) allows using simply structured commit messages to better understand the commit (for other developers as well as the author at a later point), without reading all the code lines of a commit.

12.11.2012 - NullPointerException when starting JSF 2

The exception

Below is one of the exceptions I saw while migrating to JSF 2.x. Unlike others it was tricky to track the error down. This specific exception occured during startup when deploying the application to the server:

  at com.sun.faces.config.InitFacesContext.cleanupInitMaps(InitFacesContext.java:278)
  at com.sun.faces.config.InitFacesContext.(InitFacesContext.java:102)
  at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:156)

The analysis

As this is a showstopper for even starting the application it is pretty visible. But what exactly is happening here? Let’s take a deeper look at the method in question which can be found in the JSF-reference implementation Mojarra:

static Map getInitContextServletContextMap() {
  ConcurrentHashMap initContextServletContext = null;
  try {
    Field initContextMap = FacesContext.class.getDeclaredField("initContextServletContext");
    initContextServletContext = (ConcurrentHashMap)initContextMap.get(null);
  } catch (Exception e) {}
  return initContextServletContext;

The first thing to see here is that the field access is done using reflection (so no compile safety applies) and the exception is ignored and null is returned. Sadly this method does not have any documentation and no clue on what the return value should be. That means the calling code must deal with a null value. But is that the case? Let’s take a closer look:

Map <InitFacesContext, ServletContext>initContextServletContext = InitFacesContext.getInitContextServletContextMap();
Set<Map.Entry<InitFacesContext, ServletContext>> entries = initContextServletContext.entrySet();

Now we can obviously see: A NullPointerException is going to happen and you have no chance to see the reason for the actual exception. Sadly it is pretty hard to work around a swallowed exception to figure out the details. Basically it is bad style to swallow exceptions. It is better to log the error, probably even with additional information like shown below:

} catch (Exception e) {
  log.error("Could not access FacesContext#initContextServletContext, " +
    "probably JSF-API and implementation mismatch!", e);

The solution

This way we would know what the error is: the JSF-API and implementation on the classpath don’t match. In my case I had more than one JSF-API as dependency, which resulted in an API-mixture. So my recommendation is to check the classpath of your Java-application for the following things:

  • Do the JSF implementation and JSF-API match (e.g. not mixing 2.0 and 2.1)?
  • Is more than one JSF implemenation on the classpath (e.g. JSF 1.x and 2.x)?
  • Is more than one JSF -API on the classpath?

22.02.2012 - Is a DLL 32 or 64 bit?

What are the necessary steps on a Windows system to figure out whether a DLL is 32 or 64 bits. Everyone knowing the file command on Linux, will look for a similar solution on Windows. When searching online for the problem, a lot of strange solutions come up … a Perl-Skript, install Visual Studio, etc.

I found the cheapest solution when browsing the ReactOS projects webpages: http://www.dependencywalker.com/. This is a very simple tool showing for each DLL symbol whether it has 64 or 32 bits. It is that simple you don’t even need to install Dependency Walker.

19.12.2011 - mod_jk vs SELinux

Tomcat cluster behind an Apache HTTPd

Several protocols exist to run one or multiple Apache Tomcats behind an Apache HTTPd. One of the commonly used protocols is AJP/1.3-Protokoll (Apache JServ Protocol), because it easily supports load balancing. Because the SSL-termination happen within the webserver, this scenario can only be used if the communication channel between the HTTPd and Tomcat is secure.

Clustered tomcat setup


Tomcat supports AJP/1.3-protocol out of the box, but HTTPd needs mod_jk. This setup is rather simple and used by many companies. At least in theory. When testing this setup on a SELinux-enabled distribution it failed to work and I found the following log entry:

connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=13)

Looking at the debug-output of mod_jk, it shows the connection to failed. The first suspect was IPv6. I went ahead and completely disabled it:

echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6

But this did not change the situation. The service was only using IPv4, telnet worked, but Apache HTTPd still could get no connection. In the end I found the following log entry:

[error] init_jk::mod_jk.c (3235): Initializing shm:/var/log/httpd/jk-runtime-status.16551 errno=13. Load balancing workers will not function properly.

A short research brought SELinux into focus. I’m still unsure why SELinux is enabled for some workstation distributions by default. It is a good security feature, but so far I found the tooling a bit hard to use. Especially for newbies it’s complicated to get an easy overview (whether using a GUI or a simple Log-file) or find errors. My very easy workaroundis to open the file ‘/etc/selinux/config’ and add the following:


This solves all SELinx related issues. Note: This completely disables SELinux. It is acceptable for a development system within a secure environment, but this workaround should never be applied to any production system.

22.09.2011 - Tracking an upstream-project using git

Software developments should never reinvent the wheel. This is already implied by the DRY-principle. I apply the principle also to external software projects, i.e. I don’t want to repeat myself and I don’t want to repeat the work of others. This is why almost every software project depends and builds on other software projects. But sometimes a dependent software project must be adapted, modified or patched before it can be used.

Patching external software projects

Let’s take an open source project for an example, that must be modified before it can be used. (Note: Prior to changing the code of other projects, make sure to check the license first!). We take a release of the project, perform your modifications and use the modified dependency. The issue at hand: How do we apply our changes to updated versions of the external project?

The solution can be found in the version control system git. The idea is really simple, but simplifies the workflow. First of all you import an initial version of the external project (“Release 1.0” in the example below). Now you go ahead and branch into 2 directions:

  1. one branch contains the official release (upstream)
  2. another branch also containing the local changes (master)

Now we go ahead and perform our changes in the “master” branch. When a new software version is released, we import it into the “upstream” branch. Now once we merge the branch “upstream” into “master”, you get an automatically patched version. In most cases a real time saver.

Note: You don’t have to perform the import manually if the external software project uses git. Other than that workflow is identical.

Unclean start

But how do we get to a clean start if we didn’t know that git could make our life that easy? Let’s take an example that we just took the upstream project and applied our patches onto them, maybe even without a history:

From here we create a branch “upstream” from the branch “master”. Now you switch to the newly created branch, wipe out the content and import the official release (the exact same version matching the current “master” branch passt, i.e. if master was a patched version of 1.0, we would import version 1.0). Now a merge from upstream into master would result in massive merge conflicts. Now we need to aid git a bit. We inform git, that everything in the “upstream” branch is already in the master branch:

git checkout master
git merge -s ours upstream

This way the setup will allow a workflow identical to the one described above, allowing you to import new releases into “upstream” and merge them into master.

This little trick reduced the duration of hours of manual comparisons to an automated process done within minutes. Note: After creating a new patched release you of course need to test it!

Give changes back to the upstream project

This workflow is pretty easy, but a bit expensive (according to how often upstream releases need to be incorporated). My personal idea is, to hand over the changes to the upstream project, even if not explicitly required by the license. First of all it reduces my personal workload, but also provides a (commonly useful) adaption to all users and motivates the projects maintainers.

25.08.2011 - ArrayIndexOutOfBoundsException im maven-project-info-reports-plugin

Maven provides lots of tools regarding Java projects. One of the features is automatically generating HTML-sites using mavens site goal.

Today I was facing a problem regarding the page generation, which was using the maven-project-info-reports-plugin. This plugin was failing somewhere deep in the internals with an ArrayIndexOutOfBoundsException in maven-project-info-reports-plugin:

java.lang.ArrayIndexOutOfBoundsException: 1
    at org.apache.maven.doxia.module.xhtml.XhtmlSink.tableCell(XhtmlSink.java:791)

A short research brought to light, that the maven-site-plugin was to old. To use a more recent version in your project, add the following to your pom.xml:

   <!-- Version 2.3 oder neuer wird für das maven-project-info-reports-plugin benötigt -->
« Previous 1 2 3 Next »
Copyright © christophbrill.de, 2002-2018.