Thursday, July 28, 2011

The curious case of pthread_atfork on PowerPC

Recently I was asked to take a look at an issue on PowerPC whereby the symbol pthread_atfork seems to be missing from the 64-bit library because the following message was output when trying to run an application that was linked with -lpthread.
undefined symbol: pthread_atfork
(Note: while this problem was encountered on PowerPC, the general concept is relevant to all platforms.)

If we compare the symbols exported in /lib/ (32-bit shared object) to those in /lib64/ (64-bit shared object) we see that there are no exported symbols for pthread_atfork in the 64-bit shared object:

nm /lib/ | grep atfork
0000c900 t __dyn_pthread_atfork
0000c900 T pthread_atfork@GLIBC_2.0
         U __register_atfork@@GLIBC_2.3.2
nm /lib64/ | grep atfork
In the 32-bit library notice that the addresses for __dyn_pthread_atfork and pthread_atfork@GLIBC_2.0 are the same!  This indicates that pthread_atfork@GLIBC_2.0 is a version tagged symbol which is not the same as a non-versioned pthread_atfork symbol.  This is telling us that there is no dynamically exported pthread_atfork in either the 32-bit or 64-bit

So what's going on?  Since pthread_atfork hasn't been deprecated from POSIX where is it?

Let's take a look at some code from GLIBC that may help clarify things:


/* Copyright (C) 2002 Free Software Foundation, Inc.
   This file is part of the GNU C Library.
   Contributed by Ulrich Drepper <>, 2002.

   The GNU C Library is free software; you can redistribute it and/or
   modify it under the terms of the GNU Lesser General Public
   License as published by the Free Software Foundation; either
   version 2.1 of the License, or (at your option) any later version.

   The GNU C Library is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   Lesser General Public License for more details.

   You should have received a copy of the GNU Lesser General Public
   License along with the GNU C Library; if not, write to the Free
   Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
   02111-1307 USA. */

#include <shlib-compat.h>

#if SHLIB_COMPAT (libpthread, GLIBC_2_0, GLIBC_2_3)
# define __pthread_atfork __dyn_pthread_atfork
# include "pthread_atfork.c"
# undef __pthread_atfork
compat_symbol (libpthread, __dyn_pthread_atfork, pthread_atfork, GLIBC_2_0);
/* Copyright (C) 2002, 2006 Free Software Foundation, Inc.
   This file is part of the GNU C Library.
   Contributed by Ulrich Drepper <>, 2002.

   The GNU C Library is free software; you can redistribute it and/or
   modify it under the terms of the GNU Lesser General Public
   License as published by the Free Software Foundation; either
   version 2.1 of the License, or (at your option) any later version.

   In addition to the permissions in the GNU Lesser General Public
   License, the Free Software Foundation gives you unlimited
   permission to link the compiled version of this file with other
   programs, and to distribute those programs without any restriction
   coming from the use of this file. (The GNU Lesser General Public
   License restrictions do apply in other respects; for example, they
   cover modification of the file, and distribution when not linked
   into another program.)

   Note that people who make modified versions of this file are not
   obligated to grant this special exception for their modified
   versions; it is their choice whether to do so. The GNU Lesser
   General Public License gives permission to release a modified
   version without this exception; this exception also makes it
   possible to release a modified version which carries forward this

   The GNU C Library is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   Lesser General Public License for more details.

   You should have received a copy of the GNU Lesser General Public
   License along with the GNU C Library; if not, write to the Free
   Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
   02111-1307 USA. */

#include "pthreadP.h"
#include <fork.h>

/* This is defined by newer gcc version unique for each module. */
extern void *__dso_handle __attribute__ ((__weak__,
                                          __visibility__ ("hidden")));

/* Hide the symbol so that no definition but the one locally in the
executable or DSO is used. */
#ifndef __pthread_atfork
/* Don't mark the compatibility function as hidden. */
__pthread_atfork (prepare, parent, child)
     void (*prepare) (void);
     void (*parent) (void);
     void (*child) (void);
return __register_atfork (prepare, parent, child,
                            &__dso_handle == NULL ? NULL : __dso_handle);
#ifndef __pthread_atfork
extern int pthread_atfork (void (*prepare) (void), void (*parent) (void),
                           void (*child) (void)) attribute_hidden;
strong_alias (__pthread_atfork, pthread_atfork)
Here we see that __dyn_pthread_atfork is a compat symbol for an older pthread_atfork implementation bound by alias to pthread_atfork@GLIBC_2.0, a versioned symbol.  Older applications that were linked against GLIBC 2.0 and pthread_atfork will be bound to __dyn_pthread_atfork when run against newer versions of GLIBC.

Looking back at our nm output, what we don't see is a non-versioned pthread_atfork symbol being exported by /lib/ (32-bit) and there is neither that nor a versioned symbol for /lib64/ (64-bit); why?
To answer the second question first; It's because PPC64 wasn't supported as a platform in GLIBC until after pthread_atfork was aliased to __dyn_pthread_atfork. So on PPC64 there isn't a dynamically exported version of pthread_atfork@GLIBC_2.0.

To get to the bottom of the first question takes a bit more analysis. The guard, #ifndef __pthread_atfork, is only true if we're building the compat version of the symbol, i.e. building "old_pthread_atfork.c".    
Otherwise, we're building pthread_atfork.c directly, in which case __pthread_atfork is marked hidden, and a strong_alias pthread_atfork is bound to it.  Curiously this non-versioned pthread_atfork alias is also marked hidden!  This implies that this will not be exported from

We then see that the nptl/Makefile explicitly omits a non-versioned pthread_atfork symbol from being exported by the shared object file by simply excluding the pthread_atfork relocatable file from this is done using eliding in the nptl/Makefile:   
libpthread-static-only-routines = pthread_atfork
Any relocatable files elided via a lib*-static-only-routines variable are left out of the lib*.so.* shared object and built into a lib*_nonshared.a archive instead.

In general the functions in a lib*_nonshared.a archive are kept hidden, hence the reason for the curious hidden attribute on the pthread_atfork strong_alias.

If we look at libpthread_nonshared.a we will indeed see the pthread_atfork symbol.
nm /usr/lib/libpthread_nonshared.a

         w __dso_handle
00000000 T __pthread_atfork
         U __register_atfork
00000000 T pthread_atfork

nm /usr/lib64/libpthread_nonshared.a

                 w __dso_handle
0000000000000000 D __pthread_atfork
                 U __register_atfork
0000000000000000 D pthread_atfork
The libraries in GLIBC that need to use pthread_atfork are each statically linked against the libpthread_nonshared.a static archive.  The pthread_atfork symbol becomes statically linked into each individual GLIBC library and need not be exported.

So how does a user application use pthread_atfork?  By linking against, which is in fact a linker script that directs the linker to dynamically link against and statically link in libpthread_nonshared.a:

cat /usr/lib64/
/* GNU ld script
Use the shared library, but some functions are only in
the static library, so try that secondarily. */
GROUP ( /lib/ /usr/lib/libpthread_nonshared.a )

cat /usr/lib64/
/* GNU ld script
Use the shared library, but some functions are only in
the static library, so try that secondarily. */
GROUP ( /lib64/ /usr/lib64/libpthread_nonshared.a )
This will automatically statically link libpthread_nonshared.a into an application at compilation and link time.  As of 2002-11-26 the pthread_atfork function is no longer a dynamically exported symbol in GLIBC.

So why are we getting undefined symbol: pthread_atfork even when we're linking with -lpthread?  The reason most likely has to due with the order of linking.

Let's say we have a relocatable object foo.o which includes a reference to pthread_atfork and we link foo.o into a shared object in the following way:

gcc -shared -o -lpthread ./foo.o

This seems like a perfectly reasonable way to link a shared object, and if we were only requesting dynamically exported symbols from it would probably work just fine.  The problem comes in when we need to use a symbol provided by libpthread_nonshared.a, a static archive.

The link editor (static linker) is generally quite smart, but it links files in sequential order as it encounters them on the command line.  When it examines and sees that it needs to statically link in libpthread_nonshared.a it searches for outstanding undefined symbol references that this archive might satisfy.  If it finds one it links it, and its dependencies into the archive shared object.  Otherwise it purges the symbols it encounters in libpthread_nonshared.a.

So when -lpthread is added before foo.o on the linker invocation, libpthread_nonshared.a is linked and pthread_atfork is purged as unneeded before foo.o is processed.  So to solve this we need to make sure that the reference to pthread_atfork by foo.o is registered before libpthread_nonshared.a is linked.  This can be accomplished by any one of the following methods.
  1. ... -o ./foo.o -lpthread
  2. ... -o -lpthread ./foo.o -Wl,-u,pthread_atfork
  3. ... -o -Wl,--whole-archive -lpthread -Wl,--no-whole-archive ./foo.o
  4. ... -o -\( -lpthread ./foo.o \)-
The first method simply makes sure we link libpthread_nonshared.a after foo.o.

The second method explicitly tells the linker to not purge the pthread_atfork symbol.

The third method is heavy handed and tells the linker to include all of the symbols in libpthread_nonshared.a and in the final link.

The fourth method tells the linker to reprocess all of the parenthesis enclosed libraries each time a new relocatable object or shared object is processed.  This means that the pthread libraries would be processed twice, once before and once after foo.o is processed.  This can increase link time significantly.

Monday, April 25, 2011

GDB Python Script To Get The Start Address Of A Loaded Library

The following GDB Python convenience function (as defined by the script) will search through the output of (gdb) info proc mappings for the library you've specified in the function parameter and return the start address of the .text segment of that library or executable.

Here is an example output of (gdb) info proc mapping with the entry for highlighted in red.  The Start Addr is what we'll have the convenience function fetch:
   Start Addr        End Addr       Size     Offset objfile
     0x100000        0x103000     0x3000   0x100000
    0x8000000       0x8029000    0x29000          0   /home/user/glibc64_power6/elf/
    0x8039000       0x803a000     0x1000    0x29000   /home/user/glibc64_power6/elf/
    0x803a000       0x803e000     0x4000    0x2a000   /home/user/glibc64_power6/elf/
   0x10000000      0x10002000     0x2000          0   /home/user/libdfp64_power6/test-isinf
   0x10011000      0x10012000     0x1000     0x1000   /home/user/libdfp64_power6/test-isinf
0x40000000000   0x40000002000     0x2000          0
0x40000002000   0x40000195000   0x193000          0   /home/user/glibc64_power6/
0x40000195000   0x400001a4000     0xf000   0x193000   /home/user/glibc64_power6/
0x400001a4000   0x400001a8000     0x4000   0x192000   /home/user/glibc64_power6/
0x400001a8000   0x400001bf000    0x17000   0x196000   /home/user/glibc64_power6/
0x400001bf000   0x400001c3000     0x4000   0x1bf000
0x400001c3000   0x4000026b000    0xa8000          0   /home/user/glibc64_power6/math/
0x4000026b000   0x4000027a000     0xf000    0xa8000   /home/user/glibc64_power6/math/
0x4000027a000   0x4000027b000     0x1000    0xa7000   /home/user/glibc64_power6/math/
0x4000027b000   0x40000285000     0xa000    0xa8000   /home/user/glibc64_power6/math/
0x40000285000   0x40000286000     0x1000   0x285000
0x40000286000   0x400002a3000    0x1d000          0   /home/user/glibc64_power6/nptl/
0x400002a3000   0x400002b2000     0xf000    0x1d000   /home/user/glibc64_power6/nptl/
0x400002b2000   0x400002b3000     0x1000    0x1c000   /home/user/glibc64_power6/nptl/
0x400002b3000   0x400002b5000     0x2000    0x1d000   /home/user/glibc64_power6/nptl/
0x400002b5000   0x400002ba000     0x5000   0x2b5000
0x400002ba000   0x4000031a000    0x60000          0   /home/user/libdfp64_power6/
0x4000031a000   0x40000329000     0xf000    0x60000   /home/user/libdfp64_power6/
0x40000329000   0x40000332000     0x9000    0x5f000   /home/user/libdfp64_power6/
0x40000332000   0x40000334000     0x2000   0x332000
0xffffff9d000   0xffffffb3000    0x16000 0xfff9d000               [stack]

Add the following to your ~.gdbinit file, or source it directly in your .gdb script:
source /path/to/
# created by Ryan S. Arnold, April 2011.  No
# attribution required or necessary to use/reuse/copy/modify this
# function/script.

import re
class StartAddress(gdb.Function):
    """Returns the start address of a library or executable from info
    proc mappings."""

    def __init__(self):
        super (StartAddress, self).__init__ ("start_address")

    def invoke(self, library):
        mappings = gdb.execute("info proc mappings", to_string=True)
        lines = mappings.split('\n')
        to_mappings = ""
        for line in lines:
            if (to_mappings != "true"):
                if ("Start Addr", line)):
                    to_mappings = "true"

                # The first match is the .text segment. Make sure to
                # match on "/" when passed "libdfp" and
                # not on "/libdfp" in the following example:
                # /home/ryanarn/libdfp/build64_power6/
                if ("/" + library.string() + "[^/]*$", line)):
                    chunks = line.split()
                    return int(str(chunks[0]),16)
        return 0x0

StartAddress ()
There is currently a bug in GDB which crashes GDB after a direct assignment from this user function, e.g., (gdb) set $foo = $start_address("libc"). To use the function to set a variable do the following:
(gdb) p $start_address("libc")
$3 = 4398046760960
(gdb) set $foo = $
(gdb) p $foo
$4 = 4398046760960
I created this function to use in the Libdfp Makefile.gdb GNU make script.  Libdfp's Make system will automatically generate .gdb scripts for each of the test cases in the test-suite.  I use this function to find the load address of the library so that I can programmaticaly load the symbol file in the correct location, as in the following snippet of test-isinf.gdb:
source /home/ryanarn/eglibc/eglibc/libdfp/trunk/tests/
p/x $start_address("libdfp")
set $libdfp_start = $
set $libdfp_text = 0x0af40
set $libdfp_addr = $libdfp_start + $libdfp_text
add-symbol-file ./ $libdfp_addr Published the Power Architecture 32-bit ELF ABI Supplement has published the Power Architecture® 32-bit Application Binary Interface Supplement 1.0 - Linux® & Embedded.
This is a specification document that I worked on for four years as part of the PowerABI TSC, under

It defines the 32-bit Power Architecture processor specific ELF ABI for the GNU/Linux Operating System and Embedded environments.

I served as the document owner, style-sheet maintainer, and TSC Chair (for the last year).  It's very rewarding (relieving) to see this finally published.  It was a pleasure to work with everyone on the TSC.  Their expertise and contributions were invaluable.

There are three versions of the document: the Linux & Embedded (Unified), the Linux, and the Embedded.  These are all generated from the same parent document and are separated for convenience of the readers.

Sunday, April 24, 2011

Embedding SVG images that use Javascript in Blogger/Blogspot Posts

In Blogger posts if you want to embed SVG images that make use of JavaScript (for animation control) you need to use the html <embed> tag since <object> and <img> don't allow scripting. You'll have to edit the HTML directly to use the <embed> tag as in the following example:
<embed src="" type="image/svg+xml" width="230" height="230"/>
The following embedded SVG example uses the <animateMotion> SVG element with repeatCount="indefinite" so that the animation continues indefinitely.  The animation is paused and restarted by calling JavaScript functions as a result of SVG element onclick event processing, e.g., onclick='Pause(evt)' and onclick='Play(evt)'.

Saturday, April 23, 2011

SVG Animation and The SVG DOM beginElement() Method

This example attempts to restart the animation using the SVG DOM beginElement() method. It keeps the bubble going back and forth along a zig-zag path.  It restarts the animation once the bubble has returned to the start position by using beginElement ( ) upon the original <animateMotion>.  The following SVG code is just the XML necessary to show the beginElement ( ) method call.

Note: If the example animation does not restart after one complete circuit it means that your browser (like Chrome) doesn't yet support the SVG DOM beginElement ( ) method.  The animation should restart properly in the Opera browser.
<script type="text/javascript" >
      function BeginAgain(bubble) {
          var bub = SVGDocument.getElementById(bubble);
<animateMotion  id="bubble1am1" ...  >
<animateMotion  id="bubble2am1"
     onend="BeginAgain('bubble1am1')" >

SVG <animateMotion> "fill" Attribute Conformance

I've been tinkering with SVG AnimateMotion fill attribute conformance in various HTML5 capable browsers to try to figure out what is supposed to be the normative behavior.

The SVG Animation specification for the "fill" attribute indicates that the possible values for the attribute are fill = "freeze" | "remove".


The animation effect F(t) is defined to freeze the effect value at the last value of the active duration. The animation effect is "frozen" for the remainder of the document duration (or until the animation is restarted

The animation effect is removed (no longer applied) when the active duration of the animation is over. After the active end of the animation, the animation no longer affects the target (unless the animation is restarted.
The specification defines fill="remove" as the default.

The following SVG Animation fragment demonstrates the use of "fill".

<use xlink:href="#engine_v2"
          keyPoints=".07; .92"
          keyTimes="0; 1"
          fill="freeze" >
                <mpath xlink:href="#simple-track-path1" />
          keyPoints=".92; .07"
          keyTimes="0; 1"
          fill="freeze" >
                <mpath xlink:href="#simple-track-path1" />

The following live SVG Animation example demonstrates the use of fill="none" (i.e., the default), fill="freeze", and fill="remove". In the animation a train car travels a path in one direction and then travels the path in the reverse direction using two consecutive AnimateMotion elements.
Note: You will only see this SVG image animated if your browser supports SVG Animation, even then your mileage may vary as the animation may have run and completed before it displays.  Click on the image to view it

The fill attribute for the first AnimateMotion element doesn't seem to matter when there are two consecutive AnimateMotion elements.  Only the second fill attribute matters and it determines the image state following the completion of the animation.

The Chrome browser, as of version 11.0.696.16 beta, finishes the animation in the following way:
Screenshot of Chrome

The Opera browser, as of version 11.01, finishes the animation in the following way:
Screenshot of Opera
Based on the reading of the specification I believe that Opera is performing correctly in this instance.

Unfortunately the behavior of the fill attribute changes when there aren't two consecutive elements of the same type.  The following example shows an animateTransform following an animateMotion as demonstrated by the following SVG example:

In this example the train car is supposed to travel along the path, turns around 180 degrees, and then travel back along the path.
This example only works under the Opera browser where the fill attribute is used with the animateTransform element, e.g., <animateTransform fill="freeze">.  This seems to be the correct behavior.  When fill="remove" the orientation of the train car returns to the original orientation, i.e., not rotated, after the animateTransform element rotates it.  With fill="freeze" the train car retains the rotate orientation.
Screenshot of Opera
Under the Chrome browser the train disappears following the <animateTransform fill="freeze"> element.  I believe this is a bug in the Chrome browser.  It seems to behave as expected in the <animateTransform fill="remove"> example.
Screenshot of Chrome

Using Google's App Engine to serve SVG files

I've been tinkering with SVG Animation support in HTML5 lately and since I don't like to maintain web servers myself I needed somewhere convenient to host my Javascript and SVG creations. My friend Jeff Bailey suggested that I check out Google's App Engine.

While App Engine may be a bit overkill for serving static content, recent additions to SVG allow animation elements. What appears to be a static SVG file may in-fact have embedded Javascript and SVG Animation elements.

App Engine has proven to be a great environment to work with but in order for it to properly serve SVG files such that a browser can display the images rather than the XML data the server needs to export the proper MIME type information for SVG files. This is accomplished by telling the App Engine server how to serve the SVG files.

App Engine supports both Java and Python servers. I use the Python server and the server configuration files are in YAML. The following YAML fragment is added to the project's app.yaml configuration file to support SVG MIME types:
- url: /(.*\.svg)
static_files: static/\1
upload: static/(.*\.svg)
mime_type: image/svg+xml

The following is a little tech demo I've made using App Engine:

Note: SVG Animation support is still in development by most browsers. The ones I've tested are Chrome, Opera, and Firefox. The level of conformance for each of these browsers vary. This demo works best on Chrome 11.0.696.16 beta. The spacing of the train-cars is a little off in Opera. I'm not sure which browser is the most spec conformant.

The following app.yaml file is complete example that is used for the rsasandbox project:
application: rsasandbox
version: 1
runtime: python
api_version: 1

- url: /(.*\.(html|png|jpg|gif))
static_files: static/\1
upload: static/(.*\.(html|gif|png|jpg))

- url: /(.*\.svg)
static_files: static/\1
upload: static/(.*\.svg)
mime_type: image/svg+xml

- url: /(.*\.xml)
static_files: static/\1
upload: static/(.*\.xml)