Development: Buildbot

From WxWiki
Revision as of 03:45, 13 April 2015 by VZ (talk | contribs) (Update for the new buildbot master)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Our Buildbot status page:

There is also a second status page with access to run builds manually:

The Buildbot documentation for the version currently used:


Things to do

  • Set up slaves.
  • Update the wx website to mention the buildbot.
  • Use 'make install' and build and run programs against the installed libs and headers.
  • Test wx-config, try compiling using it and take it through its basic options.
  • Add binary compatibility tests, test programs from one version against libs from another with ldd -r.
  • Set up wxPython builds if they are wanted.
  • We need to be able to see config.log when configure fails.

Editing the Configuration

Master configuration lives in


Setting up a Slave

Instructions for each platform are here:

Instructions for setting up a Windows slave are here:

Your slave needs to be able to make outgoing connections to the build master on, and to the SVN server or the York tarball site. It doesn't need to be able to accept incoming connections, the slave initiates the connection to the master rather than the other way around. Therefore buildbot slaves don't require a fixed IP address and can be behind a NAT router.

The build master is using SSL, which is not directly supported by buildbot slaves at the moment. However you can connect to it by installing stunnel

You choose your own slave name and password and let Vadim know these and the IP address(es) you will be connecting from.

The simplest approach in most cases for setting up the right PATH and environment variables for the compiler you will be using, is to do it as part of the slave's startup. Either set up the environment before running buildbot, or you can do it by modifying the os.environ dict in buildbot.tac.

If you will need more than one environment on the same machine, say you will be doing both VC++ and Watcom builds for example, then setting up multiple slaves on the same machine is simplest in most cases.

As the buildbot docs say, the slave should have its own user account. This prevents the buildbot from modifying anyone else's files, or interfering with other processes by doing things like sending signals or using debugging APIs on them.

The build slave will be running scripts and programs out of the wxWidgets repository unchecked.

WARNING: Sandbox support is not implemented yet in the new master configuration, currently all commands are sent directly to the slave, ignore the rest of this section.

To help with this the build master prefixes all commands it sends with the word 'sandbox'. This allows you do implement a program or script 'sandbox' on your slave to run the SVN code safely. This can change to another user account from the build slave so that it cannot do things like access the build slave's password or reconfigure it to take commands from another build master.

Here are some examples of no-op sandbox scripts you can use initially. Once you have your slave working you can replace the no-op script with one that gives you more security if you require it, and post it here if it will be useful to others.

For unix systems you can use:

exec sh -c "$@"

For Windows you can use sandbox.bat:

@%COMSPEC% /c %*

For Cygwin, you could run buildbot under Cygwin python. In that case it behaves as a unix system and you could use the unix sandbox script above.

Alternatively, you could set buildbot up as a Windows slave running under the Windows version of python, then for sandbox.bat:

@c:\cygwin\bin\sh -c %*

This approach has the advantage that you can use the buildbot Windows service, and would be more efficient especially if you are running more slave processes for other compilers on the same machine.

Similarly for an MSYS sandbox.bat:

@c:\msys\1.0\bin\sh -c %*

Any <sandbox> elements in your builds are sent as extra parameters to your sandbox script, and it is entirely up to you what the sandbox does with them. For example, if you wanted to use multiple compilers on the same machine, say VC++ and Watcom, then the recommended approach is to set up multiple slaves on the machine. An alternative approach with just one slave process would be for your sandbox.bat to call either VC's vcvars32.bat or Watcom's setvars.bat to set up the build environment depending on the value of the first parameter. Then in your builds you could put <sandbox>MSVC</sandbox> or <sandbox>Watcom</sandbox> to select between them.

Commands sent to your build slave for execution as part of a build step are formed as follows: arg[0] is the word 'sandbox', the content of each of the build's <sandbox> elements is appended as a separate argument and all the <command> elements of the build step are concatenated with '\n' and appended as a single argument. This is then executed without shell expansion. The <sandbox> and <command> arguments are in the order they appear in the build.

For example:



then for the <compile/> build step your slave receives:

   'sandbox', 'foo', 'make all', 'bar'

Note that having multiple <command> elements in a step is not usually useful as not all platforms can handle '\n' in parameters.

Sending in Logs from Builds Done Elsewhere

WARNING: Support for this is not implemented in the new master configuration.

If running a full build slave is not an option (e.g. because you are behind a firewall that does not allow the necessary outgoing connections) but you would still like to do regular test builds, then it is possible to do the builds elsewhere and then upload the logs to the buildbot.

For each column you will require in the waterfall display add something like the following to push.xml in SVN under trunk/build/buildbot/config:

    <name>wxOS2 Stable gcc</name>

        <show log="update"/>
        <show log="configure"/>
        <show log="compile"/>
        <show log="demos"/>
        <show log="samples"/>

There is a 'buildbot' category in the wxWidgets patch manager that you can use if you are doing this by supplying patches, there should be no delay applying changes to your own configuration.

You should have one column for each OS/Toolkit/Branch/Compiler combination you are building. The <name> and <builddir> must be unique across all build slaves.

Then when you do your test builds, you should generate two files for each <show log=...> in your configuration: a '.log' file containing the log itself, and a '.err' file which is a text file contain just the decimal exit code of that build step.

For example, for <show log="configure"> you should generate configure.log and configure.err.

These should then be zipped or tarred and gzipped into a single archive file.

To upload your logs go to the buildbot admin page, and click on your build's name at the top of its column. In there you will find a form which will allow you to upload your log archive file.

In the required 'revision' field you should supply the SVN revision of the sources you have built. If you are getting the sources with 'svn update', then it is reported at the end of the update. Or you can check it later using 'svn info'. The 'branch' can be left blank since it is indicated in the <name>.

Uploading can also be done programmatically using curl, for example:

$ curl -u user:pass --cacert tt.cacert.pem \
    -F revision=r12345 -F [email protected] \

Note that the <name> of your build appears in the url.

The certificate file tt.cacert.pem is:

$ openssl s_client -connect < /dev/null | \
        openssl x509 > tt.cacert.pem
$ cat tt.cacert.pem

Setting up your automatic builds for sending in Logs

While I am setting up some automatic builds, I'll add some notes on what needs to be done to try and simplify others' life.

If you are not setting up a build slave, chances are that you are behind a firewall with a HTTP proxy. In that case, you might have other problems as well. For me, the first one was to access svn. Your proxy needs to support some extensions to allow svn to work, e.g. squid appears to require an additional configuration line like:


Supposedly, setting up desproxy( might be a solution as well, however for me it failed to work, always claiming that refused the connection, so I attempted to just use our local proxy which seemed to work fine.

A completely different and maybe easier to handle solution would be to download the daily tarball(s) of the branch(es) you are interested in, however currently only HEAD seems to be available in this way.

Anyway, once you do have the (initial) sources you want to build, create a build directory (e.g. build/solaris), go to that directory and do your build, e.g. by doing:

  ../../configure --with-motif > configure.log 2>&1 
  echo $? > configure.err
  make > make.log 2>&1 
  echo $? > make.err
  # Optionally build demos and/or samples
  tar cvf sol-motif.tar configure.log \
      configure.err make.log make.err \
      #optionally more files ...
  gzip -9 sol-motif.tar

(together with a command to update your sources, you could put this in a script to be run by a cron demon). Finally upload the generated tar.gz to the buildbot admin page as described above.

Notes for specific platforms

OSX (10.5)


Download of Twisted 8.0.1 here

Download of sources of buildbot and building according to README with python ./ install

Install 4 way binary of cppunit:

$ ./configure --disable-dependency-tracking CXXFLAGS="-arch ppc -arch i386 -arch ppc64 -arch x86_64 -gdwarf-2 -O2"
$ make AM_LDFLAGS="-XCClinker -arch -XCClinker ppc -XCClinker -arch -XCClinker i386 -XCClinker -arch -XCClinker ppc64 -XCClinker -arch -XCClinker x86_64"
$ sudo make install

in the stunnel 4 configuration file I've had to set a path with write access for th pid, some socket infos

socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

and then the service configuration for the buildbot

; Service-level configuration

client = yes
accept  = 9989
CAfile = /Users/sandboxuser/stunnel/bb.cert.pem
verify = 2
connect =
sslVersion = SSLv3

I've put both the .conf and the .pem file into a folder stunnel in the user's home, so I can startup the connection using

stunnel ~/stunnel/stunnel.conf

if you have problems then add the following debug settings at the beginning of the file

debug = 7
foreground = yes

you will then be able to read the output directly in Terminal


A really clear guide to setting up and running a Windows Buildbot for wxWidgets is available here.

Debian Based Linux Systems

Currently writing the instructions: here.