Development: Buildbot

From WxWiki
Jump to navigation Jump to search

Our Buildbot status page: http://buildbot.tt-solutions.com/wx/

There is also a second status page with access to run builds manually: https://www.tt-solutions.com:81/buildbot/wx-admin/

The Buildbot documentation for the version currently used: http://docs.buildbot.net/0.8.9/manual/

Master

Things to do

  • Set up slaves.
  • Update the wx website to mention the buildbot.
  • Use 'make install' and build and run programs against the installed libs and headers.
  • Test wx-config, try compiling using it and take it through its basic options.
  • Add binary compatibility tests, test programs from one version against libs from another with ldd -r.
  • Set up wxPython builds if they are wanted.
  • We need to be able to see config.log when configure fails.

Editing the Configuration

Master configuration lives in https://github.com/wxWidgets/buildbot


Slave

Setting up a Slave

Instructions for each platform are here: http://buildbot.net/trac/wiki#DownloadInstall

Instructions for setting up a Windows slave are here: http://buildbot.net/trac/wiki/RunningBuildbotOnWindows

Your slave needs to be able to make outgoing connections to the build master on buildbot.tt-solutions.com:9990, and to the SVN server or the York tarball site. It doesn't need to be able to accept incoming connections, the slave initiates the connection to the master rather than the other way around. Therefore buildbot slaves don't require a fixed IP address and can be behind a NAT router.

The build master is using SSL, which is not directly supported by buildbot slaves at the moment. However you can connect to it by installing stunnel http://www.stunnel.org/.

You choose your own slave name and password and let Vadim know these and the IP address(es) you will be connecting from.

The simplest approach in most cases for setting up the right PATH and environment variables for the compiler you will be using, is to do it as part of the slave's startup. Either set up the environment before running buildbot, or you can do it by modifying the os.environ dict in buildbot.tac.

If you will need more than one environment on the same machine, say you will be doing both VC++ and Watcom builds for example, then setting up multiple slaves on the same machine is simplest in most cases.

As the buildbot docs say, the slave should have its own user account. This prevents the buildbot from modifying anyone else's files, or interfering with other processes by doing things like sending signals or using debugging APIs on them.

The build slave will be running scripts and programs out of the wxWidgets repository unchecked.

WARNING: Sandbox support is not implemented yet in the new master configuration, currently all commands are sent directly to the slave, ignore the rest of this section.

To help with this the build master prefixes all commands it sends with the word 'sandbox'. This allows you do implement a program or script 'sandbox' on your slave to run the SVN code safely. This can change to another user account from the build slave so that it cannot do things like access the build slave's password or reconfigure it to take commands from another build master.

Here are some examples of no-op sandbox scripts you can use initially. Once you have your slave working you can replace the no-op script with one that gives you more security if you require it, and post it here if it will be useful to others.

For unix systems you can use: 

#!/bin/sh
exec sh -c "$@"

For Windows you can use sandbox.bat: 

@%COMSPEC% /c %*

For Cygwin, you could run buildbot under Cygwin python. In that case it behaves as a unix system and you could use the unix sandbox script above.

Alternatively, you could set buildbot up as a Windows slave running under the Windows version of python, then for sandbox.bat: 

@c:\cygwin\bin\sh -c %*

This approach has the advantage that you can use the buildbot Windows service, and would be more efficient especially if you are running more slave processes for other compilers on the same machine.

Similarly for an MSYS sandbox.bat: 

@c:\msys\1.0\bin\sh -c %*

Any <sandbox> elements in your builds are sent as extra parameters to your sandbox script, and it is entirely up to you what the sandbox does with them. For example, if you wanted to use multiple compilers on the same machine, say VC++ and Watcom, then the recommended approach is to set up multiple slaves on the machine. An alternative approach with just one slave process would be for your sandbox.bat to call either VC's vcvars32.bat or Watcom's setvars.bat to set up the build environment depending on the value of the first parameter. Then in your builds you could put <sandbox>MSVC</sandbox> or <sandbox>Watcom</sandbox> to select between them.

Commands sent to your build slave for execution as part of a build step are formed as follows: arg[0] is the word 'sandbox', the content of each of the build's <sandbox> elements is appended as a separate argument and all the <command> elements of the build step are concatenated with '\n' and appended as a single argument. This is then executed without shell expansion. The <sandbox> and <command> arguments are in the order they appear in the build.

For example: 

<build>
    ...
    <sandbox>foo</sandbox>
    ...
    <steps>
        ...
        <compile/>
        ...
    </steps>

    <sandbox>bar</sandbox>
</build>

then for the <compile/> build step your slave receives: 

   'sandbox', 'foo', 'make all', 'bar'

Note that having multiple <command> elements in a step is not usually useful as not all platforms can handle '\n' in parameters.

Sending in Logs from Builds Done Elsewhere

WARNING: Support for this is not implemented in the new master configuration.

If running a full build slave is not an option (e.g. because you are behind a firewall that does not allow the necessary outgoing connections) but you would still like to do regular test builds, then it is possible to do the builds elsewhere and then upload the logs to the buildbot.

For each column you will require in the waterfall display add something like the following to push.xml in SVN under trunk/build/buildbot/config: 

<build>
    <name>wxOS2 Stable gcc</name>
    <builddir>psh_os2_stable</builddir>

    <steps>
        <extractlogs/>
        <show log="update"/>
        <show log="configure"/>
        <show log="compile"/>
        <show log="demos"/>
        <show log="samples"/>
    </steps>
</build>

There is a 'buildbot' category in the wxWidgets patch manager that you can use if you are doing this by supplying patches, there should be no delay applying changes to your own configuration.

You should have one column for each OS/Toolkit/Branch/Compiler combination you are building. The <name> and <builddir> must be unique across all build slaves.

Then when you do your test builds, you should generate two files for each <show log=...> in your configuration: a '.log' file containing the log itself, and a '.err' file which is a text file contain just the decimal exit code of that build step.

For example, for <show log="configure"> you should generate configure.log and configure.err.

These should then be zipped or tarred and gzipped into a single archive file.

To upload your logs go to the buildbot admin page, https://www.tt-solutions.com:81/buildbot/wx-admin and click on your build's name at the top of its column. In there you will find a form which will allow you to upload your log archive file.

In the required 'revision' field you should supply the SVN revision of the sources you have built. If you are getting the sources with 'svn update', then it is reported at the end of the update. Or you can check it later using 'svn info'. The 'branch' can be left blank since it is indicated in the <name>.

Uploading can also be done programmatically using curl, for example: 

$ curl -u user:pass --cacert tt.cacert.pem \
    -F revision=r12345 -F [email protected] \
    https://www.tt-solutions.com:81/buildbot/wx-admin/wxOS2%20Stable%20gcc/push

Note that the <name> of your build appears in the url.

The certificate file tt.cacert.pem is: 

$ openssl s_client -connect www.tt-solutions.com:81 < /dev/null | \
        openssl x509 > tt.cacert.pem
$ cat tt.cacert.pem
-----BEGIN CERTIFICATE-----
MIIDnDCCAwWgAwIBAgIJANQkXozX62c5MA0GCSqGSIb3DQEBBQUAMIGRMQswCQYD
VQQGEwJGUjEKMAgGA1UECBMBIDEQMA4GA1UEBxMHQWNoZXJlczEaMBgGA1UEChMR
VFQtU29sdXRpb25zIFNBUkwxHTAbBgNVBAMTFHd3dy50dC1zb2x1dGlvbnMuY29t
MSkwJwYJKoZIhvcNAQkBFhp3ZWJtYXN0ZXJAdHQtc29sdXRpb25zLmNvbTAeFw0w
ODA1MTYyMzQ0MThaFw0xODA1MTQyMzQ0MThaMIGRMQswCQYDVQQGEwJGUjEKMAgG
A1UECBMBIDEQMA4GA1UEBxMHQWNoZXJlczEaMBgGA1UEChMRVFQtU29sdXRpb25z
IFNBUkwxHTAbBgNVBAMTFHd3dy50dC1zb2x1dGlvbnMuY29tMSkwJwYJKoZIhvcN
AQkBFhp3ZWJtYXN0ZXJAdHQtc29sdXRpb25zLmNvbTCBnzANBgkqhkiG9w0BAQEF
AAOBjQAwgYkCgYEA0xC9fy7RW6vbaUV7Ymq+kclDWW0hfMU6ueWL69FKqmWhFVqw
hQge3oYMM62NaeFaRMWLoXI/Md+uWY1/NK/8hJXhTxpC4XfP6iTfCuo/mlOPoM/o
NzWDKLxqMsf9qBMzvo1Ln57QeBY9mZNOemXpHjLC38kM7CImdc/XC/vDaxUCAwEA
AaOB+TCB9jAdBgNVHQ4EFgQUQieYsnkkxvJdzY4trVVASkJt7cswgcYGA1UdIwSB
vjCBu4AUQieYsnkkxvJdzY4trVVASkJt7cuhgZekgZQwgZExCzAJBgNVBAYTAkZS
MQowCAYDVQQIEwEgMRAwDgYDVQQHEwdBY2hlcmVzMRowGAYDVQQKExFUVC1Tb2x1
dGlvbnMgU0FSTDEdMBsGA1UEAxMUd3d3LnR0LXNvbHV0aW9ucy5jb20xKTAnBgkq
hkiG9w0BCQEWGndlYm1hc3RlckB0dC1zb2x1dGlvbnMuY29tggkA1CRejNfrZzkw
DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQUFAAOBgQBA4fCC/1E5YD1XztAFiQtp
yR5ubFnZwU4MgCoGigoG5bGIekf9Ot6h9qXBBPQH20LMvGGGj0odVlHpO7DFfnFC
XlhO1klpgclpJIeALt+PSqT4Nb1PnIA0gDN7prbMVxsVhpP+qAvlGmii113sTIc8
yHZiHiKFLL0x7pIjD8n0dA==
-----END CERTIFICATE-----

Setting up your automatic builds for sending in Logs

While I am setting up some automatic builds, I'll add some notes on what needs to be done to try and simplify others' life.

If you are not setting up a build slave, chances are that you are behind a firewall with a HTTP proxy. In that case, you might have other problems as well. For me, the first one was to access svn. Your proxy needs to support some extensions to allow svn to work, e.g. squid appears to require an additional configuration line like: 

extension_methods REPORT MERGE MKACTIVITY CHECKOUT

Supposedly, setting up desproxy(http://www.sourceforge.net/projects/desproxy) might be a solution as well, however for me it failed to work, always claiming that svn.wxWidgets.org refused the connection, so I attempted to just use our local proxy which seemed to work fine.

A completely different and maybe easier to handle solution would be to download the daily tarball(s) of the branch(es) you are interested in, however currently only HEAD seems to be available in this way.

Anyway, once you do have the (initial) sources you want to build, create a build directory (e.g. build/solaris), go to that directory and do your build, e.g. by doing: 

  ../../configure --with-motif > configure.log 2>&1 
  echo $? > configure.err
  make > make.log 2>&1 
  echo $? > make.err
  # Optionally build demos and/or samples
  tar cvf sol-motif.tar configure.log \
      configure.err make.log make.err \
      #optionally more files ...
  gzip -9 sol-motif.tar

(together with a command to update your sources, you could put this in a script to be run by a cron demon). Finally upload the generated tar.gz to the buildbot admin page as described above.

Notes for specific platforms

OSX (10.5)

[CS]

Download of Twisted 8.0.1 here

Download of sources of buildbot and building according to README with python ./setup.py install

Install 4 way binary of cppunit: 

$ ./configure --disable-dependency-tracking CXXFLAGS="-arch ppc -arch i386 -arch ppc64 -arch x86_64 -gdwarf-2 -O2"
$ make AM_LDFLAGS="-XCClinker -arch -XCClinker ppc -XCClinker -arch -XCClinker i386 -XCClinker -arch -XCClinker ppc64 -XCClinker -arch -XCClinker x86_64"
$ sudo make install
stunnel

in the stunnel 4 configuration file I've had to set a path with write access for th pid, some socket infos 

/Users/sandboxuser/stunnel4.pid
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

and then the service configuration for the buildbot 

; Service-level configuration

[buildbot]
client = yes
accept  = 9989
CAfile = /Users/sandboxuser/stunnel/bb.cert.pem
verify = 2
connect = tt-solutions.com:9990
sslVersion = SSLv3

I've put both the .conf and the .pem file into a folder stunnel in the user's home, so I can startup the connection using 

stunnel ~/stunnel/stunnel.conf

if you have problems then add the following debug settings at the beginning of the file 

debug = 7
foreground = yes

you will then be able to read the output directly in Terminal

Windows

A really clear guide to setting up and running a Windows Buildbot for wxWidgets is available here.

Debian Based Linux Systems

Currently writing the instructions: here.

Retrieved from "https://wiki.wxwidgets.org/index.php?title=Development:_Buildbot&oldid=10306"