Build Czar Duties

Th main duties of the Build Czar are summarized as follows

  • Monitor the builds using the Scoreboard as one of the primary source of information.
  • Notify developers who broke compilation to fix the errors as soon as possible. A red color in the "Compile" column is not at all acceptable. Build Czar should ensure that. If possible, let the developer know what the source of problem could be. It is quite possible that the developer who checked in the code or the user who provided the patch may not have resources to investigate the issues. Builds Czar's help is an absolute necessity then.
  • Keep an eye on the tests that are run in every build. Anything abnormal needs to be notified to the right developer. The Build Czar should try helping the developer by providing stack traces (in case of crashes) or other details like printouts with debugging level turned on.
  • Some tests fail in the daily builds for many reasons like known bugs, transient timeouts etc. Make sure that no new test failures show up. This guy knows most of the information. Ask him to help you out with known problems.
  • Keep an eye on the footprint and performance stats. Any obnormal changes needs to be brought to the attention of the developer resposible for it or to the DEVO group .
  • Keep the builds ticking. Any red on the "Last Finished" column in the Scoreboard needs to be attended to. The link to the "Build Name" would indicate the machine where the build is being run.
  • The builds don't cover all the possible configurations. If you get a bug report about a compile error in a particular configuration, try setting up a build to make sure that it doesnt show up again if it has been fixed.
  • Keep an eye on the bugzilla entries that are registered by users and developers. Decide on the bugs that need to be fixed for the beta and pain developers for an ETA.

    The document here talks about the powers of a build Czar.

    The Build Czar has the power to set up more builds on his own for his convinience. This page has a step by step instructions on how to do that.

    The build czar can get the build configuration by looking at the config portion of the scoreboard

    Any pro-active involvement by the build czar is welcomed. Being a pro-active build czar will mean that you watch the subversion archive carefully and respond quickly to suspected changes. At the end it will safe time because the repo stays stable.


    Recipe for Cutting a Beta/Minor Kit

    The build czar is also incharge for the release of the beta. Cutting a beta is as simple as cutting butter if things go well. Here is the procedure followed while cutting a beta:

    1. The whole process takes somewhere between 8-9 hours, about 2 hours for making the release itself and the remaining time for generating the doxygen documentation.
    2. Prior to starting this, gather aggregate release notes from all developers. This is usually in the form of an email plea asking for a writeup of significant changes since the last beta. Add these notes to the NEWS files before cutting the release so that all notes are part of the release.
    3. Checkout a new workspace on deuce.doc.wustl.edu
    4. Do not forget to checkout your MPC repository. Do it like this: cvs -z 9 co -P ACE_MPC
    5. If the repository is frozen, remove all users from that file and place your user name in the FROZEN file
    6. Set $ACE_ROOT, $TAO_ROOT and $CIAO_ROOT to point to the new workspace you checked out. You must also set $MPC_ROOT to be $ACE_ROOT/MPC.
    7. Set an environment variable SIGNATURE indicating your full name. This is used to fill the ChangeLog entry.
    8. Set an environment variable MAILID indicating your mail id. This is used to fill the mail id portion of the ChangeLog entry.
    9. Change directories to to $ACE_ROOT
    10. Do a make -f Release allsources to create a beta kit
    11. Or do a make -f Release allsources REL=minor to create a minor kit
    12. To summarise, the following is a transcript of the steps up to this point executing successfully (assumes /project/acetmp/sm exists and is empty):

      sm@beatrice ~
      $ ssh bugzilla@deuce.doc.wustl.edu
      No default printer
      bugzilla@deuce cs/group/bugzilla> cd /project/acetmp/sm
      bugzilla@deuce /project/acetmp/sm> setenv CVSROOT bugzilla@cvs.doc.wustl.edu:/project/cvs-repository
      bugzilla@deuce /project/acetmp/sm> setenv CVS_RSH ssh
      bugzilla@deuce /project/acetmp/sm> setenv ACE_ROOT $PWD/ACE_wrappers
      bugzilla@deuce /project/acetmp/sm> setenv TAO_ROOT $PWD/ACE_wrappers/TAO
      bugzilla@deuce /project/acetmp/sm> setenv CIAO_ROOT $PWD/ACE_wrappers/TAO/CIAO
      bugzilla@deuce /project/acetmp/sm> setenv MPC_ROOT $PWD/ACE_wrappers/MPC
      bugzilla@deuce /project/acetmp/sm> setenv SIGNATURE "Simon McQueen"
      bugzilla@deuce /project/acetmp/sm> setenv MAILID sm@prismtech.com
      bugzilla@deuce /project/acetmp/sm> cvs -z 9 checkout ACE_wrappers
      bugzilla@deuce /project/acetmp/sm> cvs -z 9 co -P ACE_MPC
      bugzilla@deuce /project/acetmp/sm> cd ACE_wrappers/
      bugzilla@deuce acetmp/sm/ACE_wrappers> make -f Release allsources | & tee ../release.log

      Feel free to cut and paste with suitable edits.

    13. If something goes wrong and the build needs to be restarted for some reason the following files must be returned to the state they were in before the release process started and then checked back into CVS:
      ACE_wrappers/ChangeLog
      ACE_wrappers/PROBLEM-REPORT-FORM
      ACE_wrappers/VERSION
      ACE_wrappers/TAO/ChangeLog
      ACE_wrappers/TAO/PROBLEM-REPORT-FORM
      ACE_wrappers/TAO/VERSION
      ACE_wrappers/TAO/CIAO/ChangeLog
      ACE_wrappers/TAO/CIAO/PROBLEM-REPORT-FORM
      ACE_wrappers/TAO/CIAO/VERSION
      ACE_wrappers/TAO/CIAO/ciao/Version.h
      ACE_wrappers/TAO/tao/Version.h
      ACE_wrappers/ace/Version.h

      The following tags will also need to be removed: CIAO-0_X_Y, TAO-1_X_Y, and ACE-5_X_Y (wher X and Y are the minor and beta release numbers of the release that is to be restarted).

      E.g.:
      cvs tag -d CIAO-0_4_8
      cvs tag -d TAO-1_4_8
      cvs tag -d ACE-5_4_8

    14. When ready, create a manual tag ACE+TAO+CIAO-X_Y_Z, where X_Y_Z must be replaced by major/minor/beta version (for example 1_4_7). For example, the tag might be: ACE+TAO+CIAO-1_4_8
    15. Once the distribution is ready, get ready for creating doxygen documentation. This is slightly complicated than it requires. We will address the complexity soon.
    16. Login to naboo.dre.vanderbilt.edu as bczar
    17. go to /web/users/isisbuilds/tmp/ACE_wrappers
    18. Set the environment variables pointed above.
    19. Update the workspace with the right version tag
    20. Run doxygen --version within the shell.
    21. Open up $ACE_ROOT/bin/generate_rel_mangpages in your favorite editor.
    22. Search for the string 'doxy_version'.
    23. Check the version specified here. If it is the same as the one you got using doxygen --version then you don't have to worry.
    24. If it is different change the value of the doxy_version to the one installed on naboo.dre.vanderbilt.edu.
    25. Now you are ready to create documentation
    26. Run make -f Release manpages to create the doxygen documentation.
    27. While doxygen churns, format a release announcement, including the release notes gathered from developers. You can use these as an example.
      • Let Doug Schmidt review these before you do anything with them.
    28. Check the file, generate_rel_manpages into the repository if you have made some changes to it.
    29. Make sure the new version is available in Bugzilla.
    30. Now update the documentation at www.dre.vanderbilt.edu/Doxygen.
    31. Mail the approved release announcement out to, at minimum the following: ciao-users@cs.wustl.edu, tao-users@cs.wustl.edu, tao-announce@cs.wustl.edu, ace-users@cs.wustl.edu, ace-announce@cs.wustl.edu. Do this as yourself (not as bugzilla). N.B. You will not be able to post to the users' lists unless you are subscribed to them. Odds are you will not be able to post to the announce lists at all. Ask someone else (like Doug) to do this step.


    Tips to being a Build Czar

    1. Trust no one.
    2. Be careful with this guy, he is notorious in breaking builds (and fixing them as well...Rumour has it that it's actually a super-scalar, super-pipelined processor capable of out-of-order execution, in human incarnation).
    3. Don't forgive people who break ACE :-)
    4. If a build hasn't run in a long time (symptoms are a "red" in the Last Run column of the build scoreboard), delete the .disable file in /path/to/build/directory/BUILD_NAME/ by hand.
    5. Think of the group who wrote the scoreboard update script, everytime you catch an otherwise not so obvious error with the help of the scoreboard. Tell DEVO group about it.
    6. Add $CVSROOT/CVSROOT/etc/FROZEN to freeze the repo
    7. Add names of people who need to be given permission and make sure that you add your name so that you can see what is being checked in.
    8. Leave a line at the end of the FROZEN file
    9. Compile once on Win32, Linux and Solaris before cutting a beta.
    10. Trust the release script when making a release. Don't make tar balls by hand. Make sure that the public ftp directories (/project/beguine/ftp/pub/ACE+TAO-distribution and /project/beguine/ftp/pub/ACE+TAO-distribution/diffs) have the right permissions, so that the release script can copy the tar balls.
    11. When making a release, make sure that all the auto_compiles on that machine (deuce.doc.wustl.edu) are stopped. Also make sure that there is enough space in /tmp on that machine.
    12. When all hell breaks loose, don't wait for the nightly builds to monitor improvement. Instead manually start the builds.
    13. Maintain private up-to-date workspaces for problem platforms (read as Solaris).
    14. Don't hesitate to ask for help.
    15. When you get an account to acess the cvs repo, make sure you are added to the correct groups, for example, gid=100(users),5000(doc),5002(acetaodev),5003(cvs). Otherwise you will have problem to checkout various modules.
    16. Install your public key to the different machines you have frequent access to avoid typing password.
    17. Update this page if you have any more tips for future build czars :-). This page is bugzilla@deuce.doc.wustl.edu:~/.www-docs/index.html

    The Realm of the Build Czar


    Build Czar Arthur

    Many years have passes since the days of the legendary Build Czar Arthur. His duties were given to him by the mystical Lady of the Lake, who outlined the first responsibilities of the Build Czar.


    Then bespake the Lady of the Lake,
    And these were the words said shee:
    "I knoweth of thy deeds, thou noble Arthur,
    but thy task hath not finished for thee"

    "Thou shalt feitch thy trusty steed,
    And cleanse thy builds againe;
    Then shallt thy ryde hath finnished,
    When new kits released thee cann."

    Then bespake him noble Arthur
    And these were the words said he:
    "With what weapons shallt I use,
    To asure these from the devil free?"

    Then appeered before noble Arthur,
    Uppon the ground a sacred scroll
    Conjurred by the Lady of the Lake
    Borne of the earth in a roll.

    She saies, "Clasp this to thine selfe
    For thee shallt find need for it.
    It shall keep others in the cold,
    Only to be ressurected when thee sees fit."

    "Others shall join thy person,
    To ryde with thee in thy quest;
    Thee shallt be thankful of theire help,
    And to alsoe hold them steadfast."

    "But if theire talke too lodly rise,
    And causeth much damage to thine cuntry,
    He must come forth, and make proclamation,
    For the next one he shall be."

    So hath Arthur to the Lady spoke:
    "For I sweare, and save my othe,
    While enimes and evils I seeke,
    I shall fight against them bothe.