-
Notifications
You must be signed in to change notification settings - Fork 0
Developer FAQ
This FAQ (formerly called Internal FAQ) is intended for developers. Note that some actions require special permissions like e.g. updating the website.
TODO: add anchors and links, find better structure of document
How to update the openms.de website?
Can I use QT designer to create GUI widgets?
Can the START_SECTION-macro not handle template methods that have two or more arguments?
Are there binary installers created?
How do I generate a build-system for Eclipse, KDevelop, CodeBlocks etc?
What are user definable CMake Cache Variables?
Can I use another solver than GLPK?
How do I switch to Debug/Release configuration?
I changed the contrib path, but re-running CMake won't change the library paths?
What are the most useful (make) targets?
CMake can't seem to find a Qt library (usually QtCore)! What now?
(Windows) What version of Visual Studio should I use?
How do I add a new class MyClass to the build system?
[How to add a new TOPP tool?] (#add_tool)
[How to add a new TOPP util?] (#add_util)
How do I add a new GUI test (for QT Gui classes) for the class MyClass?
(Linux) When executing 'make test', all tests fail.
How can I speed up the compile process of OpenMS?
- Check out the developement version of OpenMS (see website).
- Try and build OpenMS according to the installation instructions.
- Read the OpenMS Coding Convention.
- Read the OpenMS Tutorial.
- Create a GitHub account
- Register to the open-ms-general and open-ms-developers mailing list. (You can see the developers list only if you are logged in to Sourceforge and if you are a OpenMS developer).
How do I update the www.openms.de website?
Login to the wordpress admin area at www.openms.de/wp-admin with your username and password assigned by the current Homepage maintainers.
Yes! If you want to create a class called Widget: Create .ui-File with QT designer and store it as Widget.ui. Add the class to the sources.cmake. From the .ui-File the file include/OpenMS/VISUAL/UIC/ClassTemplate.h is generated by the build system. DO NOT CHECK-IN THIS FILE, AS IT IS GENERATED AUTOMATICALLY, WHEN NEEDED!!! Derive the class Widget from WidgetTemplate. You need to check in the Widget.h and Widget.C files.
Put round brackets around the method declaration. Then it should work.
Currently binary installers from the HEAD are available only for Windows: http://ftp.mi.fu-berlin.de/OpenMS/nightly_binaries/. Please verify the creation date of the individual installers, as there may have been an error while creating the installer. Check also the accompanying log files.
- CMake builds BuildSystems for different platforms, e.g. VisualStudio Solutions on Windows, Makefiles on Linux etc. This allows us to define in one central location (namely CMakeLists.txt) how OpenMS is build and have the platform specific stuff handled by CMake. See http://www.cmake.org for more information.
- See Installation instructions for your platform. In general, you call CMake(.exe) with some parameters to create the native build-system. Afterwards you can (but usually don't have to edit the current configuration using a GUI named ccmake (or CMake-GUI in Windows), which ships with CMake). Note: whenever ccmake is mentionend in this document, substitute this by CMake-GUI if your OS is Windows. You can also edit the CMakeCache.txt file directly.
- Type cmake into a console. This will list the available code generators available on your platform, which you can pass to CMake using the -G option.
They allow the user to pass options to CMake which will influence the build system. The most important option which should be given when calling CMake.exe is:
CMAKE_FIND_ROOT_PATH, which is where CMake will search for additional libraries if they are not found in the default system paths. By default we add OpenMS/contrib. If your have installed all libraries on your system already there is no need to change CMAKE_FIND_ROOT_PATH. If you need the contrib, you will need to set this variable. On Windows, you always need the contrib, as there are no system developer packages. To pass this variable to CMake use the -D switch e.g. cmake -D CMAKE_FIND_ROOT_PATH:PATH="D:\somepath\contrib" Everything else can be edited using ccmake afterwards.
The following options are of interest:
-
CMAKE_BUILD_TYPEDefine if you want to buildDebugorReleaseversion of OpenMS.Releaseis the default. -
CMAKE_FIND_ROOT_PATHThe path to the contrib libraries. Note that you can also provide more then one value here (e.g.,-D CMAKE_FIND_ROOT_PATH="/path/to/contrib;/usr/" will search in your contrib path and in/usr` for the required libraries) -
STL_DEBUGEnables STL debug mode. -
DB_TEST(deprecated) Enables database testing. -
QT_DB_PLUGIN(deprecated) Defines the db plugin used by Qt. -
MT_CUDA_BUILD_TYPE...
Their description will be displayed when you call ccmake.
Yes, but by default the build system only links against GLPK (this is how OpenMS binary packages must be build!). To use another solver try cmake ... -D USE_COINOR=1 .... And look at the documentation of the LPWrapper class.
For Makefile generators (typically on Linux) you can set the CMAKE_BUILD_TYPE variable to either Debug or Release by calling ccmake. For Visual Studio, this is not necessary as all configurations are generated and you can choose the one you like within the IDE itself. The 'Debug' configuration enabled debug information. The 'Release' configuration disables debug information and enables optimization.
Once a library is found and its location is stored in a cache variable, it will only be searched again if the corresponding entry in the cache file is set to false. You can simply delete the CMakeCache.txt, but all other custom settings will be lost as well.
In Visual Studio you can see all targets on the left. For Makefiles type make help. However, this list is quite long. The most useful targets will be shown to you by calling the targets target, i.e. make targets.
CMake finds QT by looking for 'qmake' in your PATH or for the Environment Variable QTDIR! Set these accordingly. If the problem still persists: do you have a second installation of Qt (especially the MinGW version?)? This might lead CMake to the wrong path (it's searching for the Qt*.lib files). You should only move/delete the offending Qt version if you know what you are doing! A save workaround is to edit the CMakeCache file (e.g. via ccmake) and set all paths relating to QT (e.g. QT_LIBRARY_DIR) manually.
Use the latest if you can. Get the latest CMake, as its generator needs to support your VS. If your VS is too new and there is no CMake for that yet, you're gonna be faced with a lot of conversion issues. This happens whenever the Build-System calls CMake (which can be quite often, e.g., after changes to CMakeLists.txt).
- Create the new class in the corresponding sub-folder of the sub-project. The header has to be created in
src/<sub-project>/include/OpenMSand thecppfile insrc/<sub-project>/source, e.g.,src/openms/include/OpenMS/FORMAT/NewFileFormat.handsrc/openms/source/FORMAT/NewFileFormat.cpp. - Add both to the respective
sources.cmakefile in the same directory (e.g.,src/openms/source/FORMAT/andsrc/openms/include/OpenMS/FORMAT/). - Add the corresponding class test to
src/tests/class_tests/<sub-project>/(e.g.,src/tests/class_tests/openms/source/NewFileFormat_test.cpp) - Add the test to the
executables.cmakefile in the test folder. - Add them to git
git add
- Create two new
sources.cmakefiles (one forsrc/<sub-project>/include/OpenMS/MYDIR, one forsrc/<sub-project>/source/MYDIR), using existingsources.cmakefiles as template. - Add the new
sources.cmakefiles tosrc/<sub-project>/includes.cmake - If you created a new directory directly under
src/openms/source, then have a look atsrc/tests/class_tests/openms/executables.cmake - add a new section that makes the unit testing system aware of the new (upcoming) tests,
- look at the very bottom and augment
TEST_executables, - add a new group target to
src/tests/class_tests/openms/CMakeLists.txt
You should always add a test alongside every new class added to OpenMS.
- Add the class test to
src/tests/class_tests/<sub-project>/(e.g.,src/tests/class_tests/openms/source/NewFileFormat_test.cpp) - Add the test to the
executables.cmakefile in the test folder. - Add them to git
git add
A test template for your specific class can be generated by the create_test.php script found in tools/.
Add commands to src/tests/topp/CMakeLists.txt (where it fits alphabetically).
- Create the
MyClass_test.cppinsrc/tests/class_tests/openms_gui/source - Add it to
src/tests/class_tests/openms_gui/CMakeLists.txtin the GUI section. - Have a look at existing GUI tests, as they use the QT TestLib framework and not the OpenMS macros.
Please check the LD_LIBRARY_PATH environment variable:
You can print the LD_LIBRARY_PATH with echo $LD_LIBRARY_PATH.
If your /lib/ folder is included, check that libOpenMS.so is present.
With the ldd command, you can show the libraries used by an executable, e.g. 'ldd /bin/ClassTest_test'.
Build with several threads. If you have several pocessors/cores you can build OpenMS classes/tests and TOPP tools in in several threads. On Linux use the make option -j: make -j8 OpenMS TOPP test_build
On Windows, Visual Studio solution files are automatically build with the /MP flag, such that VS uses all available cores of the machine.
All source files added to an IDE are associated with their targets. You can find the source files for each test within its own subproject. The same is true for the TOPP and UTILS classes.
[Visual Studio] I'm getting the error "Error C2471: cannot update program database". This is a bug in Visual Studio and there is a bugfix: http://code.msdn.microsoft.com/KB946040 Only apply it if you encounter the error. The bugfix might have unwanted side effects!
[Eclipse CDT] The indexer gets stuck at some file which #includes seqan It seems that SeqAn code is just too confusing for older eclipse C++ indexers. You should upgrade to eclipse galileo (CDT 6.0.x). Also, increase the available memory limit in eclipse.ini, e.g. -Xmx1024m for one gig. [Eclipse CDT] The parser is confused after OPENMS_DLLAPI and does not recognize standard C++ headers Go to Project -> Properties -> C/C++ Include Paths and Preprocessor Symbols -> Add Preprocessor symbol -> "OPENMS_DLLAPI=". This tells eclipse that the macro is defined empty. In the same dialog you can also add an external include path to e.g. /usr/include/c++/4.3.3/, etc. The issue with C++ headers was fixed in the latest galileo release. Hints to resolve the OPENMS_DLLAPI issue using the cmake generator are welcome!
You can can execute an OpenMS class test using the CTest regular expressions:
> ctest -V -R "^<class>_test"
To build a class test, you simply call the respective make target in ./source/TEST:
> make <class>_test
> ctest -V -R "TOPP_<tool>"
To build the tool, use:
> make <tool>
There is a mechanism to have a core dumped if an uncaught exception occurs.
To enable it, the environment variable OPENMS_DUMP_CORE has to be set.
Each time an uncaught exception occures, the OPENMS_DUMP_CORE variable is checked and a segmentation fault is caused, if it is set.
Try the ulimit -c unlimited command. It sets the maximum size of a core to unlimited.
Note: We observed that, on some systems, no core is dumped even if the size of the core file is set to unlimited. We are not sure what causes this problem.
Imagine you want to debug the TOPPView application and you want it to stop at line 341 of
SpectrumMDIWindow.C.
Run gdb:
shell> gdb TOPPView
Start the application (and close it):
gdb> run [arguments]
Set the breakpoint:
gdb> break SpectrumMDIWindow.C:341
Start the application again (with the same arguments):
gdb> run
Linux: ldd
Windows (Visual studio console): Try "Dependency Walker"(http://www.dependencywalker.com/) (use x86 for 32bit builds and the x64 version for 64bit builds. Using the wrong version of depends.exe will give wrong results!) or dumpbin /DEPENDENTS OpenMS.dll
Linux: nm Use nm -C to switch on demangling of low-level symbols into their C++-equivalent names. nm also accepts .a and .o files. Windows (Visual studio console): dumpbin /ALL You can use dumpbin on object files (.o) or (shared) library files (.lib) or the DLL itself e.g. dumpbin /EXPORTS OpenMS.dll
OpenMS runs on three major platforms, each one having its own ways of doing things. Here are the most prominent causes of "it runs on Platform A, but not on B. What now?"
Reading/Writing binary files causes different behaviour ... Usually Linux does not make a difference between text-mode and binary-mode when reading files. This is quite different on Windows as some bytes are interpreted as EOF, which lead might to a premature end of the reading process. Thus, if reading binary files make sure that you explicitly state that the file is binary when opening it!
During writing in text-mode on windows a line-break (\n) is expanded to (\r\n). Keep this in mind or use the eol-style property of subversion to ensure that line endings are correctly checked out on non-Windows systems.
unsigned int vs size_t (UInt and Size) UInt and Size are the same on Linux GCC (i.e. both have the same size, 32bit on 32bit systems, 64bit on 64 bit systems), however on Windows this only holds for 32bit. On a 64bit Windows the UInt type is still 32bit, Size is (obviously) 64bit. This might lead to warnings (at best) or overflows and other nasty stuff. So make sure you do not rely on UInt being equal to Size - because they're not.
Paths and system functions... This is trivial but hardcoding something like
String tmp_dir = "/tmp"; is a big no-no! This must fail on Windows! Use Qt's QDir to get a path to the systems temporary directory if required. Also calling things like uname which are only available on Linux: don't! When working with files or directories, it is usually safe to use "/" on all platforms. Even Windows understands that. Take care of spaces in directory names though. You should always quote paths if they are used in a system call to ensure that the subsequent interpreter takes the spaced path as a single entity.
OpenMS/doc/doxygen/public/Main.doxygen
OpenMS/doc/doxygen/public/Modules.doxygen
You have to add your class to the program OpenMS/doc/doxygen/parameters/DefaultParamHandlerDocumenter.C. This program generates a html table with the parameters. This table can then be included into the class documentation using the following doxygen command:
@htmlinclude OpenMS_.parameters Note that parameter documentation is automatically generated for TOPP/UTILS included in the static ToolHandler.C tools list. To include TOPP/UTILS parameter documentation use following doxygen command:
@htmlinclude TOPP_.parameters or
@htmlinclude UTILS_.parameters You can test if everything worked by calling make doc_param_internal . The parameters documentation is written to OpenMS/doc/doxygen/parameters/output/.
The program OpenMS/doc/doxygen/parameters/TOPPDocumenter.C creates the command line documentation for all classes that are included in the static ToolHandler.C tools list. It can be included into the documentation using the following doxygen command:
@verbinclude TOPP_.cli You can test if everything worked by calling make doc_param_internal . The command line documentation is written to OpenMS/doc/doxygen/parameters/output/.
OpenMS tutorial: OpenMS/doc/OpenMS_tutorial/refman_overwrite.tex.in (for PDF tutorials) OpenMS/doc/doxygen/public/OpenMS_Tutorial_html.doxygen (for html tutorials)
TOPP and TOPPView tutorial: OpenMS/doc/TOPP_tutorial/refman_overwrite.tex.in (for PDF tutorials) OpenMS/doc/doxygen/public/TOPP_Tutorial_html.doxygen (for html tutorials)
- Submit the bug as a github issue
- Create a feature branch (e.g. feature/fix_missing_filename_issue_615) from your (up-to-date) develop branch in your fork of OpenMS
- Fix the bug and add a test for it
- Create a pull request for your branch.
- After approval and merge make sure the issue is closed
-
Create release branch
-
Add release branch to CDASH
-
Distribute tasks among developers:
-
Fix automatic tests
-
Fix release critical bugs
-
Fix compile warnings
-
Fix checker violations
-
Fix code analysis (e.g. cpplint, cppcheck) violations
-
Update the list of active maintainers in tools/ACTIVE_MAINTAINERS
-
Declare release candidate and distribute tasks: installation on target platforms check if all installed tools can be executed check if documentation is available and correct (e.g. version numbers) check if pipelines run check if interactive applications work correctly (e.g. TOPPView, TOPPAS, INIFileEditor, IDEvalutator ...) KNIME installation on target platforms check if automatic update in KNIME works
-
Organise packaging
Each maintainer must check his/her classes and TOPP tools. Keep in mind that missing documention scares off potential users!
Here is a list of what is to do:
-
Remove classes not ready for release from the release branch.
-
Update the CHANGELOG file (also document all parameter names of TOPP tools which have changed! the easiest way to do this: take all INI files from previous release and call: " -write_ini bla.ini -ini <previous.ini>" and look at output)
-
Check the test builds of the latest gcc/cl version for compiler warnings (32 and 64 bit)
-
Check if all examples in OpenMS/doc/code_examples/ compile.
-
Make sure the binary packages can be compiled for all platforms (see below). Between releases, the build system usually catches a bug, and you need to fix this!
-
Correct errors found by tools/checker.php. The option -u restricts the output to one maintainer More options are given with –help
-
Documentation:
OpenMS
Write / correct tutorial
Write / correct class documentation
Complete method documentation
TOPP
Write / correct tutorial
Check if each TOPP/UTIL is listed on the TOPP/UTIL documentation page
Write / correct doxygen documentation of each tool
Proof-read the output of –help
Check if all parameters and sections are documented in INI files (-write_ini and INIFileEditor).
-
Create the binary/source release packages (see below)
-
Upload the Source and Binary packages to SourceForge:
-
Go to the OpenMS' sourceforge site & log in
-
Create a new folder for the release, e.g., OpenMS-1.9 and upload files to this folder
-
Copy the HTML documentation and PDF tutorials of the release to your own FTP server (e.g., http://ftp.mi.fu-berlin.de/OpenMS/doc-1.9-official).
-
Update the redirection rule on the website for http://www.openms.de/current_doxygen to point to the current documentation
-
Check the function of the public search engine at http://www-bs2.informatik.uni-tuebingen.de/services/OpenMS-release/search.php
-
Update the website (news, changelog, XML schema files, links for docu and download)
-
Send a mail notification to the announcements mailing list and the general mailing list
-
How is the platform-independent source release package created?
-
Year numbers update in: /source/COPYRIGHT All headers and C files and License.txt (adapt years accordingly) cd find . -name ".h" -o -name ".C" -o -name ".txt" | xargs sed -i 's/(C) 2003-2011/(C) 2003-2012/g' Version Number update in: CMakeLists.txt of contrib. CMakeLists.txt of OpenMS (three OPENMS_PACKAGE_VERSION_ variables, and the contrib version check) source/TEST/VersionInfo_test.C TOPPView and TOPPAS splash screen source/VISUAL/ICONS/. The GIMP/PhotoShop files can be found in the images SVN module. Run cmake Make sure that the tutorial examples in doc/code_examples/ can be compiled and executed (target 'Tutorials_build' and 'Tutorials_exec') Commit the changes to the SVN repository Build the documentation that is needed for the release Check out a fresh working copy of OpenMS Build the library, and build the documentation using make doc (not doc_internal!). Build the PDF tutorials using make doc_tutorials. Make sure that the Parameter documentation was created (see TOPP documentation of a arbitrary TOPP tool). Make sure the include paths in the documentation are correct. Run the script tools/make_source_package.sh according to the help shown when executed. Test installation of contrib and OpenMS from release package How is the Windows binary release package created?
-
Check out the Windows installer scripts from https://open-ms.svn.sourceforge.net/svnroot/open-ms/win_installer see win_installer/README for further instructions
Prerequisites: make sure the 'rpm' or respective package creation utilities are installed. Check out the release branch and contrib Build the contrib libraries Configure OpenMS using
cmake -D INSTALL_PREFIX=/usr -D PACKAGE_TYPE=rpm/deb ... Set the environment variable 'OPENMS_DATA_PATH' to your OpenMS share directory, e.g. /home/myusername/binary_release/OpenMS/share/OpenMS. Explanation follows (just read if interested): When using INSTALL_PREFIX, running any OpenMS executable before installation will result in an error because the SHARE/OpenMS directory will not be found (it is already hardcoded to the future install directory). As 'make doc' etc require OpenMS executables (TOPPDocumenter,...) this step is required. Build OpenMS: make TOPP UTILS doc doc_tutorials Build the package using make package Rename the package to this naming schema: OpenMS---.<rpm/deb> Profiling and memory leaks and strange behaviour
First check that the executable you are calling actually is the correct one (in case you have multiple OpenMS version installed), and also that it uses the correct OpenMS library (check PATH (Win) or LD_LIBRARY_PATH (Linux)). Otherwise you might get the strangest access violations, pointing to arbitrary code.
Try IBM's profiler, available for all platforms (and free for academic use): Purify(Plus) and/or Quantify. Windows: this is directly supported by Visual Studio (Depending on the edition: Team and above). Follow their documentation.
Linux:
- build OpenMS in debug mode (set CMAKE_BUILD_TYPE to 'Debug')
- call the executable with valgrind: 'valgrind –tool=callgrind ' Note: other processes running on the same machine can influence the profiling. Make sure your application gets enough resources (memory, CPU time).
- You can start and stop the profiling while the executable is running e.g. to skip initialization steps:
- start valgrind with the option –instr-atstart=no
- call 'callgrind -i [on|off]' to start/stop the profiling
- The output can be viewed with 'kcachegrind callgrind.out.'
- build OpenMS in debug mode (set
CMAKE_BUILD_TYPEtoDebug) - call the executable with valgrind:
valgrind --suppressions=OpenMS/tools/valgrind/openms_external.supp –leak-check=full <executable> <parameters>
Common errors are:
-
'Invalid write/read ...'- Violation of container boundaries -
'... depends on uninitialized variable'- Uninitialized variables: -
'... definitely lost'- Memory leak that has to be fixed -
'... possibly lost'- Possible wemory leak, so have a look at the code
For more information see the valgrind documentation at http://valgrind.org/docs/manual/
- Create an .ico file: first, you need some graphics program (The GIMP is recommended) think of a motive and remind yourself that you have limited space. http://msdn2.microsoft.com/en-us/library/ms997636.aspx gives some useful hints. create at least a 16x16, 32x32, 48x48 and 64x64 pixel version and save each of them in a separate layer of the respective size. Do not add any larger sized layers, since Win XP will not display any icon then. when saving your image as type .ico the GIMP will ask you for the color depth of each layer. As it is recommended to have multiple color depths of each icon-size, go back to the layers and duplicate each layer twice. That should give you 12 layers. now save the image as .ico (e.g. TOPPView.ico) file, giving each group of equal sized layers a 32bit (8bit transparency), 8bit (1bit transparency), 4bit (1bit transparency) color depth. Attention: make sure to assign the higher color depth to the upper layers as Windows will not pick the highest possible color otherwise.
- Create a resource file: Create a text file named .rc (e.g. TOPPView.rc) Insert the following line: 101 ICON "TOPPView.ico" , replacing TOPPView with your binary name. Put both files in OpenMS/source/APPLICATIONS/TOPP/ (you should find some similar files for other TOPP tools already present). Re-run cmake and re-link your TOPP tool. Voila. You should have an iconized TOPP tool.
- Add the code to
src/topp/and register it insrc/topp/executables.cmake - Add your tool (with the correct category) to
getTOPPToolList()insrc/openms/source/APPLICATIONS/ToolHandler.cppThis creates a doxygen page with the–helpoutput of the tool (usingTOPPDocumenter). This page must be included at the end of the doxygen documentation of your tool (see other tools for an example). - Add it to the TOPP docu page (in
doc/doxygen/public/TOPP.doxygen) - Add the name to
src/topp/executables.cmake - Write a TOPP test (add it to
src/tests/topp/CMakeLists.txt)
Warning: handle any kind of input files to your TOPP tool via command line flags and use the ${DATA_DIR_TOPP} prefix. You can use ini-files to specify output-files, but not input-files. Doing otherwise will break out-of-source builds.
Hint: add -test to the call of your TOPP tool and also create the expected output that you put in src/tests/topp with that flag active. The flag ensures that UniqueId's, dates etc are equal no matter where and when the tool is run.
- Add the code to
src/utils/and register it insrc/utils/executables.cmake - Add your tool to
getUtilList()insrc/openms/source/APPLICATIONS/ToolHandler.cppThis creates a doxygen page with the –help output of the tool (using TOPPDocumenter). This page must be included at the end of the doxygen documentation of your tool (see other tools for an example). - Add it to the UTILS docu page (in
doc/doxygen/public/UTILS.doxygen) - Write a test (this is optional for UTILS). See TOPP tools above and add the test to the bottom of
src/tests/topp/CMakeLists.txt.