Utility modules

    This is a list of relatively low-level general utility modules which are used by many higher-level modules, but they should not automatically become dependencies of everything else in the code. We should use them wisely when needed.


    Using external libraries for some of this functionality should be considered, so here requirements should be documented to help in evaluating external libraries. However, there are significant costs in terms of portability and configuration-checks when using external libraries (not to mention user build work), so we should try not getting carried away in terms of long wish-lists of features that would be cool to have - let's limit it to what we need at this stage!


    Module: Error Handling


    Common framework for handling errors.


    • Easy to put debugger breakpoints on all errors (assertions, fatal errors, preferably also non-fatal errors) at the point where all context is still intact.
    • User-friendly error messages that contain enough information for developers as well.
      • Encourage developers to write descriptive error messages that contain instructions for users on how to resolve errors.
      • Provide canned explanations for common causes of errors, e.g., simulation instabilities, that can be appended to more detailed error messages.
      • Possibility to print stack traces on segmentation faults, assertions, and fatal errors (possibly only on debug builds).
    • Possibility to disable some error checking in performance-critical code on release builds.
    • Simple and easy to use for developers (both reporting errors and handling them).
    • This module should typically not handle the actual printing of errors (apart from a possible simple default handler), but report the error to a user tool or calling routine (e.g. a GUI) that decides how to show the error to the user. We want to develop the core of Gromacs into a proper library, and then we shouldn't do our own text IO. The exception is hard critical internal errors where we have no chance of handling it gracefully, but this should be an exception.
    • Question: Should we consider splitting of the entire reporting part (and formatting) into a separate message module, which also takes care of non-error messages?


    Additional info: A common policy for handling errors is at least as important as a framework; see Handling Errors for discussion.


    Error Handling using C++ Exceptions

    original version: RS110409

    This approaches aims to follow standard C++ practices as closely as possible. It is based largely on Scott Meyers: "Effective C++". 

    • Code throws exception at location where error is detected and provides the information available at that point. Exception are thrown with GMX_THROW_EXCEPTION (equal to BOOST_THROW_EXCEPTION, see threads for reason, own name to allow extended implementation in the future).
    • Exceptions are categorized in groups allowing to handle different kind of exceptions differently (same categories as those error codes proposed at Handling Error section 2) . 
    • When code higher up in the stack needs to add information to an exception, the boost excpetion mechanism is used. To make this possible, all exception classes derive from boost::exception. 
    • All new code has to be exception safe. Old code only has to be if it is using excpetions or callign code which is. See the C++ style guide for a proposal of how to achive this. 
    • Short running threaded code (e.g. OpenMP) should never throw an exception (thus is only allowed to call nothrow functions)
    • Long running threads: Exception are propagated to calling thread. For C++0x this is planned with N2179 but not supported yet by GCC. Thus the boost mechanism is used. This requires the BOOST_THROW_EXCEPTION macro. The main thread should notify all other threads when an exception cannot be handled and the program should exist.
    • Runtime checks are done with GMX_ASSERT and GMX_RELEASE_ASSERT. The condition for GMX_ASSERT is only evaluated in a Release build. It can be used for performance critical tests. Both macros require a message which should explain the reason why the condition always has to be true. Because it is not meant for the user, it does not have to include a message explaining what went wrong (like an error message would). (Both macros are identical to Teemu's proposal)
    • Exceptions should not be used for excepted problems (as part of the control flow). If a function can encounter a problem under normal conditions this function should preferable be designed so that it handles this problem. If the calling context has to know about the problem the class should provide a method to check for the condition prior to calling the method (e.g. is_empty()). Good guideline is: "Do we want stack unwinding here? Can I afford stack unwinding here?  Is an exception here the exception, not the rule?"
    • Warning/Hints/Info/... will be reported with a logging framework. Might be good to look at Boost.Log when designing it. 

    (Notice only for exceptions propagation to calling thread and ammending additional information boost features are used. The rest of the exception handling is the standard C++ exception feature)


    How this solves the requirements (in order as above)
    • Debugging of exceptions is supported by gdb using catchpoints
    • User-friendly error messages:
      • descriptive error messages are made possible because additional information can be added to the message at each point in the stack 
      • canned explanation can be based on the exception type
      • the functionality to debug stack traces is independent of exceptions. It can be done but is system dependent. See below under "Things to consider" whether we want this system dependent code
    • Easy to use
      • to report error e.g.: GMX_THROW_EXCEPTION(instability_error("Too many LINCS warnings")); Additional information can be ammended (on any level) by e<<o, where e is the exception and o is any information object
      • handle error
        • only error which are recoverable have to be handled! No error codes needed and thus don't have to be checked.
        • if code wants to handle a possible eror it can do so with "catch". Additional information can be accesed. This includes file and line number and potentially stacktrace.
    • (unhandled) errors are only shown to the user in one place. For the command line tool this would be in the main function. This allows to easily write a different output method for e.g. GUI code. This place can also write common recommendations for users depending on the kind of error.
    Important advantages
    • Code doesn't have to check the return value of functions (no error code needed)! In my (RS) opion this simplifies the code significant
    Things to consider
    • It is possible to print the stack trace with all error. But this requires platform specific code; e.g. for windows: StackWalker, for libc backtrace(3) or stacktrace. If we don't mind to have this additional (optional) dependency we can print the stack trace. If we don't want this dependency we can use the debugger to get the stack-trace. It is probably not a good idea to add the stack trace to the exception (the exception object should be kept as simple as possible). In the future, we might write the stack trace for all excpetion to the logger when GROMACS is compiled with debugging symbols. This would than be added to GMX_THROW_EXCEPTION.
    Migration to using Exceptions:


    • All new code should be exception safe!
    • For legacy code exception safety is added progressively as it is needed:
      • Functions don't have to be exception safe as long as it doesn't call any functions throwing exceptions (nothrow) - it should be commented whether functions are exception safe and whether and which exception they might throw (see why exception specifications aren't recommended)
      • When an exception is added (e.g. gmx_fatal is replaced) one needs to make sure that all calling functions are exception safe (important to comment after checking - otherwise we'll recheck functions many times).  If it is too difficult to make these function exception safe the logger could be used instead. But than one has to make sure that no function (indirectly) calling this function ignores the exit code or leaks resources depending on the exit code. This will be often as difficult and thus it is recommended to make the functions exception safe instead.
      • gmx_fatal will stay as a deprecated function as long as it is not feasible to make all required code exception safe.
      • The same should be done for functions which want to report more detail about the error than the error code (and for which an exception is appropriate (see above for guidelines) . If the logger is used instead to report additional information (because exception safety is impractical), it cannot be (easily) avoided that the error is printed even if the error is handled.


    • Exceptions don't work (well) with shared libraries, if the library is compiled with a different compiler than the application. (This has is also a problem with some other C++ features)
    • C++ Exceptions
    • Boost Exceptions (for thread propagation, amending information)
    • System dependent stack library (if stack trace is wanted - see above)
    Things to clarify
    • A hirachy for the GROMACS exception classes (to make handling of errors easy) 
    • A list of GROMACS errinfo typedefs, which defines the data which can be added to an exception (see the list of boost predefined ones)

    Module: String Formatting


    Provides facilities to easily and safely format strings.  At a minimum, could be a wrapper around snprintf() to produce std::string objects, but a more complete C++ external library could be very useful here.

    EL110409: If we can integrate and a slightly fancier semi-external module in a separate directory and distribute it with Gromacs that could be nice, but I'm hesitant whether added fanciness in string handling is critical enough that it warrants an additional dependency for the core Gromacs library, in particular if such a library isn't available by default on many platforms?  

    RS110410: I don't understand why we don't use IOStreams. The only performance critical IO is the trajectory writing which will be special anyhow because it needs to supports MPI/IO not just stdio. Thus allowing iostream for all but trajectories would not cause any performance, nor would introduce an artificial seperation which woundn't otherwise exist. For most of the exisiting code it would be trivial to change the existing IO wrapper funtions to use the IOStream backend instead.


    Module: Unit Testing Against Reference Data (current implementation in src/testutils/)


    Makes it easy to write unit tests for regression testing and/or for comparing complex output. Provides methods that depending on the mode the test is running in, either write their argument into a reference data file, or compare the argument against data in the reference data file. If someone finds a good framework for this, I'm happy to discard my current implementation.

    EL110409: Unit testing only has to work for developers, not end users, so we're a bit more flexible if we want to use something like Google test or Boost for this.

    TM110409: This is on a higher level than, e.g., Google Test, but ideally it should integrate with our unit testing of framework of choice, which currently is Google Test.



    Module: I/O

    Description: Does file I/O. At the lowest level deal with operating system requirements for file I/O. All file I/O should be symmetric such that a file can be read into a datastructure and all essential information can be written out again. How about comments?

    EL110409: I don't think it's a very good idea to have a common module responsible for reading/writing all data structures? This module will have tons of dependencies, and just keep growing.  I think it's a better approach to limit this module to provide basic IO services (such that we e.g. have a list of all files that are presently open when doing checkpointing), but leave the actual operations to the modules responsible e.g. for energy or trajectory data. 

    Depend: XML library, MPI for parallel I/O.
    Input: File or Data
    Output: Data or File


    System specifics


    Handles machine or operation system specific details, such as file/directory existence, constructing paths, etc.




    System capabilities


    Detects hardware and software of the system, e.g. SSE, cycle counters, ...



    Name: Options (current implementation in src/gromacs/options/)


    Provides a framework for declaring options, grouping them into a hierarchy of sections, and for assigning values to them. Also implements basic option types (string, integer, double, boolean), and provides interfaces that can be used to implement custom types.  The framework is general to allow it to be used in different contexts, but does not implement all fancy features.  Idea is to reuse code for, e.g., command-line parsing, mdp option parsing, providing a Python interface, and the selection parser, and extend it in these modules.

    EL110409: To keep things focused, I think we should focus this module on storing and querying options, not processing them to/from text strings or handling how options are computed. Having completely general custom types can be a can of worms that is really hard to debug, so I would probably prefer to limit it to a set of basic types (up to, say, tensors) and then just have mechanisms to group options. Then we would have a separate module for advanced string processing, e.g. for the selection parser.

    TM110409: If the module declares types for options, I think it would be very artificial to separate the type conversions somewhere else, because they also depend on the type. And writing a framework that only supports a limited set of types instead of using templates to support arbitrary types would actually be more work. This does not mean that the module should implement more than basic types; indeed, currently it only implements very few types, but allows other modules to add to the list. If the modules does not support all of the things mentioned here, the whole thing has to be reimplemented for the selection parser (as it is now), defeating most of the purpose of the module. And if there's an implementation for the selection parser that is possible to easily use for other purposes as well, I don't see why not.

    • To be useful in, e.g., Python wrappers, it should be possible to assign values to options from arbitrary types, not just strings (e.g., directly from an integer type).
    • To be useful for the selection parser, it should support "reverse" assignment, meaning that assigning object A as a value for option B would make A responsible for later computing the value of B.
      EL110409: Not sure what this really means. While I understand this concept is useful for the selection parser, isn't this a more specific requirement for how to expand text strings in selections, rather than option storage in general?
      TM110409: To clarify, it expands the first requirement. For the selection parsing, at the point where the value of an option is assigned, we are no longer dealing with string values, but instead evaluation trees. For example, a selection like "com of (resnr 1 to 3 and x < 5)" would at one point during parsing assign the result of evaluating "resnr 1 to 3 and x < 5" to the "of" option of the "com" selection method. And since that evaluating takes place later, the computation of the value has to be delegated. The evaluation tree for the "resnr 1 to 3 and x < 5" has to be constructed while it is being read, so we no longer have the string available when the assignment is done (it would be very difficult to parse the context otherwise, and reconstructing the string and reparsing it after assignment would just complicate things unnecessarily). I'm not saying that this module should implement this behavior, but the implementation should allow the selection module to provide a custom conversion routine that does this.
    • Support for multiple values per option.
      EL110409: Disagree. This will make life an order-of-magnitude harder for all modules that query an option object. Every time we check for something we would now have to check how many options there are, if we even allow multiple choices here, and whether this particular combination is OK or not. I think it's better to have a separate list format for the relatively few cases when it really is justified with multiple values for an options.
      TM110409: I'm not implying that every option should support multiple values, but if the framework actively disallows them, it will just complicate life, and again make it unusable in the selection parser. And if the framework supports multiple values for one option type, why not make it general, because it probably won't be any more work? Already now, multiple values are rejected unless explicitly allowed for an option.
    • Possibility to declare options locally where they are needed, and providing the values at a higher level. 
      EL110409: Not sure if this is practically possible (even if it would be nice).
      TM110409: Of course this can't be done in every case, but take, e.g., a look in how options are declared in modules under src/gromacs/trajectoryanalysis/modules/. This is what I mean here, and the same concept should be possible to use elsewhere as well.
    • Support for printing out help using the option declarations. Again, may not be implemented in this module, but should provide an interface for looping through the available options and getting out information that's necessary for writing out the help. For some discussion on the format for help output, see Redmine issue #690.
      TM110410: This is here to keep existing functionality, and from the maintenance point of view, the current system seems to work relatively well, although it could be a lot simpler, e.g., by using an external program to produce different formats from one single format produced by the program.


    EL110409: Although not very fancy, one easy and very extensible solution could be to have a relatively free options format where we don't have to preset option strings. The user simply says that option "Electrostatics" should have the (string) value "PME", and in various parts of the code we can ask what value the option "Electrostatics" in the "InteractionSettings" section has.  The drawback is of course that we won't catch typos or overlapping names until runtime, but it avoids having a complex mechanism for deciding what options are allowed. In particular if we combine this with an XML schema there will still be a clear list of what options are allowed?

    TM110409: This can of course be implemented in much less code that the current code, but such a bare-bones implementation would not be useful for the selection engine. And then conversion code from strings would be scattered _everywhere_.

    RS110410: If we decided that we wanted a simpler implementation for mdp/command-line than the one for selection parsing, I would suggest to look into Boost Program-Options before starting to design a 2nd GROMACS Options Module. 

    TM110412: I actually took (a very long) look at boost.program_options when considering the implementation of the options module, and the overall design of my module is actually quite similar, although the approach taken (in terms of which C++ features to use) is completely different.  boost.program_options only supports parsing values from std::strings, but otherwise it's very extensible.  But I'm not very fond of it's type-based validation routines and the fact that it pulls in a big part of the boost library with it (all kinds of template metaprogramming stuff, for example).  Somehow descriptive is that to write a custom value converter/validator, one has to overload a method with suitable types, and the signature of that method has not one, but two dummy parameters "to work around partial template specialization limitations in some compilers".  It might be that it works on the compilers we want to support, but does it really make it easily understandable for people not so familiar with C++?



    Uses error handling modules to report errors during option value assignment.

    EL110409: In particular, this module should be generic and not depend on specifics of mdp files, topologies, our selection parsers, etc.

    TM110409: It already is. But it should provide extension points that allow implementing all those different use cases, otherwise it's worse than useless, because the same functionality has to be implemented in multiple places.


    Provided interfaces:

    • Declaring option sets, to be used in self-contained objects that require user input.
    • Assignment of option values to a set of options, to be used in high-level modules that create objects using the first interface.
    • Creation of custom option types, to be used by modules that provide support for more complex user input.
      EL110409: Can we get an example of this?
      TM110409: See src/gromacs/selection/selectionoption.*, which extends the option module to parse strings directly into selection objects.


    For the first two requirements, some additional stuff (Redmine issue #653) would be needed.


    Generic message passing


    Passes messages over threads and/or processes. Should be used for most communication outside of the main md loop.

    At least scatter and reduce are required. Point to point might not be needed.

    Should support seneding of static and non-static data. Static data should be passed by pointer over threads.



    MPI and/or thread MPI and/or thread library.



    Do we also want to use this for (nearly) all communication in mdrun?
    In that case we could also implement lower-level Infiniband communication through this module.

    Page last modified 14:44, 11 May 2011 by hess