ccf
Wiki: BestPractices

Edit this page | Links to this page | Page information | Attachments | Refresh page

 

Best Practices

Motivation

This wiki page captures "best practices" for the CCF. Best practices are practices that have been found to be very helpful to avoid troubles when setting up and running CCF integration scenarios. If you miss certain practices that have helped you to get CCF scenarios up and running, do not hesitate and add them to this wiki page.

Only work on copies of the sample scenarios

The various example scenarios in the CCF downloads (directory sample) are the best point to start at when being new to the CCF. There is a high chance that the scenario you like to achieve is probably already realized as an example scenario.

When doing your own first steps with the CCF it is very recommended not to modify the samples directly but to make a copy of the sample scenario you are interested in and modify this. This has multiple advantages: You always have a clean, working template you can use to copy from, updates of the CCF won't override your changes and you have the possibility to try out multiple versions of your integration scenario. All sample scenarios are configured to just use relative pathes, so making a copy is no problem at all. All samples will share the same database by default, so all your scenarios can refer to the same data.

Test your scenario using a test repository before going into production

Setting up an integration is a challenging process: You have to make sure that all artifact fields and there values have been mapped correctly. Typically, a bidirectional synchronization is not just set up once and will start to work without any problems immediately. Setting up working integrations is more likely an iterative process. Some artifacts will be successfully synchronized immediately, others will contain user names, release ids or priorities you have not yet correctly mapped or created in the other system.

Therefore, it is highly recommended to first create a dry-run test project (for both the source system and the target syetsm) with the same structure as your production repository and similar data in it. Afterwards, configure your wiring, run the CCF, look at the result, adjust the wiring, run it again, correct the problems, ... and so forth.

During this iterative process you might want to delete all artifacts created so far from the dry-run repository. If you do so, do not forget to delete them from the identity mapping table as well. In most cases, deletion is not necessary since you can always update an existing repository. Feel free to use the replay scenario to replay failed artifact updates in the target system from the artifact hospital. This option only makes sense if your XSLT scripts are already correct but the target system was still missing user ids, releases or other specific field values. In other cases, it might be a better idea to do a "fake update" (change the last modification date) on all artifacts in the repository of the source system to force a resynch. The same behavior can be enforced by if you clear the last synchronized artifact id, timestamp and version in the synchronization status table.

Once your integration scenario works perfectly for your test repositories, make a copy of this scenario and adjust the properties and rename the xslt scripts so that it can now be used in production.

Turn off tracker workflow enforcement rules during initial import

CCF only extracts the most up to date version of an source artifact and transports it into the target system. Otherwise, imports from older repositories would consume too much time (every version of every artifact in the correct order would have to replayed). Since the whole history of an initially imported artifact is not reproduced due to performance reasons, the tracker workflow engine of the target system might deny the import of artifacts that are already in a status that cannot be reached at ordinary artifact creation time.

Therefore, it is highly recommended to turn off the tracker workflow engine for the connector user (at least at initial import time) to allow an initial import. Furthermore, the tracker workflow rules of source and target system should be identical at least when it comes to status preconditions and status invariants that have to be enforced. Otherwise, tracker workflow related problems will show up again and again during production. At this point, you might also want to run your CCF scenario as a service (this wiki provides further information on how to do that).

Figure out the right log level/log methodology for you

CCF uses log4j for all its logging. The log4j.xml file is located in the lib/ directory of your CCF download. In default mode, log4j send critical CCF errors to an e-mail address specified in this file, will log warnings into a file called ccf.log and will discard all log messages with log levels below INFO. At the beginning, CCF might be perceived not chatty enough for your testing purposes, so feel free to set the log level to DEBUG instead or turn on logging messages for core openadaptor components. However, if you like to turn your wiring into production, make sure that your logs are not too verbose in order not to run out of disk space. Feel also free to create a different log4j file for every scenario (the file that is found first in the class path will be used).

Define a process how to change repository meta-data and provide fallback value mappings

Once you have your synchronization up and running you might think that the integration job is done forever. However, artifact integration typically turns out to be a dynamic process. New releases and users will be added on both ends making your xslt files deprecated. Unfortunately, most syetsm do not provide sufficient API's to detect changes in the meta-data on the fly, therefore CCF cannot adapt to meta-data changes automatically.

However, there are some best practices available to manage meta-data change as well. First of all, make sure that for every field that is likely to change, you provide a fallback mapping, i.e., whenever your xslt script does not know how to map a certain field value it uses a fallback value instead. For instance, if a user id is not known, just map it to a default user. You might define some queries for your repository that query for these fallback values to find out immediately when to change your xslt scripts again. By using this methodology you will avoid failed artifacts that will show up in the artifact hospital.

Furthermore, it might help to define a process within your organization how to change meta-data. Whenever meta-data is changed in the source and/or the target system, this change has to be approved by the synchronization manager. He/she will only approve the change after having adjusted the xslt scripts of the CCF scenario and added the same property in the other system as well. If you can establish such a process, you will never be surprised by failed artifact updates again.

Get used to the artifact hospital and the replay option

CCF is designed in a way that it never loses artifact updates. Whenever an artifact cannot be updated in the target system (this might even be the case if an artifact data conflict was detected and you setup the quarantineArtifact conflict resolution policy) the artifact is automatically stored into the hospital data base table. The content of the hospital can be examined with our CCF GUI.

You can sort the failed artifacts according to the scenario that caused the artifact update to fail, the component that was not able to handle the artifact any more, the type of the error that occurred, ... You can examine the actual content of the failed artifact. You can have a look at the stack trace and the detailed exception message. And finally, you can use the replay option to replay an artifact from the hospital.

Delete quarantined artifacts from the hospital from time to time

The replay scenario will never delete non-replayed quarantined artifact out of the hospital. Since the hospital table also contains the failed artifact itself (remember: attachments are also artifacts), the hospital data base table may become very big.

Therefore, make sure to delete unused entries in the hospital from time to time.

Get familiar with CCF's documentation and participate in our community

The CCF download itself does only contain the most important information in order to get you started. On CCF's web site you will find tons of further documentation for the CCF. You will be surprised how many things will work with the CCF out of the box. Always feel free to pose tricky questions on our mailing list. There are high chances that the feature you are asking for can already be realized with the CCF and open adaptor core components. We are also open to your own code and documentation contributions as well as to plain feedback, bug reports and feature requests.

Decide which fields to map bidirectional and which not (how to cope with multi-select field vs. single select field problems)

In a perfect world, the repository layout of the source repository and the target repository would be the same and every source field can be mapped to the logical counterpart of the target field. However, quite often one of the fields you like to map is a single select field in the source repository and a multi-select field in the target system or vice versa. The default behavior of CCF is to keep both repositories in synch. This might lead to situations you might not expect at the first place. For instance, if you delete the only entry of the single select field, the CCF will prune all selected values in the respective field of the other system so that both fields contain exactly the same value. If this is not the behavior you expect, it might be a better idea to map every multi select field on another multi select field (recommended best practice). Another approach would be to just map the field in one direction (from the multi-select field to the single select field). Of course you can also configure further behavior in the XSLT mapping scripts and use more exotic field actions like "append" or "delete". However, most systems currently supported by the CCF do only understand the "replace" field action that will substitute the current values of a field with the new values you provide.

Our best practice is to map multi select fields to multi select fields and if that is not possible, do the mapping in just one direction as long as the system with the single select field does not support the "append" or "delete" field action and you like to spend some time configuring custom behavior in the XSLT files.

If dealing with artifact hierarchies, synchronize using a top down approach

Since version 1.3.0, CCF supports parent child relationships over multiple hierarchies. This feature only works properly if the parent of an unmapped artifact is already known to the entity service. Otherwise, the entity service will have to quarantine the artifact (no worries, you can replay it once the parent artifact has been created in the target system). Hence it makes a lot of sense to synchronize the top level hierarchies first. Doing so, only artifacts with no parent artifacts have to be created. Once these artifacts are registered in the entity service, you can proceed with the hierarchy level beneath the top level hierarchy. These artifacts can now be created as well since their parent artifacts are already known to the entity service. Once this is done, just proceed with the next level.

If you have completed the initial synchronization process, you do not have to care about this approach any more since the parents to newly created artifacts will already be known, no matter, on which hierarchy level you create them.

Avoid synchronization problems due to locked and checked out artifacts

Whenever a system you synchronize supports artifact locking or version control and you like to use these features, you have to be aware of the implications. If an artifact is locked/checked out while CCF is trying to update it, CCF has no choice but to quarantine the artifact. You can configure CCF that it automatically replays quarantined artifacts of certain types (e. g. locked/checked out artifacts) but this cannot ensure that field changes are synchronized in the order they actually happened: If a comment has been added in system foo before the artifact has been locked in system bar, all changes made in bar before the artifact will be unlocked again in bar will seem to have happened before the comment from foo. Consequently, you should try to minimize artifact locking and check out periods as best as you can. In systems where opening an artifact edit dialog automatically locks it, please ensure that this edit dialog is only opened if you actually intend to do a change and closed immediately after the change has been done. Only check out artifacts if you like to change them and check them in once you applied the change. Tell your users about the synchronization consequences if they do not follow this policy. If you do not rely on base lining features, you might want to disable version control features completely.

BestPractices (last edited 2010-02-05 14:12:28 -0800 by jonico)