IBM Global Mailbox - Technical overview

• Mailbox types
  Mailboxes are a point of integration for applications to access the same message.

• Messages
  

  A message consists of a payload and metadata.

  Message metadata includes message name (file name), size of the file (that is the size of the payload), payload type (inline storage or shared file system storage), and depending on the payload type, the payload itself (inline) or a payload reference identifier. The metadata is stored in Cassandra. The payload is an actual business document or file. A message can have only one payload.

  In Global Mailbox, you can configure a threshold for payload size. If the size of the payload is more than the threshold, the payload is stored in a shared file system storage. If the size is less than the threshold size, it is stored inline with the metadata in Cassandra, as a blob.

  Message replication is the replication of both the message metadata and the message payload. Cassandra handles the metadata replication transparently. If the payload of a message is inline, replication of payload is handled by Cassandra. If the payload is not inline, its replication is performed by a replication server.

  Global Mailbox administrators can manage messages from the Mailbox Explorer page. Global Mailbox administrators can delete messages or view all messages that a specific Global Mailbox contains.

• Virtual roots
  

  Each user must be assigned a virtual root to access Global Mailbox.

  The virtual root is the first level of the directory path for a user when they are navigating the mailbox navigation pane.

  To support limited visibility into the mailbox hierarchy, mailboxes are visible to the user as a relative path, while administrators see the mailbox in an absolute or full path. This concept is referred to as the virtual root .

  An administrator user always has a virtual root of / (slash). If a standard user is changed to an admin user, that user is assigned a virtual root value of / (slash). No default virtual root is assigned to users. An admin must assign a virtual root before users can access Global Mailbox.

• Dead Letter mailbox
  

  A Dead Letter mailbox is a mailbox that contains AS2 messages that might not be routed for some reason. The Dead Letter mailbox is available by default in Global Mailbox under the root mailbox as /DeadLetter. A Dead Letter mailbox cannot be deleted or renamed.

  If you already have a mailbox that is called Dead Letter in your setup, you must delete or rename the mailbox before you upgrade to v6.0.1 or later. Otherwise, the existing Dead Letter mailbox is overwritten with the new Dead Letter mailbox and your existing messages are lost.

  You can create submailboxes under the Dead Letter mailbox. You can also move messages from the Dead Letter mailbox or delete them. While moving messages to a target mailbox, if the target mailbox has an event rule that matches the messages, events are generated and the files are routed.

  An administrator can check the messages in the Dead Letter mailbox periodically for the messages that are not routed correctly and resolve them.

• Duplicate messages
  

  The Global Mailbox supports uploading and downloading of messages with same names (duplicate) through SFTP and FTP.

  Duplicate message support is case insensitive. For example, if you have a message with name test, and another one with TEST or teST then the messages are considered as duplicate of each other. When you download a message with duplicate name, the newest extractable message is downloaded. If extraction count is configured for the newest file, and if the extraction count of the file reaches 0 remaining extractions, then the next download fetches the next newest duplicate file.

  To upload or download a duplicate message, you must set the com.ibm.mailbox.messages.allowDuplicates property to true when you register Sterling B2B Integrator with Global Mailbox. If com.ibm.mailbox.messages.allowDuplicates property is set to false, and you upload a message with the same name, the new message overwrites the existing message.

  Important: If you set com.ibm.mailbox.messages.allowDuplicates to true and upload files with same name, duplicate files with different timestamp are created. Later if you set com.ibm.mailbox.messages.allowDuplicates to false, then every time a business process runs, a file with the duplicate name is deleted until the data in the mailbox aligns with the configuration of the com.ibm.mailbox.messages.allowDuplicates property.

  The listDuplicates property in the ftpserver.properties and sftpserver.properties files controls how the FTP Server adapter and the SFTP Server adapter handle presence of multiple files with same name in a mailbox. When the property is set to true, the FTP and SFTP Server adapters list all the duplicate messages. When the property is set to false, only the newest file is shown in the list.

  Additionally, for FTP and SFTP transfers, you can configure the Support for concurrent duplicate named file transfers parameter, which is available in the FTP and SFTP Server adapter configuration wizards. Currently, Limited and Full (resume of file transfers not supported) options are supported for Global Mailbox. For more information about the parameter and supported options, see the FTP Server Adapter topic.

  For FTP and SFTP transfers, if you configure the FTP or SFTP Server adapter to enable Global Mailbox, and also select the option Full, concatenate duplicate-named files on a GET (resume of file transfers not supported), the adapter ignores the concatenation request when operating on Global Mailbox. Instead, the adapter returns the newest message with the name that is specified in the request. If you enable Global Mailbox and concatenation, a warning message, stating that the concatenation setting is ignored, is printed in the adapter logs.

• Message collections
  

  Multiple messages with the same message name are processed to present the correct message or messages to the user.

  When multiple messages with the same message name are received into Global Mailbox, how they are handled depends on the following criteria:

  • Which application created this mailbox
  • Whether that application supports duplicate messages (a setting in the application)
    • If yes, lists all messages that match that message name, in the default order
    • If no, lists the most recently created message with the message name
    • If switched to No after previously setting Yes, lists the most recently created message with the message name

  To support the feature of having messages with the same names, set the property com.ibm.mailbox.messages.allowDuplicates. This property is specified per application. Valid values are true and false (default). To specify a value for the property, use the command-line application registration utility. By default the property is set to false during installation. You can update the configuration later as required.

  To update the property, enter the following command:

  ./appConfigUtility.sh updateAppConfig --appName=Sterling Integrator Instance 1 --Pcom.ibm.mailbox.messages.allowDuplicates=true

(Linux® or UNIX)

appConfigUtility.bat updateAppConfig --appName=Sterling Integrator Instance 1 --Pcom.ibm.mailbox.messages.allowDuplicates=true

Global Mailbox administrators can manage mailboxes, messages, permissions, virtual roots, and event rules in the Global Mailbox management tool.

Mailbox Explorer

From the Mailbox Explorer page, Global Mailbox administrators can manage all messages, submailboxes, permissions, and event rules for all mailboxes in the Global Mailbox system. The mailbox navigation tree allows Global Mailbox administrators to view all mailboxes and submailboxes on the Mailbox Explorer page.

User Explorer

Event Explorer

With the Events tab on the Event Explorer page, Global Mailbox administrators can resend all events by application that match the filter criteria that is specified. For example, if one or more events were unsuccessfully processed, Global Mailbox administrators can resend events for processing by specifying filter criteria that matches the unsuccessfully processed events.

Administrative users in Sterling B2B Integrator (admins) are granted all permissions in Global Mailbox.

When a service or adapter is configured with a repository type of Global Mailbox, a user designated as a Sterling B2B Integrator administrator is granted all rights and permissions to Global Mailbox resources, without having to specifically add them. There is no scenario whereby a Sterling B2B Integrator administrator gets a permission denied or authorization exception type error because they have all permissions within Global Mailbox.

The virtual root must be set for each admin user in Global Mailbox. If an admin user without a Global Mailbox virtual root accesses an FTP or SFTP server adapter that is enabled for Global Mailbox, the user is directed to traditional mailboxes and not to Global Mailbox.

Admins can directly access the Global Mailbox management tool from Sterling B2B Integrator if single sign-on is configured. Admins must have the necessary permissions in Sterling B2B Integrator to access the Global Mailbox management tool by single sign-on.

In Global Mailbox, admins are not listed on pages where permissions are assigned. Permissions for admins cannot be restricted.

• Event processing
  

  When a file is uploaded, events are posted to the B2Bi application queue immediately. The file is uploaded to the local data center without waiting for being replicated to other data centers.

  For event processing to be successful, Cassandra is required in the local data center.

  In event processing, protocol traffic is automatically routed to a remote data center if a WebSphere MQ outage occurs on the local data center. Files are not processed more than once. In case of failures, all files are automatically processed.

  If event processing fails because the queue manager failed or the WebSphere MQ service failed, the files are processed when the events are resent through the UI or through the eventUtility script. The files are resent to the data center from which the files originated, and the files are then processed.

• Payload replication
  Replication is the process of creating a copies of a message that is added to Global Mailbox in a data center. The copies are created in other data centers.
• Message replication configuration trade-offs
  Immediate replication and delayed replication provide different levels of availability, durability, and performance functions. Based on your business requirements, you must carefully evaluate the required functions and accordingly configure payload replication.
• Default consistency settings in Global Mailbox
  To maximize availability and consistency where required, different default consistency levels are specified for different components in the Global Mailbox system.

In the Global Mailbox system, storage (file system) implementation is based on the concept of storage buckets. The buckets are containers (logical groups) in the file system, which are configured according to business requirements based on security and retention.

The storage system includes buckets, which store blobs. Buckets include variants, which are versions of the buckets. Blobs are stored in the variants. You must configure at least one bucket and variant for storage to operate.

A variant can have a different configuration (for example, encryption settings). Each variant within a bucket is identified by a unique variant identifier (0 - 63). Variants can be marked as retired after which a variant becomes read only. The data in blobs is distributed among the active variants of the bucket. During a READ or GET operation, the blob is retrieved from the variant it exists in.

File transfer process flow

Storage configuration

When you install the initial Global Mailbox node, storage buckets (1st_provisioned and global_mbx) and the first variant (0) are created. By default, Global Mailbox uses the global_mbx bucket to store message payload. When installing the initial Global Mailbox node, you must also specify the shared storage path for all other data centers. The configuration information (1st_provisioned and global_mbx buckets, global.properties and installinfo.properties files), is copied to the shared storage path specified for other data centers after installing the initial Global Mailbox node.

When you configure your storage system, you must decide how you want to use buckets and variants to store the different kinds of information that flow through the Global Mailbox.

Pause and resume

If resumption is configured for a file that is uploaded, the file is broken into segments.

If a file is broken into four segments, a total of five files, one for each segment, and a stub file, are created and stored on the disk. The stub file contains metadata with information to reassemble the segments.

If the upload is interrupted (either by pause or loss of network connection) in the middle of uploading a segment, the whole segment is removed, and upload is resumed when the error is corrected.

• Purge
  

  Purging of data follows a regular, automated schedule to maintain Global Mailbox system performance. You can modify the configuration of purge jobs and purge job triggers with the schedulerConfigUtility script.

  Purge jobs delete the following types of data on a regular schedule:

  • Messages that are not referenced or have no parent
  • Messages with partial or incomplete payloads
  • System artifacts that have no user or application
  • Payloads with no message
  • Mailboxes that have no parent
  • Events by processing status

  Removing unneeded items prevents allocation of resources and prevents performance deterioration.

  Items that are purged cannot be recovered.

You can convert Sterling File Gateway partners and mailboxes to use Global Mailbox. You can create new accounts in Global Mailbox for Sterling B2B Integrator users and mailboxes and use adapters that are enabled for Global Mailbox. You cannot migrate users and mailboxes from one Global Mailbox system to another.

After Global Mailbox is installed and configured, Sterling File Gateway partners can be converted to enable Global Mailbox functionality. After a partner is converted in one data center, export the related resources and import them to the other data centers to take advantage of the features and architecture of Global Mailbox.

To use the features and advantages of Global Mailbox, you can convert the existing Sterling File Gateway partners (producer and consumer) to use Global Mailbox. You can only convert a single partner at a time. A Sterling File Gateway route provisioner, administrator, architect, or operator can convert a partner to use Global Mailbox. The conversion is irreversible. If for any reason you no longer want to use Global Mailbox, you must re-create the partner and routes in Sterling File Gateway.

When you convert the partners to use Global Mailbox, mailboxes that are equivalent to the existing traditional mailboxes are created in the Global Mailbox realm for the partner. User permissions and virtual roots are also created in the Global Mailbox realm. The user accounts associated with a Sterling File Gateway partner remain in Sterling B2B Integrator when the partner is converted to use Global Mailbox. Permissions for the user on certain mailboxes are granted within the Global Mailbox system, but the user account remains within Sterling B2B Integrator.

Traditional mailboxes continue to exist, but are no longer used. You can move the messages in traditional mailboxes to Global Mailbox at the time you convert the partner. After a partner is converted to use Global Mailbox, new messages for that partner are written to the mailboxes in Global Mailbox.

By creating your mailboxes and virtual roots in Global Mailbox, you can gain Global Mailbox advantages while continuing to use your existing Sterling B2B Integrator business processes. If you want to limit partner-facing impact, you may want to re-direct your load balancers to new Global Mailbox enabled adapters. Some downtime would be required while you adjust your adapters, but by re-using the externally facing ports, you may not need to have partners make additional adjustments.

With file transfer resumptions, active file transfers that are interrupted, disrupted, or stopped can be resumed because the Global Mailbox saves data prior to the occurrence of errors.

If an active file transfer is interrupted due to a network error, the protocol adapter getting disrupted or stopped, or the Sterling B2B Integrator node going down, the Global Mailbox saves the data that was uploaded before the error occurred.

To support transfer resumption, the server adapters in Sterling B2B Integrator store the incomplete files in a temporary document staging area. This allows FTP and SFTP clients to resume a transfer. File transfer is resumed on another server in the same data center.

With Load balancer, if an FTP server adapter from a specific data center goes down, an error occurs. When you resume the file transfer, a connection is established to the existing data center if the data center is not down and the file upload is successfully resumed.

However, if the data center is down, and a connection to the other data center is attempted, then resume fails. In this case, file resumption is possible only in the data center in which the file was initially uploaded.

The Global Mailbox implementation supports REST services to view and consume some resources in the Sterling B2B Integrator and Sterling File Gateway.

To establish a connection to the Global Mailbox system, Sterling B2B Integrator is registered with Global Mailbox by using the appConfigUtility during the installation. When registering Sterling B2B Integrator, the REST services are also registered with Global Mailbox. The registration is automated and does not require any manual intervention. However, if you need to change any parameters for user lookup or event data lookup, then you can run the following commands after installing Global Mailbox and Sterling B2B Integrator:

Security for your Global Mailbox system must be planned and implemented as an integrated aspect of your deployment, and carefully monitored and administered to continue evolving to meet new risks as they form. By taking appropriate steps, your organization can minimize and mitigate threats.

Global Mailbox relies on the connecting applications, such as Sterling B2B Integrator and Sterling File Gateway to authorize end users. Multiple applications can access the same mailboxes, depending on permissions. Careful management of users in all applications is necessary to ensure security.

Administrators are authenticated with a user ID and password. Certain parts of the Global Mailbox management tool can only be accessed by administrators. Because of the extensive privileges granted to administrators, the user ID and password must be carefully controlled, changed frequently, and adhere to strict rules for complexity. Delete the default administrator after you have created a unique one for your system.