Skip to main content
Version: 723.SP02

Configuration

The following steps need to be performed to implement your custom logic when writing to a target system.

Topics


Define an application for the BAdI implementation

You specify the application when you create the configuration; the application links the configuration to a specific BAdI implementation. To do so:

  1. In the SAP LT Replication Server system, in transaction SM30, enter the table IUUC_REPL_APPL, and choose the button Maintain.
  2. On the next screen choose New Entries and enter:
    • Name: ZLNKAWS_DATALAKES
    • Description: Cxlink Datalakes BAdI or a description of your choice. BADI_APP
  3. Save the changes in a new transport request for later transportation.

Create a new configuration

The application values that you specify here will be later assigned to a configuration that wants to use the custom logic implemented here

To configure your :

  1. Go to transaction LTRC
  2. Select Create Configuration to start a configuration wizard. LTRC New Configuration
  3. In Specify General Data section add:
    1. Configuration Name: Select a descriptive name that helps you easily identify the configuration purpose.
    2. Description: Specify a more descriptive message for the configuration.
    3. Authorization Group: Leave empty. LTRC New Configuration
  4. Select Next.
  5. In the Specify Source System section define:
    1. RFC Destination: Select the RFC that connects to the SM59.
    2. Allow Multiple Usage: Keep checked to allow the configuration to be used in multiple environments.
    3. Read from Single Client: Check if you want to restrict the usage of this configuration to the current client. Leave unchecked if no restrictions apply. LTRC New Configuration
  6. Select Next.
  7. In the Specify Target System section define:
    1. Select Other from the option list.
    2. Select SLT SDK in the Scenario for RFC Communication dropdown. LTRC New Configuration
  8. In the Specify Transfer Settings section define:
    1. In Application select the previously created application for BAdI implementation.
    2. In Job Options, set 3 for all values for testing purposes or adapt them to your needs.
    3. Leave the rest of options by default or adapt them to your needs. LTRC New Configuration
  9. Select Next.
  10. In the Review screen, check the values and select Create to finish the process.
  11. Back in the SAP LT Replication Server - Cockpit: Overview screen, you should be able to see the new Mass Transfer. LTRC List
  12. Ensure that the configuration is deactivated before continuing. LTRC deactivate

Now that you have created the SLT standard configuration, you will need to create the relationship with the storage target for the data. To do so:

  1. Go to transaction /n/LNKAWS/SLT_LTR.

  2. In the CxLink Datalakes Configuration - Overview screen, select Import Mass Transfer ID or (F9).

  3. Select the previously generated Mass Transfer ID* and select Import Mass Transfer ID. Cxlink Datalakes - Import MTID

  4. Back in the Cxlink Datalakes Configuration - Overview screen, find the imported MTID and open it. Cxlink Datalakes - Select MTID

  5. Select New to create a new configuration.

  6. In General data section define:

    1. Description: A descriptive text for this configuration.
    2. Cloud Service Provider: Select the desired cloud provider target for the data. Cxlink Datalakes - New Config
    3. Select Next.
  7. In Access Control section define the Credential to be used:

  8. In Specify Transfer Method define:

    1. Method: You can define a target for specific operations or the whole process:

      1. Load: Only Initial Load will be sent to the specified target.
      2. Replication: Only Replication data will be sent to the specific target.
      3. Both: Initial Load and Replication data will be sent to the specific target.
    2. Output Format: Choose between JSON or CSV as output format.

      1. JSON: Check Athena compatible if you want to export the data in JSONL format (JSON Lines).
      2. CSV: Specify the ASCII representation of the separator you want to use in the data. (https://www.asciitable.com/)
    3. Target table name conversion. Tables with "/" in their names can provoke path alterations when stored in Amazon S3, as they can be assumed as folder separation instead of a single table name. To avoid this behaviour, you can select to replace the '/' symbol for "-" or "__" depending on your needs.

    4. In the Output Settings section:

      Amazon S3 as target

      1. Prefix: You can use the listed dynamic env vars, to generate a custom path based on the values of the exported data.
      2. Custom filename: In the same way, you can use the listed dynamic env vars to build the file names generated to meet your requirements.
      3. Compression: You can enable the compression option to save some space while at rest. You should be able to uncompress the data from your consumers. Cxlink Datalakes - New Config

      Amazon Kinesis as target

      1. Partition Key: You can use the listed dynamic env vars, to generate a custom partition key for kinesis data stream.
      2. Inform of structure changes: If checked, Cxlink Datalakes will inform of any
        Cxlink Datalakes - Transfer Kinesis
    5. Select Next.

    6. Back to the CxLink Datalakes Configuration - Overview screen, you should see all the configuration values listed. Cxlink Datalakes - Config list

  9. In the Review section, check the values and select Create to finish the process.

  10. Repeat the same steps to create as many configurations as needed.


Advanced Configuration

In this section you will find all advance configuration needed for Datalakes to work properly.

Inform of structure changes

Cxlink Datalake offers the capability to notify users when modifications are made to the structure of the source system table. This option can be accessed via the transaction /LNKAWS/SLT_LTR in the configuration wizard, under the 'Inform of structure changes' step.

SLT Inform Structure Changes

Upon activation of this option and the addition of a new field to a database table affected by this configuration, two distinct files are generated. One file contains the information pertaining solely to the newly added field.

Example
US_OPERATIONUS_MTIDUS_SIDUS_TABNAMETABNAMEFIELDNAME
I
00X
SLT
ZCSKS
TEST25

The second file contains the final structure of the table, including all the fields.

Example
US_OPERATIONUS_MTIDUS_SIDUS_TABNAMETABNAMEFIELDNAME
L
00X
SLT
ZCSKS
TEST1
L
00X
SLT
ZCSKS
TEST2
L
00X
SLT
ZCSKS
TEST3
L
00X
SLT
ZCSKS
TEST4
L
00X
SLT
ZCSKS
TEST5

Both files have the same structure:

  1. US_OPERATION: As a usual replication, it’s the first field included in the file.
    1. Value ‘L’ → For the file with the final structure.
    2. Value ‘I’ → For the file with only the new field.
  2. US_MTID: Mass Transfer ID identificator from SAP LT Replication Server.
  3. US_SID: SAP System ID
  4. US_TABNAME: Name of the table with the change.
  5. Structure DD03P: SAP Standard structure that contains all the technical information about the file.

SLT_DD03P

Further information about the DD03P structure can be obtained by executing the SE11 transaction and specifying the DD_GET_DD03P function module and parameter DD03P_TAB for a specific table.

SLT_DD_GET_DD03P

Ones of the most important fields are:

FieldnameTypeDescription
TABNAMETABNAMEIn this case, empty
FIELDNAMEFIELDNAMEName of the new field
DDLANGUAGEDDLANGUAGELanguage key
POSITIONTABFDPOSPosition of the field
KEYFLAGKEYFLAGIdentify key fields of the table with ‘X’
MANDATORYMANDATORYIdentify if the field is required
DATATYPEDATATYPE_DDatatype in Dictionary
LENGDDLENGLength ( Nº of characters )
DECIMALSDECIMALSNumber of decimals
DOMNAME3LDOMNAMEDomain name

Kinesis Data Limitation

If you are using Kinesis as a target for data replication, you will need to add a custom configuration to limit the number of records per data chunk to avoid Kinesis limits.

To do so:

  1. Go to transaction LTRS
  2. Select the desired Mass Transfer ID and double-click to open its properties.
  3. Right-click on Performance Options and Add table. LTRS Add table
  4. Go to Replication Options tab and set the Portion Size (Records) to 500. LTRS Add limit
  5. Save to finish the configuration.

Health Checks

You can perform some health checks over the configuration created to ensure that everything is correctly configured before start replication tables.

To perform the checks:

  1. Go to transaction /n/LNKAWS/SLT_LTR.
  2. In the CxLink Datalakes Configuration - Overview screen, go to the Health check tab, select Configuration Health Check and Execute. SLT Health Check
  3. Double click on the nodes under AWS specific configuration and check the logs.
  4. For S3 Checks output should look similar to: LTRC Amazon S3
  5. For Kinesis checks output should look similar to: LTRC Kinesis