Configuration

The following steps need to be performed to implement your custom logic when writing to a target system.

Topics

• Define an application for the BAdI implementation
• Create a new configuration
• Associate Cxlink Datalakes to the configuration
• Advanced Configuration
  • Inform of structure changes
  • Kinesis Data Limitation
• Health Checks

---

Define an application for the BAdI implementation

You specify the application when you create the configuration; the application links the configuration to a specific BAdI implementation. To do so:

1. In the SAP LT Replication Server system, in transaction SM30, enter the table IUUC_REPL_APPL, and choose the button Maintain.
2. On the next screen choose New Entries and enter:
   • Name: ZLNKAWS_DATALAKES
   • Description: Cxlink Datalakes BAdI or a description of your choice.
3. Save the changes in a new transport request for later transportation.

---

Create a new configuration

The application values that you specify here will be later assigned to a configuration that wants to use the custom logic implemented here

To configure your :

1. Go to transaction LTRC
2. Select Create Configuration to start a configuration wizard.
3. In Specify General Data section add:
   1. Configuration Name: Select a descriptive name that helps you easily identify the configuration purpose.
   2. Description: Specify a more descriptive message for the configuration.
   3. Authorization Group: Leave empty.
4. Select Next.
5. In the Specify Source System section define:
   1. RFC Destination: Select the RFC that connects to the SM59.
   2. Allow Multiple Usage: Keep checked to allow the configuration to be used in multiple environments.
   3. Read from Single Client: Check if you want to restrict the usage of this configuration to the current client. Leave unchecked if no restrictions apply.
6. Select Next.
7. In the Specify Target System section define:
   1. Select Other from the option list.
   2. Select SLT SDK in the Scenario for RFC Communication dropdown.
8. In the Specify Transfer Settings section define:
   1. In Application select the previously created application for BAdI implementation.
   2. In Job Options, set 3 for all values for testing purposes or adapt them to your needs.
   3. Leave the rest of options by default or adapt them to your needs.
9. Select Next.
10. In the Review screen, check the values and select Create to finish the process.
11. Back in the SAP LT Replication Server - Cockpit: Overview screen, you should be able to see the new Mass Transfer.
12. Ensure that the configuration is deactivated before continuing.

---

Associate Cxlink Datalakes to the configuration

Now that you have created the SLT standard configuration, you will need to create the relationship with the storage target for the data. To do so:

1. Go to transaction /n/LNKAWS/SLT_LTR.

2. In the CxLink Data Lakes Configuration - Overview screen, select Import Mass Transfer ID or (F9).

3. Select the previously generated Mass Transfer ID* and select Import Mass Transfer ID.

4. Back in the Cxlink Datalakes Configuration - Overview screen, find the imported MTID and open it.

5. Select New to create a new configuration.

6. In General data section define:

   1. Description: A descriptive text for this configuration.
   2. Cloud Service Provider: Select the desired cloud provider target for the data.
   3. Select Next.

7. In Access Control section define the Credential to be used:

8. In Specify Transfer Method define:

   1. Method: You can define a target for specific operations or the whole process:

      1. Load: Only Initial Load will be sent to the specified target.
      2. Replication: Only Replication data will be sent to the specific target.
      3. Both: Initial Load and Replication data will be sent to the specific target.

   2. Output Format: Choose between JSON or CSV as output format.

      1. JSON: Check Athena compatible if you want to export the data in JSONL format (JSON Lines).
      2. CSV: Specify the ASCII representation of the separator you want to use in the data. (https://www.asciitable.com/)

   3. Target table name conversion. Tables with "/" in their names can provoke path alterations when stored in Amazon S3, as they can be assumed as folder separation instead of a single table name. To avoid this behaviour, you can select to replace the '/' symbol for "-" or "__" depending on your needs.

   4. In the Output Settings section:

      Amazon S3 as target

      1. Prefix: You can use the listed dynamic env vars, to generate a custom path based on the values of the exported data.
      2. Custom filename: In the same way, you can use the listed dynamic env vars to build the file names generated to meet your requirements.
      3. Compression: You can enable the compression option to save some space while at rest. You should be able to uncompress the data from your consumers.

      Amazon Kinesis as target

      1. Partition Key: You can use the listed dynamic env vars, to generate a custom partition key for kinesis data stream.
      2. Inform of structure changes: If checked, Cxlink Datalakes will inform of any
         

   5. Select Next.

   6. Back to the CxLink Data Lakes Configuration - Overview screen, you should see all the configuration values listed.

9. In the Review section, check the values and select Create to finish the process.

10. Repeat the same steps to create as many configurations as needed.

---

Advanced Configuration

In this section you will find all advance configuration needed for Datalakes to work properly.

Inform of structure changes

Cxlink Datalake offers the capability to notify users when modifications are made to the structure of the source system table. This option can be accessed via the transaction /LNKAWS/SLT_LTR in the configuration wizard, under the 'Inform of structure changes' step.

Upon activation of this option and the addition of a new field to a database table affected by this configuration, two distinct files are generated. One file contains the information pertaining solely to the newly added field.

Example

US_OPERATION	US_MTID	US_SID	US_TABNAME	TABNAME	FIELDNAME
I	00X	SLT	ZCSKS		TEST25

The second file contains the final structure of the table, including all the fields.

Example

US_OPERATION	US_MTID	US_SID	US_TABNAME	TABNAME	FIELDNAME
L	00X	SLT	ZCSKS		TEST1
L	00X	SLT	ZCSKS		TEST2
L	00X	SLT	ZCSKS		TEST3
L	00X	SLT	ZCSKS		TEST4
L	00X	SLT	ZCSKS		TEST5

Both files have the same structure:

1. US_OPERATION: As a usual replication, it’s the first field included in the file.
   1. Value ‘L’ → For the file with the final structure.
   2. Value ‘I’ → For the file with only the new field.
2. US_MTID: Mass Transfer ID identificator from SAP LT Replication Server.
3. US_SID: SAP System ID
4. US_TABNAME: Name of the table with the change.
5. Structure DD03P: SAP Standard structure that contains all the technical information about the file.

Further information about the DD03P structure can be obtained by executing the SE11 transaction and specifying the DD_GET_DD03P function module and parameter DD03P_TAB for a specific table.

Ones of the most important fields are:

Fieldname	Type	Description
TABNAME	TABNAME	In this case, empty
FIELDNAME	FIELDNAME	Name of the new field
DDLANGUAGE	DDLANGUAGE	Language key
POSITION	TABFDPOS	Position of the field
KEYFLAG	KEYFLAG	Identify key fields of the table with ‘X’
MANDATORY	MANDATORY	Identify if the field is required
DATATYPE	DATATYPE_D	Datatype in Dictionary
LENG	DDLENG	Length ( Nº of characters )
DECIMALS	DECIMALS	Number of decimals
DOMNAME3L	DOMNAME	Domain name

Kinesis Data Limitation

If you are using Kinesis as a target for data replication, you will need to add a custom configuration to limit the number of records per data chunk to avoid Kinesis limits.

To do so:

1. Go to transaction LTRS
2. Select the desired Mass Transfer ID and double-click to open its properties.
3. Right-click on Performance Options and Add table.
4. Go to Replication Options tab and set the Portion Size (Records) to 500.
5. Save to finish the configuration.

---

Health Checks

You can perform some health checks over the configuration created to ensure that everything is correctly configured before start replication tables.

To perform the checks:

1. Go to transaction /n/LNKAWS/SLT_LTR.
2. In the CxLink Data Lakes Configuration - Overview screen, go to the Health check tab, select Configuration Health Check and Execute.
3. Double click on the nodes under AWS specific configuration and check the logs.
4. For S3 Checks output should look similar to:
5. For Kinesis checks output should look similar to: