Skip to end of metadata
Go to start of metadata

 Extract/Transform/Load jobs, or ETL, are the basic steps needed to import data from one system into another. During extraction the incoming data (CSV or XML) is parsed into generic in-memory records. The extracted data is then transformed by converting the generic records into domain objects understood by the target system. Finally, this data is loaded by persisting the domain objects in the target system.

This Article Explains

This article details how enable the ETL Gear then test and deploy ETL feeds. Follow feed file standards to deploy any type of feed.

Sections in this Article

Default CSV Formatting

If you follow the default CSV formatting expected by the default Feed Component then you will be able to test your files without making any code changes. For more information on Evergage default CSV format standards please refer to the CSV Feed Requirements article in this knowledge base.

Enable Gear to View Feed Component

Your new feed component will only appear on the Feeds Dashboard if the Gear has been enabled.

  1. Select Gears > Gears List
  2. Locate the gear you want to enable
  3. Click  next to the gear
  4. Select Enabled
  5. You'll receive a message that the gear is enabled
  6. Click SAVE

You cannot disable or delete Gears that contain enabled feed components 

Test the ETL Feed

You'll need a sample file to use to test that the feed is configured correctly. The file can be any size but will be truncated to 10KB on upload.

  1. From the main navigation at the left, select Gears > Feeds Dashboard
  2. Located the feed component on the Feeds Dashboard
  3. Click TEST to open the component class inside the Gear Editor, ready for testing
  4. There are two ways to use a sample file
    1. Click UPLOAD SAMPLE to select and upload a new file you will use to test the feed
    2. Click SELECT FILE to select an already uploaded file from the Evergage SFTP site
  5. Click the SAMPLE FILE tab in the bottom panel to preview the uploaded file
  6. Click to commence a test against the currently loaded sample file
  7. You will receive a confirmation message once the file has launched

Review the ETL Test Batch

Once your test has completed you can review the batch information in the bottom panes. The Summary tab contains information on the current batch, and the Modified Rows tab provides a visual representation of the changes between what is currently in the dataset and what will be modified by the sample file. 

You will see one of the following indicators next to each row:

  • Blue pencil icon indicates an entry that has been changed
  • Green plus icon indicates that a new object entry has been detected
  • Red minus icon indicates that an object entry will be removed by this test batch

Rows that do not have an icon do not contain any changes.

Review Batch History

You can view both test and live batch history on the Batch History tab on the Feeds Dashboard.

  1. From the main navigation at the left, select Gears > Feeds Dashboard
  2. Select the Batch History tab
  3. History is listed in reverse chronological order, so the most recent batches will appear at the top
  4. Select a batch to view details in the bottom pane
  5. Click VIEW BATCH to see the entire batch object in the Gear Editor so you can review the log, see a summary of the batch, confirm modified rows, and test again
  6. Click EXPORT to download the component files in a ZIP file
  7. Click EXIT to return to the Gears List

View Feed Files

  1. Select Gears > Feeds Dashboard
  2. On the Files tab, you can view test and live files currently present on the dataset
  3. Use the folders in the file explorer to determine the processing state of the files:
    1. Failed - processing failed
    2. Inbound - not been processed
    3. Outbound - currently being processed
    4. Processed - completed processing
    5. Testing - currently in being tested
  4. The Files tab may only be used to review files and does not allow files to be moved, modified, or delete

Commit a Batch

Once you have tested the ETL job, you can commit your changes to the dataset.

  1. Select Gears > Feeds Dashboard
  2. Locate the feed component on the Available Feeds tab
  3. Click TEST
  4. Select  to commit the changes to your dataset
  5. Click YES, COMMIT BATCH when prompted
  6. You will receive a message confirming that the batch was committed successfully
Please review your changes in the Modified Rows tab or in the exported CSV to confirm your changes before committing