Integrate Snowfakery with SFDX Data Move Utility (SFDMU) and Automate Salesforce Scratch Org Data Seeding Using Generated Data

When it comes to seeding data for Salesforce scratch orgs and sandboxes, two trendy choices for developers are Snowfakery with CumulusCI and the SFDX Data Move Utility (SFDMU). Each has its unique strengths and weaknesses.

Snowfakery is an excellent tool for creating complex, realistic test data. It can efficiently generate large volumes of data and integrates seamlessly with CumulusCI, which streamlines data generation and loading processes within CI/CD pipelines. Snowfakery is highly customizable. However, its dependence on CumulusCI, which is not widely adopted, can be a drawback for a lot of users.

On the other hand, SFDX Data Move Utility (SFDMU) is a Salesforce CLI plugin that excels in handling complex data migrations between Salesforce orgs. It allows for customizable data migration and can also be integrated into CI/CD pipelines. SFDMU's strength lies in its independence from other tools, but it relies heavily on Salesforce org environments to provide data sources.

Summary of Main Pros and Cons for Each Tool

Snowfakery:

Pros:

• Complex Data Creation: Capable of generating realistic and complex test data.

• Independence from Salesforce Data: Does not rely on existing Salesforce org data.

Cons:

• Dependency on CumulusCI: Requires CumulusCI to upload data into Salesforce orgs, which may not be widely adopted.

SFDX Data Move Utility (SFDMU):

Pros:

• Tool Independence: No dependency on other tools for data migration.

• Complex Data Migration: Capable of handling complex data migrations between Salesforce orgs.

• Integration: Can be integrated into CI/CD pipelines for automated data movement.

Cons:

• Dependency on Salesforce Orgs: Relies on Salesforce orgs to provide data sources.

Is it possible to combine the strengths of both tools to avoid their individual limitations? Absolutely! By using Snowfakery without CumulusCI and SFDMU together, you can generate complex test data without relying on existing Salesforce org data, and seamlessly upload this data into Salesforce. Here is a high level diagram to show how I integrate snowfakery with SFDMU using a python script.

To demonstrate this integration, I have prepared a sample app (Github Repo) and outlined the steps below:

Step 1: Installing Snowfakery, SFDMU and clone sample app

To get started, you'll need to install both Snowfakery and SFDMU. Follow the links below for detailed installation instructions:

• Snowfakery Installation: Snowfakery Installation Guide

• SFDMU Installation: SFDMU Installation Guide

• Run Command: git clone https://github.com/junliu724515/snowfakery_with_sfdmu

Step 2: Generating Test Data with Snowfakery

• In this step, you can generate test data using the two Snowfakery recipe templates located in the Snowfakery folder within your project. If you need to construct custom Snowfakery recipe YAML files to create specific test data, please refer to the Snowfakery Documentation for detailed guidance.

• Run the command in your terminal under the project folder: snowfakery snowfakery/Account_Opportunity_Contact.recipe.yml --output-format csv --output-folder test-data/

The test csv data files with metadata file will be saved in test-data folder

Step 3: Convert Data and Generate export.json Metadata File for SFDMU

• To make your CSV file compatible with SFDMU, you'll need to replace the lowercase "id" column name with "Id" and generate an export.json file for SFDMU to upload the data files. I have prepared a Python script under scripts folder called generate_export_json.py to automate these tasks.

• Run the command - python3 scripts/generate_export_json.py test-data to convert the test CSV file located in the test-data folder

Step 4: Upload Test Data into Salesforce Org (Scratch Org or Sandbox) Using SFDMU

• Run the following command: sf sfdmu run --sourceusername csvfile --targetusername {{your target org}} --path test-data

• Make sure you replace {{your target org}} with your target org alias

The test data will now be uploaded into your org. , let's write a simple shell script to automate a scratch org creation with the test data uploaded using Snowfakery and SFDMU. You can find the detailed script in scripts/spin-up-scratch-org.sh.

#!/bin/bash

# Step 1: Create a new scratch org
echo "Creating a new scratch org..."
sf org create scratch --definition-file config/project-scratch-def.json --alias SnowfakeryDemo

## Step 2: deploy code
sf project deploy start  --target-org SnowfakeryDemo

# Step 3: Generate fake data using Snowfakery
echo "Generating fake data..."
snowfakery snowfakery/Account_Contact_Case_Opportunity_Role_Reservation.yml --output-format csv --output-folder test-data/

# Step 4: Use the Python script to prepare the data for upload
echo "Preparing data and Generating export.json for upload..."
python3 scripts/generate_export_json.py test-data

# Step 5: Upload the data to the scratch org using SFDMU
echo "Uploading data to the scratch org..."
sf sfdmu run --sourceusername csvfile  --targetusername SnowfakeryDemo --path test-data

Integrating Snowfakery with SFDMU offers a powerful solution for Salesforce scratch org and sandbox data seeding. By leveraging Snowfakery's ability to generate complex, realistic test data and SFDMU's robust data migration capabilities, you can streamline your development and testing workflows. This combined approach ensures efficient data management, reduces dependencies, and enhances the overall consistency of your Salesforce environments. Your questions and suggestions are always welcome! Please feel free to contact me.