Advanced Features

This documentation describes the advanced features of Laravel Ingest that significantly improve the developer experience and testability.

Validators
Transformation Pipelines
Conditional Mappings
Custom Data Sources
Import Events
Nested Mappings
Schema Validation
Debugging & Tracing

Validators

Validators are to validation logic what transformers are to transformation logic: reusable, testable, and declarative.

Basic Principle

use LaravelIngest\IngestConfig;
use LaravelIngest\Validators\EmailValidator;
use LaravelIngest\Validators\RequiredValidator;
use LaravelIngest\Validators\RangeValidator;

IngestConfig::for(Product::class)
    ->mapAndValidate('email', 'email', EmailValidator::class)
    ->mapAndValidate('price', 'price', [
        RequiredValidator::class,
        new RangeValidator(min: 0, max: 10000),
    ]);

Built-in Validators

RequiredValidator

Checks whether a field contains a value.

use LaravelIngest\Validators\RequiredValidator;

->mapAndValidate('name', 'product_name', RequiredValidator::class)

Validation fails for null, empty strings, and empty arrays.

EmailValidator

Validates email formats.

use LaravelIngest\Validators\EmailValidator;

->mapAndValidate('email', 'customer_email', EmailValidator::class)

Empty values are considered valid (null, '').

RangeValidator

Checks numeric values against min/max boundaries.

use LaravelIngest\Validators\RangeValidator;

// Minimum only
->mapAndValidate('price', 'price', new RangeValidator(min: 0))

// Min and Max
->mapAndValidate('quantity', 'qty', new RangeValidator(min: 1, max: 100))

// With custom error message
->mapAndValidate('discount', 'discount_percent', 
    new RangeValidator(min: 0, max: 100, message: 'Discount must be between 0% and 100%'))

RegexValidator

Validates against a regex pattern.

use LaravelIngest\Validators\RegexValidator;

// Postal code validation (5 digits)
->mapAndValidate('zip', 'postal_code', 
    new RegexValidator('/^\d{5}$/', 'Must be a 5-digit postal code'))

InArrayValidator

Checks whether a value is contained in an allowed list.

use LaravelIngest\Validators\InArrayValidator;

->mapAndValidate('status', 'status', 
    new InArrayValidator(['active', 'inactive', 'pending']))

// With strict mode (type checking)
->mapAndValidate('type', 'type', 
    new InArrayValidator(['1', '2', '3'], strict: true))

DateValidator

Validates date formats.

use LaravelIngest\Validators\DateValidator;

// Default: Y-m-d
->mapAndValidate('date', 'order_date', new DateValidator())

// Custom format
->mapAndValidate('date', 'order_date', new DateValidator('d/m/Y'))

Combined Transformation + Validation

->mapTransformAndValidate(
    'price_cents',
    'price',
    [new NumericTransformer(decimals: 2)],
    [new RangeValidator(min: 0)]
)

Transformation Pipelines

Apply multiple transformations in sequence:

use LaravelIngest\Transformers\TrimTransformer;
use LaravelIngest\Transformers\SlugTransformer;
use LaravelIngest\Transformers\DefaultValueTransformer;

// Pipeline: Trim -> Slug -> Default
->mapAndTransform('title', 'slug', [
    new TrimTransformer(),
    new SlugTransformer(),
    new DefaultValueTransformer('untitled'),
])

Additional Built-in Transformers

TrimTransformer

use LaravelIngest\Transformers\TrimTransformer;

->mapAndTransform('name', 'name', new TrimTransformer())

// With custom characters
->mapAndTransform('code', 'code', new TrimTransformer('x'))

SlugTransformer

use LaravelIngest\Transformers\SlugTransformer;

->mapAndTransform('title', 'slug', new SlugTransformer())

// With underscore instead of hyphen
->mapAndTransform('title', 'slug', new SlugTransformer('_'))

MapTransformer

Map values from a lookup table:

use LaravelIngest\Transformers\MapTransformer;

->mapAndTransform('status', 'status_code', new MapTransformer([
    'active' => 1,
    'inactive' => 0,
    'pending' => 2,
], default: -1))

BooleanTransformer

Convert string values to boolean (0/1):

use LaravelIngest\Transformers\BooleanTransformer;

// Default truthy: yes, true, 1, on, y
// Default falsy: no, false, 0, off, n
->mapAndTransform('is_active', 'is_active', new BooleanTransformer())

// Custom truthy/falsy values
->mapAndTransform('status', 'is_active', new BooleanTransformer(
    truthyValues: ['aktiv', 'active', '1'],
    falsyValues: ['inaktiv', 'inactive', '0'],
    default: 0
))

Empty or null values are mapped to default (null by default).

DateTransformer

Parse and reformat date values:

use LaravelIngest\Transformers\DateTransformer;

// Input: "31.12.2024", Output: "2024-12-31"
->mapAndTransform('date', 'created_at', new DateTransformer(
    inputFormat: 'd.m.Y',
    outputFormat: 'Y-m-d'
))

// Parse ISO date and store as timestamp
->mapAndTransform('date', 'created_at', new DateTransformer(
    inputFormat: DateTimeInterface::ATOM,
    outputFormat: 'Y-m-d H:i:s'
))

Invalid or empty values return default (null by default).

ConcatTransformer

Merge multiple fields:

use LaravelIngest\Transformers\ConcatTransformer;

// Build a full name from different columns
->mapAndTransform(null, 'full_name', new ConcatTransformer(
    ['first_name', 'last_name'], 
    separator: ' '
))

DefaultValueTransformer

Replace empty values with a default:

use LaravelIngest\Transformers\DefaultValueTransformer;

->mapAndTransform('description', 'description', new DefaultValueTransformer('No description available'))

// With custom "empty" values
->mapAndTransform('status', 'status', new DefaultValueTransformer(
    'unknown', 
    ['null', 'NULL', '']
))

Conditional Mappings

Apply mappings only when a condition is met:

use LaravelIngest\IngestConfig;
use LaravelIngest\Transformers\NumericTransformer;

// Different status fields depending on type
IngestConfig::for(Transaction::class)
    ->mapWhen('status', 'order_status', 
        fn($row) => $row['type'] === 'order',
        new MapTransformer(['pending' => 1, 'completed' => 2])
    )
    ->mapWhen('status', 'refund_status',
        fn($row) => $row['type'] === 'refund',
        new MapTransformer(['requested' => 1, 'processed' => 2])
    );

Use ConditionalMappingInterface for complex logic:

use LaravelIngest\Contracts\ConditionalMappingInterface;

class OrderStatusMapping implements ConditionalMappingInterface
{
    public function shouldApply(array $rowContext): bool
    {
        return $rowContext['type'] === 'order';
    }

    public function getSourceField(): string
    {
        return 'status';
    }

    public function getModelAttribute(): string
    {
        return 'order_status';
    }

    public function getTransformer(): ?TransformerInterface
    {
        return new MapTransformer(['pending' => 1, 'completed' => 2]);
    }

    public function getValidator(): ?ValidatorInterface
    {
        return null;
    }
}

// Usage
->mapWhen('status', 'order_status', new OrderStatusMapping())

Custom Data Sources

Use SourceInterface for external data sources:

use LaravelIngest\Contracts\SourceInterface;
use Generator;

class ShopifyProductSource implements SourceInterface
{
    public function __construct(
        private string $shopDomain,
        private string $apiKey
    ) {}

    public function read(): Generator
    {
        $client = new ShopifyClient($this->shopDomain, $this->apiKey);
        
        foreach ($client->getProducts() as $product) {
            yield [
                'id' => $product['id'],
                'title' => $product['title'],
                'price' => $product['variants'][0]['price'] ?? null,
                'sku' => $product['variants'][0]['sku'] ?? null,
            ];
        }
    }

    public function getSchema(): array
    {
        return [
            'id' => ['type' => 'integer', 'required' => true],
            'title' => ['type' => 'string', 'required' => true],
            'price' => ['type' => 'numeric', 'required' => false],
            'sku' => ['type' => 'string', 'required' => false],
        ];
    }

    public function getSourceMetadata(): array
    {
        return [
            'source_type' => 'shopify',
            'shop_domain' => $this->shopDomain,
        ];
    }
}

// Usage
IngestConfig::for(Product::class)
    ->fromSource(new ShopifyProductSource($shopDomain, $apiKey));

Import Events

Hook into the import lifecycle with event handlers:

use LaravelIngest\Contracts\ImportEventHandlerInterface;
use LaravelIngest\DTOs\RowData;
use LaravelIngest\Models\IngestRun;
use LaravelIngest\ValueObjects\ImportStats;

class SlackNotificationHandler implements ImportEventHandlerInterface
{
    public function beforeImport(IngestRun $run): void
    {
        Log::info("Starting import {$run->id}");
    }

    public function onRowProcessed(IngestRun $run, RowData $row, object $model): void
    {
        // Per row logging (use sparingly!)
    }

    public function onError(IngestRun $run, RowData $row, \Throwable $error): void
    {
        Log::error("Import error in row {$row->rowNumber}: {$error->getMessage()}");
    }

    public function afterImport(IngestRun $run, ImportStats $stats): void
    {
        $successRate = $stats->successRate();
        
        if ($stats->wasFullySuccessful()) {
            Slack::send("Import completed successfully: {$stats->successCount} rows");
        } else {
            Slack::send("Import completed with {$stats->failureCount} errors ({$successRate}% success)");
        }
    }
}

// Register
IngestConfig::for(Product::class)
    ->withEventHandler(new SlackNotificationHandler())
    ->fromSource(SourceType::UPLOAD);

ImportStats

The ImportStats object contains:

$stats->totalRows           // Total count
$stats->successCount        // Successfully processed
$stats->failureCount        // Failed rows
$stats->createdCount        // Newly created records
$stats->updatedCount        // Updated records
$stats->skippedCount()      // Skipped rows
$stats->successRate()       // Success rate in %
$stats->wasFullySuccessful() // True if no errors
$stats->duration            // Duration in seconds
$stats->toArray()           // As array for JSON

Nested Mappings

For complex, nested data structures:

use LaravelIngest\IngestConfig;
use LaravelIngest\NestedIngestConfig;
use LaravelIngest\Transformers\NumericTransformer;

IngestConfig::for(Order::class)
    ->map('order_id', 'id')
    ->map('customer_email', 'email')
    ->nest('line_items', function (NestedIngestConfig $nested) {
        $nested->map('sku', 'product_sku')
               ->map('name', 'product_name')
               ->mapAndTransform('qty', 'quantity', NumericTransformer::class)
               ->mapAndTransform('unit_price', 'price', [
                   new NumericTransformer(decimals: 2),
                   new RangeValidator(min: 0),
               ])
               ->keyedBy('sku');
    });

Input:

{
    "order_id": "123",
    "customer_email": "test@example.com",
    "line_items": [
        {"sku": "ABC-001", "name": "Widget", "qty": "2", "unit_price": "19.99"},
        {"sku": "DEF-002", "name": "Gadget", "qty": "1", "unit_price": "29.99"}
    ]
}

Schema Validation

Define the expected schema for better error messages:

IngestConfig::for(Product::class)
    ->expectSchema([
        'id' => ['type' => 'integer', 'required' => true],
        'name' => ['type' => 'string', 'required' => true],
        'price' => ['type' => 'numeric', 'required' => true],
        'description' => ['type' => 'string', 'required' => false, 'nullable' => true],
    ])
    ->fromSource(SourceType::UPLOAD);

Debugging & Tracing

Enable detailed logs for debugging:

IngestConfig::for(Product::class)
    ->withTracing()                    // Trace everything
    // or:
    ->traceTransformations()          // Only transformations
    ->traceMappings()                  // Only mappings

Tracing logs:

Input/output of each transformation
Which fields were mapped where
Which conditional mappings were active

Access traces:

$service = app(DataTransformationService::class);
$traces = $service->getTraceLog();

// [
//     'price' => [
//         ['step' => 'input', 'value' => '123.456'],
//         ['step' => 'NumericTransformer', 'value' => 123.46],
//         ['step' => 'DefaultValueTransformer', 'value' => 123.46],
//     ]
// ]

Summary

Feature	Purpose	API
Validators	Reusable, testable validation	`mapAndValidate()`
Pipelines	Multiple transformations in sequence	Array to `mapAndTransform()`
Conditional Mappings	Context-dependent mapping logic	`mapWhen()`
Custom Sources	External data sources (APIs, etc.)	`SourceInterface`
Events	Hook into the import lifecycle	`ImportEventHandlerInterface`
Nested Mappings	Nested data structures	`nest()`
Schema	Input validation	`expectSchema()`
Tracing	Debugging information	`withTracing()`

All features follow the same pattern: interface-based, testable, declarative.