drop_field_conditional

The drop_field_conditional processor drops a field from DataEntities or DataWindows based on the specified conditions. The conditions can be either a regex or a Field Validation function. There is also an invert option to drop fields that don't match a regex or don't pass the validation function. Only a regex or a validation can be specified, if both are configured the job will throw an error.

Usage

Conditionally Drop a field from a document

Example of a job using the drop_field_conditional processor and a regex configured

{
    "name" : "testing",
    "workers" : 1,
    "slicers" : 1,
    "lifecycle" : "once",
    "assets" : [
        "standard"
    ],
    "operations" : [
        {
            "_op": "test-reader"
        },
        {
            "_op": "drop_field_conditional",
            "field": "name",
            "regex": "/?.lly/i"
        }
    ]
}

Output of example job

const data = [
    DataEntity.make({ name: 'lilly', number_field: 1 }),
    DataEntity.make({ name: 'willy', number_field: 2  }),
    DataEntity.make({ name: 'billy', number_field: 3  }),
    DataEntity.make({ name: 'ron', number_field: 4  }),
]

const results = await processor.run(data);

DataEntity.make({ number_field: 1 }),
DataEntity.make({ number_field: 2  }),
DataEntity.make({ number_field: 3  }),
DataEntity.make({ name: 'ron', number_field: 4  }),

Example of a job using a Field Validation function

{
    "name" : "testing",
    "workers" : 1,
    "slicers" : 1,
    "lifecycle" : "once",
    "assets" : [
        "standard"
    ],
    "operations" : [
        {
            "_op": "test-reader"
        },
        {
            "_op": "drop_field_conditional",
            "field": "number_field",
            "validation_method": "inNumberRange",
            "validation_args": { "min": 2, "max": 4 }
        }
    ]
}

Output of example job

const data = [
    DataEntity.make({ name: 'lilly', number_field: 1 }),
    DataEntity.make({ name: 'willy', number_field: 2 }),
    DataEntity.make({ name: 'billy', number_field: 3 }),
    DataEntity.make({ name: 'ron', number_field: 4 }),
]

const results = await processor.run(data);

DataEntity.make({ name: 'lilly', number_field: 1 }),
DataEntity.make({ name: 'willy', number_field: 2 }),
DataEntity.make({ name: 'billy' }),
DataEntity.make({ name: 'ron', number_field: 4 }),

Example Job with a Field Validation function and invert set to true

{
    "name" : "testing",
    "workers" : 1,
    "slicers" : 1,
    "lifecycle" : "once",
    "assets" : [
        "standard"
    ],
    "operations" : [
        {
            "_op": "test-reader"
        },
        {
            "_op": "drop_field_conditional",
            "field": "number_field",
            "validation_method": "inNumberRange",
            "validation_args": { "min": 2, "max": 4 },
            "invert": true
        }
    ]
}

Output of example job

const data = [
    DataEntity.make({ name: 'lilly', number_field: 1 }),
    DataEntity.make({ name: 'willy', number_field: 2  }),
    DataEntity.make({ name: 'billy', number_field: 3  }),
    DataEntity.make({ name: 'ron', number_field: 4  }),
]

const results = await processor.run(data);

DataEntity.make({ name: 'lilly' }),
DataEntity.make({ name: 'willy' }),
DataEntity.make({ name: 'billy', number_field: 3 }),
DataEntity.make({ name: 'ron' }),

Parameters

Configuration	Description	Type	Notes
_op	Name of operation, it must reflect the exact name of the file	String	required
field	Name of field to remove	String	required, no default
regex	regex value to check with field values	String	regex must be in format /REGEX/FLAGS, with beginning and ending /. Flags are optional. For more details see javascript regex's
validation_method	Name of validation method to apply to field value	String	see Field Validations for list of available functions
validation_args	some validations require args	Object	optional
invert	When set to true, the processor drops the value if it doesn't match the regex or if it doesn't pass the validation	Boolean	defaults to false