@terascope/job-components / index
index
Index
Namespaces
Namespace | Description |
---|---|
Logger | - |
Enumerations
Enumeration | Description |
---|---|
DataEncoding | Available data encoding types for a DataEntity |
RecoveryCleanupType | - |
StringEntropy | - |
Classes
Class | Description |
---|---|
BigLRUMap | - |
BigMap | Avoid v8 maximum size for Map by spreading the cache across multiple Maps. This class has the same API as Map but minus more differences in ->set and ->forEach |
BigSet | Avoid v8 maximum size for Set by spreading the cache across multiple Sets. This class has the same API as Set but minus more differences in ->add and ->forEach |
Collector | An in-memory record collector, useful for batch data to a certain size or after a certain amount of time has passed. |
DataEntity | A wrapper for data that can hold additional metadata properties. A DataEntity should be essentially transparent to use within operations. |
EventLoop | A simple class for detecting when the event loop is blocked. The recommend use is to call EventLoop.init(logger) and then await EventLoop.wait() where you want to slow down potentially long running synchronous code |
FlexibleArray | This used to ignore the init size of the array in LRUMap. Doing this will avoid running out of memory at 12 million records. |
Logger | - |
Queue | A basic FIFO queue |
TSError | A custom Error class with additional properties, like statusCode and fatalError |
Interfaces
Interface | Description |
---|---|
APIConfig | APIConfig is the configuration for loading APIs and Observers into a ExecutionContext. |
AnyObject | A simple object with any values |
ConnectionConfig | - |
DataEntityMetadata | DataEntities have conventional metadata properties that can track source, destination and other process information. |
ElasticsearchError | - |
EmptyObject | - |
EncodingConfig | an encoding focused interfaces |
ExecutionConfig | - |
InNumberRangeArg | - |
JobConfig | - |
JobConfigParams | - |
ListOfRecursiveArraysOrValues | A simple definitions of array |
Many | A simple definitions of array |
OpConfig | OpConfig is the configuration that user specifies for a Operation. The only required property is _op since that is used to find the operation. |
PRetryConfig | - |
RecursiveArray | A simple definitions of array |
RouteSenderAPI | Used for sending data to particular index/topic/file/table in a storage system. This is used by the routed sender in the standard-assets |
Slice | A trackable set of work to be preformed by a "Worker" |
SliceAnalyticsData | - |
SliceRequest | The metadata created by the Slicer and ran through a job pipeline |
SysConfig | - |
TSErrorConfig | - |
TSErrorContext | A simple object with any values |
TerafoundationConfig | - |
TerasliceConfig | - |
ValidatedJobConfig | - |
References
APIConstructor
Re-exports APIConstructor
APICore
Renames and re-exports default
APICoreConstructor
Re-exports APICoreConstructor
APIFactory
Renames and re-exports default
APIFactoryRegistry
Re-exports APIFactoryRegistry
APIModule
Re-exports APIModule
ASSET_KEYWORD
Re-exports ASSET_KEYWORD
AssetBundleType
Re-exports AssetBundleType
AssetRepository
Re-exports AssetRepository
AssetRepositoryKey
Re-exports AssetRepositoryKey
AssetsAPI
Re-exports AssetsAPI
BatchProcessor
Renames and re-exports default
BundledAPIOperation
Re-exports BundledAPIOperation
BundledObserverOperation
Re-exports BundledObserverOperation
BundledProcessorOperation
Re-exports BundledProcessorOperation
BundledReaderOperation
Re-exports BundledReaderOperation
CachedClients
Re-exports CachedClients
CollectProcessor
Renames and re-exports default
Context
Re-exports Context
ConvictSchema
Renames and re-exports default
CoreOperation
Re-exports CoreOperation
CrossValidationFn
Re-exports CrossValidationFn
DelayProcessor
Renames and re-exports default
EachProcessor
Renames and re-exports default
EventHandlers
Re-exports EventHandlers
ExecutionContext
Re-exports ExecutionContext
ExecutionContextAPI
Re-exports ExecutionContextAPI
ExecutionContextAPIs
Re-exports ExecutionContextAPIs
ExecutionContextConfig
Re-exports ExecutionContextConfig
ExecutionStats
Re-exports ExecutionStats
Fetcher
Renames and re-exports default
FetcherConstructor
Re-exports FetcherConstructor
FetcherCore
Renames and re-exports default
FilterProcessor
Renames and re-exports default
FindOperationResults
Re-exports FindOperationResults
GetClientConfig
Re-exports GetClientConfig
JobAPIInstance
Re-exports JobAPIInstance
JobAPIInstances
Re-exports JobAPIInstances
JobObserver
Renames and re-exports default
JobRunnerAPI
Re-exports JobRunnerAPI
JobValidator
Re-exports JobValidator
LoaderOptions
Re-exports LoaderOptions
MapProcessor
Renames and re-exports default
MetadataFns
Re-exports MetadataFns
MockPromMetrics
Re-exports MockPromMetrics
NoopProcessor
Renames and re-exports default
Observer
Renames and re-exports default
ObserverConstructor
Re-exports ObserverConstructor
OpAPI
Re-exports OpAPI
OpAPIFn
Re-exports OpAPIFn
OpAPIInstance
Re-exports OpAPIInstance
OpRunnerAPI
Re-exports OpRunnerAPI
OpTypeToRepositoryKey
Re-exports OpTypeToRepositoryKey
OperationAPI
Renames and re-exports default
OperationAPIConstructor
Re-exports OperationAPIConstructor
OperationAPIType
Re-exports OperationAPIType
OperationCore
Renames and re-exports default
OperationCoreConstructor
Re-exports OperationCoreConstructor
OperationLifeCycle
Re-exports OperationLifeCycle
OperationLoader
Re-exports OperationLoader
OperationLocationType
Re-exports OperationLocationType
OperationModule
Re-exports OperationModule
OperationResults
Re-exports OperationResults
OperationTypeName
Re-exports OperationTypeName
ParallelSlicer
Renames and re-exports default
ParallelSlicerConstructor
Re-exports ParallelSlicerConstructor
ParseNameResponse
Re-exports ParseNameResponse
ProcessorConstructor
Re-exports ProcessorConstructor
ProcessorCore
Renames and re-exports default
ProcessorFn
Re-exports ProcessorFn
ProcessorModule
Re-exports ProcessorModule
ReaderFn
Re-exports ReaderFn
ReaderModule
Re-exports ReaderModule
RepositoryOperation
Re-exports RepositoryOperation
RunSliceResult
Re-exports RunSliceResult
SchemaConstructor
Re-exports SchemaConstructor
SchemaCore
Renames and re-exports default
SchemaModule
Re-exports SchemaModule
SelfValidationFn
Re-exports SelfValidationFn
SingleSlicerConstructor
Re-exports SingleSlicerConstructor
SliceResult
Re-exports SliceResult
SliceStatus
Re-exports SliceStatus
Slicer
Renames and re-exports default
SlicerConstructor
Re-exports SlicerConstructor
SlicerCore
Renames and re-exports default
SlicerCoreConstructor
Re-exports SlicerCoreConstructor
SlicerExecutionContext
Re-exports SlicerExecutionContext
SlicerFn
Re-exports SlicerFn
SlicerFns
Re-exports SlicerFns
SlicerOperationLifeCycle
Re-exports SlicerOperationLifeCycle
SlicerOperations
Re-exports SlicerOperations
SlicerRecoveryData
Re-exports SlicerRecoveryData
SlicerResult
Re-exports SlicerResult
TestClientConfig
Re-exports TestClientConfig
TestClients
Re-exports TestClients
TestClientsByEndpoint
Re-exports TestClientsByEndpoint
TestContext
Re-exports TestContext
TestContextAPIs
Re-exports TestContextAPIs
TestContextApis
Re-exports TestContextApis
TestContextOptions
Re-exports TestContextOptions
TestContextType
Re-exports TestContextType
TestReaderFetcher
Renames and re-exports default
TestReaderSlicer
Renames and re-exports default
ValidLoaderOptions
Re-exports ValidLoaderOptions
WorkerExecutionContext
Re-exports WorkerExecutionContext
WorkerOperationLifeCycle
Re-exports WorkerOperationLifeCycle
WorkerOperations
Re-exports WorkerOperations
WorkerSliceState
Re-exports WorkerSliceState
WorkerStatus
Re-exports WorkerStatus
addFormats
Re-exports addFormats
apiSchema
Re-exports apiSchema
formats
Re-exports formats
getAssetPath
Re-exports getAssetPath
getClient
Re-exports getClient
getMetric
Re-exports getMetric
getOpConfig
Re-exports getOpConfig
getOperationAPIType
Re-exports getOperationAPIType
isOperationAPI
Re-exports isOperationAPI
isPromAvailable
Re-exports isPromAvailable
isSlicerExecutionContext
Re-exports isSlicerExecutionContext
isWorkerExecutionContext
Re-exports isWorkerExecutionContext
jobSchema
Re-exports jobSchema
makeContextLogger
Re-exports makeContextLogger
makeExContextLogger
Re-exports makeExContextLogger
makeExecutionContext
Re-exports makeExecutionContext
makeJobSchema
Re-exports makeJobSchema
newTestExecutionConfig
Re-exports newTestExecutionConfig
newTestJobConfig
Re-exports newTestJobConfig
newTestSlice
Re-exports newTestSlice
opSchema
Re-exports opSchema
parseName
Re-exports parseName
registerApis
Re-exports registerApis
sliceAnalyticsMetrics
Re-exports sliceAnalyticsMetrics
validateAPIConfig
Re-exports validateAPIConfig
validateJobConfig
Re-exports validateJobConfig
validateOpConfig
Re-exports validateOpConfig
Type Aliases
AdjustDateArgs
AdjustDateArgs:
object
|object
Source
packages/utils/dist/src/dates.d.ts:128
Assignment
Assignment:
"assets_service"
|"cluster_master"
|"node_master"
|"execution_controller"
|"worker"
Source
packages/types/dist/src/teraslice.d.ts:446
ClusterManagerType
ClusterManagerType:
"native"
|"kubernetes"
|"kubernetesV2"
Source
packages/types/dist/src/teraslice.d.ts:2
CreateClientFactoryFn()
CreateClientFactoryFn: (
config
,logger
,options
) =>Promise
<ConnectorOutput
>
Parameters
Parameter | Type |
---|---|
config | Record <string , any > |
logger | Logger |
options | ConnectionConfig |
Returns
Promise
<ConnectorOutput
>
Source
packages/types/dist/src/terafoundation.d.ts:56
DataArrayInput
Source
packages/utils/dist/src/entities/data-entity.d.ts:152
DataEntityMetadataValue<M>
DataEntityMetadataValue<
M
>: keyofDataEntityMetadata
| keyofM
|string
|number
Type parameters
Type parameter |
---|
M |
Source
packages/utils/dist/src/entities/interfaces.d.ts:8
DataInput
DataInput:
Record
<string
,any
> |DataEntity
Source
packages/utils/dist/src/entities/data-entity.d.ts:151
DeadLetterAPIFn()
DeadLetterAPIFn: (
input
,err
) =>void
A supported DeadLetterAPIFn
Parameters
Parameter | Type |
---|---|
input | any |
err | Error |
Returns
void
Source
packages/types/dist/src/teraslice.d.ts:297
DeadLetterAction
DeadLetterAction:
"throw"
|"log"
|"none"
|string
available dead letter queue actions
Source
packages/types/dist/src/teraslice.d.ts:295
Diff<T, U>
Diff<
T
,U
>:T
extendsU
?never
:T
Remove types from T that are assignable to U
Type parameters
Type parameter |
---|
T |
U |
Source
packages/types/dist/src/utility.d.ts:74
EntityMetadataValue<M, K>
EntityMetadataValue<
M
,K
>:K
extends keyofDataEntityMetadata
?DataEntityMetadata
[K
] :K
extends keyofM
?M
[K
] :any
Type parameters
Type parameter |
---|
M |
K |
Source
packages/utils/dist/src/entities/interfaces.d.ts:9
Filter<T, U>
Filter<
T
,U
>:T
extendsU
?T
:never
Remove types from T that are NOT assignable to U
Type parameters
Type parameter |
---|
T |
U |
Source
packages/types/dist/src/utility.d.ts:78
FilteredResult<T, I, E>
FilteredResult<
T
,I
,E
>:{ [P in keyof T]: P extends I ? T[P] : P extends E ? never : T[P] }
Filters the keys of an object (T), by list of included keys (I) and excluded (E)
Type parameters
Type parameter |
---|
T |
I extends keyof T |
E extends keyof T |
Source
packages/types/dist/src/utility.d.ts:86
LifeCycle
LifeCycle:
"once"
|"persistent"
Source
packages/types/dist/src/teraslice.d.ts:176
Nil
Nil:
null
|undefined
Source
packages/types/dist/src/utility.d.ts:43
Omit<T, K>
Omit<
T
,K
>:Pick
<T
,Exclude
<keyofT
,K
>>
Omit the properties available to type. Useful for excluding properties from a type
Example
`Omit<{ a: number, b: number, c: number }, 'b'|'c'> // => { a: 1 }`
Type parameters
Type parameter |
---|
T |
K extends keyof T |
Source
packages/types/dist/src/utility.d.ts:7
Optional<T, K>
Optional<
T
,K
>: { [P in keyof T]: P extends K ? NonNullable<T[P]> | undefined : NonNullable<T[P]> }
Like Partial but makes certain properties optional
Example
`Optional<{ a: number, b: number }, 'b'>`
Type parameters
Type parameter |
---|
T |
K extends keyof T |
Source
packages/types/dist/src/utility.d.ts:40
Override<T1, T2>
Override<
T1
,T2
>:{ [P in keyof T1]: P extends keyof T1 ? T2[P] : T1[P] }
Override specific properties on a type
Example
`Override<{ a: number, b: number }, { b: string }>`
Type parameters
Type parameter |
---|
T1 |
T2 extends { [P in keyof T2]: P extends keyof T1 ? T2[P] : never } |
Source
packages/types/dist/src/utility.d.ts:22
Overwrite<T1, T2>
Overwrite<
T1
,T2
>:{ [P in Exclude<keyof T1, keyof T2>]: T1[P] }
&T2
Overwrite a simple type with different properties. Useful changing and adding additional properties
Example
`Overwrite<{ a: number, b: number }, { b?: number }>`
Type parameters
Type parameter |
---|
T1 |
T2 |
Source
packages/types/dist/src/utility.d.ts:14
PWhileOptions
PWhileOptions:
object
Type declaration
enabledJitter?
optional
enabledJitter:boolean
enable jitter to stagger requests
error?
optional
error:string
error that can be expressed if the timer ends because the function does not return true
maxJitter?
optional
maxJitter:number
the maximum the jitter wait time will be (in milliseconds), defaults to 3x the minJitter setting, but less than timeoutMs
minJitter?
optional
minJitter:number
the minimum the jitter wait time will be (in milliseconds), defaults to 100ms
name?
optional
name:string
Defaults
to "Request"
timeoutMs?
optional
timeoutMs:number
Defaults
to -1 which is never
Source
packages/utils/dist/src/promises.d.ts:54
PartialDeep<T>
PartialDeep<
T
>:{ [P in keyof T]?: PartialDeep<T[P]> }
A deep partial object
Type parameters
Type parameter |
---|
T |
Source
packages/types/dist/src/utility.d.ts:68
Required<T, K>
Required<
T
,K
>: { [P in keyof T]: P extends K ? NonNullable<T[P]> : NonNullable<T[P]> | undefined }
Like Partial but makes certain properties required
Example
`Required<{ a: number, b: number }, 'b'>`
Type parameters
Type parameter |
---|
T |
K extends keyof T |
Source
packages/types/dist/src/utility.d.ts:32
StringEntropyFN()
StringEntropyFN: (
input
) =>number
Parameters
Parameter | Type |
---|---|
input | unknown |
Returns
number
Source
packages/utils/dist/src/strings.d.ts:268
TYPE_ENTITY_METADATA_KEY
TYPE_ENTITY_METADATA_KEY:
"___EntityMetadata"
Source
packages/utils/dist/src/entities/interfaces.d.ts:3
TYPE_IS_DATAENTITY_KEY
TYPE_IS_DATAENTITY_KEY:
"__isDataEntity"
Source
packages/utils/dist/src/entities/interfaces.d.ts:2
Unpacked<T>
Unpacked<
T
>:T
extends infer U[] ?U
:T
extends (...args
) => infer U ?U
:T
extendsPromise
<infer U> ?U
:T
From https://www.typescriptlang.org/docs/handbook/advanced-types.html#type-inference-in-conditional-types
Type parameters
Type parameter |
---|
T |
Source
packages/types/dist/src/utility.d.ts:92
ValueOf<T>
ValueOf<
T
>:T
[keyofT
]
Get the types object (the opposite of keyof)
Type parameters
Type parameter |
---|
T |
Source
packages/types/dist/src/utility.d.ts:82
WithoutNil<T>
WithoutNil<
T
>:{ [P in keyof T]: T[P] extends Nil ? never : T[P] }
Without null or undefined properties
Type parameters
Type parameter |
---|
T |
Source
packages/types/dist/src/utility.d.ts:49
_DataEntityMetadata<M>
_DataEntityMetadata<
M
>:M
&DataEntityMetadata
&AnyObject
Type parameters
Type parameter |
---|
M |
Source
packages/utils/dist/src/entities/interfaces.d.ts:7
_DataEntityMetadataType
_DataEntityMetadataType:
DataEntityMetadata
|AnyObject
Source
packages/utils/dist/src/entities/interfaces.d.ts:6
Variables
LOWER_CASE_CHARS
const
LOWER_CASE_CHARS:object
Type declaration
a
readonly
a:true
b
readonly
b:true
c
readonly
c:true
d
readonly
d:true
e
readonly
e:true
f
readonly
f:true
g
readonly
g:true
h
readonly
h:true
i
readonly
i:true
j
readonly
j:true
k
readonly
k:true
l
readonly
l:true
m
readonly
m:true
n
readonly
n:true
o
readonly
o:true
p
readonly
p:true
q
readonly
q:true
r
readonly
r:true
s
readonly
s:true
t
readonly
t:true
u
readonly
u:true
v
readonly
v:true
w
readonly
w:true
x
readonly
x:true
y
readonly
y:true
z
readonly
z:true
Source
packages/utils/dist/src/strings.d.ts:75
MAX_STRING_LENGTH_BEFORE_MD5
const
MAX_STRING_LENGTH_BEFORE_MD5:1024
=1024
If we have a hash that is a long value we want to ensure that the value doesn't explode the memory since we may be using that value as a key. So when a string exceeds this specified length we can reduce its length to 35 characters by using md5
Source
packages/utils/dist/src/type-coercion.d.ts:24
NUM_CHARS
const
NUM_CHARS:object
Type declaration
0
readonly
0:true
1
readonly
1:true
2
readonly
2:true
3
readonly
3:true
4
readonly
4:true
5
readonly
5:true
6
readonly
6:true
7
readonly
7:true
8
readonly
8:true
9
readonly
9:true
Source
packages/utils/dist/src/strings.d.ts:131
STATUS_CODES
const
STATUS_CODES:object
Type declaration
100
100:
string
101
101:
string
102
102:
string
103
103:
string
200
200:
string
201
201:
string
202
202:
string
203
203:
string
204
204:
string
205
205:
string
206
206:
string
207
207:
string
208
208:
string
226
226:
string
300
300:
string
301
301:
string
302
302:
string
303
303:
string
304
304:
string
305
305:
string
307
307:
string
308
308:
string
400
400:
string
401
401:
string
402
402:
string
403
403:
string
404
404:
string
405
405:
string
406
406:
string
407
407:
string
408
408:
string
409
409:
string
410
410:
string
411
411:
string
412
412:
string
413
413:
string
414
414:
string
415
415:
string
416
416:
string
417
417:
string
418
418:
string
421
421:
string
422
422:
string
423
423:
string
424
424:
string
425
425:
string
426
426:
string
428
428:
string
429
429:
string
431
431:
string
451
451:
string
500
500:
string
501
501:
string
502
502:
string
503
503:
string
504
504:
string
505
505:
string
506
506:
string
507
507:
string
508
508:
string
509
509:
string
510
510:
string
511
511:
string
Source
packages/utils/dist/src/status-codes.d.ts:1
UPPER_CASE_CHARS
const
UPPER_CASE_CHARS:object
Type declaration
A
readonly
A:true
B
readonly
B:true
C
readonly
C:true
D
readonly
D:true
E
readonly
E:true
F
readonly
F:true
G
readonly
G:true
H
readonly
H:true
I
readonly
I:true
J
readonly
J:true
K
readonly
K:true
L
readonly
L:true
M
readonly
M:true
N
readonly
N:true
O
readonly
O:true
P
readonly
P:true
Q
readonly
Q:true
R
readonly
R:true
S
readonly
S:true
T
readonly
T:true
U
readonly
U:true
V
readonly
V:true
W
readonly
W:true
X
readonly
X:true
Y
readonly
Y:true
Z
readonly
Z:true
Source
packages/utils/dist/src/strings.d.ts:103
WORD_CHARS
const
WORD_CHARS:object
Type declaration
0
readonly
0:true
1
readonly
1:true
2
readonly
2:true
3
readonly
3:true
4
readonly
4:true
5
readonly
5:true
6
readonly
6:true
7
readonly
7:true
8
readonly
8:true
9
readonly
9:true
A
readonly
A:true
B
readonly
B:true
C
readonly
C:true
D
readonly
D:true
E
readonly
E:true
F
readonly
F:true
G
readonly
G:true
H
readonly
H:true
I
readonly
I:true
J
readonly
J:true
K
readonly
K:true
L
readonly
L:true
M
readonly
M:true
N
readonly
N:true