@terascope/file-asset-apis / base/chunked-file-sender / ChunkedFileSender

Class: abstract ChunkedFileSender

Defined in: base/chunked-file-sender.ts:50

Extended by

• FileSender
• HDFSSender
• S3Sender

Constructors

Constructor

new ChunkedFileSender(type, config, logger): ChunkedFileSender

Defined in: base/chunked-file-sender.ts:59

Parameters

Parameter	Type
type	FileSenderType
config	ChunkedFileSenderConfig
logger	Logger

Returns

ChunkedFileSender

Properties

Property	Modifier	Type	Default value	Defined in
config	readonly	ChunkedFileSenderConfig	undefined	base/chunked-file-sender.ts:56
formatter	protected	Formatter	undefined	base/chunked-file-sender.ts:53
logger	readonly	Logger	undefined	base/chunked-file-sender.ts:57
pathList	readonly	Map<string, boolean>	undefined	base/chunked-file-sender.ts:54
sliceCount	protected	number	-1	base/chunked-file-sender.ts:51
type	readonly	FileSenderType	undefined	base/chunked-file-sender.ts:55

Accessors

concurrency

Get Signature

get concurrency(): number

Defined in: base/chunked-file-sender.ts:103

Returns

number

---

filePerSlice

Get Signature

get filePerSlice(): boolean

Defined in: base/chunked-file-sender.ts:111

Returns

boolean

---

format

Get Signature

get format(): Format

Defined in: base/chunked-file-sender.ts:115

Returns

Format

---

id

Get Signature

get id(): string

Defined in: base/chunked-file-sender.ts:91

Returns

string

---

isRouter

Get Signature

get isRouter(): boolean

Defined in: base/chunked-file-sender.ts:99

Returns

boolean

---

lineDelimiter

Get Signature

get lineDelimiter(): string

Defined in: base/chunked-file-sender.ts:107

Returns

string

---

nameOptions

Get Signature

get nameOptions(): NameOptions

Defined in: base/chunked-file-sender.ts:119

Returns

NameOptions

---

path

Get Signature

get path(): string

Defined in: base/chunked-file-sender.ts:95

Returns

string

Methods

createFileDestinationName()

createFileDestinationName(filePath): Promise<string>

Defined in: base/chunked-file-sender.ts:154

Ensures the root destination file path exists and returns the full destination file path.

Parameters

Parameter	Type
filePath	string

Returns

Promise<string>

Note

this API might change since it should not have knowledge of the different sender types

---

incrementCount()

protected incrementCount(): void

Defined in: base/chunked-file-sender.ts:236

Increases the internal sliceCount used to create filenames, this is already executed in the "send" method, do not use unless you are implementing your own send logic

Returns

void

---

joinPath()

protected joinPath(route?): string

Defined in: base/chunked-file-sender.ts:177

Method to help create proper file paths, mainly used in the abstract "verify" method

Parameters

Parameter	Type
route?	string

Returns

string

---

prepareDispatch()

prepareDispatch(data): Promise<SendBatchConfig[]>

Defined in: base/chunked-file-sender.ts:192

Batches records in a slice into groups based on the "path" config or by the DataEntity metadata 'standard:route' override if dynamic routing is being used, this is called in the "send" method

Parameters

Parameter	Type
data	SendRecords

Returns

Promise<SendBatchConfig[]>

---

send()

send(records): Promise<number>

Defined in: base/chunked-file-sender.ts:251

Write data to file, uses parent "sendToDestination" method to determine location

Parameters

Parameter	Type
records	SendRecords

Returns

Promise<number>

Example

s3Sender.send([{ some: 'data' }]) => Promise<void>
  s3Sender.send([DataEntity.make({ some: 'data' })]) => Promise<void>

---

sendToDestination()

abstract protected sendToDestination(config): Promise<void>

Defined in: base/chunked-file-sender.ts:240

Parameters

Parameter	Type
config	SendBatchConfig

Returns

Promise<void>

---

simpleSend()

simpleSend(records): Promise<void>

Defined in: base/chunked-file-sender.ts:283

Write data to file, uses parent "sendToDestination" method to determine location. Use this to avoid having to bucket the data into different paths. Normally you don't want this but it can be used in specific case.

Parameters

Parameter	Type
records	SendRecords

Returns

Promise<void>

Example

s3Sender.simpleSend([{ some: 'data' }]) => Promise<void>
  s3Sender.simpleSend([DataEntity.make({ some: 'data' })]) => Promise<void>

---

verify()

abstract verify(path): Promise<void>

Defined in: base/chunked-file-sender.ts:129

Parameters

Parameter	Type
path	string

Returns

Promise<void>