@terascope/file-asset-apis / hdfs/hdfs-sender / HDFSSender

Class: HDFSSender

Defined in: hdfs/hdfs-sender.ts:11

Extends

• ChunkedFileSender

Implements

• RouteSenderAPI

Constructors

Constructor

new HDFSSender(client, config, logger): HDFSSender

Defined in: hdfs/hdfs-sender.ts:14

Parameters

Parameter	Type
client	AnyObject
config	ChunkedFileSenderConfig
logger	Logger

Returns

HDFSSender

Overrides

ChunkedFileSender.constructor

Properties

Property	Modifier	Type	Default value	Inherited from	Defined in
client	public	AnyObject	undefined	-	hdfs/hdfs-sender.ts:12
config	readonly	ChunkedFileSenderConfig	undefined	ChunkedFileSender.config	base/chunked-file-sender.ts:56
formatter	protected	Formatter	undefined	ChunkedFileSender.formatter	base/chunked-file-sender.ts:53
logger	readonly	Logger	undefined	ChunkedFileSender.logger	base/chunked-file-sender.ts:57
pathList	readonly	Map<string, boolean>	undefined	ChunkedFileSender.pathList	base/chunked-file-sender.ts:54
sliceCount	protected	number	-1	ChunkedFileSender.sliceCount	base/chunked-file-sender.ts:51
type	readonly	FileSenderType	undefined	ChunkedFileSender.type	base/chunked-file-sender.ts:55

Accessors

concurrency

Get Signature

get concurrency(): number

Defined in: base/chunked-file-sender.ts:103

Returns

number

Inherited from

ChunkedFileSender.concurrency

---

filePerSlice

Get Signature

get filePerSlice(): boolean

Defined in: base/chunked-file-sender.ts:111

Returns

boolean

Inherited from

ChunkedFileSender.filePerSlice

---

format

Get Signature

get format(): Format

Defined in: base/chunked-file-sender.ts:115

Returns

Format

Inherited from

ChunkedFileSender.format

---

id

Get Signature

get id(): string

Defined in: base/chunked-file-sender.ts:91

Returns

string

Inherited from

ChunkedFileSender.id

---

isRouter

Get Signature

get isRouter(): boolean

Defined in: base/chunked-file-sender.ts:99

Returns

boolean

Inherited from

ChunkedFileSender.isRouter

---

lineDelimiter

Get Signature

get lineDelimiter(): string

Defined in: base/chunked-file-sender.ts:107

Returns

string

Inherited from

ChunkedFileSender.lineDelimiter

---

nameOptions

Get Signature

get nameOptions(): NameOptions

Defined in: base/chunked-file-sender.ts:119

Returns

NameOptions

Inherited from

ChunkedFileSender.nameOptions

---

path

Get Signature

get path(): string

Defined in: base/chunked-file-sender.ts:95

Returns

string

Inherited from

ChunkedFileSender.path

Methods

createFileDestinationName()

createFileDestinationName(filePath): Promise<string>

Defined in: base/chunked-file-sender.ts:154

Ensures the root destination file path exists and returns the full destination file path.

Parameters

Parameter	Type
filePath	string

Returns

Promise<string>

Note

this API might change since it should not have knowledge of the different sender types

Inherited from

ChunkedFileSender.createFileDestinationName

---

incrementCount()

protected incrementCount(): void

Defined in: base/chunked-file-sender.ts:236

Increases the internal sliceCount used to create filenames, this is already executed in the "send" method, do not use unless you are implementing your own send logic

Returns

void

Inherited from

ChunkedFileSender.incrementCount

---

joinPath()

protected joinPath(route?): string

Defined in: base/chunked-file-sender.ts:177

Method to help create proper file paths, mainly used in the abstract "verify" method

Parameters

Parameter	Type
route?	string

Returns

string

Inherited from

ChunkedFileSender.joinPath

---

prepareDispatch()

prepareDispatch(data): Promise<SendBatchConfig[]>

Defined in: base/chunked-file-sender.ts:192

Batches records in a slice into groups based on the "path" config or by the DataEntity metadata 'standard:route' override if dynamic routing is being used, this is called in the "send" method

Parameters

Parameter	Type
data	SendRecords

Returns

Promise<SendBatchConfig[]>

Inherited from

ChunkedFileSender.prepareDispatch

---

send()

send(records): Promise<number>

Defined in: base/chunked-file-sender.ts:251

Write data to file, uses parent "sendToDestination" method to determine location

Parameters

Parameter	Type
records	SendRecords

Returns

Promise<number>

Example

s3Sender.send([{ some: 'data' }]) => Promise<void>
  s3Sender.send([DataEntity.make({ some: 'data' })]) => Promise<void>

Implementation of

RouteSenderAPI.send

Inherited from

ChunkedFileSender.send

---

sendToDestination()

protected sendToDestination(__namedParameters): Promise<void>

Defined in: hdfs/hdfs-sender.ts:25

This is a low level API, it is not meant to be used externally, please use the "send" method instead.

Parameters

Parameter	Type
__namedParameters	SendBatchConfig

Returns

Promise<void>

Todo

THIS PROBABLY NOT WORK and is missing some logic in the file sender

Overrides

ChunkedFileSender.sendToDestination

---

simpleSend()

simpleSend(records): Promise<void>

Defined in: base/chunked-file-sender.ts:283

Write data to file, uses parent "sendToDestination" method to determine location. Use this to avoid having to bucket the data into different paths. Normally you don't want this but it can be used in specific case.

Parameters

Parameter	Type
records	SendRecords

Returns

Promise<void>

Example

s3Sender.simpleSend([{ some: 'data' }]) => Promise<void>
  s3Sender.simpleSend([DataEntity.make({ some: 'data' })]) => Promise<void>

Inherited from

ChunkedFileSender.simpleSend

---

verify()

verify(route): Promise<void>

Defined in: hdfs/hdfs-sender.ts:49

Parameters

Parameter	Type
route	string

Returns

Promise<void>

Implementation of

RouteSenderAPI.verify

Overrides

ChunkedFileSender.verify