@terascope/file-asset-apis / base/slice

base/slice

Functions

canReadFile()

canReadFile(fileName): boolean

Defined in: base/slice.ts:110

Determines if a file name or file path can be processed, it will return false if the path includes a segment that starts with a "."

Parameters

Parameter	Type
fileName	string

Returns

boolean

Example

canReadFile('file.txt')  => true
  canReadFile('some/path/file.txt')  => true
  canReadFile('some/.private_path/file.txt')  => false

---

getOffsets()

getOffsets(size, total, delimiter): Offsets[]

Defined in: base/slice.ts:71

Used to calculate the offsets of a file, this is a lower level API, prefer to use segmentFile over this as it is a higher level API

Parameters

Parameter	Type
size	number
total	number
delimiter	string

Returns

Offsets[]

---

parsePath()

parsePath(objPath): object

Defined in: base/slice.ts:125

Parses the provided path and translates it to a bucket/prefix combo

Primarily a helper for s3 operations

Parameters

Parameter	Type
objPath	string

Returns

object

bucket

bucket: string

prefix

prefix: string

---

segmentFile()

segmentFile(file, config): FileSlice[]

Defined in: base/slice.ts:38

Used to slice up a file based on the configuration provided, this is a higher level API, prefer this over getOffsets as that is a lower level API

The returned results can be used directly with any "read" method of a reader API

Parameters

Parameter	Type
file	{ path: string; size: number; }
file.path	string
file.size	number
config	SliceConfig

Returns

FileSlice[]

Example

const slice = { path: 'some/path', size: 1000 };
    const config = {
       file_per_slice: false,
       line_delimiter: '\n',
       size: 300,
       format: Format.ldjson
    };

    const results = segmentFile(slice, config);

    // outputs => [
       {
           offset: 0, length: 300, path: 'some/path', total: 1000
       },
           offset: 299, length: 301, path: 'some/path', total: 1000
       },
           offset: 599, length: 301, path: 'some/path', total: 1000
       },
       {
           offset: 899, length: 101, path: 'some/path', total: 1000
       }
    ]