Version bump: 1.1.0 Fortune for collect's current commit: Half curse − 半凶
|6 months ago|
|src||6 months ago|
|.gitignore||6 months ago|
|Cargo.lock||6 months ago|
|Cargo.toml||6 months ago|
|LICENSE||7 months ago|
|README.md||7 months ago|
|TODO||6 months ago|
collect - Collect all input until it's closed, then output it all at once.
This small tool can be used to ensure all data between pipes is synchronised, and/or to ensure the 2nd program in the pipe doesn't start processing before the first one has finished outputting her data.
For example, in the pipeline
x | collect | y, where
x is a program who's output is sporadic (something like a network connection, reading and processing a segmented file, etc)
y will receive all of
xs output at once as soon as
x closes her standard output pipe. So
y will not start processing until
x has completed hers.
There are no runtime flags (unless logging is enabled, in which case, see below), it simply reads from
stdin and writes to
stdout. (When logging is enabled, and the log-level is set to a level that will enabled common info logging, it is written to
stderr only to not interfere with the data collected from
When compiled with the
logging feature (default), you can control the log level with the
RUST_LOG environment variable (the default for release builds is
info, for debug builds,
To set the level, run with
RUST_LOG= one of the below values:
trace- The lowest level of logging, all information will be printed.
debug- The 2nd lowest level, debugging-relevent information (such as buffer sizes, file descriptor numbers/names, read/write segment sizes, allocations, etc.) will be printed. (default for
info- Will print information when collection has started, finished, and output is over. (default for
warn- Will print only warnings. Most of these that will be seen will be related to additionally required syscalls for fd-size truncation, which are only efficiency-related and not warnings to the user herself's use of the program. But some will be.
error- Only print error messages.
off- Print no messages at all.
To build with the default configuration:
$ cargo build --release
Will build the binary into
To create a debug build:
$ cargo build
Will build the binary into
logging feature is enabled, the default logging level will be
debug instead of
To create a release build that is not symbol-stripped:
$ cargo build --profile symbols
Will build the binary into
Modes & features
There are two major operative modes:
mode-memfile (default [+
These are collections of features specific to each operating mode.
Each mode feature can be chosen by building with a
Cargo incantation in the following format:
$ cargo build --release --no-default-features --features mode-<name>[,logging]
mode-memfile- This is the default used mode, which will use the feature
memfile-preallocate. NOTE: The default enabled features chooses this mode and the
mode-buffered- This will use
bytes-allocated buffers instead of file-descriptors.
NOTE: If both modes are specified at once,
mode-memfile will take precidence by the program, and
mode-buffered will not be used.
The user can also compile the program with individual features specific to her needs.
They can be specified as such:
||Use an in-memory file-descriptor pointing to unmapped physical pages of memory. This will allow the program to make use of the more efficient
||WARNING: Can potentially cause a full system OOM if used incorrectly or irresponsibly. (See below)|
||Some crude benchmarks have shown this to be mildly more efficient in
||Removes all runtime logging code. Span-traces are still captured, however, they just are never used.||This won't save you much compared to just disabling the
||Enable the capture and reporting of span-traces and events. (See the section on logging above.)||This does cause a slowdown, but can provide useful information to the user about error locations, warnings, when and where input and output have finished and the sizes of both, etc. If you're only using it in scripts however, it'd be better to disable. (default enabled)|
memfile is enabled, and the input size can be determined by the program, it will preallocate the required space for the input.
If this input were to exceed the amount of physical memory available (since this is unpaged memory being allocated,) it could hang and/or then cause the kernel to OOMkill basically everything except
Please note however, this would only typically happen in instances where a file is passed as input (where the length can be determined, the source it usually not segmented at all); in which case
collect is just going to slow down your pipe. (It is still worth using for scripts where the script doesn't know if the standatd input is a file or not.)
In the current version, this is not yet accounted for, so passing massive files, for example:
$ collect <10-gb-file | wc -c
Will try to allocate 10GB of physical memory for the collection.
In future versions, a warning for large known-size inputs will be displayed, and an error for known-size inputs so large they would cause an OOM. (Same for unknown-sized inputs that grow the backing memfd to a size that would start to become an issue or would use too much physical memory.)
But currently, this is a pitfall of the
memfile mode that, while very unlikely to ever be encountered, could still bite the user if it is encountered.
If something like this may be a concern for your usecase, please fall-back to using the
buffered mode instead, which, while significantly slower, will only OOM itself if the input is too large and cannot eat physical memory directly, only its already-large VM page maps which are, for most instances, mostly empty.
CPL'd with <3