part: Fixed bug where `cap` may grow above buffer-halve length (in `SearchSeq` and `SearchPar`.)
part: TODO: Added test skeleton for testing `SearchSeq` and `SearchPar` correctness. Implement these tests!
Fortune for reverse's current commit: Small blessing − 小吉
Started module `part`: Partitioning of memory areas (halve-then-`{r,m}emchr("\n")` pivot calculation) (parallelised via rayon + crossbeam-queue (+maybe tokio_uring?) if `threads` feature enabled.
Fortune for reverse's current commit: Middle blessing − 中吉
# Enables parallelising the operation where possible.
#
# Cli config options can control this, but in default auto setting the process will decide if parallelisation is worth using on the dimensions of the provided input and system.
# Attempt `mmap()`ping the input instead of reading it.
# If the output can be mapped, it may also be.
#
# Useable on unix-family systems. (TODO: Make this dependency target-dependent for unix instead of a manual feature addition like this.)
mapped-file=["dep:mapped-file"]
# TODO: Make this non-dependent feature of `threads`: use `tokio_uring` tasks for I/O, and pull in `tokio` **instead of** rayon for operations. For now, we'll see if we can get away with single-threaded async for `tokio_uring` usage and `rayon`'s auto parallisation for the chunk partitioning; if not, change this to not be internal.
#
# We are using `tokio_uring` & `tokio-stream` to allow parallel queued writes. (TODO: If we end up using (or needing) this feature for partitioning (e.g. we need a controlled IO thread that takes the data gathered by the parallelised partition system or we need a task pool (NOTE: the task pool will execute on tokio_uring's thread only; rayon's thread pool is being used for auto parallelisation currently) to spawn reverse-reader tasks per partition.) then: Remove the use of crossbeam (XXX: and maybe rayon?), and use Tokio runtime for manual parallelised partitioning/line-reversing.)
#TODO: Move these features to Clap runtime arg parse options.
#TODO: Add feature 'mapped-file': Attempt to map input file and process data backwards from end if possible.
#TODO: Add feature 'threads': Use rayon to parallelise line-searching in buffer (or mapped file ^) (partition in half, move partition line to closest {r,}memchr('\n'); repeat N times recursively on each side (parallel) where N is number of logical CPU cores and/or to get the chunks below/in a certain size range.
# Output as lines instead of `["2", "1", "0"]`
output-lines=[]
#XXX output-lines = []
# Print each string escaped
output-quoted=[]
#XXX output-quoted = []
# Buffer output to conserve syscalls, useful for very large inputs (can cause higher memory usage, but generally speeds output up considerably)
buffer-output=[]
#XXX buffer-output = []
# Do not attempt to handle output errors.
# Disable this if you are writing to a faulty device or expect some output operations to stdout to fail.
ignore-output-errors=[]
#XXX ignore-output-errors = []
# Ignore invalid arguments instead of removing invalid UTF8 characters if they exist in the argument
ignore-invalid-args=[]
#XXX ignore-invalid-args = []
# Operate on raw input byte arrays instead of strings; so non-utf8 characters will be preserved in both input and output
# NOTE: May cause collecting from stdin to be *slightly* slower when enabled; so only enable if you intend to be operating on non-utf8 strings (which is usually unlikely)
# NOTE: `ignore-invalid-args` will do nothing if this is enabled.
/// To use the `unwrap_infallible()`-like interface, functions that return `-> !` should be changed to `-> Never`.
/// When `unstable` is enabled, this is an alias to `!` and `-> !` is not special cased.
///
/// # As return argument
/// When feature `unstable` is enabled, `into_unreachable()` may not be required to ensure propogation to `!` from a function returning `-> Never`.
#[cfg(feature="unstable")]
pubtypeNever=!;
/// The default bottom type.
///
/// To use the `unwrap_infallible()`-like interface, functions special cased to `-> !` should be changed to `-> Never`.
///
/// # As return argument
/// When feature `unstable` is not enabled, `into_unreachable()` may be required to be used when dealing with return bottom types other than the special case `-> !`.
/// This is a current limitation of the type system.
#[cfg(not(feature="unstable"))]
pubtypeNever=Infallible;
/// Contractually ensures this type cannot exist (i.e. it is a bottom type.)
///
/// # Safety
/// Instances of the impl type **cannot exist**.
/// They must be bottom types (i.e. empty enums, types contatining an `Infallible` / `!` object, etc.)
///
/// # Auto-impl
/// This trait is not intended to be implemented on any user-defined type other than empty enums.
///
/// By default it is implemented for the following types:
/// - `core::convert::Infallible`
/// - `!` (**feature**: `unstable`)
/// - `Box<T>` *where* `T: ?Sized + Unreachable`
pubunsafetraitUnreachable{
/// Force control flow to terminate type checking here.
///
/// # Note
/// This function will never be executed, it is used to terminate the value's existence in the type system, by converting it from any `Unreachable` type into the bottom return type `!`.
/// If this function ever **can** be called at all, it is undefined behaviour.
#[inline]
#[cold]
fninto_unreachable(self)-> !
whereSelf: Sized
{
ifcfg!(debug_assertions){
unreachable!("Unreachable conversion from {}!",std::any::type_name::<Self>())
}else{
// SAFETY: Contractually enforced by the trait impl itself.
letmax_cap=matchget_max_pivot_search_area(haystack.len()>(DEFAULT_PIVOT_MAX_SEARCH_AREA*DEFAULT_MEM_DETECT_HUGE_SIZE_PAGES)){// Assume huge-page memory if len is larger than 4 pages.
// On debug builds, cap search area to one system page only.
// let value_cont = (parking_lot::Condvar::new(), parking_lot::FairMutex::new(None::<&'a u8>));
let(muthb,muthf)=haystack.split_at(begin);
letmax_cap=matchget_max_pivot_search_area(hf.len()>(DEFAULT_PIVOT_MAX_SEARCH_AREA*DEFAULT_MEM_DETECT_HUGE_SIZE_PAGES)){// Assume huge-page memory if len is larger than 4 pages.
// On debug builds, cap search area to one system page only.
}).expect("Failed to spawn forward-searcher thread"))
}else{
None
};
//NOTE: There is no need to spawn another thread for the 2nd operation, since they are both join()'d at the end regardless and both already communicate completion.
letbackward=ifhb.len()>0{
letcap=cap;
letsf=&self;
letcomplete=&complete;
// Main thread: Backwards search.
move||-> Option<_>{
letmutcap=std::cmp::min(cap,hb.len());
letlen=hb.len();
// Check completion before starting loop too.
ifcomplete.load(){
returnNone;
}else{
// Allow previous thread to run if it is not.
std::thread::yield_now();
}
whilecap<=len{
// If `cap` is larger than the buffer `hb`, truncate it.
todo!("Perform one single buffer partition partition (buffer/2.at_nearest_mpr(needle)) (using `method.search_combined()`) and return its parts. If we can fast-path skip the `search_combined()` then that is okay (e.g. if the buffer/2 is small enough that we should just use `SearchSeq`, we can use `SearchSeq` instead of `S`, and so on.) (XXX: Also see below about thread spawning on parallelised partitions and re-using thread pools (we may be able to do this manually with crossbeam, or we might just have to embrace using `spawn_blocking()` async/a default tokio multithreaded-runtime) since parallel partitions needs at least two threads to search both directions at a time.)")
}
//XXX: Should we add a `SearchAsync`? Or an impl for SearchPar that uses already-spawned threads? TODO: It would be best if we could re-use extant threads instead of spawning two on each partition...
//Parallel (feature="threads") byte area partition-on-nearest-newline-to-halve impl, and non-parallel (default) impl. These impls can differ in their desired depths of partitioning (using parallel impls should balance num of partitions to num of logical cpus & input(/desired chunk) size.)
//TODO: Add tests for `Search{Seq,Par}` partitioning methods.
//TODO: Thread-reusing parallel `MidpointFBSearcher` (SearchSeq is thread-*spawning*; heavy.) This may require we use async and tasks. If it does, we should also create a `SearchAsync` partitioner (XXX: MidpointFBSearcher is currently a synchonous-only interface; a pure-async pivot finder may require a refactor.)
Avril