TODO: Make that calculation configuration compile-time env-var configurable in debug & test builds (for benchmarking.)
Fortune for reverse's current commit: Future small blessing − 末小吉
part: Fixed bug where `cap` may grow above buffer-halve length (in `SearchSeq` and `SearchPar`.)
part: TODO: Added test skeleton for testing `SearchSeq` and `SearchPar` correctness. Implement these tests!
Fortune for reverse's current commit: Small blessing − 小吉
Started module `part`: Partitioning of memory areas (halve-then-`{r,m}emchr("\n")` pivot calculation) (parallelised via rayon + crossbeam-queue (+maybe tokio_uring?) if `threads` feature enabled.
Fortune for reverse's current commit: Middle blessing − 中吉
# Enables parallelising the operation where possible.
#
# Cli config options can control this, but in default auto setting the process will decide if parallelisation is worth using on the dimensions of the provided input and system.
# Attempt `mmap()`ping the input instead of reading it.
# If the output can be mapped, it may also be.
#
# Useable on unix-family systems. (TODO: Make this dependency target-dependent for unix instead of a manual feature addition like this.)
mapped-file=["dep:mapped-file"]
# TODO: Make this non-dependent feature of `threads`: use `tokio_uring` tasks for I/O, and pull in `tokio` **instead of** rayon for operations. For now, we'll see if we can get away with single-threaded async for `tokio_uring` usage and `rayon`'s auto parallisation for the chunk partitioning; if not, change this to not be internal.
#
# We are using `tokio_uring` & `tokio-stream` to allow parallel queued writes. (TODO: If we end up using (or needing) this feature for partitioning (e.g. we need a controlled IO thread that takes the data gathered by the parallelised partition system or we need a task pool (NOTE: the task pool will execute on tokio_uring's thread only; rayon's thread pool is being used for auto parallelisation currently) to spawn reverse-reader tasks per partition.) then: Remove the use of crossbeam (XXX: and maybe rayon?), and use Tokio runtime for manual parallelised partitioning/line-reversing.)
#TODO: Move these features to Clap runtime arg parse options.
#TODO: Add feature 'mapped-file': Attempt to map input file and process data backwards from end if possible.
#TODO: Add feature 'threads': Use rayon to parallelise line-searching in buffer (or mapped file ^) (partition in half, move partition line to closest {r,}memchr('\n'); repeat N times recursively on each side (parallel) where N is number of logical CPU cores and/or to get the chunks below/in a certain size range.
# Output as lines instead of `["2", "1", "0"]`
output-lines=[]
#XXX output-lines = []
# Print each string escaped
output-quoted=[]
#XXX output-quoted = []
# Buffer output to conserve syscalls, useful for very large inputs (can cause higher memory usage, but generally speeds output up considerably)
buffer-output=[]
#XXX buffer-output = []
# Do not attempt to handle output errors.
# Disable this if you are writing to a faulty device or expect some output operations to stdout to fail.
ignore-output-errors=[]
#XXX ignore-output-errors = []
# Ignore invalid arguments instead of removing invalid UTF8 characters if they exist in the argument
ignore-invalid-args=[]
#XXX ignore-invalid-args = []
# Operate on raw input byte arrays instead of strings; so non-utf8 characters will be preserved in both input and output
# NOTE: May cause collecting from stdin to be *slightly* slower when enabled; so only enable if you intend to be operating on non-utf8 strings (which is usually unlikely)
# NOTE: `ignore-invalid-args` will do nothing if this is enabled.
//XXX: I don't think not replacing cloneable variants is a good or useful default. (see below V)
//// Clone non-data-holding variants.
//Self::Park => Self::Park,
//// If the ctx has already been taken, return `None`.
//Self::Taken => return None,
// Move out of data-holding variants.
_=>mem::replace(self,Self::Taken)
})
}
}
/// A loanable thread that belongs to a pool.
///
/// # Pooling
/// When taken from a `ThreadPool`, the object is *moved* out of the thread-pool's ring-buffer and then pinned in return (See `PooledThread`.)
///
/// To return the thread to the pool, the `callback_passage` writes a `Park` command to the thread, unpins itself, and then on drop moves itself back into the start of the ring-buffer of the thread pool `owner`
/// The backing thread, which when started parks itself.
/// The `PooledThread` object will unpark the thread when the object is loaned (moved) out of the pool, and re-parked when enter
///
/// Will wait on `callback_passage?.0` for `Execute` directives, pass them back to the handle, and when a `Park` directive is recieved (i.e. when moved back into the pool) the thread will park itself and wait to be unparked when moved out of the pool or closed again.
///
/// If `callback_message` is dropped, the thread will exit when unparked (dropped from pool.)
//TODO: Build a ThreadPool<'scope = 'static> using crossbeam::ArrayQueue<...> (XXX: See phone for details on how pool should use un/parking for loaning; and CondVar for passing tasks.) Also, add a `Loan` trait with assoc type `Item<'owner>` and methods `try_loan(&self) -> Result<Self::Item<'_>, ...>`, `return(&self, item: Self::Item<'_>) -> Result<(), ...>` and `try_loan_owned(self: Arc<Self>) -> `MovedResult<Arc<Self>, OwnedLoan<'static, Self>, ...>` (struct OwnedLoad<'scope, L>{ item: Option<L::Item<'scope>> } )`, etc.
/// Add forwarding borrow + deref (+ optional `into_inner()`) impls for a type.
///
/// # Usage
/// For a mutable forwarding newtype:
/// ```
/// # use crate::forward_newtype;
/// # use core::borrow::*;
///
/// /// A mutable buffer newtype over an array.
/// struct Buffer([u8; 16]);
/// forward_newtype!(mut Buffer => [u8], 0); // Generates `Borrow<[u8]>`, `BorrowMut<[u8]>`, `Deref<Target=[u8]>`, and `DerefMut` impls for `Buffer` that return `<&[mut] self.>0` (as specified by `0`.)
/// ```
///
/// For an immutable forwarding newtype:
/// ```
/// # use crate::forward_newtype;
/// # use core::borrow::*;
///
/// /// A mutable buffer newtype over an array.
/// struct Data([u8; 16]);
/// forward_newtype!(ref Buffer => [u8], 0); // Generates `Borrow<[u8]>` and `Deref<Target=[u8]>` impls for `Buffer` that return `<& self.>0` (as specified by `0`.) Immutable access only is specified by `ref`.
/// ```
///
/// ## Consuming into inner
/// To generate an `into_inner(self) -> T` inherent impl for the type, the syntax `forward_newtype!(move [const] Type => Inner, accessor)` can be used.
/// If `const` is passed, then the `into_inner()` function will be a `const fn`, if not, then it won't be.
///
/// To combine with ref-forwarding accessors, the syntax `forward_newtype!(move [const] {ref/mut} Type => Inner, accessor)` can be used to generate them all; the `Borrow`, `BorrowMut`, `Deref`, `DerefMut` and `pub [const] fn into_inner()`.
/// This is the most likely to be useful.
///
/// If you need a seperate `into_inner()` impl, you can either not use the `move` declarator, or use the `ref`/`mut` accessor generator in a different statement than the `move` one:
/// ```
/// # use crate::forward_newtype;
/// # use core::borrow::*;
///
/// /// A mutable buffer newtype over an array.
/// struct Buffer([u8; 16]);
/// forward_newtype!(mut Buffer => [u8], 0); // Generate a mutable & immutable forwarding ref to a slice of bytes.
/// forward_newtype!(move const Buffer => [u8; 16], 0); // Generate a seperately typed `into_inner()` that returns the sized array.
/// To use the `unwrap_infallible()`-like interface, functions that return `-> !` should be changed to `-> Never`.
/// When `unstable` is enabled, this is an alias to `!` and `-> !` is not special cased.
///
/// # As return argument
/// When feature `unstable` is enabled, `into_unreachable()` may not be required to ensure propogation to `!` from a function returning `-> Never`.
#[cfg(feature="unstable")]
pubtypeNever=!;
/// The default bottom type.
///
/// To use the `unwrap_infallible()`-like interface, functions special cased to `-> !` should be changed to `-> Never`.
///
/// # As return argument
/// When feature `unstable` is not enabled, `into_unreachable()` may be required to be used when dealing with return bottom types other than the special case `-> !`.
/// This is a current limitation of the type system.
#[cfg(not(feature="unstable"))]
pubtypeNever=Infallible;
/// Contractually ensures this type cannot exist (i.e. it is a bottom type.)
///
/// # Safety
/// Instances of the impl type **cannot exist**.
/// They must be bottom types (i.e. empty enums, types contatining an `Infallible` / `!` object, etc.)
///
/// # Auto-impl
/// This trait is not intended to be implemented on any user-defined type other than empty enums.
///
/// By default it is implemented for the following types:
/// - `core::convert::Infallible`
/// - `!` (**feature**: `unstable`)
/// - `Box<T>` *where* `T: ?Sized + Unreachable`
pubunsafetraitUnreachable{
/// Force control flow to terminate type checking here.
///
/// # Note
/// This function will never be executed, it is used to terminate the value's existence in the type system, by converting it from any `Unreachable` type into the bottom return type `!`.
/// If this function ever **can** be called at all, it is undefined behaviour.
#[inline]
#[cold]
fninto_unreachable(self)-> !
whereSelf: Sized
{
ifcfg!(debug_assertions){
unreachable!("Unreachable conversion from {}!",std::any::type_name::<Self>())
}else{
// SAFETY: Contractually enforced by the trait impl itself.
}).expect("Failed to spawn forward-searcher thread"))
}else{
None
};
//NOTE: There is no need to spawn another thread for the 2nd operation, since they are both join()'d at the end regardless and both already communicate completion.
letbackward=ifhb.len()>0{
letcap=cap;
letsf=&self;
letcomplete=&complete;
// Main thread: Backwards search.
move||-> Option<_>{
letmutcap=std::cmp::min(cap,hb.len());
letlen=hb.len();
// Check completion before starting loop too.
ifcomplete.load(){
returnNone;
}else{
// Allow previous thread to run if it is not.
std::thread::yield_now();
}
whilecap<=len{
// If `cap` is larger than the buffer `hb`, truncate it.
todo!("Perform one single buffer partition partition (buffer/2.at_nearest_mpr(needle)) (using `method.search_combined()`) and return its parts. If we can fast-path skip the `search_combined()` then that is okay (e.g. if the buffer/2 is small enough that we should just use `SearchSeq`, we can use `SearchSeq` instead of `S`, and so on.) (XXX: Also see below about thread spawning on parallelised partitions and re-using thread pools (we may be able to do this manually with crossbeam, or we might just have to embrace using `spawn_blocking()` async/a default tokio multithreaded-runtime) since parallel partitions needs at least two threads to search both directions at a time.)")
}
//XXX: Should we add a `SearchAsync`? Or an impl for SearchPar that uses already-spawned threads? TODO: It would be best if we could re-use extant threads instead of spawning two on each partition...
//Parallel (feature="threads") byte area partition-on-nearest-newline-to-halve impl, and non-parallel (default) impl. These impls can differ in their desired depths of partitioning (using parallel impls should balance num of partitions to num of logical cpus & input(/desired chunk) size.)
//TODO: Add tests for `Search{Seq,Par}` partitioning methods.
#[cfg(test)]
modtest
{
usesuper::*;
usestd::hint::black_box;
//TODO: Add a generic randomised lorem-ipsum-like text data generator & a generic assertion tester that can take a unique `MidpointFBSearcher`.
//TODO: Thread-reusing parallel `MidpointFBSearcher` (SearchSeq is thread-*spawning*; heavy.) This may require we use async and tasks. If it does, we should also create a `SearchAsync` partitioner (XXX: MidpointFBSearcher is currently a synchonous-only interface; a pure-async pivot finder may require a refactor.)
Avril