* shuffle3-lean
  Redegisn/upgrade of =shuffle3=

* Goals
  - [X] Functioning in-place shuffle/unshuffle
    - [X] Shuffle
    - [X] Unshuffle
  - [X] Usable in-place s/us from command line
    - [X] Shuffle
    - [X] Unshuffle
  - [ ] Functioning out-of-place/in-memory shuffle/unshuffle
    - [ ] Shuffle
    - [ ] Unshuffle
  - [ ] Usable out-of-place s/us from command line
    - [ ] Shuffle
    - [ ] Unshuffle

** NO compatibility with =shuffle3=
   =shuffle3='s ~drng~ PRNG algorithm uses an outdated global state backend. We don't want to reuse this.
   As a result, output from =shuffle3= and =shuffle3-lean= is different.

* Improvements
  - *~70-80x* speedup from shuffle3 1.0
  - Huge reduction in syscalls
  - Takes advantage of the kernel's fs cache
  - Can properly handle large files without core dumping
  - Doesn't dump huge amounts of trash onto each stack frame

** Performance
   [[https://github.com/sharkdp/hyperfine][hyperfine]]  reports a *700-800%* speedup over =v1=.

   It's easy to see why.
*** V1 flamegraph
    V1 uses a pesudo-array adaptor to perform filesystem reads, seeks, and writes. This causes a massive syscall overhead.
    [[./profiling/release-flame-old.svg]]
*** V2 flamegraph
    Whereas V2 uses a single ~mmap()~.
    [[./profiling/release-flame.svg]]
    
** Memory usage
   The [[https://www.systutorials.com/docs/linux/man/1-memusage/][memusage]] graph for =v1= shows extremely inefficient stack usage.
   [[./profiling/old-mem.png]]
   ( the green is supposed to be a line, not a bar)
   This is due to how the unshuffler buffers RNG results.

   =v1= naively used VLAs to store this buffer, which can baloon to 8 times the size of the file being unshuffled.
   It dumps this massive buffer onto the stack frame of a function that is called multiple times, causing massive and inefficient stack usage.

   This can cause a segfault when attempting to unshuffle a large file, while shuffling a file of the same size might succeed.

*** V2 improvement
    The ~memusage~ graph for =v2= is a lot more sane.
    [[./profiling/mem.png]]
    
    ~v2~ instead allocates this buffer on the heap. Note the stable stack and heap usage.
* Todo
  - [X] impl rng
  - [X] impl shuffling
  - [ ] impl out-of-place shuffling
  - [-] arg parsing and dispatch
    - [X] simple parsing
    - [ ] complex parsing