Fortune for shuffle3's current commit: Future small blessing − 末小吉
|2 weeks ago|
|include||2 weeks ago|
|old||2 years ago|
|profiling||2 years ago|
|src||2 weeks ago|
|.gitignore||2 years ago|
|LICENSE||2 years ago|
|Makefile||3 weeks ago|
|README.md||2 years ago|
|TODO.org||2 years ago|
|test.sh||3 weeks ago|
shuffle3-lean - Improved 3 stage byte shuffler
Deterministically and reversably shuffle a file's bytes around.
Shuffle a file in place
$ shuffle3 -s file
Unshuffle a file in place
$ shuffle3 -u file
-h for more options.
- ~70-80x speedup from shuffle3 v1.0
- Huge reduction in syscalls
- Takes advantage of the kernel's fs cache
- Can properly handle large files without core dumping
- Doesn't dump huge amounts of trash onto each stack frame
hyperfine reports a 700-800% speedup over
It's easy to see why.
The memusage graph for
v1 shows extremely inefficient stack usage.
( the green is supposed to be a line, not a bar! )
This is due to how the unshuffler buffers RNG results.
v1 naively used VLAs to store this buffer, which can baloon to 8 times the size of the file being unshuffled.
It dumps this massive buffer onto the stack frame of a function that is called multiple times, causing massive and inefficient stack usage.
This can cause a segfault when attempting to unshuffle a large file, while shuffling a file of the same size might succeed.
v2 instead allocates this buffer on the heap. Note the stable stack and heap usage.
make to build the normal binary. It will output to
release (default) target uses the variables
RELEASE_LDFLAGS to specify opitimisations, as well as the
OPT_FLAGS variable. These can be set by you if you wish.
OPT_FLAGS contains the flag
-march=native. This may be underisable for you, in which case set the variable or modify the makefile to remove it.
To build with debug information, run
make debug. Extra debug flags can be provided with the
DEBUG_LDFLAGS variables which have default values in the Makefile.
The build and unstripped binary will be
To build with Profile Guided Optimisation run
make pgo, the stripped and optimised binary will be output to
Before switching between
debug targets, remember to run
To disable stripping of release build binaries, run with
make STRIP=: release
There are some build-time flags you can switch while building by appending to the
||Pretend we're building a debug release even though we're not.|
||Spill buffers into a file if they grow over a threshold. Can cause massive slowdowns but prevent OOMs while unshuffling on systems with low available memory. See shuffle3.h for more details|
||Same as above except allocates memory dynamically. Might be faster.|
||Same as above except it calls
GPL'd with <3