Compare commits
73 Commits
@ -1,3 +1,5 @@
|
|||||||
*~
|
*~
|
||||||
obj
|
obj
|
||||||
build
|
build/
|
||||||
|
test/
|
||||||
|
shuffle3-*
|
||||||
|
@ -0,0 +1,674 @@
|
|||||||
|
GNU GENERAL PUBLIC LICENSE
|
||||||
|
Version 3, 29 June 2007
|
||||||
|
|
||||||
|
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||||
|
Everyone is permitted to copy and distribute verbatim copies
|
||||||
|
of this license document, but changing it is not allowed.
|
||||||
|
|
||||||
|
Preamble
|
||||||
|
|
||||||
|
The GNU General Public License is a free, copyleft license for
|
||||||
|
software and other kinds of works.
|
||||||
|
|
||||||
|
The licenses for most software and other practical works are designed
|
||||||
|
to take away your freedom to share and change the works. By contrast,
|
||||||
|
the GNU General Public License is intended to guarantee your freedom to
|
||||||
|
share and change all versions of a program--to make sure it remains free
|
||||||
|
software for all its users. We, the Free Software Foundation, use the
|
||||||
|
GNU General Public License for most of our software; it applies also to
|
||||||
|
any other work released this way by its authors. You can apply it to
|
||||||
|
your programs, too.
|
||||||
|
|
||||||
|
When we speak of free software, we are referring to freedom, not
|
||||||
|
price. Our General Public Licenses are designed to make sure that you
|
||||||
|
have the freedom to distribute copies of free software (and charge for
|
||||||
|
them if you wish), that you receive source code or can get it if you
|
||||||
|
want it, that you can change the software or use pieces of it in new
|
||||||
|
free programs, and that you know you can do these things.
|
||||||
|
|
||||||
|
To protect your rights, we need to prevent others from denying you
|
||||||
|
these rights or asking you to surrender the rights. Therefore, you have
|
||||||
|
certain responsibilities if you distribute copies of the software, or if
|
||||||
|
you modify it: responsibilities to respect the freedom of others.
|
||||||
|
|
||||||
|
For example, if you distribute copies of such a program, whether
|
||||||
|
gratis or for a fee, you must pass on to the recipients the same
|
||||||
|
freedoms that you received. You must make sure that they, too, receive
|
||||||
|
or can get the source code. And you must show them these terms so they
|
||||||
|
know their rights.
|
||||||
|
|
||||||
|
Developers that use the GNU GPL protect your rights with two steps:
|
||||||
|
(1) assert copyright on the software, and (2) offer you this License
|
||||||
|
giving you legal permission to copy, distribute and/or modify it.
|
||||||
|
|
||||||
|
For the developers' and authors' protection, the GPL clearly explains
|
||||||
|
that there is no warranty for this free software. For both users' and
|
||||||
|
authors' sake, the GPL requires that modified versions be marked as
|
||||||
|
changed, so that their problems will not be attributed erroneously to
|
||||||
|
authors of previous versions.
|
||||||
|
|
||||||
|
Some devices are designed to deny users access to install or run
|
||||||
|
modified versions of the software inside them, although the manufacturer
|
||||||
|
can do so. This is fundamentally incompatible with the aim of
|
||||||
|
protecting users' freedom to change the software. The systematic
|
||||||
|
pattern of such abuse occurs in the area of products for individuals to
|
||||||
|
use, which is precisely where it is most unacceptable. Therefore, we
|
||||||
|
have designed this version of the GPL to prohibit the practice for those
|
||||||
|
products. If such problems arise substantially in other domains, we
|
||||||
|
stand ready to extend this provision to those domains in future versions
|
||||||
|
of the GPL, as needed to protect the freedom of users.
|
||||||
|
|
||||||
|
Finally, every program is threatened constantly by software patents.
|
||||||
|
States should not allow patents to restrict development and use of
|
||||||
|
software on general-purpose computers, but in those that do, we wish to
|
||||||
|
avoid the special danger that patents applied to a free program could
|
||||||
|
make it effectively proprietary. To prevent this, the GPL assures that
|
||||||
|
patents cannot be used to render the program non-free.
|
||||||
|
|
||||||
|
The precise terms and conditions for copying, distribution and
|
||||||
|
modification follow.
|
||||||
|
|
||||||
|
TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
0. Definitions.
|
||||||
|
|
||||||
|
"This License" refers to version 3 of the GNU General Public License.
|
||||||
|
|
||||||
|
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||||
|
works, such as semiconductor masks.
|
||||||
|
|
||||||
|
"The Program" refers to any copyrightable work licensed under this
|
||||||
|
License. Each licensee is addressed as "you". "Licensees" and
|
||||||
|
"recipients" may be individuals or organizations.
|
||||||
|
|
||||||
|
To "modify" a work means to copy from or adapt all or part of the work
|
||||||
|
in a fashion requiring copyright permission, other than the making of an
|
||||||
|
exact copy. The resulting work is called a "modified version" of the
|
||||||
|
earlier work or a work "based on" the earlier work.
|
||||||
|
|
||||||
|
A "covered work" means either the unmodified Program or a work based
|
||||||
|
on the Program.
|
||||||
|
|
||||||
|
To "propagate" a work means to do anything with it that, without
|
||||||
|
permission, would make you directly or secondarily liable for
|
||||||
|
infringement under applicable copyright law, except executing it on a
|
||||||
|
computer or modifying a private copy. Propagation includes copying,
|
||||||
|
distribution (with or without modification), making available to the
|
||||||
|
public, and in some countries other activities as well.
|
||||||
|
|
||||||
|
To "convey" a work means any kind of propagation that enables other
|
||||||
|
parties to make or receive copies. Mere interaction with a user through
|
||||||
|
a computer network, with no transfer of a copy, is not conveying.
|
||||||
|
|
||||||
|
An interactive user interface displays "Appropriate Legal Notices"
|
||||||
|
to the extent that it includes a convenient and prominently visible
|
||||||
|
feature that (1) displays an appropriate copyright notice, and (2)
|
||||||
|
tells the user that there is no warranty for the work (except to the
|
||||||
|
extent that warranties are provided), that licensees may convey the
|
||||||
|
work under this License, and how to view a copy of this License. If
|
||||||
|
the interface presents a list of user commands or options, such as a
|
||||||
|
menu, a prominent item in the list meets this criterion.
|
||||||
|
|
||||||
|
1. Source Code.
|
||||||
|
|
||||||
|
The "source code" for a work means the preferred form of the work
|
||||||
|
for making modifications to it. "Object code" means any non-source
|
||||||
|
form of a work.
|
||||||
|
|
||||||
|
A "Standard Interface" means an interface that either is an official
|
||||||
|
standard defined by a recognized standards body, or, in the case of
|
||||||
|
interfaces specified for a particular programming language, one that
|
||||||
|
is widely used among developers working in that language.
|
||||||
|
|
||||||
|
The "System Libraries" of an executable work include anything, other
|
||||||
|
than the work as a whole, that (a) is included in the normal form of
|
||||||
|
packaging a Major Component, but which is not part of that Major
|
||||||
|
Component, and (b) serves only to enable use of the work with that
|
||||||
|
Major Component, or to implement a Standard Interface for which an
|
||||||
|
implementation is available to the public in source code form. A
|
||||||
|
"Major Component", in this context, means a major essential component
|
||||||
|
(kernel, window system, and so on) of the specific operating system
|
||||||
|
(if any) on which the executable work runs, or a compiler used to
|
||||||
|
produce the work, or an object code interpreter used to run it.
|
||||||
|
|
||||||
|
The "Corresponding Source" for a work in object code form means all
|
||||||
|
the source code needed to generate, install, and (for an executable
|
||||||
|
work) run the object code and to modify the work, including scripts to
|
||||||
|
control those activities. However, it does not include the work's
|
||||||
|
System Libraries, or general-purpose tools or generally available free
|
||||||
|
programs which are used unmodified in performing those activities but
|
||||||
|
which are not part of the work. For example, Corresponding Source
|
||||||
|
includes interface definition files associated with source files for
|
||||||
|
the work, and the source code for shared libraries and dynamically
|
||||||
|
linked subprograms that the work is specifically designed to require,
|
||||||
|
such as by intimate data communication or control flow between those
|
||||||
|
subprograms and other parts of the work.
|
||||||
|
|
||||||
|
The Corresponding Source need not include anything that users
|
||||||
|
can regenerate automatically from other parts of the Corresponding
|
||||||
|
Source.
|
||||||
|
|
||||||
|
The Corresponding Source for a work in source code form is that
|
||||||
|
same work.
|
||||||
|
|
||||||
|
2. Basic Permissions.
|
||||||
|
|
||||||
|
All rights granted under this License are granted for the term of
|
||||||
|
copyright on the Program, and are irrevocable provided the stated
|
||||||
|
conditions are met. This License explicitly affirms your unlimited
|
||||||
|
permission to run the unmodified Program. The output from running a
|
||||||
|
covered work is covered by this License only if the output, given its
|
||||||
|
content, constitutes a covered work. This License acknowledges your
|
||||||
|
rights of fair use or other equivalent, as provided by copyright law.
|
||||||
|
|
||||||
|
You may make, run and propagate covered works that you do not
|
||||||
|
convey, without conditions so long as your license otherwise remains
|
||||||
|
in force. You may convey covered works to others for the sole purpose
|
||||||
|
of having them make modifications exclusively for you, or provide you
|
||||||
|
with facilities for running those works, provided that you comply with
|
||||||
|
the terms of this License in conveying all material for which you do
|
||||||
|
not control copyright. Those thus making or running the covered works
|
||||||
|
for you must do so exclusively on your behalf, under your direction
|
||||||
|
and control, on terms that prohibit them from making any copies of
|
||||||
|
your copyrighted material outside their relationship with you.
|
||||||
|
|
||||||
|
Conveying under any other circumstances is permitted solely under
|
||||||
|
the conditions stated below. Sublicensing is not allowed; section 10
|
||||||
|
makes it unnecessary.
|
||||||
|
|
||||||
|
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||||
|
|
||||||
|
No covered work shall be deemed part of an effective technological
|
||||||
|
measure under any applicable law fulfilling obligations under article
|
||||||
|
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||||
|
similar laws prohibiting or restricting circumvention of such
|
||||||
|
measures.
|
||||||
|
|
||||||
|
When you convey a covered work, you waive any legal power to forbid
|
||||||
|
circumvention of technological measures to the extent such circumvention
|
||||||
|
is effected by exercising rights under this License with respect to
|
||||||
|
the covered work, and you disclaim any intention to limit operation or
|
||||||
|
modification of the work as a means of enforcing, against the work's
|
||||||
|
users, your or third parties' legal rights to forbid circumvention of
|
||||||
|
technological measures.
|
||||||
|
|
||||||
|
4. Conveying Verbatim Copies.
|
||||||
|
|
||||||
|
You may convey verbatim copies of the Program's source code as you
|
||||||
|
receive it, in any medium, provided that you conspicuously and
|
||||||
|
appropriately publish on each copy an appropriate copyright notice;
|
||||||
|
keep intact all notices stating that this License and any
|
||||||
|
non-permissive terms added in accord with section 7 apply to the code;
|
||||||
|
keep intact all notices of the absence of any warranty; and give all
|
||||||
|
recipients a copy of this License along with the Program.
|
||||||
|
|
||||||
|
You may charge any price or no price for each copy that you convey,
|
||||||
|
and you may offer support or warranty protection for a fee.
|
||||||
|
|
||||||
|
5. Conveying Modified Source Versions.
|
||||||
|
|
||||||
|
You may convey a work based on the Program, or the modifications to
|
||||||
|
produce it from the Program, in the form of source code under the
|
||||||
|
terms of section 4, provided that you also meet all of these conditions:
|
||||||
|
|
||||||
|
a) The work must carry prominent notices stating that you modified
|
||||||
|
it, and giving a relevant date.
|
||||||
|
|
||||||
|
b) The work must carry prominent notices stating that it is
|
||||||
|
released under this License and any conditions added under section
|
||||||
|
7. This requirement modifies the requirement in section 4 to
|
||||||
|
"keep intact all notices".
|
||||||
|
|
||||||
|
c) You must license the entire work, as a whole, under this
|
||||||
|
License to anyone who comes into possession of a copy. This
|
||||||
|
License will therefore apply, along with any applicable section 7
|
||||||
|
additional terms, to the whole of the work, and all its parts,
|
||||||
|
regardless of how they are packaged. This License gives no
|
||||||
|
permission to license the work in any other way, but it does not
|
||||||
|
invalidate such permission if you have separately received it.
|
||||||
|
|
||||||
|
d) If the work has interactive user interfaces, each must display
|
||||||
|
Appropriate Legal Notices; however, if the Program has interactive
|
||||||
|
interfaces that do not display Appropriate Legal Notices, your
|
||||||
|
work need not make them do so.
|
||||||
|
|
||||||
|
A compilation of a covered work with other separate and independent
|
||||||
|
works, which are not by their nature extensions of the covered work,
|
||||||
|
and which are not combined with it such as to form a larger program,
|
||||||
|
in or on a volume of a storage or distribution medium, is called an
|
||||||
|
"aggregate" if the compilation and its resulting copyright are not
|
||||||
|
used to limit the access or legal rights of the compilation's users
|
||||||
|
beyond what the individual works permit. Inclusion of a covered work
|
||||||
|
in an aggregate does not cause this License to apply to the other
|
||||||
|
parts of the aggregate.
|
||||||
|
|
||||||
|
6. Conveying Non-Source Forms.
|
||||||
|
|
||||||
|
You may convey a covered work in object code form under the terms
|
||||||
|
of sections 4 and 5, provided that you also convey the
|
||||||
|
machine-readable Corresponding Source under the terms of this License,
|
||||||
|
in one of these ways:
|
||||||
|
|
||||||
|
a) Convey the object code in, or embodied in, a physical product
|
||||||
|
(including a physical distribution medium), accompanied by the
|
||||||
|
Corresponding Source fixed on a durable physical medium
|
||||||
|
customarily used for software interchange.
|
||||||
|
|
||||||
|
b) Convey the object code in, or embodied in, a physical product
|
||||||
|
(including a physical distribution medium), accompanied by a
|
||||||
|
written offer, valid for at least three years and valid for as
|
||||||
|
long as you offer spare parts or customer support for that product
|
||||||
|
model, to give anyone who possesses the object code either (1) a
|
||||||
|
copy of the Corresponding Source for all the software in the
|
||||||
|
product that is covered by this License, on a durable physical
|
||||||
|
medium customarily used for software interchange, for a price no
|
||||||
|
more than your reasonable cost of physically performing this
|
||||||
|
conveying of source, or (2) access to copy the
|
||||||
|
Corresponding Source from a network server at no charge.
|
||||||
|
|
||||||
|
c) Convey individual copies of the object code with a copy of the
|
||||||
|
written offer to provide the Corresponding Source. This
|
||||||
|
alternative is allowed only occasionally and noncommercially, and
|
||||||
|
only if you received the object code with such an offer, in accord
|
||||||
|
with subsection 6b.
|
||||||
|
|
||||||
|
d) Convey the object code by offering access from a designated
|
||||||
|
place (gratis or for a charge), and offer equivalent access to the
|
||||||
|
Corresponding Source in the same way through the same place at no
|
||||||
|
further charge. You need not require recipients to copy the
|
||||||
|
Corresponding Source along with the object code. If the place to
|
||||||
|
copy the object code is a network server, the Corresponding Source
|
||||||
|
may be on a different server (operated by you or a third party)
|
||||||
|
that supports equivalent copying facilities, provided you maintain
|
||||||
|
clear directions next to the object code saying where to find the
|
||||||
|
Corresponding Source. Regardless of what server hosts the
|
||||||
|
Corresponding Source, you remain obligated to ensure that it is
|
||||||
|
available for as long as needed to satisfy these requirements.
|
||||||
|
|
||||||
|
e) Convey the object code using peer-to-peer transmission, provided
|
||||||
|
you inform other peers where the object code and Corresponding
|
||||||
|
Source of the work are being offered to the general public at no
|
||||||
|
charge under subsection 6d.
|
||||||
|
|
||||||
|
A separable portion of the object code, whose source code is excluded
|
||||||
|
from the Corresponding Source as a System Library, need not be
|
||||||
|
included in conveying the object code work.
|
||||||
|
|
||||||
|
A "User Product" is either (1) a "consumer product", which means any
|
||||||
|
tangible personal property which is normally used for personal, family,
|
||||||
|
or household purposes, or (2) anything designed or sold for incorporation
|
||||||
|
into a dwelling. In determining whether a product is a consumer product,
|
||||||
|
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||||
|
product received by a particular user, "normally used" refers to a
|
||||||
|
typical or common use of that class of product, regardless of the status
|
||||||
|
of the particular user or of the way in which the particular user
|
||||||
|
actually uses, or expects or is expected to use, the product. A product
|
||||||
|
is a consumer product regardless of whether the product has substantial
|
||||||
|
commercial, industrial or non-consumer uses, unless such uses represent
|
||||||
|
the only significant mode of use of the product.
|
||||||
|
|
||||||
|
"Installation Information" for a User Product means any methods,
|
||||||
|
procedures, authorization keys, or other information required to install
|
||||||
|
and execute modified versions of a covered work in that User Product from
|
||||||
|
a modified version of its Corresponding Source. The information must
|
||||||
|
suffice to ensure that the continued functioning of the modified object
|
||||||
|
code is in no case prevented or interfered with solely because
|
||||||
|
modification has been made.
|
||||||
|
|
||||||
|
If you convey an object code work under this section in, or with, or
|
||||||
|
specifically for use in, a User Product, and the conveying occurs as
|
||||||
|
part of a transaction in which the right of possession and use of the
|
||||||
|
User Product is transferred to the recipient in perpetuity or for a
|
||||||
|
fixed term (regardless of how the transaction is characterized), the
|
||||||
|
Corresponding Source conveyed under this section must be accompanied
|
||||||
|
by the Installation Information. But this requirement does not apply
|
||||||
|
if neither you nor any third party retains the ability to install
|
||||||
|
modified object code on the User Product (for example, the work has
|
||||||
|
been installed in ROM).
|
||||||
|
|
||||||
|
The requirement to provide Installation Information does not include a
|
||||||
|
requirement to continue to provide support service, warranty, or updates
|
||||||
|
for a work that has been modified or installed by the recipient, or for
|
||||||
|
the User Product in which it has been modified or installed. Access to a
|
||||||
|
network may be denied when the modification itself materially and
|
||||||
|
adversely affects the operation of the network or violates the rules and
|
||||||
|
protocols for communication across the network.
|
||||||
|
|
||||||
|
Corresponding Source conveyed, and Installation Information provided,
|
||||||
|
in accord with this section must be in a format that is publicly
|
||||||
|
documented (and with an implementation available to the public in
|
||||||
|
source code form), and must require no special password or key for
|
||||||
|
unpacking, reading or copying.
|
||||||
|
|
||||||
|
7. Additional Terms.
|
||||||
|
|
||||||
|
"Additional permissions" are terms that supplement the terms of this
|
||||||
|
License by making exceptions from one or more of its conditions.
|
||||||
|
Additional permissions that are applicable to the entire Program shall
|
||||||
|
be treated as though they were included in this License, to the extent
|
||||||
|
that they are valid under applicable law. If additional permissions
|
||||||
|
apply only to part of the Program, that part may be used separately
|
||||||
|
under those permissions, but the entire Program remains governed by
|
||||||
|
this License without regard to the additional permissions.
|
||||||
|
|
||||||
|
When you convey a copy of a covered work, you may at your option
|
||||||
|
remove any additional permissions from that copy, or from any part of
|
||||||
|
it. (Additional permissions may be written to require their own
|
||||||
|
removal in certain cases when you modify the work.) You may place
|
||||||
|
additional permissions on material, added by you to a covered work,
|
||||||
|
for which you have or can give appropriate copyright permission.
|
||||||
|
|
||||||
|
Notwithstanding any other provision of this License, for material you
|
||||||
|
add to a covered work, you may (if authorized by the copyright holders of
|
||||||
|
that material) supplement the terms of this License with terms:
|
||||||
|
|
||||||
|
a) Disclaiming warranty or limiting liability differently from the
|
||||||
|
terms of sections 15 and 16 of this License; or
|
||||||
|
|
||||||
|
b) Requiring preservation of specified reasonable legal notices or
|
||||||
|
author attributions in that material or in the Appropriate Legal
|
||||||
|
Notices displayed by works containing it; or
|
||||||
|
|
||||||
|
c) Prohibiting misrepresentation of the origin of that material, or
|
||||||
|
requiring that modified versions of such material be marked in
|
||||||
|
reasonable ways as different from the original version; or
|
||||||
|
|
||||||
|
d) Limiting the use for publicity purposes of names of licensors or
|
||||||
|
authors of the material; or
|
||||||
|
|
||||||
|
e) Declining to grant rights under trademark law for use of some
|
||||||
|
trade names, trademarks, or service marks; or
|
||||||
|
|
||||||
|
f) Requiring indemnification of licensors and authors of that
|
||||||
|
material by anyone who conveys the material (or modified versions of
|
||||||
|
it) with contractual assumptions of liability to the recipient, for
|
||||||
|
any liability that these contractual assumptions directly impose on
|
||||||
|
those licensors and authors.
|
||||||
|
|
||||||
|
All other non-permissive additional terms are considered "further
|
||||||
|
restrictions" within the meaning of section 10. If the Program as you
|
||||||
|
received it, or any part of it, contains a notice stating that it is
|
||||||
|
governed by this License along with a term that is a further
|
||||||
|
restriction, you may remove that term. If a license document contains
|
||||||
|
a further restriction but permits relicensing or conveying under this
|
||||||
|
License, you may add to a covered work material governed by the terms
|
||||||
|
of that license document, provided that the further restriction does
|
||||||
|
not survive such relicensing or conveying.
|
||||||
|
|
||||||
|
If you add terms to a covered work in accord with this section, you
|
||||||
|
must place, in the relevant source files, a statement of the
|
||||||
|
additional terms that apply to those files, or a notice indicating
|
||||||
|
where to find the applicable terms.
|
||||||
|
|
||||||
|
Additional terms, permissive or non-permissive, may be stated in the
|
||||||
|
form of a separately written license, or stated as exceptions;
|
||||||
|
the above requirements apply either way.
|
||||||
|
|
||||||
|
8. Termination.
|
||||||
|
|
||||||
|
You may not propagate or modify a covered work except as expressly
|
||||||
|
provided under this License. Any attempt otherwise to propagate or
|
||||||
|
modify it is void, and will automatically terminate your rights under
|
||||||
|
this License (including any patent licenses granted under the third
|
||||||
|
paragraph of section 11).
|
||||||
|
|
||||||
|
However, if you cease all violation of this License, then your
|
||||||
|
license from a particular copyright holder is reinstated (a)
|
||||||
|
provisionally, unless and until the copyright holder explicitly and
|
||||||
|
finally terminates your license, and (b) permanently, if the copyright
|
||||||
|
holder fails to notify you of the violation by some reasonable means
|
||||||
|
prior to 60 days after the cessation.
|
||||||
|
|
||||||
|
Moreover, your license from a particular copyright holder is
|
||||||
|
reinstated permanently if the copyright holder notifies you of the
|
||||||
|
violation by some reasonable means, this is the first time you have
|
||||||
|
received notice of violation of this License (for any work) from that
|
||||||
|
copyright holder, and you cure the violation prior to 30 days after
|
||||||
|
your receipt of the notice.
|
||||||
|
|
||||||
|
Termination of your rights under this section does not terminate the
|
||||||
|
licenses of parties who have received copies or rights from you under
|
||||||
|
this License. If your rights have been terminated and not permanently
|
||||||
|
reinstated, you do not qualify to receive new licenses for the same
|
||||||
|
material under section 10.
|
||||||
|
|
||||||
|
9. Acceptance Not Required for Having Copies.
|
||||||
|
|
||||||
|
You are not required to accept this License in order to receive or
|
||||||
|
run a copy of the Program. Ancillary propagation of a covered work
|
||||||
|
occurring solely as a consequence of using peer-to-peer transmission
|
||||||
|
to receive a copy likewise does not require acceptance. However,
|
||||||
|
nothing other than this License grants you permission to propagate or
|
||||||
|
modify any covered work. These actions infringe copyright if you do
|
||||||
|
not accept this License. Therefore, by modifying or propagating a
|
||||||
|
covered work, you indicate your acceptance of this License to do so.
|
||||||
|
|
||||||
|
10. Automatic Licensing of Downstream Recipients.
|
||||||
|
|
||||||
|
Each time you convey a covered work, the recipient automatically
|
||||||
|
receives a license from the original licensors, to run, modify and
|
||||||
|
propagate that work, subject to this License. You are not responsible
|
||||||
|
for enforcing compliance by third parties with this License.
|
||||||
|
|
||||||
|
An "entity transaction" is a transaction transferring control of an
|
||||||
|
organization, or substantially all assets of one, or subdividing an
|
||||||
|
organization, or merging organizations. If propagation of a covered
|
||||||
|
work results from an entity transaction, each party to that
|
||||||
|
transaction who receives a copy of the work also receives whatever
|
||||||
|
licenses to the work the party's predecessor in interest had or could
|
||||||
|
give under the previous paragraph, plus a right to possession of the
|
||||||
|
Corresponding Source of the work from the predecessor in interest, if
|
||||||
|
the predecessor has it or can get it with reasonable efforts.
|
||||||
|
|
||||||
|
You may not impose any further restrictions on the exercise of the
|
||||||
|
rights granted or affirmed under this License. For example, you may
|
||||||
|
not impose a license fee, royalty, or other charge for exercise of
|
||||||
|
rights granted under this License, and you may not initiate litigation
|
||||||
|
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||||
|
any patent claim is infringed by making, using, selling, offering for
|
||||||
|
sale, or importing the Program or any portion of it.
|
||||||
|
|
||||||
|
11. Patents.
|
||||||
|
|
||||||
|
A "contributor" is a copyright holder who authorizes use under this
|
||||||
|
License of the Program or a work on which the Program is based. The
|
||||||
|
work thus licensed is called the contributor's "contributor version".
|
||||||
|
|
||||||
|
A contributor's "essential patent claims" are all patent claims
|
||||||
|
owned or controlled by the contributor, whether already acquired or
|
||||||
|
hereafter acquired, that would be infringed by some manner, permitted
|
||||||
|
by this License, of making, using, or selling its contributor version,
|
||||||
|
but do not include claims that would be infringed only as a
|
||||||
|
consequence of further modification of the contributor version. For
|
||||||
|
purposes of this definition, "control" includes the right to grant
|
||||||
|
patent sublicenses in a manner consistent with the requirements of
|
||||||
|
this License.
|
||||||
|
|
||||||
|
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||||
|
patent license under the contributor's essential patent claims, to
|
||||||
|
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||||
|
propagate the contents of its contributor version.
|
||||||
|
|
||||||
|
In the following three paragraphs, a "patent license" is any express
|
||||||
|
agreement or commitment, however denominated, not to enforce a patent
|
||||||
|
(such as an express permission to practice a patent or covenant not to
|
||||||
|
sue for patent infringement). To "grant" such a patent license to a
|
||||||
|
party means to make such an agreement or commitment not to enforce a
|
||||||
|
patent against the party.
|
||||||
|
|
||||||
|
If you convey a covered work, knowingly relying on a patent license,
|
||||||
|
and the Corresponding Source of the work is not available for anyone
|
||||||
|
to copy, free of charge and under the terms of this License, through a
|
||||||
|
publicly available network server or other readily accessible means,
|
||||||
|
then you must either (1) cause the Corresponding Source to be so
|
||||||
|
available, or (2) arrange to deprive yourself of the benefit of the
|
||||||
|
patent license for this particular work, or (3) arrange, in a manner
|
||||||
|
consistent with the requirements of this License, to extend the patent
|
||||||
|
license to downstream recipients. "Knowingly relying" means you have
|
||||||
|
actual knowledge that, but for the patent license, your conveying the
|
||||||
|
covered work in a country, or your recipient's use of the covered work
|
||||||
|
in a country, would infringe one or more identifiable patents in that
|
||||||
|
country that you have reason to believe are valid.
|
||||||
|
|
||||||
|
If, pursuant to or in connection with a single transaction or
|
||||||
|
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||||
|
covered work, and grant a patent license to some of the parties
|
||||||
|
receiving the covered work authorizing them to use, propagate, modify
|
||||||
|
or convey a specific copy of the covered work, then the patent license
|
||||||
|
you grant is automatically extended to all recipients of the covered
|
||||||
|
work and works based on it.
|
||||||
|
|
||||||
|
A patent license is "discriminatory" if it does not include within
|
||||||
|
the scope of its coverage, prohibits the exercise of, or is
|
||||||
|
conditioned on the non-exercise of one or more of the rights that are
|
||||||
|
specifically granted under this License. You may not convey a covered
|
||||||
|
work if you are a party to an arrangement with a third party that is
|
||||||
|
in the business of distributing software, under which you make payment
|
||||||
|
to the third party based on the extent of your activity of conveying
|
||||||
|
the work, and under which the third party grants, to any of the
|
||||||
|
parties who would receive the covered work from you, a discriminatory
|
||||||
|
patent license (a) in connection with copies of the covered work
|
||||||
|
conveyed by you (or copies made from those copies), or (b) primarily
|
||||||
|
for and in connection with specific products or compilations that
|
||||||
|
contain the covered work, unless you entered into that arrangement,
|
||||||
|
or that patent license was granted, prior to 28 March 2007.
|
||||||
|
|
||||||
|
Nothing in this License shall be construed as excluding or limiting
|
||||||
|
any implied license or other defenses to infringement that may
|
||||||
|
otherwise be available to you under applicable patent law.
|
||||||
|
|
||||||
|
12. No Surrender of Others' Freedom.
|
||||||
|
|
||||||
|
If conditions are imposed on you (whether by court order, agreement or
|
||||||
|
otherwise) that contradict the conditions of this License, they do not
|
||||||
|
excuse you from the conditions of this License. If you cannot convey a
|
||||||
|
covered work so as to satisfy simultaneously your obligations under this
|
||||||
|
License and any other pertinent obligations, then as a consequence you may
|
||||||
|
not convey it at all. For example, if you agree to terms that obligate you
|
||||||
|
to collect a royalty for further conveying from those to whom you convey
|
||||||
|
the Program, the only way you could satisfy both those terms and this
|
||||||
|
License would be to refrain entirely from conveying the Program.
|
||||||
|
|
||||||
|
13. Use with the GNU Affero General Public License.
|
||||||
|
|
||||||
|
Notwithstanding any other provision of this License, you have
|
||||||
|
permission to link or combine any covered work with a work licensed
|
||||||
|
under version 3 of the GNU Affero General Public License into a single
|
||||||
|
combined work, and to convey the resulting work. The terms of this
|
||||||
|
License will continue to apply to the part which is the covered work,
|
||||||
|
but the special requirements of the GNU Affero General Public License,
|
||||||
|
section 13, concerning interaction through a network will apply to the
|
||||||
|
combination as such.
|
||||||
|
|
||||||
|
14. Revised Versions of this License.
|
||||||
|
|
||||||
|
The Free Software Foundation may publish revised and/or new versions of
|
||||||
|
the GNU General Public License from time to time. Such new versions will
|
||||||
|
be similar in spirit to the present version, but may differ in detail to
|
||||||
|
address new problems or concerns.
|
||||||
|
|
||||||
|
Each version is given a distinguishing version number. If the
|
||||||
|
Program specifies that a certain numbered version of the GNU General
|
||||||
|
Public License "or any later version" applies to it, you have the
|
||||||
|
option of following the terms and conditions either of that numbered
|
||||||
|
version or of any later version published by the Free Software
|
||||||
|
Foundation. If the Program does not specify a version number of the
|
||||||
|
GNU General Public License, you may choose any version ever published
|
||||||
|
by the Free Software Foundation.
|
||||||
|
|
||||||
|
If the Program specifies that a proxy can decide which future
|
||||||
|
versions of the GNU General Public License can be used, that proxy's
|
||||||
|
public statement of acceptance of a version permanently authorizes you
|
||||||
|
to choose that version for the Program.
|
||||||
|
|
||||||
|
Later license versions may give you additional or different
|
||||||
|
permissions. However, no additional obligations are imposed on any
|
||||||
|
author or copyright holder as a result of your choosing to follow a
|
||||||
|
later version.
|
||||||
|
|
||||||
|
15. Disclaimer of Warranty.
|
||||||
|
|
||||||
|
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||||
|
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||||
|
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||||
|
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||||
|
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||||
|
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||||
|
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||||
|
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||||
|
|
||||||
|
16. Limitation of Liability.
|
||||||
|
|
||||||
|
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||||
|
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||||
|
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||||
|
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||||
|
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||||
|
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||||
|
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||||
|
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||||
|
SUCH DAMAGES.
|
||||||
|
|
||||||
|
17. Interpretation of Sections 15 and 16.
|
||||||
|
|
||||||
|
If the disclaimer of warranty and limitation of liability provided
|
||||||
|
above cannot be given local legal effect according to their terms,
|
||||||
|
reviewing courts shall apply local law that most closely approximates
|
||||||
|
an absolute waiver of all civil liability in connection with the
|
||||||
|
Program, unless a warranty or assumption of liability accompanies a
|
||||||
|
copy of the Program in return for a fee.
|
||||||
|
|
||||||
|
END OF TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
How to Apply These Terms to Your New Programs
|
||||||
|
|
||||||
|
If you develop a new program, and you want it to be of the greatest
|
||||||
|
possible use to the public, the best way to achieve this is to make it
|
||||||
|
free software which everyone can redistribute and change under these terms.
|
||||||
|
|
||||||
|
To do so, attach the following notices to the program. It is safest
|
||||||
|
to attach them to the start of each source file to most effectively
|
||||||
|
state the exclusion of warranty; and each file should have at least
|
||||||
|
the "copyright" line and a pointer to where the full notice is found.
|
||||||
|
|
||||||
|
<one line to give the program's name and a brief idea of what it does.>
|
||||||
|
Copyright (C) <year> <name of author>
|
||||||
|
|
||||||
|
This program is free software: you can redistribute it and/or modify
|
||||||
|
it under the terms of the GNU General Public License as published by
|
||||||
|
the Free Software Foundation, either version 3 of the License, or
|
||||||
|
(at your option) any later version.
|
||||||
|
|
||||||
|
This program is distributed in the hope that it will be useful,
|
||||||
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
GNU General Public License for more details.
|
||||||
|
|
||||||
|
You should have received a copy of the GNU General Public License
|
||||||
|
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
Also add information on how to contact you by electronic and paper mail.
|
||||||
|
|
||||||
|
If the program does terminal interaction, make it output a short
|
||||||
|
notice like this when it starts in an interactive mode:
|
||||||
|
|
||||||
|
<program> Copyright (C) <year> <name of author>
|
||||||
|
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
|
||||||
|
This is free software, and you are welcome to redistribute it
|
||||||
|
under certain conditions; type `show c' for details.
|
||||||
|
|
||||||
|
The hypothetical commands `show w' and `show c' should show the appropriate
|
||||||
|
parts of the General Public License. Of course, your program's commands
|
||||||
|
might be different; for a GUI interface, you would use an "about box".
|
||||||
|
|
||||||
|
You should also get your employer (if you work as a programmer) or school,
|
||||||
|
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||||
|
For more information on this, and how to apply and follow the GNU GPL, see
|
||||||
|
<https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
The GNU General Public License does not permit incorporating your program
|
||||||
|
into proprietary programs. If your program is a subroutine library, you
|
||||||
|
may consider it more useful to permit linking proprietary applications with
|
||||||
|
the library. If this is what you want to do, use the GNU Lesser General
|
||||||
|
Public License instead of this License. But first, please read
|
||||||
|
<https://www.gnu.org/licenses/why-not-lgpl.html>.
|
@ -1,45 +1,177 @@
|
|||||||
SRC = $(wildcard src/*.c)
|
SRC_C = $(wildcard src/*.c)
|
||||||
INCLUDE = include/
|
SRC_CXX = $(wildcard src/*.cpp)
|
||||||
|
SRC_CXX+= $(wildcard src/rng/*.cpp)
|
||||||
|
|
||||||
|
INCLUDE = include
|
||||||
|
|
||||||
PROJECT=shuffle3
|
PROJECT=shuffle3
|
||||||
|
|
||||||
BUILD=build
|
# Link to these libraries dynamicalls
|
||||||
|
SHARED_LIBS=fmt
|
||||||
|
# Link to these libraries statically
|
||||||
|
STATIC_LIBS=
|
||||||
|
|
||||||
|
# Currently supported:
|
||||||
|
# _FS_SPILL_BUFFER: Use file backed buffer instead of memory backed one for unshuffling. See `shuffle3.h`.
|
||||||
|
# Setting the value to `DYN` enables the dynamic buffer, setting it to `MAP` enabled memory-mapped buffer. `MAP` is usually the fastest of the 3 modes.
|
||||||
|
# DEBUG: Pretend we're building a debug release even though we're not. Will enable additional output messages and may interfere with some optimisations
|
||||||
|
FEATURE_FLAGS?=
|
||||||
|
|
||||||
|
COMMON_FLAGS+= $(addprefix -D,$(FEATURE_FLAGS)) -Wall -Wstrict-aliasing $(addprefix -I,$(INCLUDE)) -fno-strict-aliasing
|
||||||
|
|
||||||
|
ARCH?=native
|
||||||
|
|
||||||
|
INTRINSICS=avx sse3
|
||||||
|
|
||||||
|
OPT_FLAGS?=-fgraphite -fopenmp -floop-parallelize-all -ftree-parallelize-loops=4 \
|
||||||
|
-floop-interchange -ftree-loop-distribution -floop-strip-mine -floop-block \
|
||||||
|
-fno-stack-check
|
||||||
|
|
||||||
|
OPT_FLAGS+=$(addprefix -m,$(INTRINSICS))
|
||||||
|
|
||||||
|
ifneq ($(ARCH),)
|
||||||
|
OPT_FLAGS+=$(addprefix -march=,$(ARCH))
|
||||||
|
endif
|
||||||
|
|
||||||
|
|
||||||
|
CXX_OPT_FLAGS?= $(OPT_FLAGS) -felide-constructors
|
||||||
|
|
||||||
|
CFLAGS += $(COMMON_FLAGS) --std=gnu11
|
||||||
|
CXXFLAGS += $(COMMON_FLAGS) --std=gnu++23 -fno-exceptions
|
||||||
|
# XXX: We might need exceptions soon, for OOP usage, because we try multiple approaches from most efficient to least.
|
||||||
|
LDFLAGS += $(addsuffix .a,$(addprefix -l:lib,$(STATIC_LIBS))) $(addprefix -l,$(SHARED_LIBS))
|
||||||
|
|
||||||
|
STRIP=strip
|
||||||
|
|
||||||
|
RELEASE_COMMON_FLAGS?= -DNOTRACE
|
||||||
|
RELEASE_COMMON_FLAGS+= -DRELEASE
|
||||||
|
|
||||||
|
RELEASE_CFLAGS?= -O3 -flto $(OPT_FLAGS) $(RELEASE_COMMON_FLAGS)
|
||||||
|
RELEASE_CXXFLAGS?= -O3 -flto $(CXX_OPT_FLAGS) $(RELEASE_COMMON_FLAGS)
|
||||||
|
RELEASE_LDFLAGS?= -Wl,-O3 -Wl,-flto
|
||||||
|
|
||||||
|
DEBUG_CFLAGS?= -Og -g -DDEBUG
|
||||||
|
DEBUG_CXXFLAGS?= $(DEBUG_CFLAGS)
|
||||||
|
DEBUG_LDFLAGS?=
|
||||||
|
|
||||||
|
# Objects
|
||||||
|
|
||||||
|
OBJ_C = $(addprefix obj/c/,$(SRC_C:.c=.o))
|
||||||
|
OBJ_CXX = $(addprefix obj/cxx/,$(SRC_CXX:.cpp=.o))
|
||||||
|
OBJ = $(OBJ_C) $(OBJ_CXX)
|
||||||
|
|
||||||
CFLAGS+= $(addprefix -I,$(INCLUDE)) -Wall -pedantic --std=gnu11
|
# PGO stuff
|
||||||
LDFLAGS+= -lm
|
|
||||||
|
|
||||||
RELEASE_CFLAGS?= -O3 -march=native -flto -fgraphite
|
PGO_OBJ_C = $(addprefix obj/pgo/c/,$(SRC_C:.c=.o))
|
||||||
RELEASE_LDFLAGS?= -O3 -flto
|
PGO_OBJ_CXX = $(addprefix obj/pgo/cxx/,$(SRC_CXX:.cpp=.o))
|
||||||
|
PGO_OBJ = $(PGO_OBJ_C) $(PGO_OBJ_CXX)
|
||||||
|
|
||||||
DEBUG_CFLAGS?= -g -O0
|
PGO_ITERATIONS=5
|
||||||
DEBUG_LDFLAGS?= -O0
|
PGO_BLOCKS={1..4}
|
||||||
|
PGO_SIZE={1024..4096}
|
||||||
|
PGO_SET_LOC?=/tmp/$(PROJECT)-pgo
|
||||||
|
|
||||||
OBJ = $(addprefix obj/,$(SRC:.c=.o))
|
PGO_FLAGS = -fprofile-generate
|
||||||
|
PGO_LDFLAGS=
|
||||||
|
|
||||||
|
# Phonies
|
||||||
|
|
||||||
.PHONY: release
|
.PHONY: release
|
||||||
release: | dirs $(BUILD)/$(PROJECT)-release
|
release: | dirs $(PROJECT)-release
|
||||||
|
|
||||||
.PHONY: debug
|
.PHONY: debug
|
||||||
debug: | dirs $(BUILD)/$(PROJECT)-debug
|
debug: | dirs $(PROJECT)-debug
|
||||||
|
|
||||||
|
.PHONY: pgo
|
||||||
|
pgo: | dirs $(PROJECT)-pgo
|
||||||
|
|
||||||
|
.PHONY: test
|
||||||
|
test: test-all
|
||||||
|
|
||||||
|
# Targets
|
||||||
|
|
||||||
dirs:
|
dirs:
|
||||||
@mkdir -p obj/src
|
@mkdir -p obj/{pgo/,}c{,xx}/src{,/rng}
|
||||||
@mkdir -p $(BUILD)
|
|
||||||
|
|
||||||
obj/%.o: %.c
|
obj/c/%.o: %.c
|
||||||
$(CC) -c $< $(CFLAGS) -o $@ $(LDFLAGS)
|
$(CC) -c $< $(CFLAGS) -o $@ $(LDFLAGS)
|
||||||
|
|
||||||
$(BUILD)/$(PROJECT)-release: CFLAGS+= $(RELEASE_CFLAGS)
|
obj/cxx/%.o: %.cpp
|
||||||
$(BUILD)/$(PROJECT)-release: LDFLAGS+= $(RELEASE_LDFLAGS)
|
$(CXX) -c $< $(CXXFLAGS) -o $@ $(LDFLAGS)
|
||||||
$(BUILD)/$(PROJECT)-release: $(OBJ)
|
|
||||||
$(CC) $^ $(CFLAGS) -o $@ $(LDFLAGS)
|
obj/pgo/c/%.o: %.c
|
||||||
strip $@
|
$(CC) -c $< $(CFLAGS) $(PGO_FLAGS) -o $@ $(LDFLAGS) $(PGO_LDFLAGS)
|
||||||
|
|
||||||
|
obj/pgo/cxx/%.o: %.cpp
|
||||||
|
$(CXX) -c $< $(CXXFLAGS) $(PGO_FLAGS) -o $@ $(LDFLAGS) $(PGO_LDFLAGS)
|
||||||
|
|
||||||
|
|
||||||
|
$(PROJECT)-release: CFLAGS+= $(RELEASE_CFLAGS)
|
||||||
|
$(PROJECT)-release: CXXFLAGS += $(RELEASE_CXXFLAGS)
|
||||||
|
$(PROJECT)-release: LDFLAGS += $(RELEASE_LDFLAGS)
|
||||||
|
$(PROJECT)-release: $(OBJ)
|
||||||
|
$(CXX) $^ $(CXXFLAGS) -o $@ $(LDFLAGS)
|
||||||
|
$(STRIP) $@
|
||||||
|
|
||||||
$(BUILD)/$(PROJECT)-debug: CFLAGS+= $(DEBUG_CFLAGS)
|
$(PROJECT)-debug: CFLAGS+= $(DEBUG_CFLAGS)
|
||||||
$(BUILD)/$(PROJECT)-debug: LDFLAGS+= $(DEBUG_LDFLAGS)
|
$(PROJECT)-debug: CXXFLAGS += $(DEBUG_CXXFLAGS)
|
||||||
$(BUILD)/$(PROJECT)-debug: $(OBJ)
|
$(PROJECT)-debug: LDFLAGS += $(DEBUG_LDFLAGS)
|
||||||
$(CC) $^ $(CFLAGS) -o $@ $(LDFLAGS)
|
$(PROJECT)-debug: $(OBJ)
|
||||||
|
$(CXX) $^ $(CXXFLAGS) -o $@ $(LDFLAGS)
|
||||||
|
|
||||||
clean:
|
pgo-generate: CFLAGS+= $(RELEASE_CFLAGS)
|
||||||
|
pgo-generate: CXXFLAGS += $(RELEASE_CXXFLAGS)
|
||||||
|
pgo-generate: LDFLAGS += $(RELEASE_LDFLAGS)
|
||||||
|
pgo-generate: $(PGO_OBJ)
|
||||||
|
$(CXX) $^ $(CXXFLAGS) $(PGO_FLAGS) -o $@ $(LDFLAGS) $(PGO_LDFLAGS)
|
||||||
|
|
||||||
|
pgo-reset:
|
||||||
|
-find ./obj/pgo -name \*.gcda -exec rm {} +
|
||||||
|
|
||||||
|
pgo-profile: | pgo-reset pgo-generate
|
||||||
|
mkdir -p $(PGO_SET_LOC)
|
||||||
|
for i in {1..$(PGO_ITERATIONS)}; do \
|
||||||
|
block=$$(rng --of $(PGO_SIZE)); \
|
||||||
|
block_count=$$(rng --of $(PGO_BLOCKS)); \
|
||||||
|
dd if=/dev/urandom of=$(PGO_SET_LOC)/small bs=$$block count=$$block_count >> /dev/null 2>&1; \
|
||||||
|
printf "Iteration $$i / $(PGO_ITERATIONS) ($$block * $$block_count)\r"; \
|
||||||
|
( echo ">> $$i" >&2; \
|
||||||
|
echo ">> $$i"; \
|
||||||
|
./pgo-generate -s $(PGO_SET_LOC)/small && \
|
||||||
|
./pgo-generate -u $(PGO_SET_LOC)/small && \
|
||||||
|
./pgo-generate -h && \
|
||||||
|
FCNT=1 ./test.sh ./pgo-generate && \
|
||||||
|
FCNT=2 ./test.sh ./pgo-generate && \
|
||||||
|
FCNT=3 ./test.sh ./pgo-generate && \
|
||||||
|
FCNT=4 ./test.sh ./pgo-generate && \
|
||||||
|
: \
|
||||||
|
) >>$(PGO_SET_LOC)/stdout.log 2>>$(PGO_SET_LOC)/stderr.log || \exit $$?; \
|
||||||
|
done
|
||||||
|
$(shell command -v bat >/dev/null && echo "bat --pager=none" || echo cat) $(PGO_SET_LOC)/stdout.log; \
|
||||||
|
$(shell command -v bat >/dev/null && echo "bat --pager=none" || echo cat) $(PGO_SET_LOC)/stderr.log >&2
|
||||||
|
rm -rf $(PGO_SET_LOC)
|
||||||
|
rm pgo-generate
|
||||||
|
|
||||||
|
pgo-use: PGO_FLAGS = -fprofile-use
|
||||||
|
pgo-use: PGO_LDFLAGS = -lgcov -fprofile-use
|
||||||
|
pgo-use: CFLAGS+= $(RELEASE_CFLAGS)
|
||||||
|
pgo-use: CXXFLAGS += $(RELEASE_CXXFLAGS)
|
||||||
|
pgo-use: LDFLAGS += $(RELEASE_LDFLAGS)
|
||||||
|
pgo-use: $(PGO_OBJ)
|
||||||
|
$(CXX) $^ $(CXXFLAGS) $(PGO_FLAGS) -o $@ $(LDFLAGS) $(PGO_LDFLAGS)
|
||||||
|
|
||||||
|
$(PROJECT)-pgo: | pgo-profile
|
||||||
|
find ./obj/pgo -name \*.o -exec rm {} +
|
||||||
|
$(MAKE) pgo-use
|
||||||
|
mv pgo-use $@
|
||||||
|
$(STRIP) $@
|
||||||
|
|
||||||
|
clean-rebuild:
|
||||||
rm -rf obj
|
rm -rf obj
|
||||||
rm -f $(BUILD)/*
|
|
||||||
|
clean: clean-rebuild
|
||||||
|
rm -f $(PROJECT)-{release,debug,pgo}
|
||||||
|
|
||||||
|
test-all:
|
||||||
|
@./test.sh ./$(PROJECT)-*
|
||||||
|
|
||||||
|
@ -0,0 +1,96 @@
|
|||||||
|
# `shuffle3-lean` - Improved 3 stage byte shuffler
|
||||||
|
|
||||||
|
Deterministically and reversably shuffle a file's bytes around.
|
||||||
|
|
||||||
|
## Shuffling
|
||||||
|
Shuffle a file in place
|
||||||
|
|
||||||
|
``` shell
|
||||||
|
$ shuffle3 -s file
|
||||||
|
```
|
||||||
|
|
||||||
|
## Unshuffling
|
||||||
|
Unshuffle a file in place
|
||||||
|
|
||||||
|
``` shell
|
||||||
|
$ shuffle3 -u file
|
||||||
|
```
|
||||||
|
|
||||||
|
## Other options
|
||||||
|
Run with `-h` for more options.
|
||||||
|
|
||||||
|
# Improvements from `v1`
|
||||||
|
* **~70-80x** speedup from shuffle3 v1.0
|
||||||
|
* Huge reduction in syscalls
|
||||||
|
* Takes advantage of the kernel's fs cache
|
||||||
|
* Can properly handle large files without core dumping
|
||||||
|
* Doesn't dump huge amounts of trash onto each stack frame
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
[hyperfine](https://github.com/sharkdp/hyperfine) reports a **700-800%** speedup over `v1`.
|
||||||
|
It's easy to see why.
|
||||||
|
|
||||||
|
### V1 flamegraph
|
||||||
|
V1 uses a pesudo-array adaptor to perform filesystem reads, seeks, and writes. This causes a massive syscall overhead.
|
||||||
|
![](./profiling/release-flame-old.png)
|
||||||
|
|
||||||
|
### V2 flamegraph
|
||||||
|
Whereas V2 uses a single `mmap()`.
|
||||||
|
![](./profiling/release-flame.png)
|
||||||
|
|
||||||
|
## Memory usage
|
||||||
|
The [memusage](https://www.systutorials.com/docs/linux/man/1-memusage/) graph for `v1` shows extremely inefficient stack usage.
|
||||||
|
![](./profiling/old-mem.png)
|
||||||
|
( the green is supposed to be a line, not a bar! )
|
||||||
|
|
||||||
|
This is due to how the unshuffler buffers RNG results.
|
||||||
|
|
||||||
|
`v1` naively used VLAs to store this buffer, which can baloon to 8 times the size of the file being unshuffled.
|
||||||
|
It dumps this massive buffer onto the stack frame of a function that is called multiple times, causing massive and inefficient stack usage.
|
||||||
|
|
||||||
|
This can cause a segfault when attempting to unshuffle a large file, while shuffling a file of the same size might succeed.
|
||||||
|
|
||||||
|
### V2 improvement
|
||||||
|
The `memusage` graph for `v2` is a lot more sane.
|
||||||
|
![](./profiling/mem.png)
|
||||||
|
|
||||||
|
`v2` instead allocates this buffer on the heap. Note the stable stack and heap usage.
|
||||||
|
|
||||||
|
|
||||||
|
# Building
|
||||||
|
Run `make` to build the normal binary. It will output to `shuffle3-release`.
|
||||||
|
|
||||||
|
## Release target
|
||||||
|
The `release` (default) target uses the variables `RELEASE_CFLAGS`, `RELEASE_CXXFLAGS` and `RELEASE_LDFLAGS` to specify opitimisations, as well as the `OPT_FLAGS` variable. These can be set by you if you wish.
|
||||||
|
|
||||||
|
### Note
|
||||||
|
The default `OPT_FLAGS` contains the flag `-march=native`. This may be underisable for you, in which case set the variable or modify the makefile to remove it.
|
||||||
|
|
||||||
|
## Debug target
|
||||||
|
To build with debug information, run `make debug`. Extra debug flags can be provided with the `DEBUG_CFLAGS`, `DEBUG_CXXFLAGS` and `DEBUG_LDFLAGS` variables which have default values in the Makefile.
|
||||||
|
|
||||||
|
The build and unstripped binary will be `shuffle3-debug`.
|
||||||
|
|
||||||
|
## PGO target
|
||||||
|
To build with Profile Guided Optimisation run `make pgo`, the stripped and optimised binary will be output to `shuffle3-pgo`.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
Before switching between `release` and `debug` targets, remember to run `make clean`.
|
||||||
|
To disable stripping of release build binaries, run with `make STRIP=: release`
|
||||||
|
|
||||||
|
### Compile-time flags
|
||||||
|
There are some build-time flags you can switch while building by appending to the `FEATURE_FLAGS` variable.
|
||||||
|
| Flag | Description |
|
||||||
|
|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||||
|
| `DEBUG` | Pretend we're building a debug release even though we're not. |
|
||||||
|
| `_FS_SPILL_BUFFER` | Spill buffers into a file if they grow over a threshold. Can cause massive slowdowns but prevent OOMs while unshuffling on systems with low available memory. See [shuffle3.h](./include/shuffle3.h) for more details |
|
||||||
|
| `_FS_SPILL_BUFFER=DYN` | Same as above except allocates memory dynamically. Might be faster. |
|
||||||
|
| `_FS_SPILL_BUFFER=MAP` | Same as above except it calls `fallocate()` and `mmap()` to prodive a buffer of the full size needed. Is usually the fastest of the options for `_FS_SPILL_BUFFER` and is preferrable if possible. |
|
||||||
|
|
||||||
|
|
||||||
|
## Gentoo ebuild
|
||||||
|
There is a gentoo ebuild for this project in the overlay [test-overlay](https://git.flanchan.moe/birb/test-overlay).
|
||||||
|
[direct link](https://git.flanchan.moe/birb/test-overlay/src/branch/master/app-misc/shuffle3/shuffle3-2.0.0.ebuild)
|
||||||
|
|
||||||
|
# License
|
||||||
|
GPL'd with <3
|
@ -0,0 +1,62 @@
|
|||||||
|
* shuffle3-lean
|
||||||
|
Redegisn/upgrade of =shuffle3=
|
||||||
|
|
||||||
|
* Goals
|
||||||
|
- [X] Functioning in-place shuffle/unshuffle
|
||||||
|
- [X] Shuffle
|
||||||
|
- [X] Unshuffle
|
||||||
|
- [X] Usable in-place s/us from command line
|
||||||
|
- [X] Shuffle
|
||||||
|
- [X] Unshuffle
|
||||||
|
- [ ] Functioning out-of-place/in-memory shuffle/unshuffle
|
||||||
|
- [ ] Shuffle
|
||||||
|
- [ ] Unshuffle
|
||||||
|
- [ ] Usable out-of-place s/us from command line
|
||||||
|
- [ ] Shuffle
|
||||||
|
- [ ] Unshuffle
|
||||||
|
|
||||||
|
** NO compatibility with =shuffle3=
|
||||||
|
=shuffle3='s ~drng~ PRNG algorithm uses an outdated global state backend. We don't want to reuse this.
|
||||||
|
As a result, output from =shuffle3= and =shuffle3-lean= is different.
|
||||||
|
|
||||||
|
* Improvements
|
||||||
|
- *~70-80x* speedup from shuffle3 1.0
|
||||||
|
- Huge reduction in syscalls
|
||||||
|
- Takes advantage of the kernel's fs cache
|
||||||
|
- Can properly handle large files without core dumping
|
||||||
|
- Doesn't dump huge amounts of trash onto each stack frame
|
||||||
|
|
||||||
|
** Performance
|
||||||
|
[[https://github.com/sharkdp/hyperfine][hyperfine]] reports a *700-800%* speedup over =v1=.
|
||||||
|
|
||||||
|
It's easy to see why.
|
||||||
|
*** V1 flamegraph
|
||||||
|
V1 uses a pesudo-array adaptor to perform filesystem reads, seeks, and writes. This causes a massive syscall overhead.
|
||||||
|
[[./profiling/release-flame-old.svg]]
|
||||||
|
*** V2 flamegraph
|
||||||
|
Whereas V2 uses a single ~mmap()~.
|
||||||
|
[[./profiling/release-flame.svg]]
|
||||||
|
|
||||||
|
** Memory usage
|
||||||
|
The [[https://www.systutorials.com/docs/linux/man/1-memusage/][memusage]] graph for =v1= shows extremely inefficient stack usage.
|
||||||
|
[[./profiling/old-mem.png]]
|
||||||
|
( the green is supposed to be a line, not a bar)
|
||||||
|
This is due to how the unshuffler buffers RNG results.
|
||||||
|
|
||||||
|
=v1= naively used VLAs to store this buffer, which can baloon to 8 times the size of the file being unshuffled.
|
||||||
|
It dumps this massive buffer onto the stack frame of a function that is called multiple times, causing massive and inefficient stack usage.
|
||||||
|
|
||||||
|
This can cause a segfault when attempting to unshuffle a large file, while shuffling a file of the same size might succeed.
|
||||||
|
|
||||||
|
*** V2 improvement
|
||||||
|
The ~memusage~ graph for =v2= is a lot more sane.
|
||||||
|
[[./profiling/mem.png]]
|
||||||
|
|
||||||
|
~v2~ instead allocates this buffer on the heap. Note the stable stack and heap usage.
|
||||||
|
* Todo
|
||||||
|
- [X] impl rng
|
||||||
|
- [X] impl shuffling
|
||||||
|
- [ ] impl out-of-place shuffling
|
||||||
|
- [-] arg parsing and dispatch
|
||||||
|
- [X] simple parsing
|
||||||
|
- [ ] complex parsing
|
@ -0,0 +1,28 @@
|
|||||||
|
// https://stackoverflow.com/a/30304782
|
||||||
|
|
||||||
|
#ifndef _COLORS_
|
||||||
|
#define _COLORS_
|
||||||
|
|
||||||
|
/* FOREGROUND */
|
||||||
|
#define RST "\x1B[0m"
|
||||||
|
#define KRED "\x1B[31m"
|
||||||
|
#define KGRN "\x1B[32m"
|
||||||
|
#define KYEL "\x1B[33m"
|
||||||
|
#define KBLU "\x1B[34m"
|
||||||
|
#define KMAG "\x1B[35m"
|
||||||
|
#define KCYN "\x1B[36m"
|
||||||
|
#define KWHT "\x1B[37m"
|
||||||
|
|
||||||
|
#define FRED(x) KRED x RST
|
||||||
|
#define FGRN(x) KGRN x RST
|
||||||
|
#define FYEL(x) KYEL x RST
|
||||||
|
#define FBLU(x) KBLU x RST
|
||||||
|
#define FMAG(x) KMAG x RST
|
||||||
|
#define FCYN(x) KCYN x RST
|
||||||
|
#define FWHT(x) KWHT x RST
|
||||||
|
|
||||||
|
#define BOLD(x) "\x1B[1m" x RST
|
||||||
|
#define UNDL(x) "\x1B[4m" x RST
|
||||||
|
|
||||||
|
#endif /* _COLORS_ */
|
||||||
|
|
@ -0,0 +1,55 @@
|
|||||||
|
#ifndef _DEBUG_H
|
||||||
|
#define _DEBUG_H
|
||||||
|
|
||||||
|
// If `TRACE` is defined, or if `NOTRACE` is defined, use simplified debug info
|
||||||
|
#if defined(TRACE) || !defined(NOTRACE)
|
||||||
|
#define D_TRACE
|
||||||
|
#define _D__FUNC __PRETTY_FUNCTION__
|
||||||
|
#else
|
||||||
|
#define _D__FUNC __func__
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
extern "C" {
|
||||||
|
#endif
|
||||||
|
|
||||||
|
struct debuginfo {
|
||||||
|
const char* file;
|
||||||
|
const char* function;
|
||||||
|
//const char* pfunction; //TODO: for `#if defined(TRACE)`, __PRETTY_FUNCTION__ goes here, __func__ goes above.
|
||||||
|
int line;
|
||||||
|
};
|
||||||
|
|
||||||
|
void _do_dprintf(struct debuginfo di, const char* fmt, ...);
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
#define _D_COMPTIME constexpr
|
||||||
|
extern "C++" {
|
||||||
|
#include <utility>
|
||||||
|
template<typename... Args>
|
||||||
|
inline void _real_dprintf(const char* file, const char* function, int line, const char* fmt, Args&&... args)
|
||||||
|
{
|
||||||
|
#ifdef DEBUG
|
||||||
|
debuginfo i = { file, function, line };
|
||||||
|
_do_dprintf(i, fmt, std::forward<Args>(args)...);
|
||||||
|
#endif
|
||||||
|
}
|
||||||
|
#define D_dprintf(fmt, ...) do { if consteval { (void)0; } else { _real_dprintf(__FILE__, _D__FUNC, __LINE__, fmt __VA_OPT__(,) __VA_ARGS__); } } while(0)
|
||||||
|
}
|
||||||
|
#else
|
||||||
|
#define _D_COMPTIME
|
||||||
|
|
||||||
|
#ifdef DEBUG
|
||||||
|
#define D_dprintf(fmt, ...) _do_dprintf( (struct debuginfo){.file = __FILE__, .function = _D__FUNC, .line = __LINE__}, fmt __VA_OPT__(,) __VA_ARGS__)
|
||||||
|
#else
|
||||||
|
static _D_COMPTIME inline void _do__nothing(const char* fmt, ...) {}
|
||||||
|
#define D_dprintf(fmt, ...) _do__nothing(fmt __VA_OPT__(,) __VA_ARGS__) //(fmt __VA_OPT__(,) __VA_ARGS__, (void)0)
|
||||||
|
#endif
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
#undef _D_COMPTIME
|
||||||
|
|
||||||
|
#endif /* _DEBUG_H */
|
@ -0,0 +1,36 @@
|
|||||||
|
#ifndef _ERROR_H
|
||||||
|
#define _ERROR_H
|
||||||
|
|
||||||
|
#ifndef $PASTE
|
||||||
|
# define $_PASTE(x,y) x ## y
|
||||||
|
# define $PASTE(x,y) $_PASTE(x,y)
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
extern "C" {
|
||||||
|
#endif
|
||||||
|
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
}
|
||||||
|
|
||||||
|
constexpr inline bool is_noexcept=
|
||||||
|
#if __cpp_exceptions
|
||||||
|
#define EXCEPT 1
|
||||||
|
//#define try try
|
||||||
|
//#define catch(...) catch(__VA_ARGS__)
|
||||||
|
false
|
||||||
|
#else
|
||||||
|
#define EXCEPT 0
|
||||||
|
#define NOEXCEPT
|
||||||
|
//#define catch(...) __try {} catch(__VA_ARGS__)
|
||||||
|
//#define try if constexpr(!is_noexcept)
|
||||||
|
//#define throw (void)0
|
||||||
|
true
|
||||||
|
#endif
|
||||||
|
;
|
||||||
|
|
||||||
|
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#endif /* _ERROR_H */
|
@ -0,0 +1,31 @@
|
|||||||
|
#ifndef _FSVEC_H
|
||||||
|
#define _FSVEC_H
|
||||||
|
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
extern "C" {
|
||||||
|
#define restrict __restrict__
|
||||||
|
#endif
|
||||||
|
|
||||||
|
// A simple file-backed back inserter
|
||||||
|
typedef struct {
|
||||||
|
FILE* backing;
|
||||||
|
size_t len;
|
||||||
|
} fvec_t;
|
||||||
|
|
||||||
|
int fvec_new(fvec_t* restrict obj, const char* path);
|
||||||
|
void fvec_close(fvec_t* restrict obj);
|
||||||
|
|
||||||
|
int fvec_pop_end(fvec_t* restrict obj, size_t sz);
|
||||||
|
void fvec_push_whole_buffer(fvec_t* restrict obj, const void* _buffer, size_t sz);
|
||||||
|
int fvec_get_whole_buffer(const fvec_t* restrict obj, void* _buffer, size_t _sz);
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
#undef restrict
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#endif /* _FSVEC_H */
|
@ -0,0 +1,324 @@
|
|||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include <cstdint>
|
||||||
|
#include <cstddef>
|
||||||
|
#include <memory>
|
||||||
|
#include <utility>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
#include <map.h>
|
||||||
|
#include <tempfile.hpp>
|
||||||
|
#include <panic.h>
|
||||||
|
#include <debug.h>
|
||||||
|
|
||||||
|
|
||||||
|
#include <shuffle3.h>
|
||||||
|
|
||||||
|
template<typename T>
|
||||||
|
struct i_back_inserter
|
||||||
|
{
|
||||||
|
virtual void push_back(T&& value) =0;
|
||||||
|
virtual void pop_back() =0;
|
||||||
|
virtual const T& back() const =0;
|
||||||
|
virtual T& back() =0;
|
||||||
|
virtual const std::size_t size() const =0;
|
||||||
|
|
||||||
|
inline bool is_empty() const { return size()==0; }
|
||||||
|
|
||||||
|
virtual inline ~i_back_inserter() =default;
|
||||||
|
};
|
||||||
|
|
||||||
|
struct i_shunt
|
||||||
|
{
|
||||||
|
virtual bool is_full() const =0;
|
||||||
|
virtual inline ~i_shunt() =default;
|
||||||
|
};
|
||||||
|
|
||||||
|
struct file_back_buffer
|
||||||
|
{
|
||||||
|
const static constexpr std::size_t DEFAULT_CAP = 1024;
|
||||||
|
typedef std::uint8_t byte;
|
||||||
|
|
||||||
|
file_back_buffer();
|
||||||
|
file_back_buffer(std::size_t cap);
|
||||||
|
file_back_buffer(const file_back_buffer& c) = delete;
|
||||||
|
file_back_buffer(file_back_buffer&& m);
|
||||||
|
|
||||||
|
void push_buf(byte* buf, std::size_t len);
|
||||||
|
bool back(byte* buf, std::size_t len) const;
|
||||||
|
bool pop_n(std::size_t len);
|
||||||
|
|
||||||
|
~file_back_buffer();
|
||||||
|
private:
|
||||||
|
struct impl;
|
||||||
|
std::unique_ptr<impl> inner;
|
||||||
|
};
|
||||||
|
|
||||||
|
namespace {
|
||||||
|
template<typename T>
|
||||||
|
inline const T* _die_if_null(const T* input, const char* msg)
|
||||||
|
{
|
||||||
|
if(!input) panic(msg);
|
||||||
|
return input;
|
||||||
|
}
|
||||||
|
template<typename T>
|
||||||
|
inline T* _die_if_null(T* input, const char* msg)
|
||||||
|
{
|
||||||
|
if(!input) panic(msg);
|
||||||
|
return input;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename T>
|
||||||
|
struct file_vector : public i_back_inserter<T>
|
||||||
|
{
|
||||||
|
inline file_vector() : file_vector(file_back_buffer::DEFAULT_CAP){}
|
||||||
|
inline file_vector(std::size_t cap) : inserter(file_back_buffer(cap)), len(0), current_back(std::vector<unsigned char>(sizeof(T))) {current_back.resize(sizeof(T));}
|
||||||
|
inline file_vector(const file_vector<T>& c) = delete;
|
||||||
|
inline file_vector(file_vector<T>&& m) : inserter(std::move(m.inserter)), len(m.len), current_back(std::move(m.current_back)){}
|
||||||
|
|
||||||
|
inline void push_back(T&& value) override
|
||||||
|
{
|
||||||
|
inserter.push_buf((file_back_buffer::byte*)&value, sizeof(T));
|
||||||
|
len += 1;
|
||||||
|
}
|
||||||
|
inline T& back() override
|
||||||
|
{
|
||||||
|
if(!len) panic("back() called on empty file_vector");
|
||||||
|
if(!inserter.back(¤t_back[0], sizeof(T))) panic("back() failed");
|
||||||
|
return *_die_if_null((T*)¤t_back[0], "file_vector::back() returned null pointer");
|
||||||
|
}
|
||||||
|
inline const T& back() const override
|
||||||
|
{
|
||||||
|
if(!len) panic("back() called on empty file_vector");
|
||||||
|
if(!inserter.back(¤t_back[0], sizeof(T))) panic("back() failed");
|
||||||
|
return *_die_if_null((const T*)¤t_back[0], "file_vector::back() (const) returned null pointer");
|
||||||
|
}
|
||||||
|
inline void pop_back() override
|
||||||
|
{
|
||||||
|
if(!len) return;
|
||||||
|
|
||||||
|
if(!inserter.pop_n(sizeof(T))) panic("pop_back(): 0 elements");
|
||||||
|
len-=1;
|
||||||
|
}
|
||||||
|
|
||||||
|
inline const std::size_t size() const override { return len; }
|
||||||
|
private:
|
||||||
|
file_back_buffer inserter;
|
||||||
|
std::size_t len=0;
|
||||||
|
mutable std::vector<unsigned char> current_back; // what an awful hack...
|
||||||
|
};
|
||||||
|
|
||||||
|
|
||||||
|
template<typename T, std::size_t Spill = FSV_DEFAULT_SPILL_AT >
|
||||||
|
requires (Spill > 0)
|
||||||
|
struct fixed_spill_vector : public i_back_inserter<T>
|
||||||
|
{
|
||||||
|
constexpr const static std::size_t SPILL_AT = Spill;
|
||||||
|
|
||||||
|
inline fixed_spill_vector() : mem(std::make_unique<std::array<T, Spill> >()){
|
||||||
|
D_dprintf("alloc cap (static): %lu", Spill);
|
||||||
|
}
|
||||||
|
inline fixed_spill_vector(const fixed_spill_vector<T>& c) = delete;
|
||||||
|
inline fixed_spill_vector(fixed_spill_vector<T>&& m)
|
||||||
|
: mem(std::move(m.mem)),
|
||||||
|
mem_fill_ptr(m.mem_fill_ptr),
|
||||||
|
fil(std::move(m.fil))
|
||||||
|
{}
|
||||||
|
inline ~fixed_spill_vector() = default;
|
||||||
|
|
||||||
|
inline void push_back(T&& value) override
|
||||||
|
{
|
||||||
|
if(mem_is_full()) {
|
||||||
|
//D_dprintf("Inserting value into fs");
|
||||||
|
fil.push_back(std::move(value));
|
||||||
|
} else {
|
||||||
|
//D_dprintf("Inserting value into memory");
|
||||||
|
(*mem)[++mem_fill_ptr] = value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
inline void pop_back() override
|
||||||
|
{
|
||||||
|
if(!size()) return;
|
||||||
|
|
||||||
|
if(fil.size()) {
|
||||||
|
//D_dprintf("Popping from fs");
|
||||||
|
fil.pop_back();
|
||||||
|
} else {
|
||||||
|
//D_dprintf("Popping from memory %ld", mem_fill_ptr);
|
||||||
|
mem_fill_ptr -= 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
inline const T& back() const override
|
||||||
|
{
|
||||||
|
if (!size()) panic("back() (const) called on no elements");
|
||||||
|
if(fil.size()) return fil.back();
|
||||||
|
else return (*mem)[mem_fill_ptr];
|
||||||
|
}
|
||||||
|
inline T& back() override
|
||||||
|
{
|
||||||
|
if (!size()) panic("back() called on no elements");
|
||||||
|
if(fil.size()) return fil.back();
|
||||||
|
else return (*mem)[mem_fill_ptr];
|
||||||
|
}
|
||||||
|
inline const std::size_t size() const override
|
||||||
|
{
|
||||||
|
return fil.size() + (std::size_t)(mem_fill_ptr+1);
|
||||||
|
}
|
||||||
|
private:
|
||||||
|
inline bool mem_is_full() const { return mem_fill_ptr >= (ssize_t)(Spill-1); }
|
||||||
|
ssize_t mem_fill_ptr=-1;
|
||||||
|
|
||||||
|
std::unique_ptr<std::array<T, Spill>> mem;
|
||||||
|
file_vector<T> fil;
|
||||||
|
};
|
||||||
|
|
||||||
|
template<typename T>
|
||||||
|
struct mapped_vector : public i_back_inserter<T>, public i_shunt
|
||||||
|
{
|
||||||
|
inline static mapped_vector<T> from_temp(std::size_t sz)
|
||||||
|
{
|
||||||
|
D_dprintf("generating with %lu size", sz);
|
||||||
|
temp_file file;
|
||||||
|
mapped_vector<T> mvec(file.full_path().c_str(), sz);
|
||||||
|
D_dprintf("generated?");
|
||||||
|
mvec.temp = std::make_unique<temp_file>(std::move(file));
|
||||||
|
return mvec;
|
||||||
|
}
|
||||||
|
inline mapped_vector(const char* file, std::size_t sz)
|
||||||
|
: sz(sz),
|
||||||
|
temp(nullptr),
|
||||||
|
map(mm::mmap::allocate(file, sz * sizeof(T))){}
|
||||||
|
inline mapped_vector(const mapped_vector<T>& c) = delete;
|
||||||
|
inline mapped_vector(mapped_vector<T>&& m)
|
||||||
|
: sz(m.sz),
|
||||||
|
fill_ptr(m.fill_ptr),
|
||||||
|
temp(std::move(m.temp)),
|
||||||
|
map(std::move(m.map)){}
|
||||||
|
inline mapped_vector() : mapped_vector(nullptr, 0)
|
||||||
|
{
|
||||||
|
panic("unsupported");
|
||||||
|
}
|
||||||
|
|
||||||
|
inline void push_back(T&& value) override
|
||||||
|
{
|
||||||
|
if(is_full()) panic("Tried to push past end of map");
|
||||||
|
else memory()[++fill_ptr] = value;
|
||||||
|
}
|
||||||
|
inline void pop_back() override
|
||||||
|
{
|
||||||
|
if(fill_ptr>=0) fill_ptr-=1;
|
||||||
|
}
|
||||||
|
|
||||||
|
inline T& back() override
|
||||||
|
{
|
||||||
|
if(fill_ptr>=0)
|
||||||
|
{
|
||||||
|
return memory()[fill_ptr];
|
||||||
|
} else panic("back() called with no elements");
|
||||||
|
}
|
||||||
|
inline const T& back() const override
|
||||||
|
{
|
||||||
|
if(fill_ptr>=0)
|
||||||
|
{
|
||||||
|
return memory()[fill_ptr];
|
||||||
|
} else panic("back() const called with no elements");
|
||||||
|
}
|
||||||
|
inline const std::size_t size() const override { return ((std::size_t)fill_ptr)+1; }
|
||||||
|
inline std::size_t cap() const { return sz; }
|
||||||
|
|
||||||
|
inline bool is_full() const { return fill_ptr >= (ssize_t)(sz-1); }
|
||||||
|
protected:
|
||||||
|
inline const span<T> memory() const { return map.as_span().reinterpret<T>(); }
|
||||||
|
inline span<T> memory() { return map.as_span().reinterpret<T>(); }
|
||||||
|
private:
|
||||||
|
std::size_t sz;
|
||||||
|
ssize_t fill_ptr=-1;
|
||||||
|
|
||||||
|
std::unique_ptr<temp_file> temp;
|
||||||
|
mm::mmap map;
|
||||||
|
};
|
||||||
|
|
||||||
|
template<typename T, typename Shunt>
|
||||||
|
requires(std::is_base_of<i_back_inserter<T>, Shunt >::value)
|
||||||
|
struct shunt : public i_back_inserter<T>, protected i_shunt
|
||||||
|
{
|
||||||
|
typedef Shunt spill_type;
|
||||||
|
|
||||||
|
inline shunt() : shunt(FSV_DEFAULT_SPILL_AT){}
|
||||||
|
inline shunt(spill_type&& into) : shunt(FSV_DEFAULT_SPILL_AT, std::move(into)){}
|
||||||
|
inline shunt(std::size_t cap) : shunt(cap, cap){}
|
||||||
|
inline shunt(std::size_t cap, spill_type&& into) : shunt(cap, cap, std::move(into)){}
|
||||||
|
inline shunt(std::size_t cap, std::size_t spill, spill_type&& into)
|
||||||
|
: _spill_at(spill), mem(std::vector<T>()), fil(std::make_unique<spill_type>(std::move(into))) {
|
||||||
|
mem.reserve(cap);
|
||||||
|
|
||||||
|
D_dprintf("alloc (explicit) cap %lu (sz %lu == 0?), spill %lu", cap, mem.size(), spill_at());
|
||||||
|
}
|
||||||
|
inline shunt(std::size_t cap, std::size_t spill) : _spill_at(spill), mem(std::vector<T>()), fil(nullptr) {
|
||||||
|
mem.reserve(cap);
|
||||||
|
D_dprintf("alloc cap %lu (sz %lu == 0?), spill %lu", cap, mem.size(), spill_at());
|
||||||
|
}
|
||||||
|
inline shunt(const shunt<T, Shunt>& c) = delete;
|
||||||
|
inline shunt(shunt<T, Shunt>&& m) :
|
||||||
|
_spill_at(m._spill_at),
|
||||||
|
mem(std::move(m.mem)),
|
||||||
|
fil(std::move(m.fil)){}
|
||||||
|
|
||||||
|
inline void push_back(T&& value) override
|
||||||
|
{
|
||||||
|
if(is_full()) {
|
||||||
|
spl()->push_back(std::move(value));
|
||||||
|
}
|
||||||
|
else mem.push_back(std::move(value));
|
||||||
|
}
|
||||||
|
inline void pop_back() override
|
||||||
|
{
|
||||||
|
if(is_full() && spl()->size()) spl()->pop_back();
|
||||||
|
else mem.pop_back();
|
||||||
|
}
|
||||||
|
inline const T& back() const override
|
||||||
|
{
|
||||||
|
if(is_full() && spl()->size()) return spl()->back();
|
||||||
|
else return mem.back();
|
||||||
|
}
|
||||||
|
inline T& back() override
|
||||||
|
{
|
||||||
|
if(is_full() && spl()->size()) return spl()->back();
|
||||||
|
else return mem.back();
|
||||||
|
}
|
||||||
|
inline const std::size_t size() const override { return mem.size() + (fil ? fil->size() : 0); }
|
||||||
|
|
||||||
|
inline const std::size_t spill_at() const { return _spill_at; }
|
||||||
|
inline bool is_spilling() const { return is_full(); }
|
||||||
|
protected:
|
||||||
|
inline bool is_full() const override { return size()>=spill_at(); }
|
||||||
|
private:
|
||||||
|
inline const spill_type* spl() const
|
||||||
|
{
|
||||||
|
if(is_full()) {
|
||||||
|
if(!fil) fil = std::make_unique<spill_type>();
|
||||||
|
return fil.get();
|
||||||
|
} else {
|
||||||
|
return fil.get();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
inline spill_type* spl()
|
||||||
|
{
|
||||||
|
if(is_full()) {
|
||||||
|
if(!fil) fil = std::make_unique<spill_type>();
|
||||||
|
return fil.get();
|
||||||
|
} else {
|
||||||
|
return fil.get();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
std::size_t _spill_at;
|
||||||
|
std::vector<T> mem;
|
||||||
|
mutable std::unique_ptr<spill_type> fil;
|
||||||
|
};
|
||||||
|
|
||||||
|
template<typename T>
|
||||||
|
using dynamic_spill_vector = shunt<T, file_vector<T> >;
|
||||||
|
|
||||||
|
template<typename T>
|
||||||
|
using mapped_spill_vector = shunt<T, mapped_vector<T> >;
|
@ -0,0 +1,202 @@
|
|||||||
|
#ifndef _MAP_H
|
||||||
|
#define _MAP_H
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
#include <stdexcept>
|
||||||
|
extern "C" {
|
||||||
|
#define restrict __restrict__
|
||||||
|
#define MIFCXX(y, n) y
|
||||||
|
#define MIFC(y, n) n
|
||||||
|
#else
|
||||||
|
#define MIFCXX(y, n) n
|
||||||
|
#define MIFC(y, n) y
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#include <stddef.h>
|
||||||
|
|
||||||
|
typedef struct mmap {
|
||||||
|
int fd;
|
||||||
|
|
||||||
|
void* ptr;
|
||||||
|
size_t len;
|
||||||
|
} mmap_t;
|
||||||
|
|
||||||
|
enum map_flags {
|
||||||
|
MMF_SHARED,
|
||||||
|
MMF_PRIVATE,
|
||||||
|
MMF_READONLY,
|
||||||
|
};
|
||||||
|
|
||||||
|
int map_raw_fd(int fd, mmap_t* restrict ptr, const size_t MIFCXX(* sz, sz[static 1]));
|
||||||
|
int open_and_alloc(const char* file, mmap_t* restrict ptr, size_t sz);
|
||||||
|
int open_and_map(const char* file, mmap_t* restrict ptr);
|
||||||
|
int unmap_and_close(mmap_t map);
|
||||||
|
int dup_map(const mmap_t *in, mmap_t* restrict out, const size_t *new_size, enum map_flags flags) __attribute__((nonnull(1)));
|
||||||
|
int map_advise_rand(mmap_t* restrict ptr, int need);
|
||||||
|
|
||||||
|
typedef void* (*map_cb)(mmap_t map, void* user);
|
||||||
|
void* map_and_then(const char* file, map_cb callback, void* user);
|
||||||
|
void* map_fd_and_then(int fd, map_cb callback, void* user);
|
||||||
|
int unmap_and_close_s(mmap_t map, int flags);
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
}
|
||||||
|
#include <panic.h>
|
||||||
|
#include "reinterpret.h"
|
||||||
|
#include <cstdint>
|
||||||
|
namespace mm {
|
||||||
|
enum class Access {
|
||||||
|
Random,
|
||||||
|
};
|
||||||
|
struct mmap;
|
||||||
|
template<typename T>
|
||||||
|
concept MemMap = std::derived_from<T, mmap>
|
||||||
|
and std::is_constructible_v<T, mmap_t>;
|
||||||
|
struct mmap {
|
||||||
|
template<MemMap M = mmap>
|
||||||
|
inline static M allocate(const char* file, std::size_t sz)
|
||||||
|
{
|
||||||
|
mmap_t map;
|
||||||
|
if(!open_and_alloc(file, &map, sz)) panic("Failed to allocmap file");
|
||||||
|
return M{map};
|
||||||
|
}
|
||||||
|
inline static mmap_t create_raw(const char* file)
|
||||||
|
{
|
||||||
|
mmap_t map;
|
||||||
|
if (!::open_and_map(file, &map)) panic("Failed to map file");
|
||||||
|
return map;
|
||||||
|
}
|
||||||
|
inline static mmap_t create_raw_fd(int fd, size_t sz)
|
||||||
|
{
|
||||||
|
mmap_t map;
|
||||||
|
if (!::map_raw_fd(fd, &map, &sz)) panic("Failed to allocmap fd");
|
||||||
|
return map;
|
||||||
|
}
|
||||||
|
inline static mmap_t create_raw_fd(int fd)
|
||||||
|
{
|
||||||
|
mmap_t map;
|
||||||
|
if (!::map_raw_fd(fd, &map, nullptr)) panic("Failed to map fd");
|
||||||
|
return map;
|
||||||
|
}
|
||||||
|
template<MemMap M = mmap>
|
||||||
|
inline static M map_raw_fd(int fd, std::size_t sz = 0)
|
||||||
|
{
|
||||||
|
if(sz)
|
||||||
|
return M{fd, sz};
|
||||||
|
else return M{fd};
|
||||||
|
}
|
||||||
|
|
||||||
|
inline explicit mmap(mmap_t raw) noexcept :inner(raw){}
|
||||||
|
inline mmap(const char* file)
|
||||||
|
: inner(create_raw(file)) {}
|
||||||
|
|
||||||
|
inline mmap(mmap&& move) : inner(move.inner)
|
||||||
|
{
|
||||||
|
auto other = const_cast<mmap_t*>(&move.inner);
|
||||||
|
other->ptr = nullptr;
|
||||||
|
}
|
||||||
|
inline mmap(const mmap& copt) = delete;
|
||||||
|
|
||||||
|
inline virtual ~mmap()
|
||||||
|
{
|
||||||
|
if (inner.ptr) {
|
||||||
|
::unmap_and_close(inner);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
inline virtual mmap&& access(Access a, bool need=false) && noexcept
|
||||||
|
{
|
||||||
|
if(inner.ptr)
|
||||||
|
static_cast<mmap&>(*this).access(a, need);
|
||||||
|
return std::move(*this);
|
||||||
|
}
|
||||||
|
inline virtual mmap& access(Access a, bool need=false) & noexcept
|
||||||
|
{
|
||||||
|
switch(a) {
|
||||||
|
case Access::Random: ::map_advise_rand(const_cast<mmap_t*>(&inner), int(need));
|
||||||
|
default: break;
|
||||||
|
}
|
||||||
|
return *this;
|
||||||
|
}
|
||||||
|
|
||||||
|
inline const span<const unsigned char> as_span() const noexcept { return span(as_ptr(), size()); }
|
||||||
|
inline span<unsigned char> as_span() noexcept { return span(as_ptr(), size()); }
|
||||||
|
|
||||||
|
inline virtual const std::uint8_t* as_ptr() const noexcept { return static_cast<const std::uint8_t*>(inner.ptr); }
|
||||||
|
inline virtual std::uint8_t* as_ptr() noexcept { return static_cast<std::uint8_t*>(inner.ptr); }
|
||||||
|
|
||||||
|
inline virtual std::size_t size() const noexcept { return inner.len; }
|
||||||
|
|
||||||
|
inline virtual int as_fd() const { return inner.fd; }
|
||||||
|
inline virtual const mmap_t& as_raw() const noexcept { return inner; }
|
||||||
|
protected:
|
||||||
|
inline mmap(int fd, size_t sz)
|
||||||
|
: inner(create_raw_fd(fd, sz)) {}
|
||||||
|
inline explicit mmap(int fd)
|
||||||
|
: inner(create_raw_fd(fd)) {}
|
||||||
|
inline virtual mmap_t& as_raw() noexcept { return const_cast<mmap_t&>( inner ); }
|
||||||
|
private:
|
||||||
|
const mmap_t inner;
|
||||||
|
};
|
||||||
|
//template mmap::map_raw_fd<mmap>(int, size_t); // Probably not needed?
|
||||||
|
//template mmap::allocate<mmap>(int, size_t);
|
||||||
|
|
||||||
|
struct vmap : public mmap {
|
||||||
|
inline explicit vmap(mmap_t m) noexcept : mmap(m) {}
|
||||||
|
|
||||||
|
vmap();
|
||||||
|
vmap(size_t);
|
||||||
|
inline explicit vmap(int fd) noexcept : mmap(fd) {}
|
||||||
|
//vmap(const char* file);
|
||||||
|
|
||||||
|
//int as_fd() const override; unneeded
|
||||||
|
|
||||||
|
inline virtual ~vmap() { /* unmap_and_close() is called by super */ }
|
||||||
|
//TODO: Implement this ^ with `memfd_create()`, etc.
|
||||||
|
};
|
||||||
|
|
||||||
|
|
||||||
|
template<MemMap M, MemMap D = M>
|
||||||
|
inline D dup_map(const M& map, size_t resize, map_flags flags =0)
|
||||||
|
{
|
||||||
|
const mmap& m = static_cast<const mmap&>(map);
|
||||||
|
mmap_t out;
|
||||||
|
if(! ::dup_map(&m.as_raw(), &out, &resize, flags)) panic("Failed to duplicate mapping");
|
||||||
|
return D{out};
|
||||||
|
}
|
||||||
|
template<MemMap M, MemMap D = M>
|
||||||
|
inline D dup_map(const M& map, map_flags flags =0)
|
||||||
|
{
|
||||||
|
const mmap& m = static_cast<const mmap&>(map);
|
||||||
|
mmap_t out;
|
||||||
|
if(! ::dup_map(&m.as_raw(), &out, nullptr, flags)) panic("Failed to duplicate mapping");
|
||||||
|
return D{out};
|
||||||
|
}
|
||||||
|
#if 0
|
||||||
|
// dup_map() deduction guids (XXX: Are these needed, or even valid?)
|
||||||
|
template<MemMap D>
|
||||||
|
dup_map(const mmap& m, size_t s, map_flags f) -> dup_map<mmap, D>;
|
||||||
|
|
||||||
|
template<>
|
||||||
|
dup_map(const mmap& m, size_t s, map_flags f) -> dup_map<mmap, mmap>;
|
||||||
|
|
||||||
|
template<MemMap M>
|
||||||
|
dup_map(const M& m, size_t s, map_flags f) -> dup_map<M, M>;
|
||||||
|
|
||||||
|
template<MemMap D>
|
||||||
|
dup_map(const mmap& m, map_flags f) -> dup_map<mmap, D>;
|
||||||
|
|
||||||
|
template<>
|
||||||
|
dup_map(const mmap& m, map_flags f) -> dup_map<mmap, mmap>;
|
||||||
|
|
||||||
|
template<MemMap M>
|
||||||
|
dup_map(const M& m, map_flags f) -> dup_map<M, M>;
|
||||||
|
#endif
|
||||||
|
}
|
||||||
|
|
||||||
|
#undef restrict
|
||||||
|
#undef MIFCXX
|
||||||
|
#undef MIFC
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#endif /* _MAP_H */
|
@ -0,0 +1,35 @@
|
|||||||
|
#ifndef _PANIC_H
|
||||||
|
#define _PANIC_H
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
extern "C" {
|
||||||
|
#endif
|
||||||
|
|
||||||
|
struct panicinfo {
|
||||||
|
const char* file;
|
||||||
|
const char* function;
|
||||||
|
int line;
|
||||||
|
};
|
||||||
|
|
||||||
|
void _do_panic(struct panicinfo pi, const char* fmt, ...) __attribute__((noreturn));
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
extern "C++" {
|
||||||
|
#include <utility>
|
||||||
|
template<typename... Args>
|
||||||
|
__attribute__((noreturn)) inline void _real_panic(const char* file, const char* function, int line, const char* fmt, Args&&... args)
|
||||||
|
{
|
||||||
|
panicinfo i = { file, function, line };
|
||||||
|
_do_panic(i, fmt, std::move(args)...);
|
||||||
|
}
|
||||||
|
#define panic(fmt, ...) _real_panic(__FILE__, __func__, __LINE__, fmt __VA_OPT__(,) __VA_ARGS__)
|
||||||
|
}
|
||||||
|
#else
|
||||||
|
#define panic(fmt, ...) _do_panic( (struct panicinfo){.file = __FILE__, .function = __func__, .line = __LINE__}, fmt __VA_OPT__(,) __VA_ARGS__)
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#endif
|
@ -0,0 +1,406 @@
|
|||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include <string>
|
||||||
|
#include <memory>
|
||||||
|
#include <utility>
|
||||||
|
#include <array>
|
||||||
|
#include <utility>
|
||||||
|
|
||||||
|
#include <cstdio>
|
||||||
|
|
||||||
|
#ifndef __cplusplus
|
||||||
|
#error "C++ header only"
|
||||||
|
#endif
|
||||||
|
|
||||||
|
namespace pr {
|
||||||
|
|
||||||
|
namespace details [[gnu::visibility("internal")]] {
|
||||||
|
/*
|
||||||
|
template<template<typename...> typename P>
|
||||||
|
struct generic_valid { template<typename... Args> constexpr static inline bool value = requires{ typename P<Args...>; }; };
|
||||||
|
template<template<typename...> class P>
|
||||||
|
struct generic : std::conditional_t<generic_valid<P>::value,
|
||||||
|
std::true_type,
|
||||||
|
std::false_type>
|
||||||
|
{ template<typename... Args> using type = P<Args...>; };
|
||||||
|
|
||||||
|
template<template<typename> typename T, typename... Args>
|
||||||
|
concept Generic = generic_valid<T>::template value<Args...>;
|
||||||
|
|
||||||
|
template<Generic T, typename... Args>
|
||||||
|
using generic_t = generic<T>::template type<Args...>;
|
||||||
|
|
||||||
|
template<template<typename> template T>
|
||||||
|
concept AnyGeneric = requires{ typename generic_valid<T>; };
|
||||||
|
*/
|
||||||
|
template<typename T>
|
||||||
|
T* take(T*&& ptr) noexcept { return std::exchange(ptr, nullptr); }
|
||||||
|
|
||||||
|
/*
|
||||||
|
template<template<typename U> Inst, typename T>
|
||||||
|
struct is_inst_of { constexpr static inline bool value = std::is_same_v<T, Inst<U>>; };
|
||||||
|
|
||||||
|
template<typename T, typename I>
|
||||||
|
constexpr inline bool is_inst_of_v = is_inst_of<I, T>::value;
|
||||||
|
*/
|
||||||
|
|
||||||
|
template<typename U, typename T>
|
||||||
|
constexpr std::unique_ptr<U> static_ucast(std::unique_ptr<T>&& from) noexcept
|
||||||
|
{ return std::unique_ptr<U>{static_cast<U*>(std::move(from).release())}; }
|
||||||
|
|
||||||
|
template<typename U, typename T>
|
||||||
|
constexpr std::shared_ptr<U> static_ucast(std::shared_ptr<T>&& from) noexcept
|
||||||
|
{ return std::static_pointer_cast<U>(std::move(from)); }
|
||||||
|
|
||||||
|
|
||||||
|
template<typename Ptr, typename From> requires(requires(From&& p) {
|
||||||
|
{ p.release() };
|
||||||
|
requires(std::is_constructible_v<Ptr, decltype(p.release())>);
|
||||||
|
})
|
||||||
|
constexpr Ptr static_pcast(From&& p) noexcept { return Ptr{p.release()}; }
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
using increment_t = long double;
|
||||||
|
|
||||||
|
template<typename T>
|
||||||
|
concept Bar = requires(std::remove_const_t<std::remove_reference_t<T>>& v) {
|
||||||
|
{ v.spin(increment_t(0)) };
|
||||||
|
{ std::as_const(v).aux() } -> std::convertible_to<std::string_view>;
|
||||||
|
//{ v.aux() } -> std::assignable_from<std::string_view>;
|
||||||
|
{ (v.aux() = std::string{}) } -> std::same_as<std::add_lvalue_reference_t<decltype(v.aux())>>;
|
||||||
|
//requires(std::assignable_from<std::add_lvalue_reference_t<decltype(v.aux())>, std::string_view>);
|
||||||
|
{ v.aux() += std::string_view{} } -> std::convertible_to<decltype(v.aux())>;
|
||||||
|
{ v.aux(std::declval<std::string&&>()) } -> std::convertible_to<std::string_view>;
|
||||||
|
};
|
||||||
|
|
||||||
|
template<Bar>
|
||||||
|
struct Carrier;
|
||||||
|
|
||||||
|
struct Dynamic {
|
||||||
|
template<Bar> friend class Carrier;
|
||||||
|
|
||||||
|
constexpr auto make(Bar auto&& bar) noexcept(std::is_nothrow_move_constructible_v<std::remove_reference_t<decltype(bar)>>);
|
||||||
|
|
||||||
|
constexpr virtual void spin(increment_t) =0;
|
||||||
|
constexpr virtual std::string& aux() =0;
|
||||||
|
constexpr virtual std::string const& aux() const noexcept =0;
|
||||||
|
constexpr virtual std::string aux(std::string&& s) { return std::exchange(aux(), std::move(s)); }
|
||||||
|
|
||||||
|
constexpr virtual ~Dynamic() {}
|
||||||
|
private:
|
||||||
|
constexpr Dynamic() noexcept {}
|
||||||
|
};
|
||||||
|
template<Bar B>
|
||||||
|
struct Carrier : public virtual Dynamic {
|
||||||
|
using held_type = B;
|
||||||
|
|
||||||
|
template<typename... Args> requires(std::constructible_from<B, Args...>)
|
||||||
|
constexpr Carrier(Args&&... args) noexcept(std::is_nothrow_constructible_v<B, Args...>)
|
||||||
|
: Dynamic(), member_(std::forward<decltype(args)>(args)...) {}
|
||||||
|
constexpr Carrier(Carrier&& m) noexcept(std::is_nothrow_move_constructible_v<B>)
|
||||||
|
requires(std::is_move_constructible_v<B>)
|
||||||
|
: Dynamic(), member_(std::move(m.member_)) {}
|
||||||
|
constexpr Carrier(Carrier const& c) noexcept(std::is_nothrow_copy_constructible_v<B>)
|
||||||
|
requires(std::is_copy_constructible_v<B>)
|
||||||
|
: Dynamic(), member_(c.member_) {}
|
||||||
|
|
||||||
|
constexpr Carrier& operator=(Carrier&& m) noexcept(std::is_nothrow_move_assignable_v<B>) requires(std::is_move_assignable_v<B>)
|
||||||
|
{ if(this != &m) member_ = std::move(m.member_); return *this; }
|
||||||
|
|
||||||
|
constexpr Carrier& operator=(Carrier const& c) noexcept(std::is_nothrow_copy_assignable_v<B>) requires(std::is_copy_assignable_v<B>)
|
||||||
|
{ if(this != &c) member_ = c.member_; return *this; }
|
||||||
|
|
||||||
|
constexpr void spin(increment_t a) override { member_.spin(a); }
|
||||||
|
|
||||||
|
constexpr std::string& aux() override { return member_.aux(); }
|
||||||
|
constexpr std::string const& aux() const noexcept override { return member_.aux(); }
|
||||||
|
constexpr std::string aux(std::string&& s) override { return member_.aux(std::move(s)); }
|
||||||
|
|
||||||
|
template<typename... Args> requires(std::is_invocable_v<B::spin, B&, increment_t, Args...>)
|
||||||
|
constexpr void spin(increment_t a, Args&&... args) noexcept(std::is_nothrow_invocable_v<B::spin, B&, increment_t, decltype(args)...>)
|
||||||
|
{ return member_.spin(a, std::forward<decltype(args)>(args)...); }
|
||||||
|
|
||||||
|
constexpr virtual ~Carrier() {}
|
||||||
|
private:
|
||||||
|
B member_;
|
||||||
|
};
|
||||||
|
template<Bar T>
|
||||||
|
Carrier(T) -> Carrier<T>;
|
||||||
|
template<Bar T>
|
||||||
|
Carrier(T const&) -> Carrier<T>;
|
||||||
|
template<Bar T>
|
||||||
|
Carrier(T&&) -> Carrier<T>;
|
||||||
|
|
||||||
|
constexpr auto Dynamic::make(Bar auto&& bar) noexcept(std::is_nothrow_move_constructible_v<std::remove_reference_t<decltype(bar)>>) { return Carrier(std::move(bar)); }
|
||||||
|
|
||||||
|
template<Bar T, typename Ptr = std::unique_ptr<Dynamic>>
|
||||||
|
constexpr Ptr make_dyn(T&& bar) noexcept(std::is_nothrow_move_constructible_v<T>)
|
||||||
|
{
|
||||||
|
using namespace details;
|
||||||
|
if constexpr(requires{
|
||||||
|
typename T::held_type;
|
||||||
|
requires(std::is_same_v<Carrier<typename T::held_type>, T>);
|
||||||
|
}) return static_pcast<Ptr>(static_ucast<Dynamic>(std::make_unique<T>(std::move(bar))));
|
||||||
|
else return static_pcast<Ptr>(static_ucast<Dynamic>(std::make_unique<Carrier<T>>(std::move(bar))));
|
||||||
|
}
|
||||||
|
|
||||||
|
template<Bar T, typename Ptr = std::unique_ptr<Dynamic>>
|
||||||
|
constexpr Ptr make_dyn(T const& bar) noexcept(std::is_nothrow_move_constructible_v<T>)
|
||||||
|
{
|
||||||
|
if constexpr(std::is_copy_constructible_v<T>) {
|
||||||
|
T nbar{bar};
|
||||||
|
return make_dyn<T, Ptr>(std::move(nbar));
|
||||||
|
} else {
|
||||||
|
struct unsafe_ref {
|
||||||
|
constexpr unsafe_ref(const T& ba) noexcept : b(std::addressof(ba)) {}
|
||||||
|
constexpr unsafe_ref(const unsafe_ref&) noexcept = default;
|
||||||
|
constexpr ~unsafe_ref() noexcept = default;
|
||||||
|
constexpr unsafe_ref(unsafe_ref&& b) noexcept : b(std::exchange(b.b, nullptr)) {}
|
||||||
|
|
||||||
|
constexpr unsafe_ref& operator=(unsafe_ref const&) noexcept = default;
|
||||||
|
constexpr unsafe_ref& operator=(unsafe_ref&& m) noexcept {
|
||||||
|
if(this != &m)
|
||||||
|
b = std::exchange(m.b, nullptr);
|
||||||
|
return *this;
|
||||||
|
}
|
||||||
|
|
||||||
|
const T* b;
|
||||||
|
|
||||||
|
constexpr operator T const&() const noexcept { return *b; }
|
||||||
|
};
|
||||||
|
unsafe_ref re{bar};
|
||||||
|
return make_dyn<unsafe_ref, Ptr>(std::move(re));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename Ptr> requires(requires(std::unique_ptr<Dynamic>&& p) { details::static_pcast<Ptr>(std::move(p)); })
|
||||||
|
constexpr Ptr make_dyn_for(Bar auto&& bar) noexcept(std::is_nothrow_move_constructible_v<std::remove_reference_t<decltype(bar)>>)
|
||||||
|
{ return make_dyn<decltype(bar), Ptr>(std::move(bar)); }
|
||||||
|
|
||||||
|
constexpr void spin(Bar auto& bar, std::convertible_to<increment_t> auto&& a) { return bar.spin(increment_t(a)); }
|
||||||
|
|
||||||
|
struct None {
|
||||||
|
constexpr None() noexcept = default;
|
||||||
|
constexpr ~None() noexcept = default;
|
||||||
|
|
||||||
|
constexpr void spin(increment_t) const noexcept {}
|
||||||
|
constexpr auto aux() noexcept { return not_string(); }
|
||||||
|
inline std::string const& aux() const noexcept { return none_; }
|
||||||
|
constexpr std::string aux(std::convertible_to<std::string> auto&& s) const noexcept { return {}; } //{ return std::exchange(none_, std::string(std::move(s))); }
|
||||||
|
private:
|
||||||
|
struct not_string {
|
||||||
|
constexpr not_string() noexcept= default;
|
||||||
|
constexpr ~not_string() noexcept= default;
|
||||||
|
|
||||||
|
[[gnu::const]]
|
||||||
|
constexpr not_string& operator+=(std::convertible_to<std::string_view> auto&&) { return *this; }
|
||||||
|
[[gnu::const]]
|
||||||
|
constexpr not_string& operator=(std::convertible_to<std::string> auto&&) { return *this; }
|
||||||
|
|
||||||
|
inline operator std::string&() noexcept { return none_; }
|
||||||
|
inline operator std::string const&() const noexcept { return none_; }
|
||||||
|
|
||||||
|
};
|
||||||
|
/*constinit*/ thread_local static inline std::string none_;
|
||||||
|
};
|
||||||
|
constexpr inline None disable{};
|
||||||
|
|
||||||
|
// A bounded progress-bar
|
||||||
|
struct Progress {
|
||||||
|
using fract_t = increment_t;
|
||||||
|
constexpr static inline bool DEFAULT_WIDTH = 50;
|
||||||
|
constexpr static inline fract_t DEFAULT_TERM_FRACT=1.0l/3.0l;
|
||||||
|
|
||||||
|
[[gnu::nonnull(1)]]
|
||||||
|
Progress(FILE* p = stdout) noexcept;
|
||||||
|
|
||||||
|
Progress(const Progress& p) noexcept;
|
||||||
|
Progress(Progress&& p) noexcept;
|
||||||
|
|
||||||
|
Progress& operator=(Progress const& c) noexcept;
|
||||||
|
Progress& operator=(Progress&& m) noexcept;
|
||||||
|
|
||||||
|
virtual ~Progress();
|
||||||
|
|
||||||
|
inline std::string& aux() noexcept { return tag(); }
|
||||||
|
inline std::string const& aux() const noexcept { return tag(); }
|
||||||
|
inline std::string aux(std::convertible_to<std::string> auto&& s)
|
||||||
|
{ return std::exchange(tag(), std::string(std::move(s))); }
|
||||||
|
protected:
|
||||||
|
std::string& tag() noexcept;
|
||||||
|
const std::string& tag() const noexcept;
|
||||||
|
public:
|
||||||
|
FILE* output() const noexcept;
|
||||||
|
FILE*& output() noexcept;
|
||||||
|
|
||||||
|
size_t& width() noexcept;
|
||||||
|
size_t width() const noexcept;
|
||||||
|
|
||||||
|
fract_t& fraction() noexcept;
|
||||||
|
fract_t fraction() const noexcept;
|
||||||
|
|
||||||
|
fract_t percentage() const noexcept;
|
||||||
|
inline auto percentage() noexcept {
|
||||||
|
struct v final {
|
||||||
|
Progress* p_;
|
||||||
|
v(Progress* p) noexcept : p_(p) {}
|
||||||
|
|
||||||
|
v(v&&) noexcept = default;
|
||||||
|
v(v const&) noexcept = default;
|
||||||
|
v& operator=(v const&) noexcept = default;
|
||||||
|
v& operator=(v&&) noexcept = default;
|
||||||
|
~v() noexcept = default;
|
||||||
|
|
||||||
|
const v& operator=(fract_t f) const noexcept { p_->percentage(f); return *this; }
|
||||||
|
operator fract_t() const noexcept { return static_cast<const Progress*>(p_)->percentage(); }
|
||||||
|
};
|
||||||
|
return v(this);
|
||||||
|
}
|
||||||
|
|
||||||
|
void spin(increment_t by, bool render=true, bool flush=true) noexcept;
|
||||||
|
void render(bool flush = true) const noexcept;
|
||||||
|
private:
|
||||||
|
void percentage(fract_t fract) noexcept;
|
||||||
|
|
||||||
|
struct _impl;
|
||||||
|
std::shared_ptr<_impl> inner_;
|
||||||
|
};
|
||||||
|
|
||||||
|
|
||||||
|
#ifdef SPINNER
|
||||||
|
class Spinner {
|
||||||
|
consteval static auto _GENERATE_MAP(auto const& map, size_t size) noexcept
|
||||||
|
-> std::array<size_t, 256>
|
||||||
|
{
|
||||||
|
static_assert(sizeof(map) == size, "Bad map size");
|
||||||
|
std::array<size_t, 256> out{};
|
||||||
|
while( size --> 0 ) out[int(map[size])] = size;
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
constexpr static increment_t range(increment_t by)
|
||||||
|
{
|
||||||
|
if(by < -1.0l) return -1.0l;
|
||||||
|
else if(by > 1.0l) return 1.0l;
|
||||||
|
return by;
|
||||||
|
}
|
||||||
|
public:
|
||||||
|
constexpr static inline auto ROUTINE = "|/-\\|/-\\|";
|
||||||
|
constexpr static inline auto ROUTINE_SIZE = sizeof(ROUTINE);
|
||||||
|
constexpr static inline auto REVERSE_MAP = _GENERATE_MAP(ROUTINE, ROUTINE_SIZE);
|
||||||
|
static_assert(ROUTINE_SIZE != sizeof(char*), "Invalid routine size");
|
||||||
|
|
||||||
|
constexpr ssize_t range(int sz) noexcept
|
||||||
|
{
|
||||||
|
/*if(__builtin_expect(sz < 0, false)) {
|
||||||
|
|
||||||
|
std::terminate(); // TODO: How to handle wrapping negatives?? Ugh.
|
||||||
|
}*/
|
||||||
|
return ssize_t(sz) % ssize_t(ROUTINE_SIZE);
|
||||||
|
}
|
||||||
|
constexpr Spinner(size_t n) noexcept
|
||||||
|
: cur_(ROUTINE[range(n)]) {}
|
||||||
|
constexpr Spinner(char c = ROUTINE[0]) noexcept
|
||||||
|
: cur_(range(REVERSE_MAP[int(c)])) {}
|
||||||
|
|
||||||
|
constexpr Spinner(Spinner const&) noexcept = default;
|
||||||
|
constexpr Spinner(Spinner &&) noexcept = default;
|
||||||
|
~Spinner();
|
||||||
|
|
||||||
|
inline void spin(int by) noexcept
|
||||||
|
{
|
||||||
|
operator+=(by);
|
||||||
|
render();
|
||||||
|
}
|
||||||
|
inline void spin(increment_by by) noexcept
|
||||||
|
{
|
||||||
|
spin(int(range(by) * increment_t(ROUTINE_SIZE)));
|
||||||
|
}
|
||||||
|
|
||||||
|
inline Spinner& operator+=(int by) noexcept
|
||||||
|
{
|
||||||
|
cur_ = ROUTINE[size_t(ssize_t(REVERSE_MAP[int(cur_)]) + range(by))];
|
||||||
|
return *this;
|
||||||
|
}
|
||||||
|
|
||||||
|
inline Spinner& operator++() noexcept { spin(1); return *this; }
|
||||||
|
inline Spinner operator++(int) noexcept { Spinner s = *this; ++s; return *this; }
|
||||||
|
constexpr Spinner& operator=(Spinner&&) noexcept = default;
|
||||||
|
constexpr Spinner& operator=(Spinner const&) noexcept = default;
|
||||||
|
|
||||||
|
constexpr char& character() noexcept { return cur_; }
|
||||||
|
constexpr char character() const noexcept { return cur_; }
|
||||||
|
|
||||||
|
void render(bool flush=true);
|
||||||
|
private:
|
||||||
|
char cur_ = '|';
|
||||||
|
};
|
||||||
|
//TODO: Spinny bar thing (unbounded)
|
||||||
|
#endif
|
||||||
|
|
||||||
|
/*
|
||||||
|
struct Bar {
|
||||||
|
Bar() noexcept = default;
|
||||||
|
Bar(const Bar&) noexcept = default;
|
||||||
|
Bar(Bar&&) noexcept = default;
|
||||||
|
Bar& operator=(const Bar&) noexcept = default;
|
||||||
|
Bar& operator=(Bar&&) noexcept = default;
|
||||||
|
virtual ~Bar() = default;
|
||||||
|
|
||||||
|
virtual void spin(int) =0;
|
||||||
|
};
|
||||||
|
|
||||||
|
enum bar_kind {
|
||||||
|
BRK_UNTRACKED,
|
||||||
|
BRK_TRACKED,
|
||||||
|
};
|
||||||
|
|
||||||
|
template<bar_kind = BRK_UNTRACKED>
|
||||||
|
struct progress;
|
||||||
|
|
||||||
|
template<>
|
||||||
|
struct progress<BRK_UNTRACKED>
|
||||||
|
: public virtual Bar {
|
||||||
|
|
||||||
|
progress(FILE* to);
|
||||||
|
progress(progress const&);
|
||||||
|
inline progress(progress&& mv) noexcept
|
||||||
|
: Bar()
|
||||||
|
, _perc(mv._perc)
|
||||||
|
, _output(details::take(mv._output)) {}
|
||||||
|
|
||||||
|
virtual ~progress();
|
||||||
|
|
||||||
|
void spin(int) override;
|
||||||
|
|
||||||
|
//TODO: this
|
||||||
|
protected:
|
||||||
|
union {
|
||||||
|
double _perc;
|
||||||
|
size_t _num; // not used here
|
||||||
|
};
|
||||||
|
FILE* _output;
|
||||||
|
};
|
||||||
|
|
||||||
|
template<>
|
||||||
|
class progress<BRK_TRACKED>
|
||||||
|
: public progress<BRK_UNTRACKED> {
|
||||||
|
using base_t = progress<BRK_UNTRACKED>;
|
||||||
|
public:
|
||||||
|
inline progress(FILE* to, size_t max) : base_t(to), _num(0), _max(max) {}
|
||||||
|
progress(const progress&) = default;
|
||||||
|
progress(progress&& m) : base_t(std::move(m)), _num(std::exchange(m._num, 0)), _max(m._max) {}
|
||||||
|
|
||||||
|
void spin(int) override;
|
||||||
|
//TODO: this
|
||||||
|
|
||||||
|
virtual ~progress() {}
|
||||||
|
|
||||||
|
protected:
|
||||||
|
using base_t::_output;
|
||||||
|
private:
|
||||||
|
using base_t::_num;
|
||||||
|
size_t _max;
|
||||||
|
};
|
||||||
|
*/
|
||||||
|
}
|
@ -0,0 +1 @@
|
|||||||
|
reinterpret.hpp
|
@ -0,0 +1,61 @@
|
|||||||
|
#ifndef _REINTERPRET_H
|
||||||
|
#define _REINTERPRET_H
|
||||||
|
|
||||||
|
#include <panic.h>
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
#define restrict __restrict__
|
||||||
|
#include <cstddef>
|
||||||
|
#include <cstdint>
|
||||||
|
#else
|
||||||
|
#include <stddef.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
template<typename T>
|
||||||
|
struct span {
|
||||||
|
inline span(T* ptr, std::size_t len) : ptr(ptr), len(len) {}
|
||||||
|
|
||||||
|
template<typename U>
|
||||||
|
inline span<U> reinterpret() const
|
||||||
|
{
|
||||||
|
auto bytes = size_bytes();
|
||||||
|
//if (len_b % sizeof(U) != 0) panic("Cannot reinterpret T to U due to unmatch sizing constraints.");
|
||||||
|
|
||||||
|
return span<U>((U*)ptr, bytes / sizeof(U));
|
||||||
|
}
|
||||||
|
|
||||||
|
inline const T& operator[](std::size_t i) const
|
||||||
|
{
|
||||||
|
if (i >= len) panic("Out of bounds access: %lu >= %lu", i, len);
|
||||||
|
return ptr[i];
|
||||||
|
}
|
||||||
|
inline T& operator[](std::size_t i)
|
||||||
|
{
|
||||||
|
if (i >= len) panic("Out of bounds access: %lu >= %lu", i, len);
|
||||||
|
return ptr[i];
|
||||||
|
}
|
||||||
|
|
||||||
|
inline const T* as_ptr() const { return ptr; }
|
||||||
|
inline T* as_ptr() { return ptr; }
|
||||||
|
|
||||||
|
inline const T* operator&() const { return as_ptr(); }
|
||||||
|
inline T* operator&() { return as_ptr(); }
|
||||||
|
|
||||||
|
inline std::size_t size_bytes() const { return len * sizeof(T); }
|
||||||
|
inline std::size_t size() const { return len; }
|
||||||
|
private:
|
||||||
|
T* const ptr;
|
||||||
|
const std::size_t len;
|
||||||
|
};
|
||||||
|
|
||||||
|
extern "C" {
|
||||||
|
#endif
|
||||||
|
uint64_t* bytes_to_long(uint8_t* ptr, size_t ptr_sz, size_t* restrict nsize);
|
||||||
|
float* bytes_to_float(uint8_t* ptr, size_t ptr_sz, size_t* restrict nsize);
|
||||||
|
#ifdef __cplusplus
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#endif /* _REINTERPRET_H */
|
@ -1,25 +1,70 @@
|
|||||||
#ifndef _RNG_H
|
#ifndef _RNG_H
|
||||||
#define _RNG_H
|
#define _RNG_H
|
||||||
|
|
||||||
typedef struct rng_algo *RNG; //RNG algorithm reference
|
#include "shuffle3.h"
|
||||||
|
|
||||||
#define rng_reinterpret(rng, value, type) (*((type*)rng_next(rng, &value, sizeof(type))))
|
#ifdef __cplusplus
|
||||||
#define rng_reinterpret_new(rng, value, type) (*((type*)rng_next(rng, malloc(sizeof(type)), sizeof(type))))
|
#include "rng/impl.hpp"
|
||||||
#define rng_reinterpret_stackalloc(rng, value, type) rng_reinterpret(rng, alloca(sizeof(type)), type)
|
extern "C" {
|
||||||
|
#endif
|
||||||
#define RNG_LEXBIND(rng, type, name) type name = rng_reinterpret(rng, name, type)
|
|
||||||
|
enum rng_kind {
|
||||||
double rng_next_double(RNG algo);
|
RNG_KIND_FRNG,
|
||||||
int rng_next_int_bounded(RNG algo, int min, int max);
|
RNG_KIND_DRNG,
|
||||||
int rng_next_int(RNG algo, int max);
|
RNG_KIND_XORNG,
|
||||||
void* rng_next(RNG algo, void* data, int len);
|
RNG_KIND_LORENZ,
|
||||||
int rng_chance(RNG algo, double d);
|
};
|
||||||
void rng_free(RNG algo);
|
|
||||||
void rng_seed(RNG algo, void* seed);
|
typedef long double rng_st_lorenz_t;
|
||||||
RNG rng_new(RNG (*instantiate)(void));
|
typedef _Complex long double rng_lorenz_t;
|
||||||
|
|
||||||
#define RNG_IMPL_DEFINITION(name) RNG __rng_impl_ ## name(void)
|
typedef struct rng_init
|
||||||
#define RNG_ALGO(name) &__rng_impl_ ## name
|
{
|
||||||
#define RNG_NEW(name) rng_new(RNG_ALGO(name))
|
enum rng_kind kind;
|
||||||
|
union {
|
||||||
|
struct {
|
||||||
|
double state[2];
|
||||||
|
} frng;
|
||||||
|
struct {
|
||||||
|
int32_t state;
|
||||||
|
} drng;
|
||||||
|
struct {
|
||||||
|
uint64_t state[2];
|
||||||
|
} xorng;
|
||||||
|
struct {
|
||||||
|
rng_lorenz_t point;
|
||||||
|
const rng_st_lorenz_t (* _UNIQUE state)[5];
|
||||||
|
uint64_t iter;
|
||||||
|
} lorenz;
|
||||||
|
} init;
|
||||||
|
|
||||||
|
} rng_init_opt;
|
||||||
|
|
||||||
|
typedef struct rng_impl* _UNIQUE rng_t;
|
||||||
|
|
||||||
|
rng_t rng_new(rng_init_opt kind);
|
||||||
|
#define RNG_INIT(_kind,...) ((rng_init_opt){.kind=(_kind), .init.__VA_ARGS__ })
|
||||||
|
void rng_free(rng_t ptr);
|
||||||
|
|
||||||
|
// Tests
|
||||||
|
extern void rng_test();
|
||||||
|
extern void rng_test_spec(rng_t rng) __attribute__((nonnull(1)));
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
}
|
||||||
|
// RNG interfaces
|
||||||
|
#include "rng/xoroshiro128plus.hpp"
|
||||||
|
#include "rng/frng.hpp"
|
||||||
|
#include "rng/drng.hpp"
|
||||||
|
#include "rng/lorenz.hpp"
|
||||||
|
|
||||||
|
namespace rng {
|
||||||
|
void test_algo(RNG&& rng);
|
||||||
|
|
||||||
|
template<class R,
|
||||||
|
typename std::enable_if<std::is_base_of<RNG, R>::value>::value __fuck>
|
||||||
|
inline void test_algo(R&& rng) { test_algo(static_cast<RNG&&>(rng)); }
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
#endif /* _RNG_H */
|
#endif /* _RNG_H */
|
||||||
|
@ -0,0 +1,22 @@
|
|||||||
|
|
||||||
|
#include "impl.hpp"
|
||||||
|
#include <debug.h>
|
||||||
|
namespace rng
|
||||||
|
{
|
||||||
|
struct drng : public RNG
|
||||||
|
{
|
||||||
|
inline drng(std::uint32_t seed) : state(seed){
|
||||||
|
D_dprintf("drng: seeded with %u", seed);
|
||||||
|
//dprintf(" dummy run sample: %f", sample());
|
||||||
|
}
|
||||||
|
inline drng() : drng(1){}
|
||||||
|
|
||||||
|
static drng from_time();
|
||||||
|
|
||||||
|
int rand();
|
||||||
|
protected:
|
||||||
|
double sample() override;
|
||||||
|
private:
|
||||||
|
std::uint32_t state;
|
||||||
|
};
|
||||||
|
}
|
@ -0,0 +1,67 @@
|
|||||||
|
|
||||||
|
#include "impl.hpp"
|
||||||
|
#include <cmath>
|
||||||
|
|
||||||
|
#include <debug.h>
|
||||||
|
|
||||||
|
namespace rng
|
||||||
|
{
|
||||||
|
struct frng : public RNG
|
||||||
|
{
|
||||||
|
template<std::size_t N>
|
||||||
|
static constexpr inline double dot(const std::array<double, N>& v, const std::array<double, N>& u)
|
||||||
|
{
|
||||||
|
double res=0;
|
||||||
|
for(std::size_t i=0;i<N;i++)
|
||||||
|
{
|
||||||
|
res += v[i] * u[i];
|
||||||
|
}
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline constexpr double fract(double x)
|
||||||
|
{
|
||||||
|
return x - floor(x);
|
||||||
|
}
|
||||||
|
static inline constexpr double sample_double(const std::array<double, 2>& state)
|
||||||
|
{
|
||||||
|
const constexpr std::array<double, 2> vec2 = { 12.9898, 78.223 };
|
||||||
|
return fract(sin(dot(state, vec2)) * 43758.5453);
|
||||||
|
}
|
||||||
|
|
||||||
|
#define P D_dprintf("frng: seeded with (%f, %f)", state[0], state[1]);
|
||||||
|
inline constexpr frng(double s1, double s2) : state({s1, s2}){P}
|
||||||
|
inline constexpr frng(const std::array<double, 2>& ar) : state(ar){P}
|
||||||
|
inline constexpr frng(std::array<double, 2>&& ar) : state(ar){P}
|
||||||
|
inline constexpr frng(const double (&ar)[2]) : state({ar[0], ar[1]}) {P}
|
||||||
|
constexpr virtual ~frng() = default;
|
||||||
|
#undef P
|
||||||
|
inline constexpr double next_double() override { return sample(); }
|
||||||
|
inline constexpr float next_float() override { return (float)sample(); }
|
||||||
|
protected:
|
||||||
|
inline constexpr double sample() override
|
||||||
|
{
|
||||||
|
double res = sample_double(state);
|
||||||
|
update_state(state, res);
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
private:
|
||||||
|
std::array<double, 2> state;
|
||||||
|
static inline constexpr void update_state(std::array<double, 2>& state, double r)
|
||||||
|
{
|
||||||
|
float v1 = (float)state[0];
|
||||||
|
float v2 = (float)state[1];
|
||||||
|
|
||||||
|
std::array<double, 2> nvec = {
|
||||||
|
r,
|
||||||
|
(double)v2,
|
||||||
|
};
|
||||||
|
|
||||||
|
state[0] = sample_double(nvec);
|
||||||
|
|
||||||
|
nvec[1] = (double)v1;
|
||||||
|
state[1] = sample_double(nvec);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,42 @@
|
|||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include <array>
|
||||||
|
#include <vector>
|
||||||
|
#include <cstdint>
|
||||||
|
|
||||||
|
/// Base class for RNG impls
|
||||||
|
struct RNG {
|
||||||
|
constexpr RNG() noexcept = default;
|
||||||
|
|
||||||
|
virtual unsigned char byte();
|
||||||
|
virtual void bytes(unsigned char* ptr, std::size_t len);
|
||||||
|
|
||||||
|
template<std::size_t N>
|
||||||
|
inline void bytes(unsigned char (&arr)[N]) { return bytes(arr, N); }
|
||||||
|
template<std::size_t N>
|
||||||
|
inline void bytes(std::array<unsigned char, N>& array) { return bytes(&array[0], N); }
|
||||||
|
inline void bytes(std::vector<unsigned char>& vec) { return bytes(&vec[0], vec.size()); }
|
||||||
|
|
||||||
|
bool chance();
|
||||||
|
virtual bool chance(double chance);
|
||||||
|
|
||||||
|
virtual std::int32_t next_int();
|
||||||
|
inline std::int32_t next_int(std::int32_t max) { return next_int(0, max); }
|
||||||
|
std::int32_t next_int(std::int32_t min, std::int32_t max);
|
||||||
|
|
||||||
|
virtual std::int64_t next_long();
|
||||||
|
inline std::int64_t next_long(std::int64_t max) { return next_long(0, max); }
|
||||||
|
std::int64_t next_long(std::int64_t min, std::int64_t max);
|
||||||
|
|
||||||
|
inline virtual float next_float() { return (float)sample(); }
|
||||||
|
inline virtual double next_double() { return sample(); }
|
||||||
|
|
||||||
|
constexpr virtual ~RNG() = default;
|
||||||
|
|
||||||
|
//explicit operator rng_t() const noexcept;
|
||||||
|
//friend operator RNG*(rng_t) noexcept;
|
||||||
|
protected:
|
||||||
|
virtual double sample() = 0;
|
||||||
|
//private:
|
||||||
|
//struct rng_impl* _UNIQUE _held = nullptr;
|
||||||
|
};
|
@ -0,0 +1,2 @@
|
|||||||
|
#pragma once
|
||||||
|
|
@ -0,0 +1,31 @@
|
|||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include "impl.hpp"
|
||||||
|
#include <debug.h>
|
||||||
|
|
||||||
|
namespace rng
|
||||||
|
{
|
||||||
|
struct xoroshiro128plus : public RNG
|
||||||
|
{
|
||||||
|
using State = std::array<std::uint64_t, 2>;
|
||||||
|
#define P D_dprintf("xorng: seeded with (%lu, %lu)", state[0], state[1]);
|
||||||
|
inline constexpr xoroshiro128plus(std::uint64_t s0, std::uint64_t s1) : RNG(), state({s0, s1}){P}
|
||||||
|
inline constexpr xoroshiro128plus(std::array<std::uint64_t, 2>&& ar) : RNG(), state(ar){P}
|
||||||
|
inline constexpr xoroshiro128plus(const std::array<std::uint64_t, 2>& ar) : RNG(), state(ar){P}
|
||||||
|
inline constexpr xoroshiro128plus(const std::uint64_t (&ar)[2]) : RNG(), state({ar[0], ar[1]}){P}
|
||||||
|
inline virtual ~xoroshiro128plus() {}
|
||||||
|
#undef P
|
||||||
|
std::uint64_t next_ulong();
|
||||||
|
using RNG::next_long;
|
||||||
|
std::int64_t next_long() override;
|
||||||
|
|
||||||
|
void jump();
|
||||||
|
void long_jump();
|
||||||
|
protected:
|
||||||
|
double sample() override;
|
||||||
|
private:
|
||||||
|
State state;
|
||||||
|
};
|
||||||
|
static_assert(std::derived_from<xoroshiro128plus, RNG>, "Wtf???");
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,76 @@
|
|||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include <rng.h>
|
||||||
|
#include <reinterpret.h>
|
||||||
|
#include <algorithm>
|
||||||
|
#include <iostream>
|
||||||
|
|
||||||
|
#include <fsvec.hpp>
|
||||||
|
|
||||||
|
#include <shuffle3.h>
|
||||||
|
|
||||||
|
namespace rng {
|
||||||
|
template<typename T, typename R>
|
||||||
|
inline void shuffle(R& rng, span<T> span)
|
||||||
|
{
|
||||||
|
if(!span.size()) return;
|
||||||
|
std::cout << " -> shuffling " << span.size() << " objects..." << std::flush;
|
||||||
|
for(std::size_t i=span.size()-1;i>0;i--)
|
||||||
|
{
|
||||||
|
auto j = rng.next_long(i);
|
||||||
|
std::swap(span[i], span[j]);
|
||||||
|
}
|
||||||
|
std::cout << " OK" << std::endl;
|
||||||
|
}
|
||||||
|
|
||||||
|
namespace {
|
||||||
|
extern "C" int _can_allocate(std::size_t bytes);
|
||||||
|
template<typename T>
|
||||||
|
inline bool can_allocate(std::size_t len)
|
||||||
|
{
|
||||||
|
return !!_can_allocate(len*sizeof(T));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename T, typename R>
|
||||||
|
inline void unshuffle(R& rng, span<T> span)
|
||||||
|
{
|
||||||
|
if(!span.size()) return;
|
||||||
|
|
||||||
|
#define DYN 2
|
||||||
|
#define MAP 3
|
||||||
|
#if defined(_FS_SPILL_BUFFER) && (_FS_SPILL_BUFFER == DYN)
|
||||||
|
D_dprintf("spill=dyn");
|
||||||
|
dynamic_spill_vector<std::size_t> rng_values =
|
||||||
|
//can_allocate<std::size_t>(span.size()) //Is there any way we can not waste this malloc() when it's valid?
|
||||||
|
span.size() <= FSV_DEFAULT_SPILL_AT
|
||||||
|
? dynamic_spill_vector<std::size_t> (span.size(), FSV_DEFAULT_SPILL_AT)
|
||||||
|
: dynamic_spill_vector<std::size_t> (FSV_DEFAULT_SPILL_AT);
|
||||||
|
#elif defined(_FS_SPILL_BUFFER) && (_FS_SPILL_BUFFER == MAP)
|
||||||
|
D_dprintf("spill=map");
|
||||||
|
mapped_spill_vector<std::size_t> rng_values =
|
||||||
|
span.size() <= FSV_DEFAULT_SPILL_AT
|
||||||
|
? mapped_spill_vector<std::size_t> (span.size(), FSV_DEFAULT_SPILL_AT)
|
||||||
|
: mapped_spill_vector<std::size_t> (FSV_DEFAULT_SPILL_AT, mapped_vector<std::size_t>::from_temp(span.size() - FSV_DEFAULT_SPILL_AT));
|
||||||
|
#elif defined(_FS_SPILL_BUFFER)
|
||||||
|
D_dprintf("spill=static");
|
||||||
|
fixed_spill_vector<std::size_t> rng_values;
|
||||||
|
#else
|
||||||
|
D_dprintf("spill=none");
|
||||||
|
std::vector<std::size_t> rng_values;
|
||||||
|
rng_values.reserve(span.size());
|
||||||
|
#endif
|
||||||
|
#undef MAP
|
||||||
|
#undef DYN
|
||||||
|
|
||||||
|
std::cout << " -> unshuffling " << span.size() << " objects..." << std::flush;
|
||||||
|
for(std::size_t i=span.size()-1;i>0;i--)
|
||||||
|
rng_values.push_back(rng.next_long(i));
|
||||||
|
|
||||||
|
for(std::size_t i=1;i<span.size();i++) {
|
||||||
|
std::swap(span[i], span[rng_values.back()]);
|
||||||
|
rng_values.pop_back();
|
||||||
|
}
|
||||||
|
std::cout << " OK" << std::endl;
|
||||||
|
}
|
||||||
|
}
|
@ -0,0 +1,42 @@
|
|||||||
|
#ifndef _SHUFFLE3_H
|
||||||
|
#define _SHUFFLE3_H
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
#define _UNIQUE __restrict
|
||||||
|
extern "C" {
|
||||||
|
#else
|
||||||
|
#define _UNIQUE restrict
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#ifdef DEBUG
|
||||||
|
#define _FORCE_INLINE __attribute__((gnu_inline)) static inline
|
||||||
|
#else
|
||||||
|
#define _FORCE_INLINE __attribute__((gnu_inline)) extern inline
|
||||||
|
#endif
|
||||||
|
|
||||||
|
//** Features **//
|
||||||
|
|
||||||
|
//#define _FS_SPILL_BUFFER /* Use a file-backed buffer when unshuffling in cases of too high memory usage. Will cause massive slowdowns but can stop OOMs when unshuffling large files */
|
||||||
|
|
||||||
|
/// When to spill a file-backed buffer onto the fs (only used when `_FS_SPILL_BUFFER` is enabled).
|
||||||
|
#define FSV_DEFAULT_SPILL_AT ((1024 * 1024) * 10) //10MB
|
||||||
|
|
||||||
|
/*
|
||||||
|
#ifdef _FS_SPILL_BUFFER
|
||||||
|
#define DYN 1
|
||||||
|
#if _FS_SPILL_BUFFER == DYN
|
||||||
|
#undef _FS_SPILL_BUFFER
|
||||||
|
#define _FS_SPILL_BUFFER DYN
|
||||||
|
#endif
|
||||||
|
#undef DYN
|
||||||
|
#endif
|
||||||
|
*/
|
||||||
|
//** Globals *//
|
||||||
|
|
||||||
|
extern const char* _prog_name;
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#endif /* _SHUFFLE3_H */
|
@ -0,0 +1,49 @@
|
|||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include <filesystem>
|
||||||
|
#include <string>
|
||||||
|
#include <utility>
|
||||||
|
#include "uuid.hpp"
|
||||||
|
|
||||||
|
#include <debug.h>
|
||||||
|
|
||||||
|
using std::size_t;
|
||||||
|
namespace fs = std::filesystem;
|
||||||
|
|
||||||
|
/// A temporary file name
|
||||||
|
struct temp_file
|
||||||
|
{
|
||||||
|
inline temp_file(const temp_file& c) = delete;
|
||||||
|
|
||||||
|
inline temp_file(temp_file&& m) : name(std::move(m.name)), _full_path(std::move(m._full_path)) {}
|
||||||
|
inline temp_file() : name(uuid::generate().to_string()+"-s3"){}
|
||||||
|
inline temp_file(const char* name) : name(name) {}
|
||||||
|
inline temp_file(std::string&& name) : name(name) {}
|
||||||
|
|
||||||
|
inline ~temp_file()
|
||||||
|
{
|
||||||
|
if(name.empty() && _full_path.empty()) return;
|
||||||
|
|
||||||
|
D_dprintf("~tempfile(): %s", _full_path.c_str());
|
||||||
|
|
||||||
|
if(!_full_path.empty() && fs::exists(_full_path) ) {
|
||||||
|
D_dprintf("tfile removing: %s", _full_path.c_str());
|
||||||
|
fs::remove(_full_path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
inline const fs::path& full_path() const
|
||||||
|
{
|
||||||
|
if(_full_path.empty()) {
|
||||||
|
_full_path = fs::absolute( fs::temp_directory_path() / name );
|
||||||
|
D_dprintf("tfile path: %s", _full_path.c_str());
|
||||||
|
}
|
||||||
|
return _full_path;
|
||||||
|
}
|
||||||
|
inline const std::string& base_name() const { return name; }
|
||||||
|
|
||||||
|
inline const fs::path* operator->() const { return &full_path(); }
|
||||||
|
private:
|
||||||
|
std::string name;
|
||||||
|
mutable fs::path _full_path;
|
||||||
|
};
|
@ -0,0 +1,32 @@
|
|||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include <array>
|
||||||
|
#include <random>
|
||||||
|
#include <string>
|
||||||
|
|
||||||
|
struct uuid //not to spec but idc
|
||||||
|
{
|
||||||
|
const static constexpr int SIZE=16;
|
||||||
|
inline static uuid generate()
|
||||||
|
{
|
||||||
|
using namespace std;
|
||||||
|
static thread_local random_device dev;
|
||||||
|
static thread_local mt19937 rng(dev());
|
||||||
|
uniform_int_distribution<int> dist(0, 15);
|
||||||
|
constexpr const char hex[] = "0123456789abcdef";
|
||||||
|
uuid id;
|
||||||
|
for(int i=0;i<SIZE;i++) id.str[i] = hex[dist(rng)];
|
||||||
|
id.str[SIZE] =0;
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
inline operator const char*() const { return &str[0]; }
|
||||||
|
inline operator const std::array<char, SIZE+1>&() const { return str; }
|
||||||
|
|
||||||
|
inline std::string to_string() const
|
||||||
|
{
|
||||||
|
return std::string(&str[0]);
|
||||||
|
}
|
||||||
|
private:
|
||||||
|
inline uuid(){}
|
||||||
|
std::array<char, SIZE+1> str;
|
||||||
|
};
|
@ -0,0 +1,52 @@
|
|||||||
|
#ifndef _WORK_H
|
||||||
|
#define _WORK_H
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
extern "C" {
|
||||||
|
#endif
|
||||||
|
|
||||||
|
enum work_buffer_opts {
|
||||||
|
WORK_BO_CPIP=0, // Copy then run in place on output file (default)
|
||||||
|
WORK_BO_BUFFERED=1, // Read whole input into memory then perform
|
||||||
|
};
|
||||||
|
|
||||||
|
typedef struct _work_args {
|
||||||
|
enum {
|
||||||
|
OP_SHUFFLE_IP, // Shuffle in place
|
||||||
|
OP_SHUFFLE_OP, // Shuffle out of place
|
||||||
|
|
||||||
|
OP_UNSHUFFLE_IP, // Unshuffle in place
|
||||||
|
OP_UNSHUFFLE_OP, // Unshuffle out of place
|
||||||
|
|
||||||
|
OP_HELP, // Print help then exit
|
||||||
|
} op;
|
||||||
|
union {
|
||||||
|
struct {
|
||||||
|
const char* file;
|
||||||
|
} op_shuffle_ip;
|
||||||
|
|
||||||
|
struct {
|
||||||
|
enum work_buffer_opts buffered;
|
||||||
|
const char* ifile;
|
||||||
|
const char* ofile;
|
||||||
|
} op_shuffle_op;
|
||||||
|
|
||||||
|
struct {
|
||||||
|
const char* file;
|
||||||
|
} op_unshuffle_ip;
|
||||||
|
|
||||||
|
struct {
|
||||||
|
enum work_buffer_opts buffered;
|
||||||
|
const char* ifile;
|
||||||
|
const char* ofile;
|
||||||
|
} op_unshuffle_op;
|
||||||
|
} data;
|
||||||
|
} work_args_t;
|
||||||
|
|
||||||
|
int do_work(const work_args_t args);
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#endif /* _WORK_H */
|
@ -0,0 +1,50 @@
|
|||||||
|
SRC = $(wildcard src/*.c)
|
||||||
|
INCLUDE = include/
|
||||||
|
|
||||||
|
PROJECT=shuffle3
|
||||||
|
|
||||||
|
BUILD=build
|
||||||
|
|
||||||
|
OPT_FLAGS?= -march=native -fgraphite -fopenmp -floop-parallelize-all -ftree-parallelize-loops=4 \
|
||||||
|
-floop-interchange -ftree-loop-distribution -floop-strip-mine -floop-block
|
||||||
|
|
||||||
|
CFLAGS+= $(addprefix -I,$(INCLUDE)) -Wall -pedantic --std=gnu11
|
||||||
|
LDFLAGS+= -lm
|
||||||
|
|
||||||
|
RELEASE_CFLAGS?= -O3 -flto $(OPT_FLAGS)
|
||||||
|
RELEASE_LDFLAGS?= -O3 -flto
|
||||||
|
|
||||||
|
DEBUG_CFLAGS?= -g -O0
|
||||||
|
DEBUG_LDFLAGS?= -O0
|
||||||
|
|
||||||
|
STRIP=strip
|
||||||
|
|
||||||
|
OBJ = $(addprefix obj/,$(SRC:.c=.o))
|
||||||
|
|
||||||
|
.PHONY: release
|
||||||
|
release: | dirs $(BUILD)/$(PROJECT)-release
|
||||||
|
|
||||||
|
.PHONY: debug
|
||||||
|
debug: | dirs $(BUILD)/$(PROJECT)-debug
|
||||||
|
|
||||||
|
dirs:
|
||||||
|
@mkdir -p obj/src
|
||||||
|
@mkdir -p $(BUILD)
|
||||||
|
|
||||||
|
obj/%.o: %.c
|
||||||
|
$(CC) -c $< $(CFLAGS) -o $@ $(LDFLAGS)
|
||||||
|
|
||||||
|
$(BUILD)/$(PROJECT)-release: CFLAGS+= $(RELEASE_CFLAGS)
|
||||||
|
$(BUILD)/$(PROJECT)-release: LDFLAGS+= $(RELEASE_LDFLAGS)
|
||||||
|
$(BUILD)/$(PROJECT)-release: $(OBJ)
|
||||||
|
$(CC) $^ $(CFLAGS) -o $@ $(LDFLAGS)
|
||||||
|
$(STRIP) $@
|
||||||
|
|
||||||
|
$(BUILD)/$(PROJECT)-debug: CFLAGS+= $(DEBUG_CFLAGS)
|
||||||
|
$(BUILD)/$(PROJECT)-debug: LDFLAGS+= $(DEBUG_LDFLAGS)
|
||||||
|
$(BUILD)/$(PROJECT)-debug: $(OBJ)
|
||||||
|
$(CC) $^ $(CFLAGS) -o $@ $(LDFLAGS)
|
||||||
|
|
||||||
|
clean:
|
||||||
|
rm -rf obj
|
||||||
|
rm -f $(BUILD)/*
|
@ -0,0 +1,25 @@
|
|||||||
|
#ifndef _RNG_H
|
||||||
|
#define _RNG_H
|
||||||
|
|
||||||
|
typedef struct rng_algo *RNG; //RNG algorithm reference
|
||||||
|
|
||||||
|
#define rng_reinterpret(rng, value, type) (*((type*)rng_next(rng, &value, sizeof(type))))
|
||||||
|
#define rng_reinterpret_new(rng, value, type) (*((type*)rng_next(rng, malloc(sizeof(type)), sizeof(type))))
|
||||||
|
#define rng_reinterpret_stackalloc(rng, value, type) rng_reinterpret(rng, alloca(sizeof(type)), type)
|
||||||
|
|
||||||
|
#define RNG_LEXBIND(rng, type, name) type name = rng_reinterpret(rng, name, type)
|
||||||
|
|
||||||
|
double rng_next_double(RNG algo);
|
||||||
|
int rng_next_int_bounded(RNG algo, int min, int max);
|
||||||
|
int rng_next_int(RNG algo, int max);
|
||||||
|
void* rng_next(RNG algo, void* data, int len);
|
||||||
|
int rng_chance(RNG algo, double d);
|
||||||
|
void rng_free(RNG algo);
|
||||||
|
void rng_seed(RNG algo, void* seed);
|
||||||
|
RNG rng_new(RNG (*instantiate)(void));
|
||||||
|
|
||||||
|
#define RNG_IMPL_DEFINITION(name) RNG __rng_impl_ ## name(void)
|
||||||
|
#define RNG_ALGO(name) &__rng_impl_ ## name
|
||||||
|
#define RNG_NEW(name) rng_new(RNG_ALGO(name))
|
||||||
|
|
||||||
|
#endif /* _RNG_H */
|
@ -0,0 +1,373 @@
|
|||||||
|
#include <stdio.h>
|
||||||
|
#include <float.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
#include <string.h>
|
||||||
|
#include <assert.h>
|
||||||
|
|
||||||
|
#include <rng.h>
|
||||||
|
#include <rng_algos.h>
|
||||||
|
#include <ptr_store.h>
|
||||||
|
#include <array.h>
|
||||||
|
|
||||||
|
_Static_assert(sizeof(float)==4, "Float must be 32 bits.");
|
||||||
|
|
||||||
|
int surpress_out=0;
|
||||||
|
|
||||||
|
/*void test_destructor(object* ptr)
|
||||||
|
{
|
||||||
|
printf("die\n");
|
||||||
|
}*/
|
||||||
|
void _rng_object_destructor(object* obj)
|
||||||
|
{
|
||||||
|
rng_free(obj->state);
|
||||||
|
}
|
||||||
|
RNG rng_object(S_LEXENV, RNG rng)
|
||||||
|
{
|
||||||
|
object proto = OBJ_PROTO;
|
||||||
|
proto.state = rng;
|
||||||
|
proto.destructor = &_rng_object_destructor;
|
||||||
|
|
||||||
|
return snew_obj(proto)->state;
|
||||||
|
}
|
||||||
|
|
||||||
|
void shuffle(RNG with, array_t data)
|
||||||
|
{
|
||||||
|
if(!surpress_out) printf(" -> shuffling %d objects...", (int)ar_size(data));
|
||||||
|
for(int i=ar_size(data)-1;i>0;i--)
|
||||||
|
{
|
||||||
|
int j = rng_next_int(with, i);
|
||||||
|
ar_swap(data, i, j);
|
||||||
|
}
|
||||||
|
if(!surpress_out) printf(" Okay\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
void unshuffle(RNG with, array_t data)
|
||||||
|
{
|
||||||
|
int rng_values[ar_size(data)-1];
|
||||||
|
int k=0;
|
||||||
|
if(!surpress_out) printf(" -> unshuffling %d objects...", (int)ar_size(data));
|
||||||
|
for(int i=ar_size(data)-1;i>0;i--)
|
||||||
|
rng_values[k++] = rng_next_int(with, i);
|
||||||
|
|
||||||
|
for(int i=1;i<ar_size(data);i++)
|
||||||
|
{
|
||||||
|
ar_swap(data, i, rng_values[--k]);
|
||||||
|
}
|
||||||
|
if(!surpress_out) printf(" Okay\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
void minmax_int64_ts(int64_t* min, int64_t* max, const array_t data)
|
||||||
|
{
|
||||||
|
for(register int i=0;i<ar_size(data);i++)
|
||||||
|
{
|
||||||
|
if(ar_get_v(data, int64_t, i) > *max) *max = ar_get_v(data, int64_t, i);
|
||||||
|
if(ar_get_v(data, int64_t, i) < *min) *min = ar_get_v(data, int64_t, i);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
int valid_float(float f)
|
||||||
|
{
|
||||||
|
return !( (f!=f) || (f< -FLT_MAX || f> FLT_MAX));
|
||||||
|
|
||||||
|
}
|
||||||
|
void minmax_floats(float* min, float* max, const array_t data)
|
||||||
|
{
|
||||||
|
for(register int i=0;i<ar_size(data);i++)
|
||||||
|
{
|
||||||
|
if(!valid_float(ar_get_v(data,float,i))) continue;
|
||||||
|
if(ar_get_v(data, float, i) > *max) *max = ar_get_v(data, float, i);
|
||||||
|
if(ar_get_v(data, float, i) < *min) *min = ar_get_v(data, float, i);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void minmax_sbytes(int8_t* min, int8_t* max, const array_t data)
|
||||||
|
{
|
||||||
|
for(register int i=0;i<ar_size(data);i++)
|
||||||
|
{
|
||||||
|
if(ar_get_v(data, int8_t, i) > *max) *max = ar_get_v(data, int8_t, i);
|
||||||
|
if(ar_get_v(data, int8_t, i) < *min) *min = ar_get_v(data, int8_t, i);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void print_array(const array_t data)
|
||||||
|
{
|
||||||
|
printf("---%d elements---\n", (int)ar_size(data));
|
||||||
|
for(register int i=0;i<ar_size(data);i++)
|
||||||
|
printf("%d ", (int)ar_get_v(data, unsigned char, i));
|
||||||
|
printf("\n---\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
void unshuffle3(S_LEXENV, array_t data)
|
||||||
|
{
|
||||||
|
if(ar_type(data)!=sizeof(int8_t))
|
||||||
|
ar_reinterpret(data, sizeof(int8_t));
|
||||||
|
RNG xoro = S_LEXCALL(rng_object, rng_new(RNG_ALGO(xoroshiro128plus)));
|
||||||
|
RNG frng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(frng)));
|
||||||
|
RNG drng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(drng)));
|
||||||
|
|
||||||
|
int8_t bmax = INT8_MIN;
|
||||||
|
int8_t bmin = INT8_MAX;
|
||||||
|
minmax_sbytes(&bmin, &bmax, data);
|
||||||
|
uint32_t seed = (0xfffa << 16) | (bmin<<8) | bmax;
|
||||||
|
rng_seed(drng, &seed);
|
||||||
|
unshuffle(drng, data);
|
||||||
|
|
||||||
|
float fmin = (float)DBL_MAX;
|
||||||
|
float fmax = (float)-DBL_MAX;
|
||||||
|
ar_reinterpret(data, sizeof(float));
|
||||||
|
minmax_floats(&fmin, &fmax, data);
|
||||||
|
double fseed[2];
|
||||||
|
fseed[0] = fmin;
|
||||||
|
fseed[1] = fmax;
|
||||||
|
rng_seed(frng, fseed);
|
||||||
|
unshuffle(frng, data);
|
||||||
|
|
||||||
|
int64_t min = INT64_MAX;
|
||||||
|
int64_t max = INT64_MIN;
|
||||||
|
ar_reinterpret(data, sizeof(int64_t));
|
||||||
|
minmax_int64_ts(&min, &max, data);
|
||||||
|
uint64_t xseed[2];
|
||||||
|
xseed[0] = min;
|
||||||
|
xseed[1] = max;
|
||||||
|
rng_seed(xoro, xseed);
|
||||||
|
unshuffle(xoro, data);
|
||||||
|
}
|
||||||
|
void shuffle3(S_LEXENV, array_t data)
|
||||||
|
{
|
||||||
|
if(ar_type(data)!=sizeof(int64_t))
|
||||||
|
ar_reinterpret(data, sizeof(int64_t));
|
||||||
|
RNG xoro = S_LEXCALL(rng_object, rng_new(RNG_ALGO(xoroshiro128plus)));
|
||||||
|
RNG frng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(frng)));
|
||||||
|
RNG drng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(drng)));
|
||||||
|
|
||||||
|
int64_t min = INT64_MAX;
|
||||||
|
int64_t max = INT64_MIN;
|
||||||
|
|
||||||
|
float fmin = (float)DBL_MAX;
|
||||||
|
float fmax = (float)-DBL_MAX;
|
||||||
|
|
||||||
|
int8_t bmax = INT8_MIN;
|
||||||
|
int8_t bmin = INT8_MAX;
|
||||||
|
|
||||||
|
minmax_int64_ts(&min, &max, data);
|
||||||
|
uint64_t xseed[2];
|
||||||
|
xseed[0] = min;
|
||||||
|
xseed[1] = max;
|
||||||
|
|
||||||
|
rng_seed(xoro, xseed);
|
||||||
|
shuffle(xoro, data);
|
||||||
|
|
||||||
|
ar_reinterpret(data, sizeof(float));
|
||||||
|
minmax_floats(&fmin, &fmax, data);
|
||||||
|
|
||||||
|
double fseed[2];
|
||||||
|
fseed[0] = fmin;
|
||||||
|
fseed[1] = fmax;
|
||||||
|
rng_seed(frng, fseed);
|
||||||
|
shuffle(frng, data);
|
||||||
|
|
||||||
|
ar_reinterpret(data,sizeof(int8_t));
|
||||||
|
minmax_sbytes(&bmin, &bmax, data);
|
||||||
|
uint32_t seed = (0xfffa << 16) | (bmin<<8) | bmax;
|
||||||
|
rng_seed(drng, &seed);
|
||||||
|
shuffle(drng, data);
|
||||||
|
}
|
||||||
|
|
||||||
|
void print_usage(char** argv)
|
||||||
|
{
|
||||||
|
printf("Usage: %s -[b]s <file> [<outfile>]\nUsage: %s -[b]u <file> [<outfile>]\n", argv[0], argv[0]);
|
||||||
|
printf("Usage: %s -S <string>|-\nUsage: %s -U <string>|-\n", argv[0], argv[0]);
|
||||||
|
printf(" <file>\t\tFile to shuffle\n");
|
||||||
|
printf(" <outfile>\tOutput file (only valid for buffered mode)\n");
|
||||||
|
printf(" <string>\tThe string to shuffle (for -(S|U)). If `-`, read from stdin\n");
|
||||||
|
printf("\ts\tShuffle\n");
|
||||||
|
printf("\tu\tReverse shuffle\n");
|
||||||
|
printf("\tS\tShuffle string\n");
|
||||||
|
printf("\tU\tReverse shuffle string\n");
|
||||||
|
printf("\n\tb\tUse buffered mode instead of shuffling in place (must specify output file)\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
array_t read_whole_file(S_NAMED_LEXENV(base), FILE* fp)
|
||||||
|
{
|
||||||
|
fseek(fp,0,SEEK_END);
|
||||||
|
int64_t sz = ftell(fp);
|
||||||
|
fseek(fp,0,SEEK_SET);
|
||||||
|
|
||||||
|
array_t ar;
|
||||||
|
MANAGED({
|
||||||
|
void* buf = smalloc(sz);
|
||||||
|
|
||||||
|
register size_t read;
|
||||||
|
register size_t full=0;
|
||||||
|
while ((read=fread(buf, 1, sz-full, fp))>0)
|
||||||
|
full+=read;
|
||||||
|
|
||||||
|
assert(full == sz);
|
||||||
|
|
||||||
|
ar = ar_create_memory_from(base, buf, 1, sz);
|
||||||
|
});
|
||||||
|
return ar;
|
||||||
|
}
|
||||||
|
|
||||||
|
#define BUFSIZE 512
|
||||||
|
#define BUFMIN 128
|
||||||
|
|
||||||
|
char* read_stdin()
|
||||||
|
{
|
||||||
|
char* input, *p;
|
||||||
|
int len,remain,n,size;
|
||||||
|
|
||||||
|
size = BUFSIZE;
|
||||||
|
input = malloc(size);
|
||||||
|
memset(input,0,size);
|
||||||
|
len=0;
|
||||||
|
remain = size;
|
||||||
|
|
||||||
|
while(!feof(stdin))
|
||||||
|
{
|
||||||
|
if(remain<= BUFMIN)
|
||||||
|
{
|
||||||
|
remain+=size;
|
||||||
|
size *=2;
|
||||||
|
p = realloc(input, size);
|
||||||
|
if(p==NULL) {
|
||||||
|
free(input);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
input = p;
|
||||||
|
}
|
||||||
|
fgets(input+len, remain, stdin);
|
||||||
|
n+=strlen(input+len);
|
||||||
|
len+=n;
|
||||||
|
remain-=n;
|
||||||
|
}
|
||||||
|
return input;
|
||||||
|
}
|
||||||
|
|
||||||
|
int string_shuffle(char* string, int un)
|
||||||
|
{
|
||||||
|
if(!string)
|
||||||
|
{
|
||||||
|
char* ptr = read_stdin();
|
||||||
|
if(!ptr) {
|
||||||
|
printf("! read from stdin failed\n");
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
int ret = string_shuffle(ptr, un);
|
||||||
|
free(ptr);
|
||||||
|
return ret;
|
||||||
|
}
|
||||||
|
else if(strcmp(string, "-") == 0)
|
||||||
|
{
|
||||||
|
return string_shuffle(NULL, un);
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
surpress_out=1;
|
||||||
|
MANAGED({
|
||||||
|
array_t ar = ar_create_memory_from(LEXENV, string, 1, strlen(string));
|
||||||
|
|
||||||
|
if(un) unshuffle3(LEXENV, ar);
|
||||||
|
else shuffle3(LEXENV, ar);
|
||||||
|
|
||||||
|
ar_reinterpret(ar, sizeof(char));
|
||||||
|
|
||||||
|
for(int i=0;i<ar_size(ar);i++)
|
||||||
|
{
|
||||||
|
printf("%c", ar_get_v(ar, char, i));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(int argc, char** argv)
|
||||||
|
{
|
||||||
|
int is_buffered=0;
|
||||||
|
int i=1;
|
||||||
|
int is_unshuffling =0;
|
||||||
|
if(!argv[1] || !argv[2] || argv[1][0] != '-')
|
||||||
|
{
|
||||||
|
print_usage(argv);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
do_switch:
|
||||||
|
switch(argv[1][i])
|
||||||
|
{
|
||||||
|
case 'b':
|
||||||
|
if(is_buffered)
|
||||||
|
{
|
||||||
|
print_usage(argv);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
is_buffered = 1;
|
||||||
|
i+=1;
|
||||||
|
goto do_switch;
|
||||||
|
case 'u':
|
||||||
|
is_unshuffling=1;
|
||||||
|
break;
|
||||||
|
case 's':
|
||||||
|
break;
|
||||||
|
case 'S':
|
||||||
|
return string_shuffle(argv[2], 0);
|
||||||
|
case 'U':
|
||||||
|
return string_shuffle(argv[2], 1);
|
||||||
|
default:
|
||||||
|
print_usage(argv);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if(is_buffered && !argv[3])
|
||||||
|
{
|
||||||
|
printf("Buffered option requires an output file\n");
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
MANAGED_RETURNABLE(int ,{
|
||||||
|
array_t array;
|
||||||
|
|
||||||
|
if(is_buffered)
|
||||||
|
{
|
||||||
|
FILE* infile = fopen(argv[2], "rb");
|
||||||
|
if(!infile)
|
||||||
|
{
|
||||||
|
printf("! could not open file for reading\n");
|
||||||
|
MANAGED_RETURN(1);
|
||||||
|
}
|
||||||
|
array = read_whole_file(LEXENV, infile);
|
||||||
|
fclose(infile);
|
||||||
|
printf(" buffered file (%ld bytes)\n", ar_size(array));
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
FILE* infile = fopen(argv[2], "r+b");
|
||||||
|
if(!infile)
|
||||||
|
{
|
||||||
|
printf("! could not open file for reading+writing\n");
|
||||||
|
MANAGED_RETURN(1);
|
||||||
|
}
|
||||||
|
array = ar_create_file(LEXENV, infile, 1, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
//print_array(array);
|
||||||
|
|
||||||
|
if(is_unshuffling) unshuffle3(LEXENV, array);
|
||||||
|
else shuffle3(LEXENV, array);
|
||||||
|
|
||||||
|
if(is_buffered)
|
||||||
|
{
|
||||||
|
FILE* outfile = fopen(argv[3], "wb");
|
||||||
|
if(!outfile)
|
||||||
|
{
|
||||||
|
printf("! could not open outfile for writing\n");
|
||||||
|
MANAGED_RETURN(1);
|
||||||
|
}
|
||||||
|
void* wbuf = smalloc(ar_full_size(array));
|
||||||
|
if(!ar_ndump(array, wbuf, ar_full_size(array), 0, ar_size(array)))
|
||||||
|
printf("W memory dump failed, continuing anyway\n");
|
||||||
|
fwrite(wbuf, 1, ar_full_size(array), outfile);
|
||||||
|
fclose(outfile);
|
||||||
|
printf(" write completed (%d bytes)\n", (int)ar_size(array));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return 0;
|
||||||
|
}
|
After Width: | Height: | Size: 5.0 KiB |
After Width: | Height: | Size: 5.1 KiB |
After Width: | Height: | Size: 41 KiB |
After Width: | Height: | Size: 25 KiB |
After Width: | Height: | Size: 42 KiB |
After Width: | Height: | Size: 51 KiB |
@ -0,0 +1,567 @@
|
|||||||
|
desc: (none)
|
||||||
|
cmd: ./shuffle3-release small-u
|
||||||
|
time_unit: i
|
||||||
|
#-----------
|
||||||
|
snapshot=0
|
||||||
|
#-----------
|
||||||
|
time=0
|
||||||
|
mem_heap_B=0
|
||||||
|
mem_heap_extra_B=0
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=1
|
||||||
|
#-----------
|
||||||
|
time=4048133
|
||||||
|
mem_heap_B=32816
|
||||||
|
mem_heap_extra_B=8
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=2
|
||||||
|
#-----------
|
||||||
|
time=4050820
|
||||||
|
mem_heap_B=8
|
||||||
|
mem_heap_extra_B=16
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=3
|
||||||
|
#-----------
|
||||||
|
time=4057134
|
||||||
|
mem_heap_B=72712
|
||||||
|
mem_heap_extra_B=24
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=4
|
||||||
|
#-----------
|
||||||
|
time=4282565
|
||||||
|
mem_heap_B=73736
|
||||||
|
mem_heap_extra_B=32
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=5
|
||||||
|
#-----------
|
||||||
|
time=4289663
|
||||||
|
mem_heap_B=74248
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=6
|
||||||
|
#-----------
|
||||||
|
time=4295729
|
||||||
|
mem_heap_B=75784
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=7
|
||||||
|
#-----------
|
||||||
|
time=4303612
|
||||||
|
mem_heap_B=77832
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=8
|
||||||
|
#-----------
|
||||||
|
time=4318766
|
||||||
|
mem_heap_B=86024
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=9
|
||||||
|
#-----------
|
||||||
|
time=4322930
|
||||||
|
mem_heap_B=81928
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=10
|
||||||
|
#-----------
|
||||||
|
time=4353188
|
||||||
|
mem_heap_B=98312
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=11
|
||||||
|
#-----------
|
||||||
|
time=4361448
|
||||||
|
mem_heap_B=90120
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=12
|
||||||
|
#-----------
|
||||||
|
time=4421914
|
||||||
|
mem_heap_B=122888
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=13
|
||||||
|
#-----------
|
||||||
|
time=4438366
|
||||||
|
mem_heap_B=122888
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=detailed
|
||||||
|
n3: 122888 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||||
|
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||||
|
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||||
|
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||||
|
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x1: ???
|
||||||
|
n1: 72704 0x1FFF000AC6: ???
|
||||||
|
n0: 72704 0x1FFF000AD9: ???
|
||||||
|
n1: 49152 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n1: 49152 0x109385: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n0: 49152 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||||
|
n0: 1032 in 3 places, all below massif's threshold (1.00%)
|
||||||
|
#-----------
|
||||||
|
snapshot=14
|
||||||
|
#-----------
|
||||||
|
time=4438366
|
||||||
|
mem_heap_B=106504
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=15
|
||||||
|
#-----------
|
||||||
|
time=4559248
|
||||||
|
mem_heap_B=172040
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=16
|
||||||
|
#-----------
|
||||||
|
time=4592084
|
||||||
|
mem_heap_B=172040
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=detailed
|
||||||
|
n3: 172040 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||||
|
n1: 98304 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n1: 98304 0x109385: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n0: 98304 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||||
|
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||||
|
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||||
|
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x1: ???
|
||||||
|
n1: 72704 0x1FFF000AC6: ???
|
||||||
|
n0: 72704 0x1FFF000AD9: ???
|
||||||
|
n0: 1032 in 3 places, all below massif's threshold (1.00%)
|
||||||
|
#-----------
|
||||||
|
snapshot=17
|
||||||
|
#-----------
|
||||||
|
time=4592084
|
||||||
|
mem_heap_B=139272
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=18
|
||||||
|
#-----------
|
||||||
|
time=4833798
|
||||||
|
mem_heap_B=270344
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=19
|
||||||
|
#-----------
|
||||||
|
time=4899402
|
||||||
|
mem_heap_B=270344
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=detailed
|
||||||
|
n3: 270344 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||||
|
n1: 196608 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n1: 196608 0x109385: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n0: 196608 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||||
|
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||||
|
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||||
|
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x1: ???
|
||||||
|
n1: 72704 0x1FFF000AC6: ???
|
||||||
|
n0: 72704 0x1FFF000AD9: ???
|
||||||
|
n0: 1032 in 3 places, all below massif's threshold (1.00%)
|
||||||
|
#-----------
|
||||||
|
snapshot=20
|
||||||
|
#-----------
|
||||||
|
time=4899402
|
||||||
|
mem_heap_B=204808
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=21
|
||||||
|
#-----------
|
||||||
|
time=5382780
|
||||||
|
mem_heap_B=466952
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=22
|
||||||
|
#-----------
|
||||||
|
time=5513920
|
||||||
|
mem_heap_B=466952
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=peak
|
||||||
|
n3: 466952 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||||
|
n1: 393216 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n1: 393216 0x109385: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n0: 393216 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||||
|
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||||
|
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||||
|
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x1: ???
|
||||||
|
n1: 72704 0x1FFF000AC6: ???
|
||||||
|
n0: 72704 0x1FFF000AD9: ???
|
||||||
|
n0: 1032 in 3 places, all below massif's threshold (1.00%)
|
||||||
|
#-----------
|
||||||
|
snapshot=23
|
||||||
|
#-----------
|
||||||
|
time=5513920
|
||||||
|
mem_heap_B=335880
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=24
|
||||||
|
#-----------
|
||||||
|
time=5892087
|
||||||
|
mem_heap_B=73736
|
||||||
|
mem_heap_extra_B=32
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=25
|
||||||
|
#-----------
|
||||||
|
time=5989421
|
||||||
|
mem_heap_B=73744
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=26
|
||||||
|
#-----------
|
||||||
|
time=5996279
|
||||||
|
mem_heap_B=73992
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=27
|
||||||
|
#-----------
|
||||||
|
time=6002698
|
||||||
|
mem_heap_B=74248
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=28
|
||||||
|
#-----------
|
||||||
|
time=6015289
|
||||||
|
mem_heap_B=75272
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=29
|
||||||
|
#-----------
|
||||||
|
time=6015414
|
||||||
|
mem_heap_B=74760
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=30
|
||||||
|
#-----------
|
||||||
|
time=6040651
|
||||||
|
mem_heap_B=76808
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=31
|
||||||
|
#-----------
|
||||||
|
time=6040828
|
||||||
|
mem_heap_B=75784
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=32
|
||||||
|
#-----------
|
||||||
|
time=6091310
|
||||||
|
mem_heap_B=79880
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=detailed
|
||||||
|
n4: 79880 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||||
|
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||||
|
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||||
|
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||||
|
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x1: ???
|
||||||
|
n1: 72704 0x1FFF000AC6: ???
|
||||||
|
n0: 72704 0x1FFF000AD9: ???
|
||||||
|
n2: 6144 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n1: 6144 0x1095C2: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n0: 6144 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||||
|
n0: 0 in 1 place, below massif's threshold (1.00%)
|
||||||
|
n1: 1024 0x4CB6E03: _IO_file_doallocate (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CC570F: _IO_doallocbuf (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CC48A7: _IO_file_overflow@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CC3955: _IO_file_xsputn@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CB8330: fwrite (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4877FD7: fmt::v7::vprint(_IO_FILE*, fmt::v7::basic_string_view<char>, fmt::v7::format_args) (in /usr/lib/libfmt.so.7.1.2)
|
||||||
|
n1: 1024 0x10A258: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n1: 1024 0x10933E: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n0: 1024 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||||
|
n0: 8 in 2 places, all below massif's threshold (1.00%)
|
||||||
|
#-----------
|
||||||
|
snapshot=33
|
||||||
|
#-----------
|
||||||
|
time=6091591
|
||||||
|
mem_heap_B=77832
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=34
|
||||||
|
#-----------
|
||||||
|
time=6192490
|
||||||
|
mem_heap_B=86024
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=35
|
||||||
|
#-----------
|
||||||
|
time=6196654
|
||||||
|
mem_heap_B=81928
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=36
|
||||||
|
#-----------
|
||||||
|
time=6398233
|
||||||
|
mem_heap_B=98312
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=37
|
||||||
|
#-----------
|
||||||
|
time=6406493
|
||||||
|
mem_heap_B=90120
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=38
|
||||||
|
#-----------
|
||||||
|
time=6809904
|
||||||
|
mem_heap_B=122888
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=39
|
||||||
|
#-----------
|
||||||
|
time=6826356
|
||||||
|
mem_heap_B=106504
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=40
|
||||||
|
#-----------
|
||||||
|
time=7633039
|
||||||
|
mem_heap_B=172040
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=41
|
||||||
|
#-----------
|
||||||
|
time=7665875
|
||||||
|
mem_heap_B=139272
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=42
|
||||||
|
#-----------
|
||||||
|
time=7951930
|
||||||
|
mem_heap_B=73736
|
||||||
|
mem_heap_extra_B=32
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=detailed
|
||||||
|
n3: 73736 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||||
|
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||||
|
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||||
|
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||||
|
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x1: ???
|
||||||
|
n1: 72704 0x1FFF000AC6: ???
|
||||||
|
n0: 72704 0x1FFF000AD9: ???
|
||||||
|
n1: 1024 0x4CB6E03: _IO_file_doallocate (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CC570F: _IO_doallocbuf (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CC48A7: _IO_file_overflow@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CC3955: _IO_file_xsputn@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CB8330: fwrite (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4877FD7: fmt::v7::vprint(_IO_FILE*, fmt::v7::basic_string_view<char>, fmt::v7::format_args) (in /usr/lib/libfmt.so.7.1.2)
|
||||||
|
n1: 1024 0x10A258: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n1: 1024 0x10933E: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n0: 1024 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||||
|
n0: 8 in 3 places, all below massif's threshold (1.00%)
|
||||||
|
#-----------
|
||||||
|
snapshot=43
|
||||||
|
#-----------
|
||||||
|
time=7971885
|
||||||
|
mem_heap_B=73744
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=44
|
||||||
|
#-----------
|
||||||
|
time=7979276
|
||||||
|
mem_heap_B=75784
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=45
|
||||||
|
#-----------
|
||||||
|
time=7985879
|
||||||
|
mem_heap_B=77832
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=46
|
||||||
|
#-----------
|
||||||
|
time=7998473
|
||||||
|
mem_heap_B=86024
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=detailed
|
||||||
|
n4: 86024 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||||
|
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||||
|
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||||
|
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||||
|
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||||
|
n1: 72704 0x1: ???
|
||||||
|
n1: 72704 0x1FFF000AC6: ???
|
||||||
|
n0: 72704 0x1FFF000AD9: ???
|
||||||
|
n2: 12288 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n1: 12288 0x10973E: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n0: 12288 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||||
|
n0: 0 in 2 places, all below massif's threshold (1.00%)
|
||||||
|
n1: 1024 0x4CB6E03: _IO_file_doallocate (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CC570F: _IO_doallocbuf (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CC48A7: _IO_file_overflow@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CC3955: _IO_file_xsputn@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4CB8330: fwrite (in /usr/lib/libc-2.32.so)
|
||||||
|
n1: 1024 0x4877FD7: fmt::v7::vprint(_IO_FILE*, fmt::v7::basic_string_view<char>, fmt::v7::format_args) (in /usr/lib/libfmt.so.7.1.2)
|
||||||
|
n1: 1024 0x10A258: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n1: 1024 0x10933E: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||||
|
n0: 1024 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||||
|
n0: 8 in 2 places, all below massif's threshold (1.00%)
|
||||||
|
#-----------
|
||||||
|
snapshot=47
|
||||||
|
#-----------
|
||||||
|
time=8002637
|
||||||
|
mem_heap_B=81928
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=48
|
||||||
|
#-----------
|
||||||
|
time=8027775
|
||||||
|
mem_heap_B=98312
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=49
|
||||||
|
#-----------
|
||||||
|
time=8036035
|
||||||
|
mem_heap_B=90120
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=50
|
||||||
|
#-----------
|
||||||
|
time=8086261
|
||||||
|
mem_heap_B=122888
|
||||||
|
mem_heap_extra_B=48
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=51
|
||||||
|
#-----------
|
||||||
|
time=8102713
|
||||||
|
mem_heap_B=106504
|
||||||
|
mem_heap_extra_B=40
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=52
|
||||||
|
#-----------
|
||||||
|
time=8147526
|
||||||
|
mem_heap_B=73736
|
||||||
|
mem_heap_extra_B=32
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=53
|
||||||
|
#-----------
|
||||||
|
time=8156172
|
||||||
|
mem_heap_B=1032
|
||||||
|
mem_heap_extra_B=24
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
||||||
|
#-----------
|
||||||
|
snapshot=54
|
||||||
|
#-----------
|
||||||
|
time=8157131
|
||||||
|
mem_heap_B=8
|
||||||
|
mem_heap_extra_B=16
|
||||||
|
mem_stacks_B=0
|
||||||
|
heap_tree=empty
|
After Width: | Height: | Size: 4.1 KiB |
After Width: | Height: | Size: 4.5 KiB |
After Width: | Height: | Size: 3.2 KiB |
After Width: | Height: | Size: 3.6 KiB |
After Width: | Height: | Size: 4.1 KiB |
After Width: | Height: | Size: 3.0 KiB |
After Width: | Height: | Size: 32 KiB |
After Width: | Height: | Size: 38 KiB |
After Width: | Height: | Size: 10 KiB |
After Width: | Height: | Size: 17 KiB |
After Width: | Height: | Size: 19 KiB |
After Width: | Height: | Size: 23 KiB |
@ -0,0 +1,87 @@
|
|||||||
|
#include <stdlib.h>
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <stdarg.h>
|
||||||
|
#include <time.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
#include <string.h>
|
||||||
|
|
||||||
|
#include <debug.h>
|
||||||
|
|
||||||
|
#include <colours.h>
|
||||||
|
|
||||||
|
enum jt_comp {
|
||||||
|
JT_COMP_UNIX,
|
||||||
|
JT_COMP_DATE,
|
||||||
|
JT_COMP_TIME,
|
||||||
|
JT_COMP_DATETIME,
|
||||||
|
};
|
||||||
|
|
||||||
|
#define CLONE(p) ({ __auto_type $__p = (p); \
|
||||||
|
__auto_type $__r = __builtin_choose_expr(__builtin_classify_type($__p) == 5, *$__p, $__p); \
|
||||||
|
/* ((__typeof__($__r) []) { $__r }); }) */ \
|
||||||
|
__typeof__($__r) $__rp[1]; \
|
||||||
|
$__rp[0] = $__r; \
|
||||||
|
&$__rp[0]; })
|
||||||
|
|
||||||
|
__attribute__((returns_nonnull, nonnull(1)))
|
||||||
|
static inline
|
||||||
|
const char* jtime(const struct tm* restrict lt, enum jt_comp kind)
|
||||||
|
{
|
||||||
|
_Thread_local static char datebuf[1
|
||||||
|
+ 4 + 2 + 4 + 1 // `dd/mm/yyyy`
|
||||||
|
+ 6 + 3 // ` hh:MM:ss`
|
||||||
|
];
|
||||||
|
_Static_assert(sizeof(datebuf) != sizeof(char*), "Unexpected decay");
|
||||||
|
|
||||||
|
//memset(datebuf, '0', sizeof(datebuf)-1);
|
||||||
|
if(!kind)
|
||||||
|
snprintf(datebuf, sizeof(datebuf), "%lu", mktime(CLONE(lt)));
|
||||||
|
else {
|
||||||
|
size_t w = 0, sz = sizeof(datebuf);
|
||||||
|
if(kind & JT_COMP_DATE)
|
||||||
|
w += snprintf(datebuf, sz, "%02u/%02u/%4.4u"
|
||||||
|
, (unsigned)lt->tm_mday
|
||||||
|
, (unsigned)lt->tm_mon
|
||||||
|
, (unsigned)(1900 + lt->tm_year));
|
||||||
|
|
||||||
|
if(kind & JT_COMP_TIME)
|
||||||
|
w += snprintf(datebuf+w, sz - w, " %02u:%02u:%02u"
|
||||||
|
, (unsigned)lt->tm_hour
|
||||||
|
, (unsigned)lt->tm_min
|
||||||
|
, (unsigned)lt->tm_sec);
|
||||||
|
}
|
||||||
|
return datebuf;
|
||||||
|
}
|
||||||
|
|
||||||
|
void _do_dprintf(struct debuginfo info, const char* fmt, ...)
|
||||||
|
{
|
||||||
|
#ifdef DEBUG
|
||||||
|
va_list li;
|
||||||
|
|
||||||
|
time_t utime;
|
||||||
|
struct tm ltime;
|
||||||
|
time(&utime);
|
||||||
|
localtime_r(&utime, <ime);
|
||||||
|
|
||||||
|
va_start(li, fmt);
|
||||||
|
|
||||||
|
#ifdef D_TRACE
|
||||||
|
#define FMT_FUNC "%s"
|
||||||
|
#else
|
||||||
|
#define FMT_FUNC "%s()"
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#define TTY_OUT "["FGRN("dbg")" (%s %s) " FYEL("@") BOLD("%s") "->" FWHT(FMT_FUNC) ":" FYEL("%d") "]: "
|
||||||
|
#define NCOL(x) x
|
||||||
|
#define FIL_OUT "["NCOL("dbg")" (%s %s) " NCOL("@") NCOL("%s") "->" NCOL(FMT_FUNC) ":" NCOL("%d") "]: "
|
||||||
|
|
||||||
|
fprintf(stderr, isatty(fileno(stderr)) ? (TTY_OUT) : (FIL_OUT)
|
||||||
|
, jtime(<ime, JT_COMP_DATETIME), ltime.tm_zone
|
||||||
|
, info.file
|
||||||
|
, info.function
|
||||||
|
, info.line);
|
||||||
|
vfprintf(stderr, fmt, li);
|
||||||
|
fprintf(stderr, "\n");
|
||||||
|
va_end(li);
|
||||||
|
#endif
|
||||||
|
}
|
@ -0,0 +1,67 @@
|
|||||||
|
#include <unistd.h>
|
||||||
|
#include <fsvec.h>
|
||||||
|
#include <panic.h>
|
||||||
|
|
||||||
|
int fvec_new(fvec_t* restrict obj, const char* path)
|
||||||
|
{
|
||||||
|
obj->backing = fopen(path, "w+b");
|
||||||
|
if(!obj->backing) {
|
||||||
|
perror("fvec_new(): Failed to open file");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
obj->len=0;
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
void fvec_close(fvec_t* restrict obj)
|
||||||
|
{
|
||||||
|
if(obj->backing) fclose(obj->backing);
|
||||||
|
obj->backing=NULL;
|
||||||
|
obj->len=0;
|
||||||
|
}
|
||||||
|
|
||||||
|
int fvec_pop_end(fvec_t* restrict obj, size_t sz)
|
||||||
|
{
|
||||||
|
if(!obj->len) return 0;
|
||||||
|
|
||||||
|
if(ftruncate(fileno(obj->backing), (obj->len-=sz))<0) {
|
||||||
|
perror("Failed to pop buffer");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
void fvec_push_whole_buffer(fvec_t* restrict obj, const void* _buffer, size_t sz)
|
||||||
|
{
|
||||||
|
size_t w= 0;
|
||||||
|
size_t c;
|
||||||
|
const unsigned char* buffer = _buffer;
|
||||||
|
while ( w<sz && (c=fwrite(buffer+w, 1, sz-w, obj->backing))>0 ) w+=c;
|
||||||
|
if(w!=sz) {
|
||||||
|
perror("Corrupted buffer state, aborting");
|
||||||
|
panic("Cannot continue");
|
||||||
|
}
|
||||||
|
obj->len += sz;
|
||||||
|
}
|
||||||
|
int fvec_get_whole_buffer(const fvec_t* restrict obj, void* _buffer, size_t _sz)
|
||||||
|
{
|
||||||
|
ssize_t sz = (ssize_t)_sz;
|
||||||
|
if(_sz != (size_t)sz) panic("Buffer size %lu too large", _sz);
|
||||||
|
|
||||||
|
if(obj->len<(size_t)sz) return 0;
|
||||||
|
if(fseek(obj->backing, -sz, SEEK_END)<0) {
|
||||||
|
perror("fvec_get_whole_buffer() failed to seek");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
size_t w=0;
|
||||||
|
ssize_t c;
|
||||||
|
unsigned char* buffer = _buffer;
|
||||||
|
while ( w<_sz && (c=fread(buffer+w, 1, sz-w, obj->backing))>0 ) w+=c;
|
||||||
|
if (w!=_sz) {
|
||||||
|
perror("Corrupted buffer state, aborting");
|
||||||
|
panic("Cannot continue on FD %d (%p)", fileno(obj->backing), obj->backing);
|
||||||
|
}
|
||||||
|
|
||||||
|
return 1;
|
||||||
|
}
|
@ -0,0 +1,73 @@
|
|||||||
|
#include <filesystem>
|
||||||
|
|
||||||
|
#include <fsvec.hpp>
|
||||||
|
#include <uuid.hpp>
|
||||||
|
#include <fsvec.h>
|
||||||
|
#include <debug.h>
|
||||||
|
|
||||||
|
#include <tempfile.hpp>
|
||||||
|
|
||||||
|
#define FB file_back_buffer
|
||||||
|
|
||||||
|
|
||||||
|
struct FB::impl
|
||||||
|
{
|
||||||
|
size_t cap;
|
||||||
|
temp_file file;
|
||||||
|
|
||||||
|
fvec_t backing;
|
||||||
|
};
|
||||||
|
|
||||||
|
FB::FB(size_t cap) : inner(std::make_unique<FB::impl>())
|
||||||
|
{
|
||||||
|
// Set cap
|
||||||
|
inner->cap = cap;
|
||||||
|
// Create then open `inner->file`
|
||||||
|
if(!fvec_new(&inner->backing, inner->file->c_str())) panic("Failed to open backing for temp file buffer");
|
||||||
|
|
||||||
|
}
|
||||||
|
FB::FB(file_back_buffer&& m) : inner(std::move(m.inner)){}
|
||||||
|
FB::FB() : FB(DEFAULT_CAP){}
|
||||||
|
FB::~FB()
|
||||||
|
{
|
||||||
|
fvec_close(&inner->backing);
|
||||||
|
}
|
||||||
|
|
||||||
|
void FB::push_buf(byte* buf, size_t len)
|
||||||
|
{
|
||||||
|
fvec_push_whole_buffer(&inner->backing, (void*)buf, len);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool FB::back(byte* buf, size_t len) const
|
||||||
|
{
|
||||||
|
return !!fvec_get_whole_buffer(&inner->backing, (void*)buf, len);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool FB::pop_n(size_t len)
|
||||||
|
{
|
||||||
|
return !!fvec_pop_end(&inner->backing, len);
|
||||||
|
}
|
||||||
|
|
||||||
|
extern "C" void _fb_run_tests()
|
||||||
|
{
|
||||||
|
fixed_spill_vector<int, 8> test;
|
||||||
|
int r0,r1=0;
|
||||||
|
for(int i=0;i<10;i++) {
|
||||||
|
D_dprintf("push: %d", (10-i));
|
||||||
|
test.push_back(10-i);
|
||||||
|
r1+= (10-i);
|
||||||
|
}
|
||||||
|
D_dprintf("r1: %d", r1);
|
||||||
|
r0=0;
|
||||||
|
while(test.size())
|
||||||
|
{
|
||||||
|
r0+=test.back();
|
||||||
|
D_dprintf("back: %d", test.back());
|
||||||
|
test.pop_back();
|
||||||
|
}
|
||||||
|
D_dprintf("r0: %d", r0);
|
||||||
|
|
||||||
|
if(r0!=r1) panic("fb failed test: %d, %d", r0, r1);
|
||||||
|
|
||||||
|
D_dprintf("test successful");
|
||||||
|
}
|
@ -1,365 +1,83 @@
|
|||||||
|
#include <stdlib.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <float.h>
|
|
||||||
#include <stdint.h>
|
|
||||||
#include <string.h>
|
|
||||||
|
|
||||||
#include <rng.h>
|
|
||||||
#include <rng_algos.h>
|
|
||||||
#include <ptr_store.h>
|
|
||||||
#include <array.h>
|
|
||||||
|
|
||||||
_Static_assert(sizeof(float)==4, "Float must be 32 bits.");
|
#include <string.h>
|
||||||
|
|
||||||
int surpress_out=0;
|
|
||||||
|
|
||||||
/*void test_destructor(object* ptr)
|
#include <shuffle3.h>
|
||||||
{
|
#include <panic.h>
|
||||||
printf("die\n");
|
#include <reinterpret.h>
|
||||||
}*/
|
#include <map.h>
|
||||||
void _rng_object_destructor(object* obj)
|
#include <rng.h>
|
||||||
{
|
#include <work.h>
|
||||||
rng_free(obj->state);
|
|
||||||
}
|
|
||||||
RNG rng_object(S_LEXENV, RNG rng)
|
|
||||||
{
|
|
||||||
object proto = OBJ_PROTO;
|
|
||||||
proto.state = rng;
|
|
||||||
proto.destructor = &_rng_object_destructor;
|
|
||||||
|
|
||||||
return snew_obj(proto)->state;
|
|
||||||
}
|
|
||||||
|
|
||||||
void shuffle(RNG with, array_t data)
|
#include <debug.h>
|
||||||
{
|
|
||||||
if(!surpress_out) printf(" -> shuffling %d objects...", (int)ar_size(data));
|
|
||||||
for(int i=ar_size(data)-1;i>0;i--)
|
|
||||||
{
|
|
||||||
int j = rng_next_int(with, i);
|
|
||||||
ar_swap(data, i, j);
|
|
||||||
}
|
|
||||||
if(!surpress_out) printf(" Okay\n");
|
|
||||||
}
|
|
||||||
|
|
||||||
void unshuffle(RNG with, array_t data)
|
#define noreturn __attribute__((noreturn)) void
|
||||||
{
|
|
||||||
int rng_values[ar_size(data)-1];
|
|
||||||
int k=0;
|
|
||||||
if(!surpress_out) printf(" -> unshuffling %d objects...", (int)ar_size(data));
|
|
||||||
for(int i=ar_size(data)-1;i>0;i--)
|
|
||||||
rng_values[k++] = rng_next_int(with, i);
|
|
||||||
|
|
||||||
for(int i=1;i<ar_size(data);i++)
|
_Static_assert(sizeof(float)==sizeof(uint32_t), "float is not 32 bits");
|
||||||
{
|
|
||||||
ar_swap(data, i, rng_values[--k]);
|
|
||||||
}
|
|
||||||
if(!surpress_out) printf(" Okay\n");
|
|
||||||
}
|
|
||||||
|
|
||||||
void minmax_int64_ts(int64_t* min, int64_t* max, const array_t data)
|
const char* _prog_name;
|
||||||
{
|
|
||||||
for(register int i=0;i<ar_size(data);i++)
|
|
||||||
{
|
|
||||||
if(ar_get_v(data, int64_t, i) > *max) *max = ar_get_v(data, int64_t, i);
|
|
||||||
if(ar_get_v(data, int64_t, i) < *min) *min = ar_get_v(data, int64_t, i);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
int valid_float(float f)
|
|
||||||
{
|
|
||||||
return !( (f!=f) || (f< -FLT_MAX || f> FLT_MAX));
|
|
||||||
|
|
||||||
}
|
extern void _fb_run_tests();
|
||||||
void minmax_floats(float* min, float* max, const array_t data)
|
|
||||||
{
|
|
||||||
for(register int i=0;i<ar_size(data);i++)
|
|
||||||
{
|
|
||||||
if(!valid_float(ar_get_v(data,float,i))) continue;
|
|
||||||
if(ar_get_v(data, float, i) > *max) *max = ar_get_v(data, float, i);
|
|
||||||
if(ar_get_v(data, float, i) < *min) *min = ar_get_v(data, float, i);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
void minmax_sbytes(int8_t* min, int8_t* max, const array_t data)
|
noreturn help_then_exit()
|
||||||
{
|
{
|
||||||
for(register int i=0;i<ar_size(data);i++)
|
fprintf(stderr, "Try passing `-h`\n");
|
||||||
{
|
exit(1);
|
||||||
if(ar_get_v(data, int8_t, i) > *max) *max = ar_get_v(data, int8_t, i);
|
|
||||||
if(ar_get_v(data, int8_t, i) < *min) *min = ar_get_v(data, int8_t, i);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
void print_array(const array_t data)
|
void usage()
|
||||||
{
|
{
|
||||||
printf("---%d elements---\n", (int)ar_size(data));
|
printf( "shuffle3-lean - improved 3 pass binary shuffler\n"
|
||||||
for(register int i=0;i<ar_size(data);i++)
|
" written by Avril <flanchan@cumallover.me> (flanchan.moe) with <3\n"
|
||||||
printf("%d ", (int)ar_get_v(data, unsigned char, i));
|
" licensed with GPL v3.0 or later\n\n"
|
||||||
printf("\n---\n");
|
"Usage: %s -s <file>\n"
|
||||||
|
"Usage: %s -u <file>\n", _prog_name, _prog_name);
|
||||||
|
printf("\nOPTIONS:\n"
|
||||||
|
" -s\tShuffle file in place\n"
|
||||||
|
" -u\tUnshuffle file in place\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
void unshuffle3(S_LEXENV, array_t data)
|
int main(int argc, char** argv)
|
||||||
{
|
{
|
||||||
if(ar_type(data)!=sizeof(int8_t))
|
_prog_name = argv[0];
|
||||||
ar_reinterpret(data, sizeof(int8_t));
|
|
||||||
RNG xoro = S_LEXCALL(rng_object, rng_new(RNG_ALGO(xoroshiro128plus)));
|
|
||||||
RNG frng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(frng)));
|
|
||||||
RNG drng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(drng)));
|
|
||||||
|
|
||||||
int8_t bmax = INT8_MIN;
|
//_fb_run_tests();
|
||||||
int8_t bmin = INT8_MAX;
|
work_args_t parsed;
|
||||||
minmax_sbytes(&bmin, &bmax, data);
|
|
||||||
uint32_t seed = (0xfffa << 16) | (bmin<<8) | bmax;
|
|
||||||
rng_seed(drng, &seed);
|
|
||||||
unshuffle(drng, data);
|
|
||||||
|
|
||||||
float fmin = (float)DBL_MAX;
|
if( !argv[1] || *(argv[1]) != '-') help_then_exit();
|
||||||
float fmax = (float)-DBL_MAX;
|
|
||||||
ar_reinterpret(data, sizeof(float));
|
|
||||||
minmax_floats(&fmin, &fmax, data);
|
|
||||||
double fseed[2];
|
|
||||||
fseed[0] = fmin;
|
|
||||||
fseed[1] = fmax;
|
|
||||||
rng_seed(frng, fseed);
|
|
||||||
unshuffle(frng, data);
|
|
||||||
|
|
||||||
int64_t min = INT64_MAX;
|
D_dprintf("Parsing `%c'", argv[1][1]);
|
||||||
int64_t max = INT64_MIN;
|
switch(argv[1][1])
|
||||||
ar_reinterpret(data, sizeof(int64_t));
|
|
||||||
minmax_int64_ts(&min, &max, data);
|
|
||||||
uint64_t xseed[2];
|
|
||||||
xseed[0] = min;
|
|
||||||
xseed[1] = max;
|
|
||||||
rng_seed(xoro, xseed);
|
|
||||||
unshuffle(xoro, data);
|
|
||||||
}
|
|
||||||
void shuffle3(S_LEXENV, array_t data)
|
|
||||||
{
|
|
||||||
if(ar_type(data)!=sizeof(int64_t))
|
|
||||||
ar_reinterpret(data, sizeof(int64_t));
|
|
||||||
RNG xoro = S_LEXCALL(rng_object, rng_new(RNG_ALGO(xoroshiro128plus)));
|
|
||||||
RNG frng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(frng)));
|
|
||||||
RNG drng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(drng)));
|
|
||||||
|
|
||||||
int64_t min = INT64_MAX;
|
|
||||||
int64_t max = INT64_MIN;
|
|
||||||
|
|
||||||
float fmin = (float)DBL_MAX;
|
|
||||||
float fmax = (float)-DBL_MAX;
|
|
||||||
|
|
||||||
int8_t bmax = INT8_MIN;
|
|
||||||
int8_t bmin = INT8_MAX;
|
|
||||||
|
|
||||||
minmax_int64_ts(&min, &max, data);
|
|
||||||
uint64_t xseed[2];
|
|
||||||
xseed[0] = min;
|
|
||||||
xseed[1] = max;
|
|
||||||
|
|
||||||
rng_seed(xoro, xseed);
|
|
||||||
shuffle(xoro, data);
|
|
||||||
|
|
||||||
ar_reinterpret(data, sizeof(float));
|
|
||||||
minmax_floats(&fmin, &fmax, data);
|
|
||||||
|
|
||||||
double fseed[2];
|
|
||||||
fseed[0] = fmin;
|
|
||||||
fseed[1] = fmax;
|
|
||||||
rng_seed(frng, fseed);
|
|
||||||
shuffle(frng, data);
|
|
||||||
|
|
||||||
ar_reinterpret(data,sizeof(int8_t));
|
|
||||||
minmax_sbytes(&bmin, &bmax, data);
|
|
||||||
uint32_t seed = (0xfffa << 16) | (bmin<<8) | bmax;
|
|
||||||
rng_seed(drng, &seed);
|
|
||||||
shuffle(drng, data);
|
|
||||||
}
|
|
||||||
|
|
||||||
void print_usage(char** argv)
|
|
||||||
{
|
|
||||||
printf("Usage: %s -[b]s <file> [<outfile>]\nUsage: %s -[b]u <file> [<outfile>]\n", argv[0], argv[0]);
|
|
||||||
printf("Usage: %s -S <string>|-\nUsage: %s -U <string>|-\n", argv[0], argv[0]);
|
|
||||||
printf(" <file>\t\tFile to shuffle\n");
|
|
||||||
printf(" <outfile>\tOutput file (only valid for buffered mode)\n");
|
|
||||||
printf(" <string>\tThe string to shuffle (for -(S|U)). If `-`, read from stdin\n");
|
|
||||||
printf("\ts\tShuffle\n");
|
|
||||||
printf("\tu\tReverse shuffle\n");
|
|
||||||
printf("\tS\tShuffle string\n");
|
|
||||||
printf("\tU\tReverse shuffle string\n");
|
|
||||||
printf("\n\tb\tUse buffered mode instead of shuffling in place (must specify output file)\n");
|
|
||||||
}
|
|
||||||
|
|
||||||
array_t read_whole_file(S_NAMED_LEXENV(base), FILE* fp)
|
|
||||||
{
|
|
||||||
fseek(fp,0,SEEK_END);
|
|
||||||
int64_t sz = ftell(fp);
|
|
||||||
fseek(fp,0,SEEK_SET);
|
|
||||||
|
|
||||||
array_t ar;
|
|
||||||
MANAGED({
|
|
||||||
void* buf = smalloc(sz);
|
|
||||||
fread(buf, 1, sz, fp);
|
|
||||||
ar = ar_create_memory_from(base, buf, 1, sz);
|
|
||||||
});
|
|
||||||
return ar;
|
|
||||||
}
|
|
||||||
|
|
||||||
#define BUFSIZE 512
|
|
||||||
#define BUFMIN 128
|
|
||||||
|
|
||||||
char* read_stdin()
|
|
||||||
{
|
|
||||||
char* input, *p;
|
|
||||||
int len,remain,n,size;
|
|
||||||
|
|
||||||
size = BUFSIZE;
|
|
||||||
input = malloc(size);
|
|
||||||
memset(input,0,size);
|
|
||||||
len=0;
|
|
||||||
remain = size;
|
|
||||||
|
|
||||||
while(!feof(stdin))
|
|
||||||
{
|
{
|
||||||
if(remain<= BUFMIN)
|
case 's':
|
||||||
{
|
parsed.op = OP_SHUFFLE_IP;
|
||||||
remain+=size;
|
if(!(parsed.data.op_shuffle_ip.file = argv[2]))
|
||||||
size *=2;
|
|
||||||
p = realloc(input, size);
|
|
||||||
if(p==NULL) {
|
|
||||||
free(input);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
input = p;
|
|
||||||
}
|
|
||||||
fgets(input+len, remain, stdin);
|
|
||||||
n+=strlen(input+len);
|
|
||||||
len+=n;
|
|
||||||
remain-=n;
|
|
||||||
}
|
|
||||||
return input;
|
|
||||||
}
|
|
||||||
|
|
||||||
int string_shuffle(char* string, int un)
|
|
||||||
{
|
|
||||||
if(!string)
|
|
||||||
{
|
|
||||||
char* ptr = read_stdin();
|
|
||||||
if(!ptr) {
|
|
||||||
printf("! read from stdin failed\n");
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
int ret = string_shuffle(ptr, un);
|
|
||||||
free(ptr);
|
|
||||||
return ret;
|
|
||||||
}
|
|
||||||
else if(strcmp(string, "-") == 0)
|
|
||||||
{
|
|
||||||
return string_shuffle(NULL, un);
|
|
||||||
}
|
|
||||||
else {
|
|
||||||
surpress_out=1;
|
|
||||||
MANAGED({
|
|
||||||
array_t ar = ar_create_memory_from(LEXENV, string, 1, strlen(string));
|
|
||||||
|
|
||||||
if(un) unshuffle3(LEXENV, ar);
|
|
||||||
else shuffle3(LEXENV, ar);
|
|
||||||
|
|
||||||
ar_reinterpret(ar, sizeof(char));
|
|
||||||
|
|
||||||
for(int i=0;i<ar_size(ar);i++)
|
|
||||||
{
|
{
|
||||||
printf("%c", ar_get_v(ar, char, i));
|
fprintf(stderr, "Error: -s expected file argument.\n");
|
||||||
}
|
|
||||||
});
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
int main(int argc, char** argv)
|
|
||||||
{
|
|
||||||
int is_buffered=0;
|
|
||||||
int i=1;
|
|
||||||
int is_unshuffling =0;
|
|
||||||
if(!argv[1] || !argv[2] || argv[1][0] != '-')
|
|
||||||
{
|
|
||||||
print_usage(argv);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
else {
|
|
||||||
do_switch:
|
|
||||||
switch(argv[1][i])
|
|
||||||
{
|
|
||||||
case 'b':
|
|
||||||
if(is_buffered)
|
|
||||||
{
|
|
||||||
print_usage(argv);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
is_buffered = 1;
|
|
||||||
i+=1;
|
|
||||||
goto do_switch;
|
|
||||||
case 'u':
|
|
||||||
is_unshuffling=1;
|
|
||||||
break;
|
|
||||||
case 's':
|
|
||||||
break;
|
|
||||||
case 'S':
|
|
||||||
return string_shuffle(argv[2], 0);
|
|
||||||
case 'U':
|
|
||||||
return string_shuffle(argv[2], 1);
|
|
||||||
default:
|
|
||||||
print_usage(argv);
|
|
||||||
return 1;
|
return 1;
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if(is_buffered && !argv[3])
|
|
||||||
{
|
|
||||||
printf("Buffered option requires an output file\n");
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
MANAGED_RETURNABLE(int ,{
|
|
||||||
array_t array;
|
|
||||||
|
|
||||||
if(is_buffered)
|
|
||||||
{
|
|
||||||
FILE* infile = fopen(argv[2], "rb");
|
|
||||||
if(!infile)
|
|
||||||
{
|
|
||||||
printf("! could not open file for reading\n");
|
|
||||||
MANAGED_RETURN(1);
|
|
||||||
}
|
}
|
||||||
array = read_whole_file(LEXENV, infile);
|
D_dprintf("parsed.op = %d", OP_SHUFFLE_IP);
|
||||||
fclose(infile);
|
break;
|
||||||
printf(" buffered file (%ld bytes)\n", ar_size(array));
|
case 'u':
|
||||||
}
|
parsed.op = OP_UNSHUFFLE_IP;
|
||||||
else {
|
if(!(parsed.data.op_unshuffle_ip.file = argv[2]))
|
||||||
FILE* infile = fopen(argv[2], "r+b");
|
|
||||||
if(!infile)
|
|
||||||
{
|
{
|
||||||
printf("! could not open file for reading+writing\n");
|
fprintf(stderr, "Error: -u expected file argument.\n");
|
||||||
MANAGED_RETURN(1);
|
return 1;
|
||||||
}
|
}
|
||||||
array = ar_create_file(LEXENV, infile, 1, 1);
|
D_dprintf("parsed.op = %d", OP_UNSHUFFLE_IP);
|
||||||
}
|
break;
|
||||||
|
case 'h':
|
||||||
//print_array(array);
|
usage();
|
||||||
|
return 0;
|
||||||
|
default:
|
||||||
|
fprintf(stderr, "Error: unknown argument `%s'\n\n", argv[1]);
|
||||||
|
help_then_exit();
|
||||||
|
panic("Unreachable");
|
||||||
|
}
|
||||||
|
|
||||||
if(is_unshuffling) unshuffle3(LEXENV, array);
|
return do_work(parsed);
|
||||||
else shuffle3(LEXENV, array);
|
|
||||||
|
|
||||||
if(is_buffered)
|
|
||||||
{
|
|
||||||
FILE* outfile = fopen(argv[3], "wb");
|
|
||||||
if(!outfile)
|
|
||||||
{
|
|
||||||
printf("! could not open outfile for writing\n");
|
|
||||||
MANAGED_RETURN(1);
|
|
||||||
}
|
|
||||||
void* wbuf = smalloc(ar_full_size(array));
|
|
||||||
if(!ar_ndump(array, wbuf, ar_full_size(array), 0, ar_size(array)))
|
|
||||||
printf("W memory dump failed, continuing anyway\n");
|
|
||||||
fwrite(wbuf, 1, ar_full_size(array), outfile);
|
|
||||||
fclose(outfile);
|
|
||||||
printf(" write completed (%d bytes)\n", (int)ar_size(array));
|
|
||||||
}
|
|
||||||
});
|
|
||||||
return 0;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -0,0 +1,171 @@
|
|||||||
|
#define _GNU_SOURCE
|
||||||
|
#include<stdio.h>
|
||||||
|
#include<stdlib.h>
|
||||||
|
#include<string.h>
|
||||||
|
#include<unistd.h>
|
||||||
|
#include <sys/types.h>
|
||||||
|
#include <sys/stat.h>
|
||||||
|
#include<sys/mman.h>
|
||||||
|
#include<fcntl.h>
|
||||||
|
|
||||||
|
#define FILEMODE S_IRWXU | S_IRGRP | S_IROTH
|
||||||
|
|
||||||
|
#include <map.h>
|
||||||
|
|
||||||
|
__attribute__((pure, nonnull(1)))
|
||||||
|
static inline int _map(mmap_t* restrict map)
|
||||||
|
{
|
||||||
|
//TODO: Based on map->len, choose (or calculate) appropriate `MAP_HUGE_*` flag
|
||||||
|
return (map->ptr = mmap(NULL, map->len, PROT_READ | PROT_WRITE, MAP_SHARED, map->fd, 0)) != MAP_FAILED;
|
||||||
|
}
|
||||||
|
//#define _map(...) _map(__VA_ARGS__, 0)
|
||||||
|
|
||||||
|
int map_advise_rand(mmap_t* restrict ptr, int need)
|
||||||
|
{
|
||||||
|
int flags = MADV_RANDOM | (need ? MADV_WILLNEED : 0);
|
||||||
|
if(madvise(ptr->ptr, ptr->len, flags)) {
|
||||||
|
perror("madvise() failed");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int map_raw_fd(int fd, mmap_t* restrict ptr, const size_t sz[static 1])
|
||||||
|
{
|
||||||
|
if(fd < 0) return 0;
|
||||||
|
struct stat st;
|
||||||
|
|
||||||
|
|
||||||
|
if(sz && ftruncate(fd, *sz)) {
|
||||||
|
perror("Failed to allocate");
|
||||||
|
return 0;
|
||||||
|
} else if(fstat(fd, &st) < 0) {
|
||||||
|
perror("Failed to stat file");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct mmap map = { .fd = fd, .ptr = NULL, .len = sz ? *sz : st.st_size };
|
||||||
|
|
||||||
|
if (!_map(&map)) {
|
||||||
|
perror("mmap() failed");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
*ptr = map;
|
||||||
|
|
||||||
|
return 1;
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
int open_and_map(const char* file, mmap_t* restrict ptr)
|
||||||
|
{
|
||||||
|
int fd;
|
||||||
|
struct stat st;
|
||||||
|
if ((fd = open(file, O_RDWR, FILEMODE)) < 0) {
|
||||||
|
perror("Failed to open file");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (fstat(fd, &st) < 0) {
|
||||||
|
perror("Failed to stat file");
|
||||||
|
close(fd);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct mmap map = { .fd = fd, .ptr = NULL, .len = st.st_size };
|
||||||
|
|
||||||
|
if (!_map(&map)) {
|
||||||
|
perror("mmap() failed");
|
||||||
|
close(fd);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
*ptr = map;
|
||||||
|
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int open_and_alloc(const char* file, mmap_t* restrict ptr, size_t sz)
|
||||||
|
{
|
||||||
|
int fd;
|
||||||
|
if ((fd = open(file, O_CREAT | O_RDWR, FILEMODE)) < 0) {
|
||||||
|
perror("Failed to open file");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if(ftruncate(fd, sz))
|
||||||
|
{
|
||||||
|
perror("Failed to allocate");
|
||||||
|
close(fd);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct mmap map = { .fd = fd, .ptr = NULL, .len = sz };
|
||||||
|
|
||||||
|
if (!_map(&map)) {
|
||||||
|
perror("mmap() failed");
|
||||||
|
close(fd);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
*ptr = map;
|
||||||
|
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
inline
|
||||||
|
int unmap_and_close_s(mmap_t map, int flags)
|
||||||
|
{
|
||||||
|
register int rval=1;
|
||||||
|
if (map.fd && msync(map.ptr, map.len, flags))
|
||||||
|
perror("msync() failed");
|
||||||
|
if (munmap(map.ptr, map.len) < 0) {
|
||||||
|
perror("munmap() failed");
|
||||||
|
rval=0;
|
||||||
|
}
|
||||||
|
if (close(map.fd) <0) {
|
||||||
|
perror("Failed to close fd");
|
||||||
|
rval=0;
|
||||||
|
}
|
||||||
|
|
||||||
|
return rval;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
int unmap_and_close(mmap_t map)
|
||||||
|
{
|
||||||
|
return unmap_and_close_s(map, MS_SYNC /* | MS_INVALIDATE XXX: I don't think this is needed, we son't be using the mapping anymore*/);
|
||||||
|
}
|
||||||
|
|
||||||
|
__attribute__((nonnull(1)))
|
||||||
|
int dup_map(const mmap_t *in, mmap_t* restrict ptr, const size_t *new_size, enum map_flags flags)
|
||||||
|
{
|
||||||
|
int fd;
|
||||||
|
if ( (fd = dup(in->fd)) < 0 )
|
||||||
|
{
|
||||||
|
perror("dup() failed");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
size_t sz;
|
||||||
|
if(new_size) {
|
||||||
|
if(ftruncate(fd, sz = *new_size))
|
||||||
|
{
|
||||||
|
perror("Failed to set cloned map size");
|
||||||
|
close(fd);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
} else sz = in->len;
|
||||||
|
|
||||||
|
struct mmap map = { .fd = fd, .ptr = NULL, .len = sz };
|
||||||
|
|
||||||
|
if ((map.ptr = mmap(NULL, map.len, PROT_READ | (flags & MMF_READONLY ? 0 : PROT_WRITE), (flags & MMF_PRIVATE) ? MAP_PRIVATE : MAP_SHARED,fd, 0)) == MAP_FAILED) {
|
||||||
|
perror("mmap() failed");
|
||||||
|
close(fd);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
*ptr = map;
|
||||||
|
return 1;
|
||||||
|
}
|
@ -0,0 +1,31 @@
|
|||||||
|
|
||||||
|
#include <sys/types.h>
|
||||||
|
#include <sys/stat.h>
|
||||||
|
#include <sys/mman.h>
|
||||||
|
#include <fcntl.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
|
||||||
|
#include <panic.h>
|
||||||
|
#include <map.h>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
namespace {
|
||||||
|
constexpr inline auto MEMFD_DEFAULT_FLAGS = MFD_CLOEXEC;
|
||||||
|
int memfd_alloc(const char* name, size_t size = 0, int flags = MEMFD_DEFAULT_FLAGS)
|
||||||
|
{
|
||||||
|
int fd = memfd_create(name, flags);
|
||||||
|
if(fd < 0) panic("memfd_create() failed");
|
||||||
|
if(size)
|
||||||
|
if(ftruncate(fd, size)) { close(fd); panic("ftruncate() failed"); }
|
||||||
|
return fd;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
namespace mm {
|
||||||
|
vmap::vmap()
|
||||||
|
: mmap(create_raw_fd(memfd_alloc(__PRETTY_FUNCTION__))) {}
|
||||||
|
vmap::vmap(size_t sz)
|
||||||
|
: mmap(create_raw_fd(memfd_alloc(__PRETTY_FUNCTION__, sz))) {}
|
||||||
|
|
||||||
|
}
|
@ -0,0 +1,17 @@
|
|||||||
|
|
||||||
|
#include <shuffle3.h>
|
||||||
|
#include <reinterpret.hpp>
|
||||||
|
#include <map.h>
|
||||||
|
|
||||||
|
|
||||||
|
extern "C" void* map_and_then(const char* file, map_cb callback, void* user)
|
||||||
|
{
|
||||||
|
mm::mmap map(file);
|
||||||
|
return callback(std::as_const(map).as_raw(), user);
|
||||||
|
}
|
||||||
|
|
||||||
|
extern "C" void* map_fd_and_then(int fd, map_cb callback, void* user)
|
||||||
|
{
|
||||||
|
auto map = mm::mmap::map_raw_fd(fd);
|
||||||
|
return callback(std::as_const(map).as_raw(), user);
|
||||||
|
}
|
@ -0,0 +1,12 @@
|
|||||||
|
#include <stdint.h>
|
||||||
|
#include <stddef.h>
|
||||||
|
#include <stdlib.h>
|
||||||
|
|
||||||
|
int _can_allocate(size_t bytes)
|
||||||
|
{
|
||||||
|
void* shim = malloc(bytes);
|
||||||
|
if(shim) {
|
||||||
|
free(shim);
|
||||||
|
return 1;
|
||||||
|
} else return 0;
|
||||||
|
}
|
@ -0,0 +1,41 @@
|
|||||||
|
#include <stdlib.h>
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
#include <stdarg.h>
|
||||||
|
|
||||||
|
#include <panic.h>
|
||||||
|
|
||||||
|
#include <colours.h>
|
||||||
|
|
||||||
|
#ifndef DEFAULT_PANIC_HANDLER
|
||||||
|
#if __cpp_exceptions || 1 /* XXX: Cannot detect this here */
|
||||||
|
# define DEFAULT_PANIC_HANDLER "_panic__start_unwind_cxx"
|
||||||
|
#else
|
||||||
|
# define DEFAULT_PANIC_HANDLER "_panic__start_unwind_abort"
|
||||||
|
#endif
|
||||||
|
#endif
|
||||||
|
|
||||||
|
|
||||||
|
extern void _panic__start_unwind_cxx(void);
|
||||||
|
static void _panic__start_unwind_abort(void*)
|
||||||
|
__attribute__((weakref("abort"), noreturn));
|
||||||
|
|
||||||
|
static void _panic__start_unwind(void* payload)
|
||||||
|
__attribute__((noreturn, weakref(DEFAULT_PANIC_HANDLER)));
|
||||||
|
|
||||||
|
__attribute__((noreturn, weak)) void _do_panic(struct panicinfo info, const char* fmt, ...)
|
||||||
|
{
|
||||||
|
va_list li;
|
||||||
|
va_start(li, fmt);
|
||||||
|
#define FMT_TTY BOLD(UNDL(FRED("[!]"))) " (" BOLD("%s") "->" BOLD(FRED("%s")) ":" FYEL("%d") ") " BOLD(FRED("fatal error")) ": "
|
||||||
|
#define FMT_FIL "[!]" " (" "%s" "->" "%s" ":" "%d" ") " "fatal error" ": "
|
||||||
|
fprintf(stderr, isatty(fileno(stderr))
|
||||||
|
? FMT_TTY
|
||||||
|
: FMT_FIL , info.file, info.function, info.line);
|
||||||
|
vfprintf(stderr, fmt, li);
|
||||||
|
fprintf(stderr, "\n");
|
||||||
|
va_end(li);
|
||||||
|
|
||||||
|
_panic__start_unwind(&info);
|
||||||
|
__builtin_unreachable();
|
||||||
|
}
|
@ -0,0 +1,47 @@
|
|||||||
|
|
||||||
|
#include <utility>
|
||||||
|
#include <stdexcept>
|
||||||
|
|
||||||
|
#include <cstdlib>
|
||||||
|
|
||||||
|
extern "C" {
|
||||||
|
struct ErrorPayload {};
|
||||||
|
}
|
||||||
|
|
||||||
|
struct FatalError {
|
||||||
|
FatalError(void* payload)
|
||||||
|
: payload(payload) {}
|
||||||
|
FatalError(FatalError&& m)
|
||||||
|
: payload(std::exchange(m.payload, nullptr)) {}
|
||||||
|
|
||||||
|
FatalError& operator=(FatalError&& m)
|
||||||
|
{
|
||||||
|
if(this != &m) {
|
||||||
|
if(payload) destroy();
|
||||||
|
payload = std::exchange(m.payload, nullptr);
|
||||||
|
} return *this;
|
||||||
|
}
|
||||||
|
virtual ~FatalError() noexcept {
|
||||||
|
destroy();
|
||||||
|
}
|
||||||
|
protected:
|
||||||
|
virtual void destroy() noexcept
|
||||||
|
{
|
||||||
|
/*if(auto* p = dynamic_cast<ErrorPayload*>(payload))
|
||||||
|
delete p;*/
|
||||||
|
}
|
||||||
|
public:
|
||||||
|
void* payload;
|
||||||
|
};
|
||||||
|
|
||||||
|
extern "C" {
|
||||||
|
[[noreturn, gnu::weak]]
|
||||||
|
void _panic__start_unwind_cxx(void* payload)
|
||||||
|
{
|
||||||
|
#if __EXCEPTIONS
|
||||||
|
throw FatalError(payload);
|
||||||
|
#else
|
||||||
|
::abort();
|
||||||
|
#endif
|
||||||
|
}
|
||||||
|
}
|
@ -0,0 +1,135 @@
|
|||||||
|
|
||||||
|
#include <string>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
#include <fmt/format.h>
|
||||||
|
|
||||||
|
#include <sys/ioctl.h>
|
||||||
|
|
||||||
|
#include <perc.h>
|
||||||
|
|
||||||
|
using fract_t = pr::Progress::fract_t;
|
||||||
|
|
||||||
|
namespace {
|
||||||
|
using namespace pr;
|
||||||
|
size_t twidth(int fd = STDIN_FILENO, size_t orr = Progress::DEFAULT_WIDTH) noexcept
|
||||||
|
{
|
||||||
|
struct winsize w;
|
||||||
|
if(ioctl(fd, TIOCGWINSZ, &w) == -1) return orr;
|
||||||
|
return size_t(w.ws_col);
|
||||||
|
}
|
||||||
|
|
||||||
|
size_t def_bar_width(size_t tw) noexcept
|
||||||
|
{
|
||||||
|
return size_t(std::round(Progress::DEFAULT_TERM_FRACT * fract_t(tw)));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
namespace pr {
|
||||||
|
|
||||||
|
struct Progress::_impl {
|
||||||
|
Progress::fract_t fract; // 0..=1
|
||||||
|
FILE* output;
|
||||||
|
size_t width;
|
||||||
|
std::string aux;
|
||||||
|
};
|
||||||
|
|
||||||
|
size_t& Progress::width() noexcept { return inner_->width; }
|
||||||
|
size_t Progress::width() const noexcept { return inner_->width; }
|
||||||
|
|
||||||
|
fract_t& Progress::fraction() noexcept { return inner_->fract; }
|
||||||
|
fract_t Progress::fraction() const noexcept { return inner_->fract; }
|
||||||
|
|
||||||
|
fract_t Progress::percentage() const noexcept { return inner_->fract * 100.0l; }
|
||||||
|
void Progress::percentage(fract_t fract) noexcept { inner_->fract = fract / 100.0l; }
|
||||||
|
|
||||||
|
std::string& Progress::tag() noexcept { return inner_->aux; }
|
||||||
|
const std::string& Progress::tag() const noexcept { return inner_->aux; }
|
||||||
|
|
||||||
|
FILE* Progress::output() const noexcept { return inner_->output; }
|
||||||
|
FILE*& Progress::output() noexcept { return inner_->output; }
|
||||||
|
|
||||||
|
void Progress::render(bool flush) const noexcept
|
||||||
|
{
|
||||||
|
constinit thread_local static std::vector<char> buffer{};
|
||||||
|
|
||||||
|
const auto& inner = *inner_;
|
||||||
|
FILE* out = inner.output;
|
||||||
|
auto wf = fract_t(inner.width) * inner.fract;
|
||||||
|
auto perc = percentage();
|
||||||
|
size_t cf = size_t(std::floor(wf));
|
||||||
|
|
||||||
|
|
||||||
|
//auto print = []<typename F> (F f, auto&&... args) {
|
||||||
|
#define print(...) fmt::format_to(std::back_inserter(buffer), __VA_ARGS__);
|
||||||
|
//};
|
||||||
|
|
||||||
|
buffer.clear();
|
||||||
|
// Render bar
|
||||||
|
buffer.push_back('[');
|
||||||
|
for(size_t i=0;i<inner.width;i++)
|
||||||
|
if(i<=cf) buffer.push_back( '#' );
|
||||||
|
else buffer.push_back( ' ' );
|
||||||
|
buffer.push_back( ']' );
|
||||||
|
print(": {:00.2}%", perc);
|
||||||
|
if(inner.aux.size())
|
||||||
|
print(" ({})", std::string_view(inner.aux));
|
||||||
|
// Print bar
|
||||||
|
fprintf(out, "\r%.*s", int(buffer.size() & INT_MAX), static_cast<const char*>(buffer.data()));
|
||||||
|
// Flush output stream
|
||||||
|
if(flush) fflush(out);
|
||||||
|
#undef print
|
||||||
|
}
|
||||||
|
void Progress::spin(increment_t by, bool r, bool f) noexcept
|
||||||
|
{
|
||||||
|
auto& inner = *inner_;
|
||||||
|
inner.fract=by;
|
||||||
|
if(inner.fract > 1.0l)
|
||||||
|
inner.fract=1.0l;
|
||||||
|
if(r) this->render(f);
|
||||||
|
}
|
||||||
|
|
||||||
|
[[gnu::nonnull(1)]]
|
||||||
|
Progress::Progress(FILE* p) noexcept
|
||||||
|
: inner_(std::make_unique<_impl>(0, p, def_bar_width(twidth(fileno(p))) )) {}
|
||||||
|
|
||||||
|
Progress::Progress(const Progress& p) noexcept
|
||||||
|
: inner_(p.inner_ ? std::make_unique<_impl>(*p.inner_) : nullptr) {}
|
||||||
|
|
||||||
|
Progress::Progress(Progress&& p) noexcept
|
||||||
|
: inner_(std::move(p.inner_)) {}
|
||||||
|
|
||||||
|
Progress& Progress::operator=(Progress&& p) noexcept
|
||||||
|
{
|
||||||
|
if(this != &p) inner_ = std::move(p.inner_);
|
||||||
|
return *this;
|
||||||
|
}
|
||||||
|
|
||||||
|
Progress& Progress::operator=(Progress const& p) noexcept
|
||||||
|
{
|
||||||
|
if(this != &p) inner_ = p.inner_;
|
||||||
|
return *this;
|
||||||
|
}
|
||||||
|
|
||||||
|
Progress::~Progress() {
|
||||||
|
if(inner_)
|
||||||
|
fputc('\n', inner_->output);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
#ifdef SPINNER
|
||||||
|
Spinner::~Spinner() {
|
||||||
|
//TODO: Add output: if(cur_) fputc('\n', output);
|
||||||
|
}
|
||||||
|
|
||||||
|
void Spinner::render(bool flush) {
|
||||||
|
/*char arr[] = {
|
||||||
|
'\r',
|
||||||
|
cur_,
|
||||||
|
0,
|
||||||
|
};
|
||||||
|
fputs(arr, output);
|
||||||
|
if(flush) fflush(output);*/
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
}
|
@ -0,0 +1,22 @@
|
|||||||
|
#include <shuffle3.h>
|
||||||
|
#include <reinterpret.hpp>
|
||||||
|
|
||||||
|
template<typename T>
|
||||||
|
static inline T* bytes_to_t(std::uint8_t* ptr, std::size_t ptr_sz, std::size_t* restrict nsize)
|
||||||
|
{
|
||||||
|
span<uint8_t> bytes(ptr, ptr_sz);
|
||||||
|
auto tout = bytes.reinterpret<T>();
|
||||||
|
*nsize = tout.size();
|
||||||
|
return tout.as_ptr();
|
||||||
|
}
|
||||||
|
|
||||||
|
extern "C" {
|
||||||
|
uint64_t* bytes_to_long(uint8_t* ptr, size_t ptr_sz, size_t* restrict nsize)
|
||||||
|
{
|
||||||
|
return bytes_to_t<uint64_t>(ptr, ptr_sz, nsize);
|
||||||
|
}
|
||||||
|
float* bytes_to_float(uint8_t* ptr, size_t ptr_sz, size_t* restrict nsize)
|
||||||
|
{
|
||||||
|
return bytes_to_t<float>(ptr, ptr_sz, nsize);
|
||||||
|
}
|
||||||
|
}
|
@ -0,0 +1,165 @@
|
|||||||
|
#include <stdexcept>
|
||||||
|
|
||||||
|
#include <iostream>
|
||||||
|
#include <cmath>
|
||||||
|
#include <sys/types.h>
|
||||||
|
|
||||||
|
#include <rng/impl.hpp>
|
||||||
|
|
||||||
|
#include <error.h>
|
||||||
|
#include <panic.h>
|
||||||
|
|
||||||
|
|
||||||
|
//#define IMPORT_ONE(NS, NAME) using NAME = NS :: NAME
|
||||||
|
|
||||||
|
//IMPORT_ONE(std, ssize_t);
|
||||||
|
|
||||||
|
template<typename To, typename T, typename Unit=unsigned char>
|
||||||
|
static inline To* offset_ptr(T* ptr, ssize_t by) noexcept
|
||||||
|
{
|
||||||
|
return reinterpret_cast<To*>(reinterpret_cast<Unit*>(ptr)+by);
|
||||||
|
}
|
||||||
|
/*
|
||||||
|
template<typename To, typename Unit>
|
||||||
|
offset_ptr(auto* p, std::ssize_t by) -> offset_ptr<To, decltype(p), Unit>;
|
||||||
|
template<typename To>
|
||||||
|
offset_ptr(auto* p, std::ssize_t by) -> offset_ptr<To, decltype(p)>;
|
||||||
|
*/
|
||||||
|
#ifdef FT_PT_OPERATORS
|
||||||
|
inline operator RNG*(rng_t ptr) noexcept
|
||||||
|
{
|
||||||
|
if(__builtin_expect(!ptr, false)) return nullptr;
|
||||||
|
#ifdef DEBUG
|
||||||
|
RNG* op =
|
||||||
|
#else
|
||||||
|
return
|
||||||
|
#endif
|
||||||
|
reinterpret_cast<RNG*>(offset_ptr<RNG>(ptr, -static_cast<ssize_t>(offsetof(RNG, _held))));
|
||||||
|
#ifdef DEBUG
|
||||||
|
if(__builtin_expect(!op, false)) return nullptr;
|
||||||
|
if(__builtin_expect(op->_held, false)) panic("Invalid rng_t -> RNG conversion");
|
||||||
|
return ptr;
|
||||||
|
#endif
|
||||||
|
}
|
||||||
|
|
||||||
|
RNG::operator rng_t() const noexcept
|
||||||
|
{
|
||||||
|
return reinterpret_cast<rng_t>(static_cast<RNG*>(this)->_held);
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
|
inline unsigned char RNG::byte()
|
||||||
|
{
|
||||||
|
return (unsigned char)next_int(255);
|
||||||
|
}
|
||||||
|
|
||||||
|
void RNG::bytes(unsigned char* ptr, std::size_t len)
|
||||||
|
{
|
||||||
|
for(std::size_t i=0;i<len;i++)
|
||||||
|
{
|
||||||
|
ptr[i] = byte();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
bool RNG::chance()
|
||||||
|
{
|
||||||
|
return chance(.5);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool RNG::chance(double d)
|
||||||
|
{
|
||||||
|
if (d<=0) return false;
|
||||||
|
return sample() <= d;
|
||||||
|
}
|
||||||
|
|
||||||
|
std::int32_t RNG::next_int(std::int32_t min, std::int32_t max)
|
||||||
|
{
|
||||||
|
return (std::int32_t)floor(sample() * (double)(max-min))+min;
|
||||||
|
}
|
||||||
|
|
||||||
|
std::int32_t RNG::next_int()
|
||||||
|
{
|
||||||
|
return (chance() ? 1 : -1) * (std::int32_t)floor(sample() * (double)INT32_MAX);
|
||||||
|
}
|
||||||
|
|
||||||
|
std::int64_t RNG::next_long(std::int64_t min, std::int64_t max)
|
||||||
|
{
|
||||||
|
return (std::int64_t)floor(sample() * (double)(max-min))+min;
|
||||||
|
}
|
||||||
|
|
||||||
|
std::int64_t RNG::next_long()
|
||||||
|
{
|
||||||
|
return (chance() ? 1l : -1l) * (std::int64_t)floor(sample() * (double)INT64_MAX);
|
||||||
|
}
|
||||||
|
|
||||||
|
#include <rng.h>
|
||||||
|
namespace { // C interface
|
||||||
|
using namespace std;
|
||||||
|
[[gnu::always_inline, gnu::gnu_inline]]
|
||||||
|
static inline RNG* extract_ptr(rng_t ptr)
|
||||||
|
{ return reinterpret_cast<RNG*> (ptr); }
|
||||||
|
static inline RNG& extract_ref(rng_t rng)
|
||||||
|
{
|
||||||
|
return *extract_ptr(rng);
|
||||||
|
}
|
||||||
|
template<std::derived_from<RNG> T>
|
||||||
|
static inline rng_t wrap_ptr(T* ptr) noexcept
|
||||||
|
{
|
||||||
|
if(__builtin_expect(!ptr, false)) return nullptr;
|
||||||
|
return reinterpret_cast<rng_t>(static_cast<RNG*>(ptr));
|
||||||
|
}
|
||||||
|
template<typename T>
|
||||||
|
static inline T* extract_downcast_ptr(rng_t rng)
|
||||||
|
{
|
||||||
|
return dynamic_cast<T*>(extract_ptr(rng));
|
||||||
|
}
|
||||||
|
|
||||||
|
extern "C"
|
||||||
|
{
|
||||||
|
rng_t rng_new(rng_init_opt opt)
|
||||||
|
{
|
||||||
|
switch(opt.kind)
|
||||||
|
{
|
||||||
|
case RNG_KIND_FRNG: return wrap_ptr( new rng::frng(opt.init.frng.state) );
|
||||||
|
case RNG_KIND_DRNG: return wrap_ptr( new rng::drng(opt.init.drng.state) );
|
||||||
|
case RNG_KIND_XORNG: return wrap_ptr( new rng::xoroshiro128plus(opt.init.xorng.state) );
|
||||||
|
//case RNG_KIND_LORENX: return static_cast<rng_t>( new rng::lorenzAttractor(opt.init.lorenz); );
|
||||||
|
default: panic("Unknown RNG init opt: %d", opt.kind);
|
||||||
|
__builtin_unreachable();
|
||||||
|
}
|
||||||
|
return nullptr;
|
||||||
|
}
|
||||||
|
|
||||||
|
void rng_free(rng_t rng)
|
||||||
|
{
|
||||||
|
RNG* ptr = extract_ptr(rng);
|
||||||
|
delete ptr;
|
||||||
|
}
|
||||||
|
|
||||||
|
[[gnu::nonnull(1)]]
|
||||||
|
void rng_test_spec(rng_t rng)
|
||||||
|
{
|
||||||
|
auto& ref = extract_ref(rng);
|
||||||
|
cout << "rng_test_spec (" << typeid(ref).name() << ")..." << std::flush;
|
||||||
|
if constexpr(!is_noexcept) {
|
||||||
|
#if EXCEPT
|
||||||
|
try {
|
||||||
|
rng::test_algo(std::move(ref));
|
||||||
|
}
|
||||||
|
catch(const std::exception& except) {
|
||||||
|
cout << "\r\r\r\tERROR: " << (except.what() ?: typeid(except).name()) << endl;
|
||||||
|
rng_free(rng);
|
||||||
|
throw;
|
||||||
|
} catch(...) {
|
||||||
|
cout << "\r\r\r\tERROR" << endl;
|
||||||
|
rng_free(rng);
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
} else rng::test_algo(std::move(ref));
|
||||||
|
|
||||||
|
cout << "\r\r\r\tOK" << endl;
|
||||||
|
rng_free(rng);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
@ -0,0 +1,21 @@
|
|||||||
|
#include <cstdlib>
|
||||||
|
#include <ctime>
|
||||||
|
#include <iostream>
|
||||||
|
|
||||||
|
#include <rng/drng.hpp>
|
||||||
|
|
||||||
|
namespace rng
|
||||||
|
{
|
||||||
|
drng drng::from_time() { return drng(time(NULL)); }
|
||||||
|
|
||||||
|
int drng::rand()
|
||||||
|
{
|
||||||
|
return rand_r(&state);
|
||||||
|
}
|
||||||
|
|
||||||
|
double drng::sample()
|
||||||
|
{
|
||||||
|
int val = rand_r(&state);
|
||||||
|
return (double)val / (double)RAND_MAX;
|
||||||
|
}
|
||||||
|
}
|
@ -0,0 +1,3 @@
|
|||||||
|
#include <rng/impl.hpp>
|
||||||
|
#include <rng/frng.hpp>
|
||||||
|
#include <iostream>
|
@ -0,0 +1,37 @@
|
|||||||
|
#include <iostream>
|
||||||
|
#include <rng.h>
|
||||||
|
|
||||||
|
namespace rng {
|
||||||
|
void test_algo(RNG&& rng)
|
||||||
|
{
|
||||||
|
using namespace std;
|
||||||
|
|
||||||
|
for(int i=0;i<10;i++) {
|
||||||
|
double d = rng.next_double();
|
||||||
|
long l = rng.next_long(-10, 10);
|
||||||
|
|
||||||
|
std::array<bool, 10> ar;
|
||||||
|
for(auto& i : ar) i = rng.chance();
|
||||||
|
|
||||||
|
cout << "\t(Sampled: " << d;
|
||||||
|
cout << ", Long: " << l;
|
||||||
|
cout << ", Bools: [ ";
|
||||||
|
for(const auto& i : ar) cout << i << " ";
|
||||||
|
cout << "])" << endl;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
extern "C" void rng_test()
|
||||||
|
{
|
||||||
|
using namespace std;
|
||||||
|
|
||||||
|
cout << "frng:" << endl;
|
||||||
|
rng::test_algo(rng::frng(1.0, 1.2));
|
||||||
|
|
||||||
|
cout << "drng:" << endl;
|
||||||
|
rng::test_algo(rng::drng(10));
|
||||||
|
|
||||||
|
cout << "xoroshiro128+:" << endl;
|
||||||
|
rng::test_algo(rng::xoroshiro128plus(100ul, 200ul));
|
||||||
|
}
|
@ -0,0 +1,86 @@
|
|||||||
|
#include <rng/xoroshiro128plus.hpp>
|
||||||
|
#include <iostream>
|
||||||
|
|
||||||
|
using u64 = std::uint64_t;
|
||||||
|
|
||||||
|
#define XO xoroshiro128plus
|
||||||
|
|
||||||
|
static inline constexpr u64 rotl(u64 x, int k)
|
||||||
|
{
|
||||||
|
return (x << k) | (x >> (64 - k));
|
||||||
|
}
|
||||||
|
|
||||||
|
namespace rng
|
||||||
|
{
|
||||||
|
inline constexpr u64 next(XO::State& s)
|
||||||
|
{
|
||||||
|
u64 s0 = s[0];
|
||||||
|
u64 s1 = s[1];
|
||||||
|
u64 result = s0 + s1;
|
||||||
|
|
||||||
|
s1 ^= s0;
|
||||||
|
s[0] = rotl(s0, 24) ^ s1 ^ (s1 << 16);
|
||||||
|
s[1] = rotl(s1, 37);
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
inline constexpr void xo_jump(XO::State& s)
|
||||||
|
{
|
||||||
|
constexpr const std::uint64_t JUMP[] = { 0xdf900294d8f554a5, 0x170865df4b3201fc };
|
||||||
|
|
||||||
|
std::uint64_t s0 = 0;
|
||||||
|
std::uint64_t s1 = 0;
|
||||||
|
for(u64 i = 0; i < sizeof JUMP / sizeof *JUMP; i++)
|
||||||
|
for(int b = 0; b < 64; b++) {
|
||||||
|
if (JUMP[i] & UINT64_C(1) << b) {
|
||||||
|
s0 ^= s[0];
|
||||||
|
s1 ^= s[1];
|
||||||
|
}
|
||||||
|
next(s);
|
||||||
|
}
|
||||||
|
|
||||||
|
s[0] = s0;
|
||||||
|
s[1] = s1;
|
||||||
|
}
|
||||||
|
|
||||||
|
inline constexpr void xo_long_jump(XO::State& s)
|
||||||
|
{
|
||||||
|
constexpr const uint64_t LONG_JUMP[] = { 0xd2a98b26625eee7b, 0xdddf9b1090aa7ac1 };
|
||||||
|
|
||||||
|
std::uint64_t s0 = 0;
|
||||||
|
std::uint64_t s1 = 0;
|
||||||
|
for(u64 i = 0; i < sizeof LONG_JUMP / sizeof *LONG_JUMP; i++)
|
||||||
|
for(int b = 0; b < 64; b++) {
|
||||||
|
if (LONG_JUMP[i] & UINT64_C(1) << b) {
|
||||||
|
s0 ^= s[0];
|
||||||
|
s1 ^= s[1];
|
||||||
|
}
|
||||||
|
next(s);
|
||||||
|
}
|
||||||
|
|
||||||
|
s[0] = s0;
|
||||||
|
s[1] = s1;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
void XO::jump() { xo_jump(state); }
|
||||||
|
void XO::long_jump() { xo_long_jump(state); }
|
||||||
|
|
||||||
|
std::uint64_t XO::next_ulong()
|
||||||
|
{
|
||||||
|
return next(state);
|
||||||
|
}
|
||||||
|
std:: int64_t XO::next_long()
|
||||||
|
{
|
||||||
|
const u64 v = next_ulong();
|
||||||
|
static_assert(sizeof(v) == sizeof(decltype(next_long())));
|
||||||
|
|
||||||
|
return *(const std::int64_t*)&v;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
double XO::sample()
|
||||||
|
{
|
||||||
|
return (next(state) & ((INT64_C(1) << 53) - 1)) * (1.00 / (INT64_C(1) << 53));
|
||||||
|
}
|
||||||
|
}
|
@ -0,0 +1,142 @@
|
|||||||
|
#include <tuple>
|
||||||
|
#include <functional>
|
||||||
|
#include <bit>
|
||||||
|
#include <cfloat>
|
||||||
|
|
||||||
|
#include <shuffle3.h>
|
||||||
|
#include <panic.h>
|
||||||
|
#include <map.h>
|
||||||
|
#include <reinterpret.h>
|
||||||
|
#include <shuffle.hpp>
|
||||||
|
|
||||||
|
#include <work.h>
|
||||||
|
#include <debug.h>
|
||||||
|
|
||||||
|
template<typename T, typename Fn>
|
||||||
|
std::tuple<T, T> minmax_t(const span<T>& array, Fn keep)
|
||||||
|
{
|
||||||
|
T highest
|
||||||
|
#ifdef DEBUG
|
||||||
|
{}
|
||||||
|
#endif
|
||||||
|
;
|
||||||
|
T lowest
|
||||||
|
#ifdef DEBUG
|
||||||
|
{}
|
||||||
|
#endif
|
||||||
|
;
|
||||||
|
bool init=false;
|
||||||
|
D_dprintf("minmax_t: %p (%lu)", array.as_ptr(), array.size());
|
||||||
|
for(std::size_t i=0;i<array.size();i++)
|
||||||
|
{
|
||||||
|
if(!keep(array[i])) continue;
|
||||||
|
|
||||||
|
auto item = array[i];
|
||||||
|
if(!init) {
|
||||||
|
init = true;
|
||||||
|
lowest = highest = item;
|
||||||
|
}
|
||||||
|
else if(item < lowest) lowest = item;
|
||||||
|
else if(item > highest) highest = item;
|
||||||
|
}
|
||||||
|
//fmt::print("MMX {}, {}\n", lowest, highest);
|
||||||
|
return {lowest, highest};
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename T>
|
||||||
|
inline std::tuple<T, T> minmax_t(const span<T>& array)
|
||||||
|
{
|
||||||
|
return minmax_t(array, [](T _val) { return true; });
|
||||||
|
}
|
||||||
|
|
||||||
|
namespace work
|
||||||
|
{
|
||||||
|
/// Shuffle or unshuffle in place
|
||||||
|
template<bool unshuffle>
|
||||||
|
int xshuffle_ip(const char* file)
|
||||||
|
{
|
||||||
|
mm::mmap map(file);
|
||||||
|
map.access(mm::Access::Random, true);
|
||||||
|
|
||||||
|
if constexpr(unshuffle)
|
||||||
|
{
|
||||||
|
auto [byte_l, byte_h] = minmax_t(map.as_span().reinterpret<std::int8_t>());
|
||||||
|
D_dprintf("MMX res (s8): %d -- %d", byte_l, byte_h);
|
||||||
|
rng::drng drng((std::int32_t) ((0xfffa << 16) | (byte_l<<7) | byte_h ));
|
||||||
|
rng::unshuffle(drng, map.as_span());
|
||||||
|
|
||||||
|
auto [float_l, float_h] = minmax_t(map.as_span().reinterpret<float>(), [](float f) -> bool { return !( (f!=f) || f < -FLT_MAX || f > FLT_MAX); });
|
||||||
|
D_dprintf("MMX res (f32): %f -- %f", float_l, float_h);
|
||||||
|
rng::frng frng(float_l, float_h);
|
||||||
|
rng::unshuffle(frng, map.as_span().reinterpret<float>());
|
||||||
|
|
||||||
|
auto [long_l, long_h] = minmax_t(map.as_span().reinterpret<std::int64_t>());
|
||||||
|
D_dprintf("MMX res (u64): %ld -- %ld", long_l, long_h);
|
||||||
|
rng::xoroshiro128plus xorng(std::bit_cast<std::uint64_t>(long_l), std::bit_cast<std::uint64_t>(long_h));
|
||||||
|
rng::unshuffle(xorng, map.as_span().reinterpret<std::int64_t>());
|
||||||
|
} else {
|
||||||
|
auto [long_l, long_h] = minmax_t(map.as_span().reinterpret<std::int64_t>());
|
||||||
|
D_dprintf("MMX res (u64): %ld -- %ld", long_l, long_h);
|
||||||
|
rng::xoroshiro128plus xorng(std::bit_cast<std::uint64_t>(long_l), std::bit_cast<std::uint64_t>(long_h));
|
||||||
|
rng::shuffle(xorng, map.as_span().reinterpret<std::int64_t>());
|
||||||
|
|
||||||
|
auto [float_l, float_h] = minmax_t(map.as_span().reinterpret<float>(), [](float f) -> bool { return !( (f!=f) || f < -FLT_MAX || f > FLT_MAX); });
|
||||||
|
D_dprintf("MMX res (f32): %f -- %f", float_l, float_h);
|
||||||
|
rng::frng frng(float_l, float_h);
|
||||||
|
rng::shuffle(frng, map.as_span().reinterpret<float>());
|
||||||
|
|
||||||
|
auto [byte_l, byte_h] = minmax_t(map.as_span().reinterpret<std::int8_t>());
|
||||||
|
D_dprintf("MMX res (s8): %d -- %d", byte_l, byte_h);
|
||||||
|
rng::drng drng((std::int32_t) ((0xfffa << 16) | (byte_l<<7) | byte_h ));
|
||||||
|
rng::shuffle(drng, map.as_span());
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Shuffle or unshuffle out of place
|
||||||
|
template<bool unshuffle>
|
||||||
|
int xshuffle_op(const char* ifile, const char* ofile, bool is_buffered)
|
||||||
|
{
|
||||||
|
//TODO: Use libcow's `cow_create_fd()`/`Cow::Cow(fd, size)` to try to map `ifile`
|
||||||
|
// Then, clone it, and use the `Fake` to do the shuffling on, and write the data to `ofile`.
|
||||||
|
// If `ifile` cannot be mapped, create a shim `Cow(size(ifile))`, `sendfile()/copy_file_range()/splice()` `ifile` into the shim memory fd, and use that (do *not* clone it to a Fake,)
|
||||||
|
// Then, perform the shuffling on the original `Cow`, and `sendfile()/copy_file_range()/splice()` the `ifile` file memory descriptor used for the `Cow` into `ofile`, and then destroy the `Cow`, **make sure to `msync()` the Cow'd range before the copy**.
|
||||||
|
mm::vmap imap; //{ifile};
|
||||||
|
|
||||||
|
if constexpr(unshuffle)
|
||||||
|
{
|
||||||
|
|
||||||
|
} else {
|
||||||
|
|
||||||
|
}
|
||||||
|
panic("Unimplemented");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
int help()
|
||||||
|
{
|
||||||
|
//Print help then exit
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
extern "C" int do_work(const work_args_t args)
|
||||||
|
{
|
||||||
|
using A = decltype(args.op);
|
||||||
|
switch (args.op) {
|
||||||
|
case A::OP_SHUFFLE_IP: return work::xshuffle_ip<false >(args.data.op_shuffle_ip.file);
|
||||||
|
case A::OP_SHUFFLE_OP: return work::xshuffle_op<false >(args.data.op_shuffle_op.ifile,
|
||||||
|
args.data.op_shuffle_op.ofile,
|
||||||
|
args.data.op_shuffle_op.buffered == WORK_BO_BUFFERED);
|
||||||
|
case A::OP_UNSHUFFLE_IP: return work::xshuffle_ip<true >(args.data.op_unshuffle_ip.file);
|
||||||
|
case A::OP_UNSHUFFLE_OP: return work::xshuffle_op<true>(args.data.op_unshuffle_op.ifile,
|
||||||
|
args.data.op_unshuffle_op.ofile,
|
||||||
|
args.data.op_unshuffle_op.buffered == WORK_BO_BUFFERED);
|
||||||
|
case A::OP_HELP: return help();
|
||||||
|
|
||||||
|
default: panic("Unknown op %d", (int)args.op);
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
@ -0,0 +1,51 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
TFILE=/tmp/shuffle3-test-$(uuidgen)
|
||||||
|
TFILE2=$TFILE-cmp
|
||||||
|
|
||||||
|
FSIZE=${FSIZE:-$(( 1024 * 1024 * 10))}
|
||||||
|
FCNT=${FCNT:-4}
|
||||||
|
|
||||||
|
TRUE_SIZE=$(( FSIZE * FCNT))
|
||||||
|
|
||||||
|
function cleanup {
|
||||||
|
rm -f $TFILE $TFILE2
|
||||||
|
}
|
||||||
|
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
stest() {
|
||||||
|
if fcmp $TFILE $TFILE2; then
|
||||||
|
echo ": shuffling $TRUE_SIZE bytes"
|
||||||
|
$1 -s $TFILE
|
||||||
|
if fcmp $TFILE $TFILE2; then
|
||||||
|
echo "Failed: Shuffle did nothing"
|
||||||
|
exit 1
|
||||||
|
else
|
||||||
|
echo ": unshuffling $TRUE_SIZE bytes"
|
||||||
|
$1 -u $TFILE
|
||||||
|
if fcmp $TFILE $TFILE2; then
|
||||||
|
echo "OK"
|
||||||
|
else
|
||||||
|
echo "Failed: Unshuffle didn't reproduce original file"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "Invalid init: Files weren't identical"
|
||||||
|
exit -1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
echo ">>> Initialising"
|
||||||
|
dd if=/dev/urandom of=$TFILE bs=$FSIZE count=$FCNT >> /dev/null 2>&1 || exit -1
|
||||||
|
cp $TFILE $TFILE2 || exit -1
|
||||||
|
|
||||||
|
for ex in "$@"; do
|
||||||
|
if [[ -f "${ex%% *}" ]] || command -v "${ex%% *}" >/dev/null 2>&1; then
|
||||||
|
echo ">>> Testing $ex"
|
||||||
|
stest "$ex" || exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "Passed."
|