Compare commits
73 Commits
@ -1,3 +1,5 @@
|
||||
*~
|
||||
obj
|
||||
build
|
||||
build/
|
||||
test/
|
||||
shuffle3-*
|
||||
|
||||
@ -0,0 +1,674 @@
|
||||
GNU GENERAL PUBLIC LICENSE
|
||||
Version 3, 29 June 2007
|
||||
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The GNU General Public License is a free, copyleft license for
|
||||
software and other kinds of works.
|
||||
|
||||
The licenses for most software and other practical works are designed
|
||||
to take away your freedom to share and change the works. By contrast,
|
||||
the GNU General Public License is intended to guarantee your freedom to
|
||||
share and change all versions of a program--to make sure it remains free
|
||||
software for all its users. We, the Free Software Foundation, use the
|
||||
GNU General Public License for most of our software; it applies also to
|
||||
any other work released this way by its authors. You can apply it to
|
||||
your programs, too.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
them if you wish), that you receive source code or can get it if you
|
||||
want it, that you can change the software or use pieces of it in new
|
||||
free programs, and that you know you can do these things.
|
||||
|
||||
To protect your rights, we need to prevent others from denying you
|
||||
these rights or asking you to surrender the rights. Therefore, you have
|
||||
certain responsibilities if you distribute copies of the software, or if
|
||||
you modify it: responsibilities to respect the freedom of others.
|
||||
|
||||
For example, if you distribute copies of such a program, whether
|
||||
gratis or for a fee, you must pass on to the recipients the same
|
||||
freedoms that you received. You must make sure that they, too, receive
|
||||
or can get the source code. And you must show them these terms so they
|
||||
know their rights.
|
||||
|
||||
Developers that use the GNU GPL protect your rights with two steps:
|
||||
(1) assert copyright on the software, and (2) offer you this License
|
||||
giving you legal permission to copy, distribute and/or modify it.
|
||||
|
||||
For the developers' and authors' protection, the GPL clearly explains
|
||||
that there is no warranty for this free software. For both users' and
|
||||
authors' sake, the GPL requires that modified versions be marked as
|
||||
changed, so that their problems will not be attributed erroneously to
|
||||
authors of previous versions.
|
||||
|
||||
Some devices are designed to deny users access to install or run
|
||||
modified versions of the software inside them, although the manufacturer
|
||||
can do so. This is fundamentally incompatible with the aim of
|
||||
protecting users' freedom to change the software. The systematic
|
||||
pattern of such abuse occurs in the area of products for individuals to
|
||||
use, which is precisely where it is most unacceptable. Therefore, we
|
||||
have designed this version of the GPL to prohibit the practice for those
|
||||
products. If such problems arise substantially in other domains, we
|
||||
stand ready to extend this provision to those domains in future versions
|
||||
of the GPL, as needed to protect the freedom of users.
|
||||
|
||||
Finally, every program is threatened constantly by software patents.
|
||||
States should not allow patents to restrict development and use of
|
||||
software on general-purpose computers, but in those that do, we wish to
|
||||
avoid the special danger that patents applied to a free program could
|
||||
make it effectively proprietary. To prevent this, the GPL assures that
|
||||
patents cannot be used to render the program non-free.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
TERMS AND CONDITIONS
|
||||
|
||||
0. Definitions.
|
||||
|
||||
"This License" refers to version 3 of the GNU General Public License.
|
||||
|
||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||
works, such as semiconductor masks.
|
||||
|
||||
"The Program" refers to any copyrightable work licensed under this
|
||||
License. Each licensee is addressed as "you". "Licensees" and
|
||||
"recipients" may be individuals or organizations.
|
||||
|
||||
To "modify" a work means to copy from or adapt all or part of the work
|
||||
in a fashion requiring copyright permission, other than the making of an
|
||||
exact copy. The resulting work is called a "modified version" of the
|
||||
earlier work or a work "based on" the earlier work.
|
||||
|
||||
A "covered work" means either the unmodified Program or a work based
|
||||
on the Program.
|
||||
|
||||
To "propagate" a work means to do anything with it that, without
|
||||
permission, would make you directly or secondarily liable for
|
||||
infringement under applicable copyright law, except executing it on a
|
||||
computer or modifying a private copy. Propagation includes copying,
|
||||
distribution (with or without modification), making available to the
|
||||
public, and in some countries other activities as well.
|
||||
|
||||
To "convey" a work means any kind of propagation that enables other
|
||||
parties to make or receive copies. Mere interaction with a user through
|
||||
a computer network, with no transfer of a copy, is not conveying.
|
||||
|
||||
An interactive user interface displays "Appropriate Legal Notices"
|
||||
to the extent that it includes a convenient and prominently visible
|
||||
feature that (1) displays an appropriate copyright notice, and (2)
|
||||
tells the user that there is no warranty for the work (except to the
|
||||
extent that warranties are provided), that licensees may convey the
|
||||
work under this License, and how to view a copy of this License. If
|
||||
the interface presents a list of user commands or options, such as a
|
||||
menu, a prominent item in the list meets this criterion.
|
||||
|
||||
1. Source Code.
|
||||
|
||||
The "source code" for a work means the preferred form of the work
|
||||
for making modifications to it. "Object code" means any non-source
|
||||
form of a work.
|
||||
|
||||
A "Standard Interface" means an interface that either is an official
|
||||
standard defined by a recognized standards body, or, in the case of
|
||||
interfaces specified for a particular programming language, one that
|
||||
is widely used among developers working in that language.
|
||||
|
||||
The "System Libraries" of an executable work include anything, other
|
||||
than the work as a whole, that (a) is included in the normal form of
|
||||
packaging a Major Component, but which is not part of that Major
|
||||
Component, and (b) serves only to enable use of the work with that
|
||||
Major Component, or to implement a Standard Interface for which an
|
||||
implementation is available to the public in source code form. A
|
||||
"Major Component", in this context, means a major essential component
|
||||
(kernel, window system, and so on) of the specific operating system
|
||||
(if any) on which the executable work runs, or a compiler used to
|
||||
produce the work, or an object code interpreter used to run it.
|
||||
|
||||
The "Corresponding Source" for a work in object code form means all
|
||||
the source code needed to generate, install, and (for an executable
|
||||
work) run the object code and to modify the work, including scripts to
|
||||
control those activities. However, it does not include the work's
|
||||
System Libraries, or general-purpose tools or generally available free
|
||||
programs which are used unmodified in performing those activities but
|
||||
which are not part of the work. For example, Corresponding Source
|
||||
includes interface definition files associated with source files for
|
||||
the work, and the source code for shared libraries and dynamically
|
||||
linked subprograms that the work is specifically designed to require,
|
||||
such as by intimate data communication or control flow between those
|
||||
subprograms and other parts of the work.
|
||||
|
||||
The Corresponding Source need not include anything that users
|
||||
can regenerate automatically from other parts of the Corresponding
|
||||
Source.
|
||||
|
||||
The Corresponding Source for a work in source code form is that
|
||||
same work.
|
||||
|
||||
2. Basic Permissions.
|
||||
|
||||
All rights granted under this License are granted for the term of
|
||||
copyright on the Program, and are irrevocable provided the stated
|
||||
conditions are met. This License explicitly affirms your unlimited
|
||||
permission to run the unmodified Program. The output from running a
|
||||
covered work is covered by this License only if the output, given its
|
||||
content, constitutes a covered work. This License acknowledges your
|
||||
rights of fair use or other equivalent, as provided by copyright law.
|
||||
|
||||
You may make, run and propagate covered works that you do not
|
||||
convey, without conditions so long as your license otherwise remains
|
||||
in force. You may convey covered works to others for the sole purpose
|
||||
of having them make modifications exclusively for you, or provide you
|
||||
with facilities for running those works, provided that you comply with
|
||||
the terms of this License in conveying all material for which you do
|
||||
not control copyright. Those thus making or running the covered works
|
||||
for you must do so exclusively on your behalf, under your direction
|
||||
and control, on terms that prohibit them from making any copies of
|
||||
your copyrighted material outside their relationship with you.
|
||||
|
||||
Conveying under any other circumstances is permitted solely under
|
||||
the conditions stated below. Sublicensing is not allowed; section 10
|
||||
makes it unnecessary.
|
||||
|
||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||
|
||||
No covered work shall be deemed part of an effective technological
|
||||
measure under any applicable law fulfilling obligations under article
|
||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||
similar laws prohibiting or restricting circumvention of such
|
||||
measures.
|
||||
|
||||
When you convey a covered work, you waive any legal power to forbid
|
||||
circumvention of technological measures to the extent such circumvention
|
||||
is effected by exercising rights under this License with respect to
|
||||
the covered work, and you disclaim any intention to limit operation or
|
||||
modification of the work as a means of enforcing, against the work's
|
||||
users, your or third parties' legal rights to forbid circumvention of
|
||||
technological measures.
|
||||
|
||||
4. Conveying Verbatim Copies.
|
||||
|
||||
You may convey verbatim copies of the Program's source code as you
|
||||
receive it, in any medium, provided that you conspicuously and
|
||||
appropriately publish on each copy an appropriate copyright notice;
|
||||
keep intact all notices stating that this License and any
|
||||
non-permissive terms added in accord with section 7 apply to the code;
|
||||
keep intact all notices of the absence of any warranty; and give all
|
||||
recipients a copy of this License along with the Program.
|
||||
|
||||
You may charge any price or no price for each copy that you convey,
|
||||
and you may offer support or warranty protection for a fee.
|
||||
|
||||
5. Conveying Modified Source Versions.
|
||||
|
||||
You may convey a work based on the Program, or the modifications to
|
||||
produce it from the Program, in the form of source code under the
|
||||
terms of section 4, provided that you also meet all of these conditions:
|
||||
|
||||
a) The work must carry prominent notices stating that you modified
|
||||
it, and giving a relevant date.
|
||||
|
||||
b) The work must carry prominent notices stating that it is
|
||||
released under this License and any conditions added under section
|
||||
7. This requirement modifies the requirement in section 4 to
|
||||
"keep intact all notices".
|
||||
|
||||
c) You must license the entire work, as a whole, under this
|
||||
License to anyone who comes into possession of a copy. This
|
||||
License will therefore apply, along with any applicable section 7
|
||||
additional terms, to the whole of the work, and all its parts,
|
||||
regardless of how they are packaged. This License gives no
|
||||
permission to license the work in any other way, but it does not
|
||||
invalidate such permission if you have separately received it.
|
||||
|
||||
d) If the work has interactive user interfaces, each must display
|
||||
Appropriate Legal Notices; however, if the Program has interactive
|
||||
interfaces that do not display Appropriate Legal Notices, your
|
||||
work need not make them do so.
|
||||
|
||||
A compilation of a covered work with other separate and independent
|
||||
works, which are not by their nature extensions of the covered work,
|
||||
and which are not combined with it such as to form a larger program,
|
||||
in or on a volume of a storage or distribution medium, is called an
|
||||
"aggregate" if the compilation and its resulting copyright are not
|
||||
used to limit the access or legal rights of the compilation's users
|
||||
beyond what the individual works permit. Inclusion of a covered work
|
||||
in an aggregate does not cause this License to apply to the other
|
||||
parts of the aggregate.
|
||||
|
||||
6. Conveying Non-Source Forms.
|
||||
|
||||
You may convey a covered work in object code form under the terms
|
||||
of sections 4 and 5, provided that you also convey the
|
||||
machine-readable Corresponding Source under the terms of this License,
|
||||
in one of these ways:
|
||||
|
||||
a) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by the
|
||||
Corresponding Source fixed on a durable physical medium
|
||||
customarily used for software interchange.
|
||||
|
||||
b) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by a
|
||||
written offer, valid for at least three years and valid for as
|
||||
long as you offer spare parts or customer support for that product
|
||||
model, to give anyone who possesses the object code either (1) a
|
||||
copy of the Corresponding Source for all the software in the
|
||||
product that is covered by this License, on a durable physical
|
||||
medium customarily used for software interchange, for a price no
|
||||
more than your reasonable cost of physically performing this
|
||||
conveying of source, or (2) access to copy the
|
||||
Corresponding Source from a network server at no charge.
|
||||
|
||||
c) Convey individual copies of the object code with a copy of the
|
||||
written offer to provide the Corresponding Source. This
|
||||
alternative is allowed only occasionally and noncommercially, and
|
||||
only if you received the object code with such an offer, in accord
|
||||
with subsection 6b.
|
||||
|
||||
d) Convey the object code by offering access from a designated
|
||||
place (gratis or for a charge), and offer equivalent access to the
|
||||
Corresponding Source in the same way through the same place at no
|
||||
further charge. You need not require recipients to copy the
|
||||
Corresponding Source along with the object code. If the place to
|
||||
copy the object code is a network server, the Corresponding Source
|
||||
may be on a different server (operated by you or a third party)
|
||||
that supports equivalent copying facilities, provided you maintain
|
||||
clear directions next to the object code saying where to find the
|
||||
Corresponding Source. Regardless of what server hosts the
|
||||
Corresponding Source, you remain obligated to ensure that it is
|
||||
available for as long as needed to satisfy these requirements.
|
||||
|
||||
e) Convey the object code using peer-to-peer transmission, provided
|
||||
you inform other peers where the object code and Corresponding
|
||||
Source of the work are being offered to the general public at no
|
||||
charge under subsection 6d.
|
||||
|
||||
A separable portion of the object code, whose source code is excluded
|
||||
from the Corresponding Source as a System Library, need not be
|
||||
included in conveying the object code work.
|
||||
|
||||
A "User Product" is either (1) a "consumer product", which means any
|
||||
tangible personal property which is normally used for personal, family,
|
||||
or household purposes, or (2) anything designed or sold for incorporation
|
||||
into a dwelling. In determining whether a product is a consumer product,
|
||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||
product received by a particular user, "normally used" refers to a
|
||||
typical or common use of that class of product, regardless of the status
|
||||
of the particular user or of the way in which the particular user
|
||||
actually uses, or expects or is expected to use, the product. A product
|
||||
is a consumer product regardless of whether the product has substantial
|
||||
commercial, industrial or non-consumer uses, unless such uses represent
|
||||
the only significant mode of use of the product.
|
||||
|
||||
"Installation Information" for a User Product means any methods,
|
||||
procedures, authorization keys, or other information required to install
|
||||
and execute modified versions of a covered work in that User Product from
|
||||
a modified version of its Corresponding Source. The information must
|
||||
suffice to ensure that the continued functioning of the modified object
|
||||
code is in no case prevented or interfered with solely because
|
||||
modification has been made.
|
||||
|
||||
If you convey an object code work under this section in, or with, or
|
||||
specifically for use in, a User Product, and the conveying occurs as
|
||||
part of a transaction in which the right of possession and use of the
|
||||
User Product is transferred to the recipient in perpetuity or for a
|
||||
fixed term (regardless of how the transaction is characterized), the
|
||||
Corresponding Source conveyed under this section must be accompanied
|
||||
by the Installation Information. But this requirement does not apply
|
||||
if neither you nor any third party retains the ability to install
|
||||
modified object code on the User Product (for example, the work has
|
||||
been installed in ROM).
|
||||
|
||||
The requirement to provide Installation Information does not include a
|
||||
requirement to continue to provide support service, warranty, or updates
|
||||
for a work that has been modified or installed by the recipient, or for
|
||||
the User Product in which it has been modified or installed. Access to a
|
||||
network may be denied when the modification itself materially and
|
||||
adversely affects the operation of the network or violates the rules and
|
||||
protocols for communication across the network.
|
||||
|
||||
Corresponding Source conveyed, and Installation Information provided,
|
||||
in accord with this section must be in a format that is publicly
|
||||
documented (and with an implementation available to the public in
|
||||
source code form), and must require no special password or key for
|
||||
unpacking, reading or copying.
|
||||
|
||||
7. Additional Terms.
|
||||
|
||||
"Additional permissions" are terms that supplement the terms of this
|
||||
License by making exceptions from one or more of its conditions.
|
||||
Additional permissions that are applicable to the entire Program shall
|
||||
be treated as though they were included in this License, to the extent
|
||||
that they are valid under applicable law. If additional permissions
|
||||
apply only to part of the Program, that part may be used separately
|
||||
under those permissions, but the entire Program remains governed by
|
||||
this License without regard to the additional permissions.
|
||||
|
||||
When you convey a copy of a covered work, you may at your option
|
||||
remove any additional permissions from that copy, or from any part of
|
||||
it. (Additional permissions may be written to require their own
|
||||
removal in certain cases when you modify the work.) You may place
|
||||
additional permissions on material, added by you to a covered work,
|
||||
for which you have or can give appropriate copyright permission.
|
||||
|
||||
Notwithstanding any other provision of this License, for material you
|
||||
add to a covered work, you may (if authorized by the copyright holders of
|
||||
that material) supplement the terms of this License with terms:
|
||||
|
||||
a) Disclaiming warranty or limiting liability differently from the
|
||||
terms of sections 15 and 16 of this License; or
|
||||
|
||||
b) Requiring preservation of specified reasonable legal notices or
|
||||
author attributions in that material or in the Appropriate Legal
|
||||
Notices displayed by works containing it; or
|
||||
|
||||
c) Prohibiting misrepresentation of the origin of that material, or
|
||||
requiring that modified versions of such material be marked in
|
||||
reasonable ways as different from the original version; or
|
||||
|
||||
d) Limiting the use for publicity purposes of names of licensors or
|
||||
authors of the material; or
|
||||
|
||||
e) Declining to grant rights under trademark law for use of some
|
||||
trade names, trademarks, or service marks; or
|
||||
|
||||
f) Requiring indemnification of licensors and authors of that
|
||||
material by anyone who conveys the material (or modified versions of
|
||||
it) with contractual assumptions of liability to the recipient, for
|
||||
any liability that these contractual assumptions directly impose on
|
||||
those licensors and authors.
|
||||
|
||||
All other non-permissive additional terms are considered "further
|
||||
restrictions" within the meaning of section 10. If the Program as you
|
||||
received it, or any part of it, contains a notice stating that it is
|
||||
governed by this License along with a term that is a further
|
||||
restriction, you may remove that term. If a license document contains
|
||||
a further restriction but permits relicensing or conveying under this
|
||||
License, you may add to a covered work material governed by the terms
|
||||
of that license document, provided that the further restriction does
|
||||
not survive such relicensing or conveying.
|
||||
|
||||
If you add terms to a covered work in accord with this section, you
|
||||
must place, in the relevant source files, a statement of the
|
||||
additional terms that apply to those files, or a notice indicating
|
||||
where to find the applicable terms.
|
||||
|
||||
Additional terms, permissive or non-permissive, may be stated in the
|
||||
form of a separately written license, or stated as exceptions;
|
||||
the above requirements apply either way.
|
||||
|
||||
8. Termination.
|
||||
|
||||
You may not propagate or modify a covered work except as expressly
|
||||
provided under this License. Any attempt otherwise to propagate or
|
||||
modify it is void, and will automatically terminate your rights under
|
||||
this License (including any patent licenses granted under the third
|
||||
paragraph of section 11).
|
||||
|
||||
However, if you cease all violation of this License, then your
|
||||
license from a particular copyright holder is reinstated (a)
|
||||
provisionally, unless and until the copyright holder explicitly and
|
||||
finally terminates your license, and (b) permanently, if the copyright
|
||||
holder fails to notify you of the violation by some reasonable means
|
||||
prior to 60 days after the cessation.
|
||||
|
||||
Moreover, your license from a particular copyright holder is
|
||||
reinstated permanently if the copyright holder notifies you of the
|
||||
violation by some reasonable means, this is the first time you have
|
||||
received notice of violation of this License (for any work) from that
|
||||
copyright holder, and you cure the violation prior to 30 days after
|
||||
your receipt of the notice.
|
||||
|
||||
Termination of your rights under this section does not terminate the
|
||||
licenses of parties who have received copies or rights from you under
|
||||
this License. If your rights have been terminated and not permanently
|
||||
reinstated, you do not qualify to receive new licenses for the same
|
||||
material under section 10.
|
||||
|
||||
9. Acceptance Not Required for Having Copies.
|
||||
|
||||
You are not required to accept this License in order to receive or
|
||||
run a copy of the Program. Ancillary propagation of a covered work
|
||||
occurring solely as a consequence of using peer-to-peer transmission
|
||||
to receive a copy likewise does not require acceptance. However,
|
||||
nothing other than this License grants you permission to propagate or
|
||||
modify any covered work. These actions infringe copyright if you do
|
||||
not accept this License. Therefore, by modifying or propagating a
|
||||
covered work, you indicate your acceptance of this License to do so.
|
||||
|
||||
10. Automatic Licensing of Downstream Recipients.
|
||||
|
||||
Each time you convey a covered work, the recipient automatically
|
||||
receives a license from the original licensors, to run, modify and
|
||||
propagate that work, subject to this License. You are not responsible
|
||||
for enforcing compliance by third parties with this License.
|
||||
|
||||
An "entity transaction" is a transaction transferring control of an
|
||||
organization, or substantially all assets of one, or subdividing an
|
||||
organization, or merging organizations. If propagation of a covered
|
||||
work results from an entity transaction, each party to that
|
||||
transaction who receives a copy of the work also receives whatever
|
||||
licenses to the work the party's predecessor in interest had or could
|
||||
give under the previous paragraph, plus a right to possession of the
|
||||
Corresponding Source of the work from the predecessor in interest, if
|
||||
the predecessor has it or can get it with reasonable efforts.
|
||||
|
||||
You may not impose any further restrictions on the exercise of the
|
||||
rights granted or affirmed under this License. For example, you may
|
||||
not impose a license fee, royalty, or other charge for exercise of
|
||||
rights granted under this License, and you may not initiate litigation
|
||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||
any patent claim is infringed by making, using, selling, offering for
|
||||
sale, or importing the Program or any portion of it.
|
||||
|
||||
11. Patents.
|
||||
|
||||
A "contributor" is a copyright holder who authorizes use under this
|
||||
License of the Program or a work on which the Program is based. The
|
||||
work thus licensed is called the contributor's "contributor version".
|
||||
|
||||
A contributor's "essential patent claims" are all patent claims
|
||||
owned or controlled by the contributor, whether already acquired or
|
||||
hereafter acquired, that would be infringed by some manner, permitted
|
||||
by this License, of making, using, or selling its contributor version,
|
||||
but do not include claims that would be infringed only as a
|
||||
consequence of further modification of the contributor version. For
|
||||
purposes of this definition, "control" includes the right to grant
|
||||
patent sublicenses in a manner consistent with the requirements of
|
||||
this License.
|
||||
|
||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||
patent license under the contributor's essential patent claims, to
|
||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||
propagate the contents of its contributor version.
|
||||
|
||||
In the following three paragraphs, a "patent license" is any express
|
||||
agreement or commitment, however denominated, not to enforce a patent
|
||||
(such as an express permission to practice a patent or covenant not to
|
||||
sue for patent infringement). To "grant" such a patent license to a
|
||||
party means to make such an agreement or commitment not to enforce a
|
||||
patent against the party.
|
||||
|
||||
If you convey a covered work, knowingly relying on a patent license,
|
||||
and the Corresponding Source of the work is not available for anyone
|
||||
to copy, free of charge and under the terms of this License, through a
|
||||
publicly available network server or other readily accessible means,
|
||||
then you must either (1) cause the Corresponding Source to be so
|
||||
available, or (2) arrange to deprive yourself of the benefit of the
|
||||
patent license for this particular work, or (3) arrange, in a manner
|
||||
consistent with the requirements of this License, to extend the patent
|
||||
license to downstream recipients. "Knowingly relying" means you have
|
||||
actual knowledge that, but for the patent license, your conveying the
|
||||
covered work in a country, or your recipient's use of the covered work
|
||||
in a country, would infringe one or more identifiable patents in that
|
||||
country that you have reason to believe are valid.
|
||||
|
||||
If, pursuant to or in connection with a single transaction or
|
||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||
covered work, and grant a patent license to some of the parties
|
||||
receiving the covered work authorizing them to use, propagate, modify
|
||||
or convey a specific copy of the covered work, then the patent license
|
||||
you grant is automatically extended to all recipients of the covered
|
||||
work and works based on it.
|
||||
|
||||
A patent license is "discriminatory" if it does not include within
|
||||
the scope of its coverage, prohibits the exercise of, or is
|
||||
conditioned on the non-exercise of one or more of the rights that are
|
||||
specifically granted under this License. You may not convey a covered
|
||||
work if you are a party to an arrangement with a third party that is
|
||||
in the business of distributing software, under which you make payment
|
||||
to the third party based on the extent of your activity of conveying
|
||||
the work, and under which the third party grants, to any of the
|
||||
parties who would receive the covered work from you, a discriminatory
|
||||
patent license (a) in connection with copies of the covered work
|
||||
conveyed by you (or copies made from those copies), or (b) primarily
|
||||
for and in connection with specific products or compilations that
|
||||
contain the covered work, unless you entered into that arrangement,
|
||||
or that patent license was granted, prior to 28 March 2007.
|
||||
|
||||
Nothing in this License shall be construed as excluding or limiting
|
||||
any implied license or other defenses to infringement that may
|
||||
otherwise be available to you under applicable patent law.
|
||||
|
||||
12. No Surrender of Others' Freedom.
|
||||
|
||||
If conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot convey a
|
||||
covered work so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you may
|
||||
not convey it at all. For example, if you agree to terms that obligate you
|
||||
to collect a royalty for further conveying from those to whom you convey
|
||||
the Program, the only way you could satisfy both those terms and this
|
||||
License would be to refrain entirely from conveying the Program.
|
||||
|
||||
13. Use with the GNU Affero General Public License.
|
||||
|
||||
Notwithstanding any other provision of this License, you have
|
||||
permission to link or combine any covered work with a work licensed
|
||||
under version 3 of the GNU Affero General Public License into a single
|
||||
combined work, and to convey the resulting work. The terms of this
|
||||
License will continue to apply to the part which is the covered work,
|
||||
but the special requirements of the GNU Affero General Public License,
|
||||
section 13, concerning interaction through a network will apply to the
|
||||
combination as such.
|
||||
|
||||
14. Revised Versions of this License.
|
||||
|
||||
The Free Software Foundation may publish revised and/or new versions of
|
||||
the GNU General Public License from time to time. Such new versions will
|
||||
be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the
|
||||
Program specifies that a certain numbered version of the GNU General
|
||||
Public License "or any later version" applies to it, you have the
|
||||
option of following the terms and conditions either of that numbered
|
||||
version or of any later version published by the Free Software
|
||||
Foundation. If the Program does not specify a version number of the
|
||||
GNU General Public License, you may choose any version ever published
|
||||
by the Free Software Foundation.
|
||||
|
||||
If the Program specifies that a proxy can decide which future
|
||||
versions of the GNU General Public License can be used, that proxy's
|
||||
public statement of acceptance of a version permanently authorizes you
|
||||
to choose that version for the Program.
|
||||
|
||||
Later license versions may give you additional or different
|
||||
permissions. However, no additional obligations are imposed on any
|
||||
author or copyright holder as a result of your choosing to follow a
|
||||
later version.
|
||||
|
||||
15. Disclaimer of Warranty.
|
||||
|
||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. Limitation of Liability.
|
||||
|
||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||
SUCH DAMAGES.
|
||||
|
||||
17. Interpretation of Sections 15 and 16.
|
||||
|
||||
If the disclaimer of warranty and limitation of liability provided
|
||||
above cannot be given local legal effect according to their terms,
|
||||
reviewing courts shall apply local law that most closely approximates
|
||||
an absolute waiver of all civil liability in connection with the
|
||||
Program, unless a warranty or assumption of liability accompanies a
|
||||
copy of the Program in return for a fee.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
state the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the program's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License
|
||||
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If the program does terminal interaction, make it output a short
|
||||
notice like this when it starts in an interactive mode:
|
||||
|
||||
<program> Copyright (C) <year> <name of author>
|
||||
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
|
||||
This is free software, and you are welcome to redistribute it
|
||||
under certain conditions; type `show c' for details.
|
||||
|
||||
The hypothetical commands `show w' and `show c' should show the appropriate
|
||||
parts of the General Public License. Of course, your program's commands
|
||||
might be different; for a GUI interface, you would use an "about box".
|
||||
|
||||
You should also get your employer (if you work as a programmer) or school,
|
||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||
For more information on this, and how to apply and follow the GNU GPL, see
|
||||
<https://www.gnu.org/licenses/>.
|
||||
|
||||
The GNU General Public License does not permit incorporating your program
|
||||
into proprietary programs. If your program is a subroutine library, you
|
||||
may consider it more useful to permit linking proprietary applications with
|
||||
the library. If this is what you want to do, use the GNU Lesser General
|
||||
Public License instead of this License. But first, please read
|
||||
<https://www.gnu.org/licenses/why-not-lgpl.html>.
|
||||
@ -1,45 +1,177 @@
|
||||
SRC = $(wildcard src/*.c)
|
||||
INCLUDE = include/
|
||||
SRC_C = $(wildcard src/*.c)
|
||||
SRC_CXX = $(wildcard src/*.cpp)
|
||||
SRC_CXX+= $(wildcard src/rng/*.cpp)
|
||||
|
||||
INCLUDE = include
|
||||
|
||||
PROJECT=shuffle3
|
||||
|
||||
BUILD=build
|
||||
# Link to these libraries dynamicalls
|
||||
SHARED_LIBS=fmt
|
||||
# Link to these libraries statically
|
||||
STATIC_LIBS=
|
||||
|
||||
# Currently supported:
|
||||
# _FS_SPILL_BUFFER: Use file backed buffer instead of memory backed one for unshuffling. See `shuffle3.h`.
|
||||
# Setting the value to `DYN` enables the dynamic buffer, setting it to `MAP` enabled memory-mapped buffer. `MAP` is usually the fastest of the 3 modes.
|
||||
# DEBUG: Pretend we're building a debug release even though we're not. Will enable additional output messages and may interfere with some optimisations
|
||||
FEATURE_FLAGS?=
|
||||
|
||||
COMMON_FLAGS+= $(addprefix -D,$(FEATURE_FLAGS)) -Wall -Wstrict-aliasing $(addprefix -I,$(INCLUDE)) -fno-strict-aliasing
|
||||
|
||||
ARCH?=native
|
||||
|
||||
INTRINSICS=avx sse3
|
||||
|
||||
OPT_FLAGS?=-fgraphite -fopenmp -floop-parallelize-all -ftree-parallelize-loops=4 \
|
||||
-floop-interchange -ftree-loop-distribution -floop-strip-mine -floop-block \
|
||||
-fno-stack-check
|
||||
|
||||
OPT_FLAGS+=$(addprefix -m,$(INTRINSICS))
|
||||
|
||||
ifneq ($(ARCH),)
|
||||
OPT_FLAGS+=$(addprefix -march=,$(ARCH))
|
||||
endif
|
||||
|
||||
|
||||
CXX_OPT_FLAGS?= $(OPT_FLAGS) -felide-constructors
|
||||
|
||||
CFLAGS += $(COMMON_FLAGS) --std=gnu11
|
||||
CXXFLAGS += $(COMMON_FLAGS) --std=gnu++23 -fno-exceptions
|
||||
# XXX: We might need exceptions soon, for OOP usage, because we try multiple approaches from most efficient to least.
|
||||
LDFLAGS += $(addsuffix .a,$(addprefix -l:lib,$(STATIC_LIBS))) $(addprefix -l,$(SHARED_LIBS))
|
||||
|
||||
STRIP=strip
|
||||
|
||||
RELEASE_COMMON_FLAGS?= -DNOTRACE
|
||||
RELEASE_COMMON_FLAGS+= -DRELEASE
|
||||
|
||||
RELEASE_CFLAGS?= -O3 -flto $(OPT_FLAGS) $(RELEASE_COMMON_FLAGS)
|
||||
RELEASE_CXXFLAGS?= -O3 -flto $(CXX_OPT_FLAGS) $(RELEASE_COMMON_FLAGS)
|
||||
RELEASE_LDFLAGS?= -Wl,-O3 -Wl,-flto
|
||||
|
||||
DEBUG_CFLAGS?= -Og -g -DDEBUG
|
||||
DEBUG_CXXFLAGS?= $(DEBUG_CFLAGS)
|
||||
DEBUG_LDFLAGS?=
|
||||
|
||||
# Objects
|
||||
|
||||
OBJ_C = $(addprefix obj/c/,$(SRC_C:.c=.o))
|
||||
OBJ_CXX = $(addprefix obj/cxx/,$(SRC_CXX:.cpp=.o))
|
||||
OBJ = $(OBJ_C) $(OBJ_CXX)
|
||||
|
||||
CFLAGS+= $(addprefix -I,$(INCLUDE)) -Wall -pedantic --std=gnu11
|
||||
LDFLAGS+= -lm
|
||||
# PGO stuff
|
||||
|
||||
RELEASE_CFLAGS?= -O3 -march=native -flto -fgraphite
|
||||
RELEASE_LDFLAGS?= -O3 -flto
|
||||
PGO_OBJ_C = $(addprefix obj/pgo/c/,$(SRC_C:.c=.o))
|
||||
PGO_OBJ_CXX = $(addprefix obj/pgo/cxx/,$(SRC_CXX:.cpp=.o))
|
||||
PGO_OBJ = $(PGO_OBJ_C) $(PGO_OBJ_CXX)
|
||||
|
||||
DEBUG_CFLAGS?= -g -O0
|
||||
DEBUG_LDFLAGS?= -O0
|
||||
PGO_ITERATIONS=5
|
||||
PGO_BLOCKS={1..4}
|
||||
PGO_SIZE={1024..4096}
|
||||
PGO_SET_LOC?=/tmp/$(PROJECT)-pgo
|
||||
|
||||
OBJ = $(addprefix obj/,$(SRC:.c=.o))
|
||||
PGO_FLAGS = -fprofile-generate
|
||||
PGO_LDFLAGS=
|
||||
|
||||
# Phonies
|
||||
|
||||
.PHONY: release
|
||||
release: | dirs $(BUILD)/$(PROJECT)-release
|
||||
release: | dirs $(PROJECT)-release
|
||||
|
||||
.PHONY: debug
|
||||
debug: | dirs $(BUILD)/$(PROJECT)-debug
|
||||
debug: | dirs $(PROJECT)-debug
|
||||
|
||||
.PHONY: pgo
|
||||
pgo: | dirs $(PROJECT)-pgo
|
||||
|
||||
.PHONY: test
|
||||
test: test-all
|
||||
|
||||
# Targets
|
||||
|
||||
dirs:
|
||||
@mkdir -p obj/src
|
||||
@mkdir -p $(BUILD)
|
||||
@mkdir -p obj/{pgo/,}c{,xx}/src{,/rng}
|
||||
|
||||
obj/%.o: %.c
|
||||
obj/c/%.o: %.c
|
||||
$(CC) -c $< $(CFLAGS) -o $@ $(LDFLAGS)
|
||||
|
||||
$(BUILD)/$(PROJECT)-release: CFLAGS+= $(RELEASE_CFLAGS)
|
||||
$(BUILD)/$(PROJECT)-release: LDFLAGS+= $(RELEASE_LDFLAGS)
|
||||
$(BUILD)/$(PROJECT)-release: $(OBJ)
|
||||
$(CC) $^ $(CFLAGS) -o $@ $(LDFLAGS)
|
||||
strip $@
|
||||
obj/cxx/%.o: %.cpp
|
||||
$(CXX) -c $< $(CXXFLAGS) -o $@ $(LDFLAGS)
|
||||
|
||||
obj/pgo/c/%.o: %.c
|
||||
$(CC) -c $< $(CFLAGS) $(PGO_FLAGS) -o $@ $(LDFLAGS) $(PGO_LDFLAGS)
|
||||
|
||||
obj/pgo/cxx/%.o: %.cpp
|
||||
$(CXX) -c $< $(CXXFLAGS) $(PGO_FLAGS) -o $@ $(LDFLAGS) $(PGO_LDFLAGS)
|
||||
|
||||
|
||||
$(PROJECT)-release: CFLAGS+= $(RELEASE_CFLAGS)
|
||||
$(PROJECT)-release: CXXFLAGS += $(RELEASE_CXXFLAGS)
|
||||
$(PROJECT)-release: LDFLAGS += $(RELEASE_LDFLAGS)
|
||||
$(PROJECT)-release: $(OBJ)
|
||||
$(CXX) $^ $(CXXFLAGS) -o $@ $(LDFLAGS)
|
||||
$(STRIP) $@
|
||||
|
||||
$(BUILD)/$(PROJECT)-debug: CFLAGS+= $(DEBUG_CFLAGS)
|
||||
$(BUILD)/$(PROJECT)-debug: LDFLAGS+= $(DEBUG_LDFLAGS)
|
||||
$(BUILD)/$(PROJECT)-debug: $(OBJ)
|
||||
$(CC) $^ $(CFLAGS) -o $@ $(LDFLAGS)
|
||||
$(PROJECT)-debug: CFLAGS+= $(DEBUG_CFLAGS)
|
||||
$(PROJECT)-debug: CXXFLAGS += $(DEBUG_CXXFLAGS)
|
||||
$(PROJECT)-debug: LDFLAGS += $(DEBUG_LDFLAGS)
|
||||
$(PROJECT)-debug: $(OBJ)
|
||||
$(CXX) $^ $(CXXFLAGS) -o $@ $(LDFLAGS)
|
||||
|
||||
clean:
|
||||
pgo-generate: CFLAGS+= $(RELEASE_CFLAGS)
|
||||
pgo-generate: CXXFLAGS += $(RELEASE_CXXFLAGS)
|
||||
pgo-generate: LDFLAGS += $(RELEASE_LDFLAGS)
|
||||
pgo-generate: $(PGO_OBJ)
|
||||
$(CXX) $^ $(CXXFLAGS) $(PGO_FLAGS) -o $@ $(LDFLAGS) $(PGO_LDFLAGS)
|
||||
|
||||
pgo-reset:
|
||||
-find ./obj/pgo -name \*.gcda -exec rm {} +
|
||||
|
||||
pgo-profile: | pgo-reset pgo-generate
|
||||
mkdir -p $(PGO_SET_LOC)
|
||||
for i in {1..$(PGO_ITERATIONS)}; do \
|
||||
block=$$(rng --of $(PGO_SIZE)); \
|
||||
block_count=$$(rng --of $(PGO_BLOCKS)); \
|
||||
dd if=/dev/urandom of=$(PGO_SET_LOC)/small bs=$$block count=$$block_count >> /dev/null 2>&1; \
|
||||
printf "Iteration $$i / $(PGO_ITERATIONS) ($$block * $$block_count)\r"; \
|
||||
( echo ">> $$i" >&2; \
|
||||
echo ">> $$i"; \
|
||||
./pgo-generate -s $(PGO_SET_LOC)/small && \
|
||||
./pgo-generate -u $(PGO_SET_LOC)/small && \
|
||||
./pgo-generate -h && \
|
||||
FCNT=1 ./test.sh ./pgo-generate && \
|
||||
FCNT=2 ./test.sh ./pgo-generate && \
|
||||
FCNT=3 ./test.sh ./pgo-generate && \
|
||||
FCNT=4 ./test.sh ./pgo-generate && \
|
||||
: \
|
||||
) >>$(PGO_SET_LOC)/stdout.log 2>>$(PGO_SET_LOC)/stderr.log || \exit $$?; \
|
||||
done
|
||||
$(shell command -v bat >/dev/null && echo "bat --pager=none" || echo cat) $(PGO_SET_LOC)/stdout.log; \
|
||||
$(shell command -v bat >/dev/null && echo "bat --pager=none" || echo cat) $(PGO_SET_LOC)/stderr.log >&2
|
||||
rm -rf $(PGO_SET_LOC)
|
||||
rm pgo-generate
|
||||
|
||||
pgo-use: PGO_FLAGS = -fprofile-use
|
||||
pgo-use: PGO_LDFLAGS = -lgcov -fprofile-use
|
||||
pgo-use: CFLAGS+= $(RELEASE_CFLAGS)
|
||||
pgo-use: CXXFLAGS += $(RELEASE_CXXFLAGS)
|
||||
pgo-use: LDFLAGS += $(RELEASE_LDFLAGS)
|
||||
pgo-use: $(PGO_OBJ)
|
||||
$(CXX) $^ $(CXXFLAGS) $(PGO_FLAGS) -o $@ $(LDFLAGS) $(PGO_LDFLAGS)
|
||||
|
||||
$(PROJECT)-pgo: | pgo-profile
|
||||
find ./obj/pgo -name \*.o -exec rm {} +
|
||||
$(MAKE) pgo-use
|
||||
mv pgo-use $@
|
||||
$(STRIP) $@
|
||||
|
||||
clean-rebuild:
|
||||
rm -rf obj
|
||||
rm -f $(BUILD)/*
|
||||
|
||||
clean: clean-rebuild
|
||||
rm -f $(PROJECT)-{release,debug,pgo}
|
||||
|
||||
test-all:
|
||||
@./test.sh ./$(PROJECT)-*
|
||||
|
||||
|
||||
@ -0,0 +1,96 @@
|
||||
# `shuffle3-lean` - Improved 3 stage byte shuffler
|
||||
|
||||
Deterministically and reversably shuffle a file's bytes around.
|
||||
|
||||
## Shuffling
|
||||
Shuffle a file in place
|
||||
|
||||
``` shell
|
||||
$ shuffle3 -s file
|
||||
```
|
||||
|
||||
## Unshuffling
|
||||
Unshuffle a file in place
|
||||
|
||||
``` shell
|
||||
$ shuffle3 -u file
|
||||
```
|
||||
|
||||
## Other options
|
||||
Run with `-h` for more options.
|
||||
|
||||
# Improvements from `v1`
|
||||
* **~70-80x** speedup from shuffle3 v1.0
|
||||
* Huge reduction in syscalls
|
||||
* Takes advantage of the kernel's fs cache
|
||||
* Can properly handle large files without core dumping
|
||||
* Doesn't dump huge amounts of trash onto each stack frame
|
||||
|
||||
## Performance
|
||||
[hyperfine](https://github.com/sharkdp/hyperfine) reports a **700-800%** speedup over `v1`.
|
||||
It's easy to see why.
|
||||
|
||||
### V1 flamegraph
|
||||
V1 uses a pesudo-array adaptor to perform filesystem reads, seeks, and writes. This causes a massive syscall overhead.
|
||||
|
||||
|
||||
### V2 flamegraph
|
||||
Whereas V2 uses a single `mmap()`.
|
||||
|
||||
|
||||
## Memory usage
|
||||
The [memusage](https://www.systutorials.com/docs/linux/man/1-memusage/) graph for `v1` shows extremely inefficient stack usage.
|
||||
|
||||
( the green is supposed to be a line, not a bar! )
|
||||
|
||||
This is due to how the unshuffler buffers RNG results.
|
||||
|
||||
`v1` naively used VLAs to store this buffer, which can baloon to 8 times the size of the file being unshuffled.
|
||||
It dumps this massive buffer onto the stack frame of a function that is called multiple times, causing massive and inefficient stack usage.
|
||||
|
||||
This can cause a segfault when attempting to unshuffle a large file, while shuffling a file of the same size might succeed.
|
||||
|
||||
### V2 improvement
|
||||
The `memusage` graph for `v2` is a lot more sane.
|
||||
|
||||
|
||||
`v2` instead allocates this buffer on the heap. Note the stable stack and heap usage.
|
||||
|
||||
|
||||
# Building
|
||||
Run `make` to build the normal binary. It will output to `shuffle3-release`.
|
||||
|
||||
## Release target
|
||||
The `release` (default) target uses the variables `RELEASE_CFLAGS`, `RELEASE_CXXFLAGS` and `RELEASE_LDFLAGS` to specify opitimisations, as well as the `OPT_FLAGS` variable. These can be set by you if you wish.
|
||||
|
||||
### Note
|
||||
The default `OPT_FLAGS` contains the flag `-march=native`. This may be underisable for you, in which case set the variable or modify the makefile to remove it.
|
||||
|
||||
## Debug target
|
||||
To build with debug information, run `make debug`. Extra debug flags can be provided with the `DEBUG_CFLAGS`, `DEBUG_CXXFLAGS` and `DEBUG_LDFLAGS` variables which have default values in the Makefile.
|
||||
|
||||
The build and unstripped binary will be `shuffle3-debug`.
|
||||
|
||||
## PGO target
|
||||
To build with Profile Guided Optimisation run `make pgo`, the stripped and optimised binary will be output to `shuffle3-pgo`.
|
||||
|
||||
## Notes
|
||||
Before switching between `release` and `debug` targets, remember to run `make clean`.
|
||||
To disable stripping of release build binaries, run with `make STRIP=: release`
|
||||
|
||||
### Compile-time flags
|
||||
There are some build-time flags you can switch while building by appending to the `FEATURE_FLAGS` variable.
|
||||
| Flag | Description |
|
||||
|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `DEBUG` | Pretend we're building a debug release even though we're not. |
|
||||
| `_FS_SPILL_BUFFER` | Spill buffers into a file if they grow over a threshold. Can cause massive slowdowns but prevent OOMs while unshuffling on systems with low available memory. See [shuffle3.h](./include/shuffle3.h) for more details |
|
||||
| `_FS_SPILL_BUFFER=DYN` | Same as above except allocates memory dynamically. Might be faster. |
|
||||
| `_FS_SPILL_BUFFER=MAP` | Same as above except it calls `fallocate()` and `mmap()` to prodive a buffer of the full size needed. Is usually the fastest of the options for `_FS_SPILL_BUFFER` and is preferrable if possible. |
|
||||
|
||||
|
||||
## Gentoo ebuild
|
||||
There is a gentoo ebuild for this project in the overlay [test-overlay](https://git.flanchan.moe/birb/test-overlay).
|
||||
[direct link](https://git.flanchan.moe/birb/test-overlay/src/branch/master/app-misc/shuffle3/shuffle3-2.0.0.ebuild)
|
||||
|
||||
# License
|
||||
GPL'd with <3
|
||||
@ -0,0 +1,62 @@
|
||||
* shuffle3-lean
|
||||
Redegisn/upgrade of =shuffle3=
|
||||
|
||||
* Goals
|
||||
- [X] Functioning in-place shuffle/unshuffle
|
||||
- [X] Shuffle
|
||||
- [X] Unshuffle
|
||||
- [X] Usable in-place s/us from command line
|
||||
- [X] Shuffle
|
||||
- [X] Unshuffle
|
||||
- [ ] Functioning out-of-place/in-memory shuffle/unshuffle
|
||||
- [ ] Shuffle
|
||||
- [ ] Unshuffle
|
||||
- [ ] Usable out-of-place s/us from command line
|
||||
- [ ] Shuffle
|
||||
- [ ] Unshuffle
|
||||
|
||||
** NO compatibility with =shuffle3=
|
||||
=shuffle3='s ~drng~ PRNG algorithm uses an outdated global state backend. We don't want to reuse this.
|
||||
As a result, output from =shuffle3= and =shuffle3-lean= is different.
|
||||
|
||||
* Improvements
|
||||
- *~70-80x* speedup from shuffle3 1.0
|
||||
- Huge reduction in syscalls
|
||||
- Takes advantage of the kernel's fs cache
|
||||
- Can properly handle large files without core dumping
|
||||
- Doesn't dump huge amounts of trash onto each stack frame
|
||||
|
||||
** Performance
|
||||
[[https://github.com/sharkdp/hyperfine][hyperfine]] reports a *700-800%* speedup over =v1=.
|
||||
|
||||
It's easy to see why.
|
||||
*** V1 flamegraph
|
||||
V1 uses a pesudo-array adaptor to perform filesystem reads, seeks, and writes. This causes a massive syscall overhead.
|
||||
[[./profiling/release-flame-old.svg]]
|
||||
*** V2 flamegraph
|
||||
Whereas V2 uses a single ~mmap()~.
|
||||
[[./profiling/release-flame.svg]]
|
||||
|
||||
** Memory usage
|
||||
The [[https://www.systutorials.com/docs/linux/man/1-memusage/][memusage]] graph for =v1= shows extremely inefficient stack usage.
|
||||
[[./profiling/old-mem.png]]
|
||||
( the green is supposed to be a line, not a bar)
|
||||
This is due to how the unshuffler buffers RNG results.
|
||||
|
||||
=v1= naively used VLAs to store this buffer, which can baloon to 8 times the size of the file being unshuffled.
|
||||
It dumps this massive buffer onto the stack frame of a function that is called multiple times, causing massive and inefficient stack usage.
|
||||
|
||||
This can cause a segfault when attempting to unshuffle a large file, while shuffling a file of the same size might succeed.
|
||||
|
||||
*** V2 improvement
|
||||
The ~memusage~ graph for =v2= is a lot more sane.
|
||||
[[./profiling/mem.png]]
|
||||
|
||||
~v2~ instead allocates this buffer on the heap. Note the stable stack and heap usage.
|
||||
* Todo
|
||||
- [X] impl rng
|
||||
- [X] impl shuffling
|
||||
- [ ] impl out-of-place shuffling
|
||||
- [-] arg parsing and dispatch
|
||||
- [X] simple parsing
|
||||
- [ ] complex parsing
|
||||
@ -0,0 +1,28 @@
|
||||
// https://stackoverflow.com/a/30304782
|
||||
|
||||
#ifndef _COLORS_
|
||||
#define _COLORS_
|
||||
|
||||
/* FOREGROUND */
|
||||
#define RST "\x1B[0m"
|
||||
#define KRED "\x1B[31m"
|
||||
#define KGRN "\x1B[32m"
|
||||
#define KYEL "\x1B[33m"
|
||||
#define KBLU "\x1B[34m"
|
||||
#define KMAG "\x1B[35m"
|
||||
#define KCYN "\x1B[36m"
|
||||
#define KWHT "\x1B[37m"
|
||||
|
||||
#define FRED(x) KRED x RST
|
||||
#define FGRN(x) KGRN x RST
|
||||
#define FYEL(x) KYEL x RST
|
||||
#define FBLU(x) KBLU x RST
|
||||
#define FMAG(x) KMAG x RST
|
||||
#define FCYN(x) KCYN x RST
|
||||
#define FWHT(x) KWHT x RST
|
||||
|
||||
#define BOLD(x) "\x1B[1m" x RST
|
||||
#define UNDL(x) "\x1B[4m" x RST
|
||||
|
||||
#endif /* _COLORS_ */
|
||||
|
||||
@ -0,0 +1,55 @@
|
||||
#ifndef _DEBUG_H
|
||||
#define _DEBUG_H
|
||||
|
||||
// If `TRACE` is defined, or if `NOTRACE` is defined, use simplified debug info
|
||||
#if defined(TRACE) || !defined(NOTRACE)
|
||||
#define D_TRACE
|
||||
#define _D__FUNC __PRETTY_FUNCTION__
|
||||
#else
|
||||
#define _D__FUNC __func__
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
struct debuginfo {
|
||||
const char* file;
|
||||
const char* function;
|
||||
//const char* pfunction; //TODO: for `#if defined(TRACE)`, __PRETTY_FUNCTION__ goes here, __func__ goes above.
|
||||
int line;
|
||||
};
|
||||
|
||||
void _do_dprintf(struct debuginfo di, const char* fmt, ...);
|
||||
|
||||
#ifdef __cplusplus
|
||||
#define _D_COMPTIME constexpr
|
||||
extern "C++" {
|
||||
#include <utility>
|
||||
template<typename... Args>
|
||||
inline void _real_dprintf(const char* file, const char* function, int line, const char* fmt, Args&&... args)
|
||||
{
|
||||
#ifdef DEBUG
|
||||
debuginfo i = { file, function, line };
|
||||
_do_dprintf(i, fmt, std::forward<Args>(args)...);
|
||||
#endif
|
||||
}
|
||||
#define D_dprintf(fmt, ...) do { if consteval { (void)0; } else { _real_dprintf(__FILE__, _D__FUNC, __LINE__, fmt __VA_OPT__(,) __VA_ARGS__); } } while(0)
|
||||
}
|
||||
#else
|
||||
#define _D_COMPTIME
|
||||
|
||||
#ifdef DEBUG
|
||||
#define D_dprintf(fmt, ...) _do_dprintf( (struct debuginfo){.file = __FILE__, .function = _D__FUNC, .line = __LINE__}, fmt __VA_OPT__(,) __VA_ARGS__)
|
||||
#else
|
||||
static _D_COMPTIME inline void _do__nothing(const char* fmt, ...) {}
|
||||
#define D_dprintf(fmt, ...) _do__nothing(fmt __VA_OPT__(,) __VA_ARGS__) //(fmt __VA_OPT__(,) __VA_ARGS__, (void)0)
|
||||
#endif
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
#undef _D_COMPTIME
|
||||
|
||||
#endif /* _DEBUG_H */
|
||||
@ -0,0 +1,36 @@
|
||||
#ifndef _ERROR_H
|
||||
#define _ERROR_H
|
||||
|
||||
#ifndef $PASTE
|
||||
# define $_PASTE(x,y) x ## y
|
||||
# define $PASTE(x,y) $_PASTE(x,y)
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
|
||||
constexpr inline bool is_noexcept=
|
||||
#if __cpp_exceptions
|
||||
#define EXCEPT 1
|
||||
//#define try try
|
||||
//#define catch(...) catch(__VA_ARGS__)
|
||||
false
|
||||
#else
|
||||
#define EXCEPT 0
|
||||
#define NOEXCEPT
|
||||
//#define catch(...) __try {} catch(__VA_ARGS__)
|
||||
//#define try if constexpr(!is_noexcept)
|
||||
//#define throw (void)0
|
||||
true
|
||||
#endif
|
||||
;
|
||||
|
||||
|
||||
#endif
|
||||
|
||||
#endif /* _ERROR_H */
|
||||
@ -0,0 +1,31 @@
|
||||
#ifndef _FSVEC_H
|
||||
#define _FSVEC_H
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <stdint.h>
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#define restrict __restrict__
|
||||
#endif
|
||||
|
||||
// A simple file-backed back inserter
|
||||
typedef struct {
|
||||
FILE* backing;
|
||||
size_t len;
|
||||
} fvec_t;
|
||||
|
||||
int fvec_new(fvec_t* restrict obj, const char* path);
|
||||
void fvec_close(fvec_t* restrict obj);
|
||||
|
||||
int fvec_pop_end(fvec_t* restrict obj, size_t sz);
|
||||
void fvec_push_whole_buffer(fvec_t* restrict obj, const void* _buffer, size_t sz);
|
||||
int fvec_get_whole_buffer(const fvec_t* restrict obj, void* _buffer, size_t _sz);
|
||||
|
||||
#ifdef __cplusplus
|
||||
#undef restrict
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* _FSVEC_H */
|
||||
@ -0,0 +1,324 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <cstddef>
|
||||
#include <memory>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
#include <map.h>
|
||||
#include <tempfile.hpp>
|
||||
#include <panic.h>
|
||||
#include <debug.h>
|
||||
|
||||
|
||||
#include <shuffle3.h>
|
||||
|
||||
template<typename T>
|
||||
struct i_back_inserter
|
||||
{
|
||||
virtual void push_back(T&& value) =0;
|
||||
virtual void pop_back() =0;
|
||||
virtual const T& back() const =0;
|
||||
virtual T& back() =0;
|
||||
virtual const std::size_t size() const =0;
|
||||
|
||||
inline bool is_empty() const { return size()==0; }
|
||||
|
||||
virtual inline ~i_back_inserter() =default;
|
||||
};
|
||||
|
||||
struct i_shunt
|
||||
{
|
||||
virtual bool is_full() const =0;
|
||||
virtual inline ~i_shunt() =default;
|
||||
};
|
||||
|
||||
struct file_back_buffer
|
||||
{
|
||||
const static constexpr std::size_t DEFAULT_CAP = 1024;
|
||||
typedef std::uint8_t byte;
|
||||
|
||||
file_back_buffer();
|
||||
file_back_buffer(std::size_t cap);
|
||||
file_back_buffer(const file_back_buffer& c) = delete;
|
||||
file_back_buffer(file_back_buffer&& m);
|
||||
|
||||
void push_buf(byte* buf, std::size_t len);
|
||||
bool back(byte* buf, std::size_t len) const;
|
||||
bool pop_n(std::size_t len);
|
||||
|
||||
~file_back_buffer();
|
||||
private:
|
||||
struct impl;
|
||||
std::unique_ptr<impl> inner;
|
||||
};
|
||||
|
||||
namespace {
|
||||
template<typename T>
|
||||
inline const T* _die_if_null(const T* input, const char* msg)
|
||||
{
|
||||
if(!input) panic(msg);
|
||||
return input;
|
||||
}
|
||||
template<typename T>
|
||||
inline T* _die_if_null(T* input, const char* msg)
|
||||
{
|
||||
if(!input) panic(msg);
|
||||
return input;
|
||||
}
|
||||
}
|
||||
|
||||
template<typename T>
|
||||
struct file_vector : public i_back_inserter<T>
|
||||
{
|
||||
inline file_vector() : file_vector(file_back_buffer::DEFAULT_CAP){}
|
||||
inline file_vector(std::size_t cap) : inserter(file_back_buffer(cap)), len(0), current_back(std::vector<unsigned char>(sizeof(T))) {current_back.resize(sizeof(T));}
|
||||
inline file_vector(const file_vector<T>& c) = delete;
|
||||
inline file_vector(file_vector<T>&& m) : inserter(std::move(m.inserter)), len(m.len), current_back(std::move(m.current_back)){}
|
||||
|
||||
inline void push_back(T&& value) override
|
||||
{
|
||||
inserter.push_buf((file_back_buffer::byte*)&value, sizeof(T));
|
||||
len += 1;
|
||||
}
|
||||
inline T& back() override
|
||||
{
|
||||
if(!len) panic("back() called on empty file_vector");
|
||||
if(!inserter.back(¤t_back[0], sizeof(T))) panic("back() failed");
|
||||
return *_die_if_null((T*)¤t_back[0], "file_vector::back() returned null pointer");
|
||||
}
|
||||
inline const T& back() const override
|
||||
{
|
||||
if(!len) panic("back() called on empty file_vector");
|
||||
if(!inserter.back(¤t_back[0], sizeof(T))) panic("back() failed");
|
||||
return *_die_if_null((const T*)¤t_back[0], "file_vector::back() (const) returned null pointer");
|
||||
}
|
||||
inline void pop_back() override
|
||||
{
|
||||
if(!len) return;
|
||||
|
||||
if(!inserter.pop_n(sizeof(T))) panic("pop_back(): 0 elements");
|
||||
len-=1;
|
||||
}
|
||||
|
||||
inline const std::size_t size() const override { return len; }
|
||||
private:
|
||||
file_back_buffer inserter;
|
||||
std::size_t len=0;
|
||||
mutable std::vector<unsigned char> current_back; // what an awful hack...
|
||||
};
|
||||
|
||||
|
||||
template<typename T, std::size_t Spill = FSV_DEFAULT_SPILL_AT >
|
||||
requires (Spill > 0)
|
||||
struct fixed_spill_vector : public i_back_inserter<T>
|
||||
{
|
||||
constexpr const static std::size_t SPILL_AT = Spill;
|
||||
|
||||
inline fixed_spill_vector() : mem(std::make_unique<std::array<T, Spill> >()){
|
||||
D_dprintf("alloc cap (static): %lu", Spill);
|
||||
}
|
||||
inline fixed_spill_vector(const fixed_spill_vector<T>& c) = delete;
|
||||
inline fixed_spill_vector(fixed_spill_vector<T>&& m)
|
||||
: mem(std::move(m.mem)),
|
||||
mem_fill_ptr(m.mem_fill_ptr),
|
||||
fil(std::move(m.fil))
|
||||
{}
|
||||
inline ~fixed_spill_vector() = default;
|
||||
|
||||
inline void push_back(T&& value) override
|
||||
{
|
||||
if(mem_is_full()) {
|
||||
//D_dprintf("Inserting value into fs");
|
||||
fil.push_back(std::move(value));
|
||||
} else {
|
||||
//D_dprintf("Inserting value into memory");
|
||||
(*mem)[++mem_fill_ptr] = value;
|
||||
}
|
||||
}
|
||||
inline void pop_back() override
|
||||
{
|
||||
if(!size()) return;
|
||||
|
||||
if(fil.size()) {
|
||||
//D_dprintf("Popping from fs");
|
||||
fil.pop_back();
|
||||
} else {
|
||||
//D_dprintf("Popping from memory %ld", mem_fill_ptr);
|
||||
mem_fill_ptr -= 1;
|
||||
}
|
||||
}
|
||||
inline const T& back() const override
|
||||
{
|
||||
if (!size()) panic("back() (const) called on no elements");
|
||||
if(fil.size()) return fil.back();
|
||||
else return (*mem)[mem_fill_ptr];
|
||||
}
|
||||
inline T& back() override
|
||||
{
|
||||
if (!size()) panic("back() called on no elements");
|
||||
if(fil.size()) return fil.back();
|
||||
else return (*mem)[mem_fill_ptr];
|
||||
}
|
||||
inline const std::size_t size() const override
|
||||
{
|
||||
return fil.size() + (std::size_t)(mem_fill_ptr+1);
|
||||
}
|
||||
private:
|
||||
inline bool mem_is_full() const { return mem_fill_ptr >= (ssize_t)(Spill-1); }
|
||||
ssize_t mem_fill_ptr=-1;
|
||||
|
||||
std::unique_ptr<std::array<T, Spill>> mem;
|
||||
file_vector<T> fil;
|
||||
};
|
||||
|
||||
template<typename T>
|
||||
struct mapped_vector : public i_back_inserter<T>, public i_shunt
|
||||
{
|
||||
inline static mapped_vector<T> from_temp(std::size_t sz)
|
||||
{
|
||||
D_dprintf("generating with %lu size", sz);
|
||||
temp_file file;
|
||||
mapped_vector<T> mvec(file.full_path().c_str(), sz);
|
||||
D_dprintf("generated?");
|
||||
mvec.temp = std::make_unique<temp_file>(std::move(file));
|
||||
return mvec;
|
||||
}
|
||||
inline mapped_vector(const char* file, std::size_t sz)
|
||||
: sz(sz),
|
||||
temp(nullptr),
|
||||
map(mm::mmap::allocate(file, sz * sizeof(T))){}
|
||||
inline mapped_vector(const mapped_vector<T>& c) = delete;
|
||||
inline mapped_vector(mapped_vector<T>&& m)
|
||||
: sz(m.sz),
|
||||
fill_ptr(m.fill_ptr),
|
||||
temp(std::move(m.temp)),
|
||||
map(std::move(m.map)){}
|
||||
inline mapped_vector() : mapped_vector(nullptr, 0)
|
||||
{
|
||||
panic("unsupported");
|
||||
}
|
||||
|
||||
inline void push_back(T&& value) override
|
||||
{
|
||||
if(is_full()) panic("Tried to push past end of map");
|
||||
else memory()[++fill_ptr] = value;
|
||||
}
|
||||
inline void pop_back() override
|
||||
{
|
||||
if(fill_ptr>=0) fill_ptr-=1;
|
||||
}
|
||||
|
||||
inline T& back() override
|
||||
{
|
||||
if(fill_ptr>=0)
|
||||
{
|
||||
return memory()[fill_ptr];
|
||||
} else panic("back() called with no elements");
|
||||
}
|
||||
inline const T& back() const override
|
||||
{
|
||||
if(fill_ptr>=0)
|
||||
{
|
||||
return memory()[fill_ptr];
|
||||
} else panic("back() const called with no elements");
|
||||
}
|
||||
inline const std::size_t size() const override { return ((std::size_t)fill_ptr)+1; }
|
||||
inline std::size_t cap() const { return sz; }
|
||||
|
||||
inline bool is_full() const { return fill_ptr >= (ssize_t)(sz-1); }
|
||||
protected:
|
||||
inline const span<T> memory() const { return map.as_span().reinterpret<T>(); }
|
||||
inline span<T> memory() { return map.as_span().reinterpret<T>(); }
|
||||
private:
|
||||
std::size_t sz;
|
||||
ssize_t fill_ptr=-1;
|
||||
|
||||
std::unique_ptr<temp_file> temp;
|
||||
mm::mmap map;
|
||||
};
|
||||
|
||||
template<typename T, typename Shunt>
|
||||
requires(std::is_base_of<i_back_inserter<T>, Shunt >::value)
|
||||
struct shunt : public i_back_inserter<T>, protected i_shunt
|
||||
{
|
||||
typedef Shunt spill_type;
|
||||
|
||||
inline shunt() : shunt(FSV_DEFAULT_SPILL_AT){}
|
||||
inline shunt(spill_type&& into) : shunt(FSV_DEFAULT_SPILL_AT, std::move(into)){}
|
||||
inline shunt(std::size_t cap) : shunt(cap, cap){}
|
||||
inline shunt(std::size_t cap, spill_type&& into) : shunt(cap, cap, std::move(into)){}
|
||||
inline shunt(std::size_t cap, std::size_t spill, spill_type&& into)
|
||||
: _spill_at(spill), mem(std::vector<T>()), fil(std::make_unique<spill_type>(std::move(into))) {
|
||||
mem.reserve(cap);
|
||||
|
||||
D_dprintf("alloc (explicit) cap %lu (sz %lu == 0?), spill %lu", cap, mem.size(), spill_at());
|
||||
}
|
||||
inline shunt(std::size_t cap, std::size_t spill) : _spill_at(spill), mem(std::vector<T>()), fil(nullptr) {
|
||||
mem.reserve(cap);
|
||||
D_dprintf("alloc cap %lu (sz %lu == 0?), spill %lu", cap, mem.size(), spill_at());
|
||||
}
|
||||
inline shunt(const shunt<T, Shunt>& c) = delete;
|
||||
inline shunt(shunt<T, Shunt>&& m) :
|
||||
_spill_at(m._spill_at),
|
||||
mem(std::move(m.mem)),
|
||||
fil(std::move(m.fil)){}
|
||||
|
||||
inline void push_back(T&& value) override
|
||||
{
|
||||
if(is_full()) {
|
||||
spl()->push_back(std::move(value));
|
||||
}
|
||||
else mem.push_back(std::move(value));
|
||||
}
|
||||
inline void pop_back() override
|
||||
{
|
||||
if(is_full() && spl()->size()) spl()->pop_back();
|
||||
else mem.pop_back();
|
||||
}
|
||||
inline const T& back() const override
|
||||
{
|
||||
if(is_full() && spl()->size()) return spl()->back();
|
||||
else return mem.back();
|
||||
}
|
||||
inline T& back() override
|
||||
{
|
||||
if(is_full() && spl()->size()) return spl()->back();
|
||||
else return mem.back();
|
||||
}
|
||||
inline const std::size_t size() const override { return mem.size() + (fil ? fil->size() : 0); }
|
||||
|
||||
inline const std::size_t spill_at() const { return _spill_at; }
|
||||
inline bool is_spilling() const { return is_full(); }
|
||||
protected:
|
||||
inline bool is_full() const override { return size()>=spill_at(); }
|
||||
private:
|
||||
inline const spill_type* spl() const
|
||||
{
|
||||
if(is_full()) {
|
||||
if(!fil) fil = std::make_unique<spill_type>();
|
||||
return fil.get();
|
||||
} else {
|
||||
return fil.get();
|
||||
}
|
||||
}
|
||||
inline spill_type* spl()
|
||||
{
|
||||
if(is_full()) {
|
||||
if(!fil) fil = std::make_unique<spill_type>();
|
||||
return fil.get();
|
||||
} else {
|
||||
return fil.get();
|
||||
}
|
||||
}
|
||||
std::size_t _spill_at;
|
||||
std::vector<T> mem;
|
||||
mutable std::unique_ptr<spill_type> fil;
|
||||
};
|
||||
|
||||
template<typename T>
|
||||
using dynamic_spill_vector = shunt<T, file_vector<T> >;
|
||||
|
||||
template<typename T>
|
||||
using mapped_spill_vector = shunt<T, mapped_vector<T> >;
|
||||
@ -0,0 +1,202 @@
|
||||
#ifndef _MAP_H
|
||||
#define _MAP_H
|
||||
|
||||
#ifdef __cplusplus
|
||||
#include <stdexcept>
|
||||
extern "C" {
|
||||
#define restrict __restrict__
|
||||
#define MIFCXX(y, n) y
|
||||
#define MIFC(y, n) n
|
||||
#else
|
||||
#define MIFCXX(y, n) n
|
||||
#define MIFC(y, n) y
|
||||
#endif
|
||||
|
||||
#include <stddef.h>
|
||||
|
||||
typedef struct mmap {
|
||||
int fd;
|
||||
|
||||
void* ptr;
|
||||
size_t len;
|
||||
} mmap_t;
|
||||
|
||||
enum map_flags {
|
||||
MMF_SHARED,
|
||||
MMF_PRIVATE,
|
||||
MMF_READONLY,
|
||||
};
|
||||
|
||||
int map_raw_fd(int fd, mmap_t* restrict ptr, const size_t MIFCXX(* sz, sz[static 1]));
|
||||
int open_and_alloc(const char* file, mmap_t* restrict ptr, size_t sz);
|
||||
int open_and_map(const char* file, mmap_t* restrict ptr);
|
||||
int unmap_and_close(mmap_t map);
|
||||
int dup_map(const mmap_t *in, mmap_t* restrict out, const size_t *new_size, enum map_flags flags) __attribute__((nonnull(1)));
|
||||
int map_advise_rand(mmap_t* restrict ptr, int need);
|
||||
|
||||
typedef void* (*map_cb)(mmap_t map, void* user);
|
||||
void* map_and_then(const char* file, map_cb callback, void* user);
|
||||
void* map_fd_and_then(int fd, map_cb callback, void* user);
|
||||
int unmap_and_close_s(mmap_t map, int flags);
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#include <panic.h>
|
||||
#include "reinterpret.h"
|
||||
#include <cstdint>
|
||||
namespace mm {
|
||||
enum class Access {
|
||||
Random,
|
||||
};
|
||||
struct mmap;
|
||||
template<typename T>
|
||||
concept MemMap = std::derived_from<T, mmap>
|
||||
and std::is_constructible_v<T, mmap_t>;
|
||||
struct mmap {
|
||||
template<MemMap M = mmap>
|
||||
inline static M allocate(const char* file, std::size_t sz)
|
||||
{
|
||||
mmap_t map;
|
||||
if(!open_and_alloc(file, &map, sz)) panic("Failed to allocmap file");
|
||||
return M{map};
|
||||
}
|
||||
inline static mmap_t create_raw(const char* file)
|
||||
{
|
||||
mmap_t map;
|
||||
if (!::open_and_map(file, &map)) panic("Failed to map file");
|
||||
return map;
|
||||
}
|
||||
inline static mmap_t create_raw_fd(int fd, size_t sz)
|
||||
{
|
||||
mmap_t map;
|
||||
if (!::map_raw_fd(fd, &map, &sz)) panic("Failed to allocmap fd");
|
||||
return map;
|
||||
}
|
||||
inline static mmap_t create_raw_fd(int fd)
|
||||
{
|
||||
mmap_t map;
|
||||
if (!::map_raw_fd(fd, &map, nullptr)) panic("Failed to map fd");
|
||||
return map;
|
||||
}
|
||||
template<MemMap M = mmap>
|
||||
inline static M map_raw_fd(int fd, std::size_t sz = 0)
|
||||
{
|
||||
if(sz)
|
||||
return M{fd, sz};
|
||||
else return M{fd};
|
||||
}
|
||||
|
||||
inline explicit mmap(mmap_t raw) noexcept :inner(raw){}
|
||||
inline mmap(const char* file)
|
||||
: inner(create_raw(file)) {}
|
||||
|
||||
inline mmap(mmap&& move) : inner(move.inner)
|
||||
{
|
||||
auto other = const_cast<mmap_t*>(&move.inner);
|
||||
other->ptr = nullptr;
|
||||
}
|
||||
inline mmap(const mmap& copt) = delete;
|
||||
|
||||
inline virtual ~mmap()
|
||||
{
|
||||
if (inner.ptr) {
|
||||
::unmap_and_close(inner);
|
||||
}
|
||||
}
|
||||
|
||||
inline virtual mmap&& access(Access a, bool need=false) && noexcept
|
||||
{
|
||||
if(inner.ptr)
|
||||
static_cast<mmap&>(*this).access(a, need);
|
||||
return std::move(*this);
|
||||
}
|
||||
inline virtual mmap& access(Access a, bool need=false) & noexcept
|
||||
{
|
||||
switch(a) {
|
||||
case Access::Random: ::map_advise_rand(const_cast<mmap_t*>(&inner), int(need));
|
||||
default: break;
|
||||
}
|
||||
return *this;
|
||||
}
|
||||
|
||||
inline const span<const unsigned char> as_span() const noexcept { return span(as_ptr(), size()); }
|
||||
inline span<unsigned char> as_span() noexcept { return span(as_ptr(), size()); }
|
||||
|
||||
inline virtual const std::uint8_t* as_ptr() const noexcept { return static_cast<const std::uint8_t*>(inner.ptr); }
|
||||
inline virtual std::uint8_t* as_ptr() noexcept { return static_cast<std::uint8_t*>(inner.ptr); }
|
||||
|
||||
inline virtual std::size_t size() const noexcept { return inner.len; }
|
||||
|
||||
inline virtual int as_fd() const { return inner.fd; }
|
||||
inline virtual const mmap_t& as_raw() const noexcept { return inner; }
|
||||
protected:
|
||||
inline mmap(int fd, size_t sz)
|
||||
: inner(create_raw_fd(fd, sz)) {}
|
||||
inline explicit mmap(int fd)
|
||||
: inner(create_raw_fd(fd)) {}
|
||||
inline virtual mmap_t& as_raw() noexcept { return const_cast<mmap_t&>( inner ); }
|
||||
private:
|
||||
const mmap_t inner;
|
||||
};
|
||||
//template mmap::map_raw_fd<mmap>(int, size_t); // Probably not needed?
|
||||
//template mmap::allocate<mmap>(int, size_t);
|
||||
|
||||
struct vmap : public mmap {
|
||||
inline explicit vmap(mmap_t m) noexcept : mmap(m) {}
|
||||
|
||||
vmap();
|
||||
vmap(size_t);
|
||||
inline explicit vmap(int fd) noexcept : mmap(fd) {}
|
||||
//vmap(const char* file);
|
||||
|
||||
//int as_fd() const override; unneeded
|
||||
|
||||
inline virtual ~vmap() { /* unmap_and_close() is called by super */ }
|
||||
//TODO: Implement this ^ with `memfd_create()`, etc.
|
||||
};
|
||||
|
||||
|
||||
template<MemMap M, MemMap D = M>
|
||||
inline D dup_map(const M& map, size_t resize, map_flags flags =0)
|
||||
{
|
||||
const mmap& m = static_cast<const mmap&>(map);
|
||||
mmap_t out;
|
||||
if(! ::dup_map(&m.as_raw(), &out, &resize, flags)) panic("Failed to duplicate mapping");
|
||||
return D{out};
|
||||
}
|
||||
template<MemMap M, MemMap D = M>
|
||||
inline D dup_map(const M& map, map_flags flags =0)
|
||||
{
|
||||
const mmap& m = static_cast<const mmap&>(map);
|
||||
mmap_t out;
|
||||
if(! ::dup_map(&m.as_raw(), &out, nullptr, flags)) panic("Failed to duplicate mapping");
|
||||
return D{out};
|
||||
}
|
||||
#if 0
|
||||
// dup_map() deduction guids (XXX: Are these needed, or even valid?)
|
||||
template<MemMap D>
|
||||
dup_map(const mmap& m, size_t s, map_flags f) -> dup_map<mmap, D>;
|
||||
|
||||
template<>
|
||||
dup_map(const mmap& m, size_t s, map_flags f) -> dup_map<mmap, mmap>;
|
||||
|
||||
template<MemMap M>
|
||||
dup_map(const M& m, size_t s, map_flags f) -> dup_map<M, M>;
|
||||
|
||||
template<MemMap D>
|
||||
dup_map(const mmap& m, map_flags f) -> dup_map<mmap, D>;
|
||||
|
||||
template<>
|
||||
dup_map(const mmap& m, map_flags f) -> dup_map<mmap, mmap>;
|
||||
|
||||
template<MemMap M>
|
||||
dup_map(const M& m, map_flags f) -> dup_map<M, M>;
|
||||
#endif
|
||||
}
|
||||
|
||||
#undef restrict
|
||||
#undef MIFCXX
|
||||
#undef MIFC
|
||||
#endif
|
||||
|
||||
#endif /* _MAP_H */
|
||||
@ -0,0 +1,35 @@
|
||||
#ifndef _PANIC_H
|
||||
#define _PANIC_H
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
struct panicinfo {
|
||||
const char* file;
|
||||
const char* function;
|
||||
int line;
|
||||
};
|
||||
|
||||
void _do_panic(struct panicinfo pi, const char* fmt, ...) __attribute__((noreturn));
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C++" {
|
||||
#include <utility>
|
||||
template<typename... Args>
|
||||
__attribute__((noreturn)) inline void _real_panic(const char* file, const char* function, int line, const char* fmt, Args&&... args)
|
||||
{
|
||||
panicinfo i = { file, function, line };
|
||||
_do_panic(i, fmt, std::move(args)...);
|
||||
}
|
||||
#define panic(fmt, ...) _real_panic(__FILE__, __func__, __LINE__, fmt __VA_OPT__(,) __VA_ARGS__)
|
||||
}
|
||||
#else
|
||||
#define panic(fmt, ...) _do_panic( (struct panicinfo){.file = __FILE__, .function = __func__, .line = __LINE__}, fmt __VA_OPT__(,) __VA_ARGS__)
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif
|
||||
@ -0,0 +1,406 @@
|
||||
#pragma once
|
||||
|
||||
#include <string>
|
||||
#include <memory>
|
||||
#include <utility>
|
||||
#include <array>
|
||||
#include <utility>
|
||||
|
||||
#include <cstdio>
|
||||
|
||||
#ifndef __cplusplus
|
||||
#error "C++ header only"
|
||||
#endif
|
||||
|
||||
namespace pr {
|
||||
|
||||
namespace details [[gnu::visibility("internal")]] {
|
||||
/*
|
||||
template<template<typename...> typename P>
|
||||
struct generic_valid { template<typename... Args> constexpr static inline bool value = requires{ typename P<Args...>; }; };
|
||||
template<template<typename...> class P>
|
||||
struct generic : std::conditional_t<generic_valid<P>::value,
|
||||
std::true_type,
|
||||
std::false_type>
|
||||
{ template<typename... Args> using type = P<Args...>; };
|
||||
|
||||
template<template<typename> typename T, typename... Args>
|
||||
concept Generic = generic_valid<T>::template value<Args...>;
|
||||
|
||||
template<Generic T, typename... Args>
|
||||
using generic_t = generic<T>::template type<Args...>;
|
||||
|
||||
template<template<typename> template T>
|
||||
concept AnyGeneric = requires{ typename generic_valid<T>; };
|
||||
*/
|
||||
template<typename T>
|
||||
T* take(T*&& ptr) noexcept { return std::exchange(ptr, nullptr); }
|
||||
|
||||
/*
|
||||
template<template<typename U> Inst, typename T>
|
||||
struct is_inst_of { constexpr static inline bool value = std::is_same_v<T, Inst<U>>; };
|
||||
|
||||
template<typename T, typename I>
|
||||
constexpr inline bool is_inst_of_v = is_inst_of<I, T>::value;
|
||||
*/
|
||||
|
||||
template<typename U, typename T>
|
||||
constexpr std::unique_ptr<U> static_ucast(std::unique_ptr<T>&& from) noexcept
|
||||
{ return std::unique_ptr<U>{static_cast<U*>(std::move(from).release())}; }
|
||||
|
||||
template<typename U, typename T>
|
||||
constexpr std::shared_ptr<U> static_ucast(std::shared_ptr<T>&& from) noexcept
|
||||
{ return std::static_pointer_cast<U>(std::move(from)); }
|
||||
|
||||
|
||||
template<typename Ptr, typename From> requires(requires(From&& p) {
|
||||
{ p.release() };
|
||||
requires(std::is_constructible_v<Ptr, decltype(p.release())>);
|
||||
})
|
||||
constexpr Ptr static_pcast(From&& p) noexcept { return Ptr{p.release()}; }
|
||||
|
||||
}
|
||||
|
||||
using increment_t = long double;
|
||||
|
||||
template<typename T>
|
||||
concept Bar = requires(std::remove_const_t<std::remove_reference_t<T>>& v) {
|
||||
{ v.spin(increment_t(0)) };
|
||||
{ std::as_const(v).aux() } -> std::convertible_to<std::string_view>;
|
||||
//{ v.aux() } -> std::assignable_from<std::string_view>;
|
||||
{ (v.aux() = std::string{}) } -> std::same_as<std::add_lvalue_reference_t<decltype(v.aux())>>;
|
||||
//requires(std::assignable_from<std::add_lvalue_reference_t<decltype(v.aux())>, std::string_view>);
|
||||
{ v.aux() += std::string_view{} } -> std::convertible_to<decltype(v.aux())>;
|
||||
{ v.aux(std::declval<std::string&&>()) } -> std::convertible_to<std::string_view>;
|
||||
};
|
||||
|
||||
template<Bar>
|
||||
struct Carrier;
|
||||
|
||||
struct Dynamic {
|
||||
template<Bar> friend class Carrier;
|
||||
|
||||
constexpr auto make(Bar auto&& bar) noexcept(std::is_nothrow_move_constructible_v<std::remove_reference_t<decltype(bar)>>);
|
||||
|
||||
constexpr virtual void spin(increment_t) =0;
|
||||
constexpr virtual std::string& aux() =0;
|
||||
constexpr virtual std::string const& aux() const noexcept =0;
|
||||
constexpr virtual std::string aux(std::string&& s) { return std::exchange(aux(), std::move(s)); }
|
||||
|
||||
constexpr virtual ~Dynamic() {}
|
||||
private:
|
||||
constexpr Dynamic() noexcept {}
|
||||
};
|
||||
template<Bar B>
|
||||
struct Carrier : public virtual Dynamic {
|
||||
using held_type = B;
|
||||
|
||||
template<typename... Args> requires(std::constructible_from<B, Args...>)
|
||||
constexpr Carrier(Args&&... args) noexcept(std::is_nothrow_constructible_v<B, Args...>)
|
||||
: Dynamic(), member_(std::forward<decltype(args)>(args)...) {}
|
||||
constexpr Carrier(Carrier&& m) noexcept(std::is_nothrow_move_constructible_v<B>)
|
||||
requires(std::is_move_constructible_v<B>)
|
||||
: Dynamic(), member_(std::move(m.member_)) {}
|
||||
constexpr Carrier(Carrier const& c) noexcept(std::is_nothrow_copy_constructible_v<B>)
|
||||
requires(std::is_copy_constructible_v<B>)
|
||||
: Dynamic(), member_(c.member_) {}
|
||||
|
||||
constexpr Carrier& operator=(Carrier&& m) noexcept(std::is_nothrow_move_assignable_v<B>) requires(std::is_move_assignable_v<B>)
|
||||
{ if(this != &m) member_ = std::move(m.member_); return *this; }
|
||||
|
||||
constexpr Carrier& operator=(Carrier const& c) noexcept(std::is_nothrow_copy_assignable_v<B>) requires(std::is_copy_assignable_v<B>)
|
||||
{ if(this != &c) member_ = c.member_; return *this; }
|
||||
|
||||
constexpr void spin(increment_t a) override { member_.spin(a); }
|
||||
|
||||
constexpr std::string& aux() override { return member_.aux(); }
|
||||
constexpr std::string const& aux() const noexcept override { return member_.aux(); }
|
||||
constexpr std::string aux(std::string&& s) override { return member_.aux(std::move(s)); }
|
||||
|
||||
template<typename... Args> requires(std::is_invocable_v<B::spin, B&, increment_t, Args...>)
|
||||
constexpr void spin(increment_t a, Args&&... args) noexcept(std::is_nothrow_invocable_v<B::spin, B&, increment_t, decltype(args)...>)
|
||||
{ return member_.spin(a, std::forward<decltype(args)>(args)...); }
|
||||
|
||||
constexpr virtual ~Carrier() {}
|
||||
private:
|
||||
B member_;
|
||||
};
|
||||
template<Bar T>
|
||||
Carrier(T) -> Carrier<T>;
|
||||
template<Bar T>
|
||||
Carrier(T const&) -> Carrier<T>;
|
||||
template<Bar T>
|
||||
Carrier(T&&) -> Carrier<T>;
|
||||
|
||||
constexpr auto Dynamic::make(Bar auto&& bar) noexcept(std::is_nothrow_move_constructible_v<std::remove_reference_t<decltype(bar)>>) { return Carrier(std::move(bar)); }
|
||||
|
||||
template<Bar T, typename Ptr = std::unique_ptr<Dynamic>>
|
||||
constexpr Ptr make_dyn(T&& bar) noexcept(std::is_nothrow_move_constructible_v<T>)
|
||||
{
|
||||
using namespace details;
|
||||
if constexpr(requires{
|
||||
typename T::held_type;
|
||||
requires(std::is_same_v<Carrier<typename T::held_type>, T>);
|
||||
}) return static_pcast<Ptr>(static_ucast<Dynamic>(std::make_unique<T>(std::move(bar))));
|
||||
else return static_pcast<Ptr>(static_ucast<Dynamic>(std::make_unique<Carrier<T>>(std::move(bar))));
|
||||
}
|
||||
|
||||
template<Bar T, typename Ptr = std::unique_ptr<Dynamic>>
|
||||
constexpr Ptr make_dyn(T const& bar) noexcept(std::is_nothrow_move_constructible_v<T>)
|
||||
{
|
||||
if constexpr(std::is_copy_constructible_v<T>) {
|
||||
T nbar{bar};
|
||||
return make_dyn<T, Ptr>(std::move(nbar));
|
||||
} else {
|
||||
struct unsafe_ref {
|
||||
constexpr unsafe_ref(const T& ba) noexcept : b(std::addressof(ba)) {}
|
||||
constexpr unsafe_ref(const unsafe_ref&) noexcept = default;
|
||||
constexpr ~unsafe_ref() noexcept = default;
|
||||
constexpr unsafe_ref(unsafe_ref&& b) noexcept : b(std::exchange(b.b, nullptr)) {}
|
||||
|
||||
constexpr unsafe_ref& operator=(unsafe_ref const&) noexcept = default;
|
||||
constexpr unsafe_ref& operator=(unsafe_ref&& m) noexcept {
|
||||
if(this != &m)
|
||||
b = std::exchange(m.b, nullptr);
|
||||
return *this;
|
||||
}
|
||||
|
||||
const T* b;
|
||||
|
||||
constexpr operator T const&() const noexcept { return *b; }
|
||||
};
|
||||
unsafe_ref re{bar};
|
||||
return make_dyn<unsafe_ref, Ptr>(std::move(re));
|
||||
}
|
||||
}
|
||||
|
||||
template<typename Ptr> requires(requires(std::unique_ptr<Dynamic>&& p) { details::static_pcast<Ptr>(std::move(p)); })
|
||||
constexpr Ptr make_dyn_for(Bar auto&& bar) noexcept(std::is_nothrow_move_constructible_v<std::remove_reference_t<decltype(bar)>>)
|
||||
{ return make_dyn<decltype(bar), Ptr>(std::move(bar)); }
|
||||
|
||||
constexpr void spin(Bar auto& bar, std::convertible_to<increment_t> auto&& a) { return bar.spin(increment_t(a)); }
|
||||
|
||||
struct None {
|
||||
constexpr None() noexcept = default;
|
||||
constexpr ~None() noexcept = default;
|
||||
|
||||
constexpr void spin(increment_t) const noexcept {}
|
||||
constexpr auto aux() noexcept { return not_string(); }
|
||||
inline std::string const& aux() const noexcept { return none_; }
|
||||
constexpr std::string aux(std::convertible_to<std::string> auto&& s) const noexcept { return {}; } //{ return std::exchange(none_, std::string(std::move(s))); }
|
||||
private:
|
||||
struct not_string {
|
||||
constexpr not_string() noexcept= default;
|
||||
constexpr ~not_string() noexcept= default;
|
||||
|
||||
[[gnu::const]]
|
||||
constexpr not_string& operator+=(std::convertible_to<std::string_view> auto&&) { return *this; }
|
||||
[[gnu::const]]
|
||||
constexpr not_string& operator=(std::convertible_to<std::string> auto&&) { return *this; }
|
||||
|
||||
inline operator std::string&() noexcept { return none_; }
|
||||
inline operator std::string const&() const noexcept { return none_; }
|
||||
|
||||
};
|
||||
/*constinit*/ thread_local static inline std::string none_;
|
||||
};
|
||||
constexpr inline None disable{};
|
||||
|
||||
// A bounded progress-bar
|
||||
struct Progress {
|
||||
using fract_t = increment_t;
|
||||
constexpr static inline bool DEFAULT_WIDTH = 50;
|
||||
constexpr static inline fract_t DEFAULT_TERM_FRACT=1.0l/3.0l;
|
||||
|
||||
[[gnu::nonnull(1)]]
|
||||
Progress(FILE* p = stdout) noexcept;
|
||||
|
||||
Progress(const Progress& p) noexcept;
|
||||
Progress(Progress&& p) noexcept;
|
||||
|
||||
Progress& operator=(Progress const& c) noexcept;
|
||||
Progress& operator=(Progress&& m) noexcept;
|
||||
|
||||
virtual ~Progress();
|
||||
|
||||
inline std::string& aux() noexcept { return tag(); }
|
||||
inline std::string const& aux() const noexcept { return tag(); }
|
||||
inline std::string aux(std::convertible_to<std::string> auto&& s)
|
||||
{ return std::exchange(tag(), std::string(std::move(s))); }
|
||||
protected:
|
||||
std::string& tag() noexcept;
|
||||
const std::string& tag() const noexcept;
|
||||
public:
|
||||
FILE* output() const noexcept;
|
||||
FILE*& output() noexcept;
|
||||
|
||||
size_t& width() noexcept;
|
||||
size_t width() const noexcept;
|
||||
|
||||
fract_t& fraction() noexcept;
|
||||
fract_t fraction() const noexcept;
|
||||
|
||||
fract_t percentage() const noexcept;
|
||||
inline auto percentage() noexcept {
|
||||
struct v final {
|
||||
Progress* p_;
|
||||
v(Progress* p) noexcept : p_(p) {}
|
||||
|
||||
v(v&&) noexcept = default;
|
||||
v(v const&) noexcept = default;
|
||||
v& operator=(v const&) noexcept = default;
|
||||
v& operator=(v&&) noexcept = default;
|
||||
~v() noexcept = default;
|
||||
|
||||
const v& operator=(fract_t f) const noexcept { p_->percentage(f); return *this; }
|
||||
operator fract_t() const noexcept { return static_cast<const Progress*>(p_)->percentage(); }
|
||||
};
|
||||
return v(this);
|
||||
}
|
||||
|
||||
void spin(increment_t by, bool render=true, bool flush=true) noexcept;
|
||||
void render(bool flush = true) const noexcept;
|
||||
private:
|
||||
void percentage(fract_t fract) noexcept;
|
||||
|
||||
struct _impl;
|
||||
std::shared_ptr<_impl> inner_;
|
||||
};
|
||||
|
||||
|
||||
#ifdef SPINNER
|
||||
class Spinner {
|
||||
consteval static auto _GENERATE_MAP(auto const& map, size_t size) noexcept
|
||||
-> std::array<size_t, 256>
|
||||
{
|
||||
static_assert(sizeof(map) == size, "Bad map size");
|
||||
std::array<size_t, 256> out{};
|
||||
while( size --> 0 ) out[int(map[size])] = size;
|
||||
return out;
|
||||
}
|
||||
constexpr static increment_t range(increment_t by)
|
||||
{
|
||||
if(by < -1.0l) return -1.0l;
|
||||
else if(by > 1.0l) return 1.0l;
|
||||
return by;
|
||||
}
|
||||
public:
|
||||
constexpr static inline auto ROUTINE = "|/-\\|/-\\|";
|
||||
constexpr static inline auto ROUTINE_SIZE = sizeof(ROUTINE);
|
||||
constexpr static inline auto REVERSE_MAP = _GENERATE_MAP(ROUTINE, ROUTINE_SIZE);
|
||||
static_assert(ROUTINE_SIZE != sizeof(char*), "Invalid routine size");
|
||||
|
||||
constexpr ssize_t range(int sz) noexcept
|
||||
{
|
||||
/*if(__builtin_expect(sz < 0, false)) {
|
||||
|
||||
std::terminate(); // TODO: How to handle wrapping negatives?? Ugh.
|
||||
}*/
|
||||
return ssize_t(sz) % ssize_t(ROUTINE_SIZE);
|
||||
}
|
||||
constexpr Spinner(size_t n) noexcept
|
||||
: cur_(ROUTINE[range(n)]) {}
|
||||
constexpr Spinner(char c = ROUTINE[0]) noexcept
|
||||
: cur_(range(REVERSE_MAP[int(c)])) {}
|
||||
|
||||
constexpr Spinner(Spinner const&) noexcept = default;
|
||||
constexpr Spinner(Spinner &&) noexcept = default;
|
||||
~Spinner();
|
||||
|
||||
inline void spin(int by) noexcept
|
||||
{
|
||||
operator+=(by);
|
||||
render();
|
||||
}
|
||||
inline void spin(increment_by by) noexcept
|
||||
{
|
||||
spin(int(range(by) * increment_t(ROUTINE_SIZE)));
|
||||
}
|
||||
|
||||
inline Spinner& operator+=(int by) noexcept
|
||||
{
|
||||
cur_ = ROUTINE[size_t(ssize_t(REVERSE_MAP[int(cur_)]) + range(by))];
|
||||
return *this;
|
||||
}
|
||||
|
||||
inline Spinner& operator++() noexcept { spin(1); return *this; }
|
||||
inline Spinner operator++(int) noexcept { Spinner s = *this; ++s; return *this; }
|
||||
constexpr Spinner& operator=(Spinner&&) noexcept = default;
|
||||
constexpr Spinner& operator=(Spinner const&) noexcept = default;
|
||||
|
||||
constexpr char& character() noexcept { return cur_; }
|
||||
constexpr char character() const noexcept { return cur_; }
|
||||
|
||||
void render(bool flush=true);
|
||||
private:
|
||||
char cur_ = '|';
|
||||
};
|
||||
//TODO: Spinny bar thing (unbounded)
|
||||
#endif
|
||||
|
||||
/*
|
||||
struct Bar {
|
||||
Bar() noexcept = default;
|
||||
Bar(const Bar&) noexcept = default;
|
||||
Bar(Bar&&) noexcept = default;
|
||||
Bar& operator=(const Bar&) noexcept = default;
|
||||
Bar& operator=(Bar&&) noexcept = default;
|
||||
virtual ~Bar() = default;
|
||||
|
||||
virtual void spin(int) =0;
|
||||
};
|
||||
|
||||
enum bar_kind {
|
||||
BRK_UNTRACKED,
|
||||
BRK_TRACKED,
|
||||
};
|
||||
|
||||
template<bar_kind = BRK_UNTRACKED>
|
||||
struct progress;
|
||||
|
||||
template<>
|
||||
struct progress<BRK_UNTRACKED>
|
||||
: public virtual Bar {
|
||||
|
||||
progress(FILE* to);
|
||||
progress(progress const&);
|
||||
inline progress(progress&& mv) noexcept
|
||||
: Bar()
|
||||
, _perc(mv._perc)
|
||||
, _output(details::take(mv._output)) {}
|
||||
|
||||
virtual ~progress();
|
||||
|
||||
void spin(int) override;
|
||||
|
||||
//TODO: this
|
||||
protected:
|
||||
union {
|
||||
double _perc;
|
||||
size_t _num; // not used here
|
||||
};
|
||||
FILE* _output;
|
||||
};
|
||||
|
||||
template<>
|
||||
class progress<BRK_TRACKED>
|
||||
: public progress<BRK_UNTRACKED> {
|
||||
using base_t = progress<BRK_UNTRACKED>;
|
||||
public:
|
||||
inline progress(FILE* to, size_t max) : base_t(to), _num(0), _max(max) {}
|
||||
progress(const progress&) = default;
|
||||
progress(progress&& m) : base_t(std::move(m)), _num(std::exchange(m._num, 0)), _max(m._max) {}
|
||||
|
||||
void spin(int) override;
|
||||
//TODO: this
|
||||
|
||||
virtual ~progress() {}
|
||||
|
||||
protected:
|
||||
using base_t::_output;
|
||||
private:
|
||||
using base_t::_num;
|
||||
size_t _max;
|
||||
};
|
||||
*/
|
||||
}
|
||||
@ -0,0 +1 @@
|
||||
reinterpret.hpp
|
||||
@ -0,0 +1,61 @@
|
||||
#ifndef _REINTERPRET_H
|
||||
#define _REINTERPRET_H
|
||||
|
||||
#include <panic.h>
|
||||
|
||||
#ifdef __cplusplus
|
||||
#define restrict __restrict__
|
||||
#include <cstddef>
|
||||
#include <cstdint>
|
||||
#else
|
||||
#include <stddef.h>
|
||||
#include <stdint.h>
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
template<typename T>
|
||||
struct span {
|
||||
inline span(T* ptr, std::size_t len) : ptr(ptr), len(len) {}
|
||||
|
||||
template<typename U>
|
||||
inline span<U> reinterpret() const
|
||||
{
|
||||
auto bytes = size_bytes();
|
||||
//if (len_b % sizeof(U) != 0) panic("Cannot reinterpret T to U due to unmatch sizing constraints.");
|
||||
|
||||
return span<U>((U*)ptr, bytes / sizeof(U));
|
||||
}
|
||||
|
||||
inline const T& operator[](std::size_t i) const
|
||||
{
|
||||
if (i >= len) panic("Out of bounds access: %lu >= %lu", i, len);
|
||||
return ptr[i];
|
||||
}
|
||||
inline T& operator[](std::size_t i)
|
||||
{
|
||||
if (i >= len) panic("Out of bounds access: %lu >= %lu", i, len);
|
||||
return ptr[i];
|
||||
}
|
||||
|
||||
inline const T* as_ptr() const { return ptr; }
|
||||
inline T* as_ptr() { return ptr; }
|
||||
|
||||
inline const T* operator&() const { return as_ptr(); }
|
||||
inline T* operator&() { return as_ptr(); }
|
||||
|
||||
inline std::size_t size_bytes() const { return len * sizeof(T); }
|
||||
inline std::size_t size() const { return len; }
|
||||
private:
|
||||
T* const ptr;
|
||||
const std::size_t len;
|
||||
};
|
||||
|
||||
extern "C" {
|
||||
#endif
|
||||
uint64_t* bytes_to_long(uint8_t* ptr, size_t ptr_sz, size_t* restrict nsize);
|
||||
float* bytes_to_float(uint8_t* ptr, size_t ptr_sz, size_t* restrict nsize);
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* _REINTERPRET_H */
|
||||
@ -1,25 +1,70 @@
|
||||
#ifndef _RNG_H
|
||||
#define _RNG_H
|
||||
|
||||
typedef struct rng_algo *RNG; //RNG algorithm reference
|
||||
|
||||
#define rng_reinterpret(rng, value, type) (*((type*)rng_next(rng, &value, sizeof(type))))
|
||||
#define rng_reinterpret_new(rng, value, type) (*((type*)rng_next(rng, malloc(sizeof(type)), sizeof(type))))
|
||||
#define rng_reinterpret_stackalloc(rng, value, type) rng_reinterpret(rng, alloca(sizeof(type)), type)
|
||||
|
||||
#define RNG_LEXBIND(rng, type, name) type name = rng_reinterpret(rng, name, type)
|
||||
|
||||
double rng_next_double(RNG algo);
|
||||
int rng_next_int_bounded(RNG algo, int min, int max);
|
||||
int rng_next_int(RNG algo, int max);
|
||||
void* rng_next(RNG algo, void* data, int len);
|
||||
int rng_chance(RNG algo, double d);
|
||||
void rng_free(RNG algo);
|
||||
void rng_seed(RNG algo, void* seed);
|
||||
RNG rng_new(RNG (*instantiate)(void));
|
||||
|
||||
#define RNG_IMPL_DEFINITION(name) RNG __rng_impl_ ## name(void)
|
||||
#define RNG_ALGO(name) &__rng_impl_ ## name
|
||||
#define RNG_NEW(name) rng_new(RNG_ALGO(name))
|
||||
#include "shuffle3.h"
|
||||
|
||||
#ifdef __cplusplus
|
||||
#include "rng/impl.hpp"
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
enum rng_kind {
|
||||
RNG_KIND_FRNG,
|
||||
RNG_KIND_DRNG,
|
||||
RNG_KIND_XORNG,
|
||||
RNG_KIND_LORENZ,
|
||||
};
|
||||
|
||||
typedef long double rng_st_lorenz_t;
|
||||
typedef _Complex long double rng_lorenz_t;
|
||||
|
||||
typedef struct rng_init
|
||||
{
|
||||
enum rng_kind kind;
|
||||
union {
|
||||
struct {
|
||||
double state[2];
|
||||
} frng;
|
||||
struct {
|
||||
int32_t state;
|
||||
} drng;
|
||||
struct {
|
||||
uint64_t state[2];
|
||||
} xorng;
|
||||
struct {
|
||||
rng_lorenz_t point;
|
||||
const rng_st_lorenz_t (* _UNIQUE state)[5];
|
||||
uint64_t iter;
|
||||
} lorenz;
|
||||
} init;
|
||||
|
||||
} rng_init_opt;
|
||||
|
||||
typedef struct rng_impl* _UNIQUE rng_t;
|
||||
|
||||
rng_t rng_new(rng_init_opt kind);
|
||||
#define RNG_INIT(_kind,...) ((rng_init_opt){.kind=(_kind), .init.__VA_ARGS__ })
|
||||
void rng_free(rng_t ptr);
|
||||
|
||||
// Tests
|
||||
extern void rng_test();
|
||||
extern void rng_test_spec(rng_t rng) __attribute__((nonnull(1)));
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
// RNG interfaces
|
||||
#include "rng/xoroshiro128plus.hpp"
|
||||
#include "rng/frng.hpp"
|
||||
#include "rng/drng.hpp"
|
||||
#include "rng/lorenz.hpp"
|
||||
|
||||
namespace rng {
|
||||
void test_algo(RNG&& rng);
|
||||
|
||||
template<class R,
|
||||
typename std::enable_if<std::is_base_of<RNG, R>::value>::value __fuck>
|
||||
inline void test_algo(R&& rng) { test_algo(static_cast<RNG&&>(rng)); }
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* _RNG_H */
|
||||
|
||||
@ -0,0 +1,22 @@
|
||||
|
||||
#include "impl.hpp"
|
||||
#include <debug.h>
|
||||
namespace rng
|
||||
{
|
||||
struct drng : public RNG
|
||||
{
|
||||
inline drng(std::uint32_t seed) : state(seed){
|
||||
D_dprintf("drng: seeded with %u", seed);
|
||||
//dprintf(" dummy run sample: %f", sample());
|
||||
}
|
||||
inline drng() : drng(1){}
|
||||
|
||||
static drng from_time();
|
||||
|
||||
int rand();
|
||||
protected:
|
||||
double sample() override;
|
||||
private:
|
||||
std::uint32_t state;
|
||||
};
|
||||
}
|
||||
@ -0,0 +1,67 @@
|
||||
|
||||
#include "impl.hpp"
|
||||
#include <cmath>
|
||||
|
||||
#include <debug.h>
|
||||
|
||||
namespace rng
|
||||
{
|
||||
struct frng : public RNG
|
||||
{
|
||||
template<std::size_t N>
|
||||
static constexpr inline double dot(const std::array<double, N>& v, const std::array<double, N>& u)
|
||||
{
|
||||
double res=0;
|
||||
for(std::size_t i=0;i<N;i++)
|
||||
{
|
||||
res += v[i] * u[i];
|
||||
}
|
||||
return res;
|
||||
}
|
||||
|
||||
static inline constexpr double fract(double x)
|
||||
{
|
||||
return x - floor(x);
|
||||
}
|
||||
static inline constexpr double sample_double(const std::array<double, 2>& state)
|
||||
{
|
||||
const constexpr std::array<double, 2> vec2 = { 12.9898, 78.223 };
|
||||
return fract(sin(dot(state, vec2)) * 43758.5453);
|
||||
}
|
||||
|
||||
#define P D_dprintf("frng: seeded with (%f, %f)", state[0], state[1]);
|
||||
inline constexpr frng(double s1, double s2) : state({s1, s2}){P}
|
||||
inline constexpr frng(const std::array<double, 2>& ar) : state(ar){P}
|
||||
inline constexpr frng(std::array<double, 2>&& ar) : state(ar){P}
|
||||
inline constexpr frng(const double (&ar)[2]) : state({ar[0], ar[1]}) {P}
|
||||
constexpr virtual ~frng() = default;
|
||||
#undef P
|
||||
inline constexpr double next_double() override { return sample(); }
|
||||
inline constexpr float next_float() override { return (float)sample(); }
|
||||
protected:
|
||||
inline constexpr double sample() override
|
||||
{
|
||||
double res = sample_double(state);
|
||||
update_state(state, res);
|
||||
return res;
|
||||
}
|
||||
private:
|
||||
std::array<double, 2> state;
|
||||
static inline constexpr void update_state(std::array<double, 2>& state, double r)
|
||||
{
|
||||
float v1 = (float)state[0];
|
||||
float v2 = (float)state[1];
|
||||
|
||||
std::array<double, 2> nvec = {
|
||||
r,
|
||||
(double)v2,
|
||||
};
|
||||
|
||||
state[0] = sample_double(nvec);
|
||||
|
||||
nvec[1] = (double)v1;
|
||||
state[1] = sample_double(nvec);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
@ -0,0 +1,42 @@
|
||||
#pragma once
|
||||
|
||||
#include <array>
|
||||
#include <vector>
|
||||
#include <cstdint>
|
||||
|
||||
/// Base class for RNG impls
|
||||
struct RNG {
|
||||
constexpr RNG() noexcept = default;
|
||||
|
||||
virtual unsigned char byte();
|
||||
virtual void bytes(unsigned char* ptr, std::size_t len);
|
||||
|
||||
template<std::size_t N>
|
||||
inline void bytes(unsigned char (&arr)[N]) { return bytes(arr, N); }
|
||||
template<std::size_t N>
|
||||
inline void bytes(std::array<unsigned char, N>& array) { return bytes(&array[0], N); }
|
||||
inline void bytes(std::vector<unsigned char>& vec) { return bytes(&vec[0], vec.size()); }
|
||||
|
||||
bool chance();
|
||||
virtual bool chance(double chance);
|
||||
|
||||
virtual std::int32_t next_int();
|
||||
inline std::int32_t next_int(std::int32_t max) { return next_int(0, max); }
|
||||
std::int32_t next_int(std::int32_t min, std::int32_t max);
|
||||
|
||||
virtual std::int64_t next_long();
|
||||
inline std::int64_t next_long(std::int64_t max) { return next_long(0, max); }
|
||||
std::int64_t next_long(std::int64_t min, std::int64_t max);
|
||||
|
||||
inline virtual float next_float() { return (float)sample(); }
|
||||
inline virtual double next_double() { return sample(); }
|
||||
|
||||
constexpr virtual ~RNG() = default;
|
||||
|
||||
//explicit operator rng_t() const noexcept;
|
||||
//friend operator RNG*(rng_t) noexcept;
|
||||
protected:
|
||||
virtual double sample() = 0;
|
||||
//private:
|
||||
//struct rng_impl* _UNIQUE _held = nullptr;
|
||||
};
|
||||
@ -0,0 +1,2 @@
|
||||
#pragma once
|
||||
|
||||
@ -0,0 +1,31 @@
|
||||
#pragma once
|
||||
|
||||
#include "impl.hpp"
|
||||
#include <debug.h>
|
||||
|
||||
namespace rng
|
||||
{
|
||||
struct xoroshiro128plus : public RNG
|
||||
{
|
||||
using State = std::array<std::uint64_t, 2>;
|
||||
#define P D_dprintf("xorng: seeded with (%lu, %lu)", state[0], state[1]);
|
||||
inline constexpr xoroshiro128plus(std::uint64_t s0, std::uint64_t s1) : RNG(), state({s0, s1}){P}
|
||||
inline constexpr xoroshiro128plus(std::array<std::uint64_t, 2>&& ar) : RNG(), state(ar){P}
|
||||
inline constexpr xoroshiro128plus(const std::array<std::uint64_t, 2>& ar) : RNG(), state(ar){P}
|
||||
inline constexpr xoroshiro128plus(const std::uint64_t (&ar)[2]) : RNG(), state({ar[0], ar[1]}){P}
|
||||
inline virtual ~xoroshiro128plus() {}
|
||||
#undef P
|
||||
std::uint64_t next_ulong();
|
||||
using RNG::next_long;
|
||||
std::int64_t next_long() override;
|
||||
|
||||
void jump();
|
||||
void long_jump();
|
||||
protected:
|
||||
double sample() override;
|
||||
private:
|
||||
State state;
|
||||
};
|
||||
static_assert(std::derived_from<xoroshiro128plus, RNG>, "Wtf???");
|
||||
}
|
||||
|
||||
@ -0,0 +1,76 @@
|
||||
#pragma once
|
||||
|
||||
#include <rng.h>
|
||||
#include <reinterpret.h>
|
||||
#include <algorithm>
|
||||
#include <iostream>
|
||||
|
||||
#include <fsvec.hpp>
|
||||
|
||||
#include <shuffle3.h>
|
||||
|
||||
namespace rng {
|
||||
template<typename T, typename R>
|
||||
inline void shuffle(R& rng, span<T> span)
|
||||
{
|
||||
if(!span.size()) return;
|
||||
std::cout << " -> shuffling " << span.size() << " objects..." << std::flush;
|
||||
for(std::size_t i=span.size()-1;i>0;i--)
|
||||
{
|
||||
auto j = rng.next_long(i);
|
||||
std::swap(span[i], span[j]);
|
||||
}
|
||||
std::cout << " OK" << std::endl;
|
||||
}
|
||||
|
||||
namespace {
|
||||
extern "C" int _can_allocate(std::size_t bytes);
|
||||
template<typename T>
|
||||
inline bool can_allocate(std::size_t len)
|
||||
{
|
||||
return !!_can_allocate(len*sizeof(T));
|
||||
}
|
||||
}
|
||||
|
||||
template<typename T, typename R>
|
||||
inline void unshuffle(R& rng, span<T> span)
|
||||
{
|
||||
if(!span.size()) return;
|
||||
|
||||
#define DYN 2
|
||||
#define MAP 3
|
||||
#if defined(_FS_SPILL_BUFFER) && (_FS_SPILL_BUFFER == DYN)
|
||||
D_dprintf("spill=dyn");
|
||||
dynamic_spill_vector<std::size_t> rng_values =
|
||||
//can_allocate<std::size_t>(span.size()) //Is there any way we can not waste this malloc() when it's valid?
|
||||
span.size() <= FSV_DEFAULT_SPILL_AT
|
||||
? dynamic_spill_vector<std::size_t> (span.size(), FSV_DEFAULT_SPILL_AT)
|
||||
: dynamic_spill_vector<std::size_t> (FSV_DEFAULT_SPILL_AT);
|
||||
#elif defined(_FS_SPILL_BUFFER) && (_FS_SPILL_BUFFER == MAP)
|
||||
D_dprintf("spill=map");
|
||||
mapped_spill_vector<std::size_t> rng_values =
|
||||
span.size() <= FSV_DEFAULT_SPILL_AT
|
||||
? mapped_spill_vector<std::size_t> (span.size(), FSV_DEFAULT_SPILL_AT)
|
||||
: mapped_spill_vector<std::size_t> (FSV_DEFAULT_SPILL_AT, mapped_vector<std::size_t>::from_temp(span.size() - FSV_DEFAULT_SPILL_AT));
|
||||
#elif defined(_FS_SPILL_BUFFER)
|
||||
D_dprintf("spill=static");
|
||||
fixed_spill_vector<std::size_t> rng_values;
|
||||
#else
|
||||
D_dprintf("spill=none");
|
||||
std::vector<std::size_t> rng_values;
|
||||
rng_values.reserve(span.size());
|
||||
#endif
|
||||
#undef MAP
|
||||
#undef DYN
|
||||
|
||||
std::cout << " -> unshuffling " << span.size() << " objects..." << std::flush;
|
||||
for(std::size_t i=span.size()-1;i>0;i--)
|
||||
rng_values.push_back(rng.next_long(i));
|
||||
|
||||
for(std::size_t i=1;i<span.size();i++) {
|
||||
std::swap(span[i], span[rng_values.back()]);
|
||||
rng_values.pop_back();
|
||||
}
|
||||
std::cout << " OK" << std::endl;
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,42 @@
|
||||
#ifndef _SHUFFLE3_H
|
||||
#define _SHUFFLE3_H
|
||||
|
||||
#ifdef __cplusplus
|
||||
#define _UNIQUE __restrict
|
||||
extern "C" {
|
||||
#else
|
||||
#define _UNIQUE restrict
|
||||
#endif
|
||||
|
||||
#ifdef DEBUG
|
||||
#define _FORCE_INLINE __attribute__((gnu_inline)) static inline
|
||||
#else
|
||||
#define _FORCE_INLINE __attribute__((gnu_inline)) extern inline
|
||||
#endif
|
||||
|
||||
//** Features **//
|
||||
|
||||
//#define _FS_SPILL_BUFFER /* Use a file-backed buffer when unshuffling in cases of too high memory usage. Will cause massive slowdowns but can stop OOMs when unshuffling large files */
|
||||
|
||||
/// When to spill a file-backed buffer onto the fs (only used when `_FS_SPILL_BUFFER` is enabled).
|
||||
#define FSV_DEFAULT_SPILL_AT ((1024 * 1024) * 10) //10MB
|
||||
|
||||
/*
|
||||
#ifdef _FS_SPILL_BUFFER
|
||||
#define DYN 1
|
||||
#if _FS_SPILL_BUFFER == DYN
|
||||
#undef _FS_SPILL_BUFFER
|
||||
#define _FS_SPILL_BUFFER DYN
|
||||
#endif
|
||||
#undef DYN
|
||||
#endif
|
||||
*/
|
||||
//** Globals *//
|
||||
|
||||
extern const char* _prog_name;
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* _SHUFFLE3_H */
|
||||
@ -0,0 +1,49 @@
|
||||
#pragma once
|
||||
|
||||
#include <filesystem>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include "uuid.hpp"
|
||||
|
||||
#include <debug.h>
|
||||
|
||||
using std::size_t;
|
||||
namespace fs = std::filesystem;
|
||||
|
||||
/// A temporary file name
|
||||
struct temp_file
|
||||
{
|
||||
inline temp_file(const temp_file& c) = delete;
|
||||
|
||||
inline temp_file(temp_file&& m) : name(std::move(m.name)), _full_path(std::move(m._full_path)) {}
|
||||
inline temp_file() : name(uuid::generate().to_string()+"-s3"){}
|
||||
inline temp_file(const char* name) : name(name) {}
|
||||
inline temp_file(std::string&& name) : name(name) {}
|
||||
|
||||
inline ~temp_file()
|
||||
{
|
||||
if(name.empty() && _full_path.empty()) return;
|
||||
|
||||
D_dprintf("~tempfile(): %s", _full_path.c_str());
|
||||
|
||||
if(!_full_path.empty() && fs::exists(_full_path) ) {
|
||||
D_dprintf("tfile removing: %s", _full_path.c_str());
|
||||
fs::remove(_full_path);
|
||||
}
|
||||
}
|
||||
|
||||
inline const fs::path& full_path() const
|
||||
{
|
||||
if(_full_path.empty()) {
|
||||
_full_path = fs::absolute( fs::temp_directory_path() / name );
|
||||
D_dprintf("tfile path: %s", _full_path.c_str());
|
||||
}
|
||||
return _full_path;
|
||||
}
|
||||
inline const std::string& base_name() const { return name; }
|
||||
|
||||
inline const fs::path* operator->() const { return &full_path(); }
|
||||
private:
|
||||
std::string name;
|
||||
mutable fs::path _full_path;
|
||||
};
|
||||
@ -0,0 +1,32 @@
|
||||
#pragma once
|
||||
|
||||
#include <array>
|
||||
#include <random>
|
||||
#include <string>
|
||||
|
||||
struct uuid //not to spec but idc
|
||||
{
|
||||
const static constexpr int SIZE=16;
|
||||
inline static uuid generate()
|
||||
{
|
||||
using namespace std;
|
||||
static thread_local random_device dev;
|
||||
static thread_local mt19937 rng(dev());
|
||||
uniform_int_distribution<int> dist(0, 15);
|
||||
constexpr const char hex[] = "0123456789abcdef";
|
||||
uuid id;
|
||||
for(int i=0;i<SIZE;i++) id.str[i] = hex[dist(rng)];
|
||||
id.str[SIZE] =0;
|
||||
return id;
|
||||
}
|
||||
inline operator const char*() const { return &str[0]; }
|
||||
inline operator const std::array<char, SIZE+1>&() const { return str; }
|
||||
|
||||
inline std::string to_string() const
|
||||
{
|
||||
return std::string(&str[0]);
|
||||
}
|
||||
private:
|
||||
inline uuid(){}
|
||||
std::array<char, SIZE+1> str;
|
||||
};
|
||||
@ -0,0 +1,52 @@
|
||||
#ifndef _WORK_H
|
||||
#define _WORK_H
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
enum work_buffer_opts {
|
||||
WORK_BO_CPIP=0, // Copy then run in place on output file (default)
|
||||
WORK_BO_BUFFERED=1, // Read whole input into memory then perform
|
||||
};
|
||||
|
||||
typedef struct _work_args {
|
||||
enum {
|
||||
OP_SHUFFLE_IP, // Shuffle in place
|
||||
OP_SHUFFLE_OP, // Shuffle out of place
|
||||
|
||||
OP_UNSHUFFLE_IP, // Unshuffle in place
|
||||
OP_UNSHUFFLE_OP, // Unshuffle out of place
|
||||
|
||||
OP_HELP, // Print help then exit
|
||||
} op;
|
||||
union {
|
||||
struct {
|
||||
const char* file;
|
||||
} op_shuffle_ip;
|
||||
|
||||
struct {
|
||||
enum work_buffer_opts buffered;
|
||||
const char* ifile;
|
||||
const char* ofile;
|
||||
} op_shuffle_op;
|
||||
|
||||
struct {
|
||||
const char* file;
|
||||
} op_unshuffle_ip;
|
||||
|
||||
struct {
|
||||
enum work_buffer_opts buffered;
|
||||
const char* ifile;
|
||||
const char* ofile;
|
||||
} op_unshuffle_op;
|
||||
} data;
|
||||
} work_args_t;
|
||||
|
||||
int do_work(const work_args_t args);
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* _WORK_H */
|
||||
@ -0,0 +1,50 @@
|
||||
SRC = $(wildcard src/*.c)
|
||||
INCLUDE = include/
|
||||
|
||||
PROJECT=shuffle3
|
||||
|
||||
BUILD=build
|
||||
|
||||
OPT_FLAGS?= -march=native -fgraphite -fopenmp -floop-parallelize-all -ftree-parallelize-loops=4 \
|
||||
-floop-interchange -ftree-loop-distribution -floop-strip-mine -floop-block
|
||||
|
||||
CFLAGS+= $(addprefix -I,$(INCLUDE)) -Wall -pedantic --std=gnu11
|
||||
LDFLAGS+= -lm
|
||||
|
||||
RELEASE_CFLAGS?= -O3 -flto $(OPT_FLAGS)
|
||||
RELEASE_LDFLAGS?= -O3 -flto
|
||||
|
||||
DEBUG_CFLAGS?= -g -O0
|
||||
DEBUG_LDFLAGS?= -O0
|
||||
|
||||
STRIP=strip
|
||||
|
||||
OBJ = $(addprefix obj/,$(SRC:.c=.o))
|
||||
|
||||
.PHONY: release
|
||||
release: | dirs $(BUILD)/$(PROJECT)-release
|
||||
|
||||
.PHONY: debug
|
||||
debug: | dirs $(BUILD)/$(PROJECT)-debug
|
||||
|
||||
dirs:
|
||||
@mkdir -p obj/src
|
||||
@mkdir -p $(BUILD)
|
||||
|
||||
obj/%.o: %.c
|
||||
$(CC) -c $< $(CFLAGS) -o $@ $(LDFLAGS)
|
||||
|
||||
$(BUILD)/$(PROJECT)-release: CFLAGS+= $(RELEASE_CFLAGS)
|
||||
$(BUILD)/$(PROJECT)-release: LDFLAGS+= $(RELEASE_LDFLAGS)
|
||||
$(BUILD)/$(PROJECT)-release: $(OBJ)
|
||||
$(CC) $^ $(CFLAGS) -o $@ $(LDFLAGS)
|
||||
$(STRIP) $@
|
||||
|
||||
$(BUILD)/$(PROJECT)-debug: CFLAGS+= $(DEBUG_CFLAGS)
|
||||
$(BUILD)/$(PROJECT)-debug: LDFLAGS+= $(DEBUG_LDFLAGS)
|
||||
$(BUILD)/$(PROJECT)-debug: $(OBJ)
|
||||
$(CC) $^ $(CFLAGS) -o $@ $(LDFLAGS)
|
||||
|
||||
clean:
|
||||
rm -rf obj
|
||||
rm -f $(BUILD)/*
|
||||
@ -0,0 +1,25 @@
|
||||
#ifndef _RNG_H
|
||||
#define _RNG_H
|
||||
|
||||
typedef struct rng_algo *RNG; //RNG algorithm reference
|
||||
|
||||
#define rng_reinterpret(rng, value, type) (*((type*)rng_next(rng, &value, sizeof(type))))
|
||||
#define rng_reinterpret_new(rng, value, type) (*((type*)rng_next(rng, malloc(sizeof(type)), sizeof(type))))
|
||||
#define rng_reinterpret_stackalloc(rng, value, type) rng_reinterpret(rng, alloca(sizeof(type)), type)
|
||||
|
||||
#define RNG_LEXBIND(rng, type, name) type name = rng_reinterpret(rng, name, type)
|
||||
|
||||
double rng_next_double(RNG algo);
|
||||
int rng_next_int_bounded(RNG algo, int min, int max);
|
||||
int rng_next_int(RNG algo, int max);
|
||||
void* rng_next(RNG algo, void* data, int len);
|
||||
int rng_chance(RNG algo, double d);
|
||||
void rng_free(RNG algo);
|
||||
void rng_seed(RNG algo, void* seed);
|
||||
RNG rng_new(RNG (*instantiate)(void));
|
||||
|
||||
#define RNG_IMPL_DEFINITION(name) RNG __rng_impl_ ## name(void)
|
||||
#define RNG_ALGO(name) &__rng_impl_ ## name
|
||||
#define RNG_NEW(name) rng_new(RNG_ALGO(name))
|
||||
|
||||
#endif /* _RNG_H */
|
||||
@ -0,0 +1,373 @@
|
||||
#include <stdio.h>
|
||||
#include <float.h>
|
||||
#include <stdint.h>
|
||||
#include <string.h>
|
||||
#include <assert.h>
|
||||
|
||||
#include <rng.h>
|
||||
#include <rng_algos.h>
|
||||
#include <ptr_store.h>
|
||||
#include <array.h>
|
||||
|
||||
_Static_assert(sizeof(float)==4, "Float must be 32 bits.");
|
||||
|
||||
int surpress_out=0;
|
||||
|
||||
/*void test_destructor(object* ptr)
|
||||
{
|
||||
printf("die\n");
|
||||
}*/
|
||||
void _rng_object_destructor(object* obj)
|
||||
{
|
||||
rng_free(obj->state);
|
||||
}
|
||||
RNG rng_object(S_LEXENV, RNG rng)
|
||||
{
|
||||
object proto = OBJ_PROTO;
|
||||
proto.state = rng;
|
||||
proto.destructor = &_rng_object_destructor;
|
||||
|
||||
return snew_obj(proto)->state;
|
||||
}
|
||||
|
||||
void shuffle(RNG with, array_t data)
|
||||
{
|
||||
if(!surpress_out) printf(" -> shuffling %d objects...", (int)ar_size(data));
|
||||
for(int i=ar_size(data)-1;i>0;i--)
|
||||
{
|
||||
int j = rng_next_int(with, i);
|
||||
ar_swap(data, i, j);
|
||||
}
|
||||
if(!surpress_out) printf(" Okay\n");
|
||||
}
|
||||
|
||||
void unshuffle(RNG with, array_t data)
|
||||
{
|
||||
int rng_values[ar_size(data)-1];
|
||||
int k=0;
|
||||
if(!surpress_out) printf(" -> unshuffling %d objects...", (int)ar_size(data));
|
||||
for(int i=ar_size(data)-1;i>0;i--)
|
||||
rng_values[k++] = rng_next_int(with, i);
|
||||
|
||||
for(int i=1;i<ar_size(data);i++)
|
||||
{
|
||||
ar_swap(data, i, rng_values[--k]);
|
||||
}
|
||||
if(!surpress_out) printf(" Okay\n");
|
||||
}
|
||||
|
||||
void minmax_int64_ts(int64_t* min, int64_t* max, const array_t data)
|
||||
{
|
||||
for(register int i=0;i<ar_size(data);i++)
|
||||
{
|
||||
if(ar_get_v(data, int64_t, i) > *max) *max = ar_get_v(data, int64_t, i);
|
||||
if(ar_get_v(data, int64_t, i) < *min) *min = ar_get_v(data, int64_t, i);
|
||||
}
|
||||
}
|
||||
int valid_float(float f)
|
||||
{
|
||||
return !( (f!=f) || (f< -FLT_MAX || f> FLT_MAX));
|
||||
|
||||
}
|
||||
void minmax_floats(float* min, float* max, const array_t data)
|
||||
{
|
||||
for(register int i=0;i<ar_size(data);i++)
|
||||
{
|
||||
if(!valid_float(ar_get_v(data,float,i))) continue;
|
||||
if(ar_get_v(data, float, i) > *max) *max = ar_get_v(data, float, i);
|
||||
if(ar_get_v(data, float, i) < *min) *min = ar_get_v(data, float, i);
|
||||
}
|
||||
}
|
||||
|
||||
void minmax_sbytes(int8_t* min, int8_t* max, const array_t data)
|
||||
{
|
||||
for(register int i=0;i<ar_size(data);i++)
|
||||
{
|
||||
if(ar_get_v(data, int8_t, i) > *max) *max = ar_get_v(data, int8_t, i);
|
||||
if(ar_get_v(data, int8_t, i) < *min) *min = ar_get_v(data, int8_t, i);
|
||||
}
|
||||
}
|
||||
|
||||
void print_array(const array_t data)
|
||||
{
|
||||
printf("---%d elements---\n", (int)ar_size(data));
|
||||
for(register int i=0;i<ar_size(data);i++)
|
||||
printf("%d ", (int)ar_get_v(data, unsigned char, i));
|
||||
printf("\n---\n");
|
||||
}
|
||||
|
||||
void unshuffle3(S_LEXENV, array_t data)
|
||||
{
|
||||
if(ar_type(data)!=sizeof(int8_t))
|
||||
ar_reinterpret(data, sizeof(int8_t));
|
||||
RNG xoro = S_LEXCALL(rng_object, rng_new(RNG_ALGO(xoroshiro128plus)));
|
||||
RNG frng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(frng)));
|
||||
RNG drng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(drng)));
|
||||
|
||||
int8_t bmax = INT8_MIN;
|
||||
int8_t bmin = INT8_MAX;
|
||||
minmax_sbytes(&bmin, &bmax, data);
|
||||
uint32_t seed = (0xfffa << 16) | (bmin<<8) | bmax;
|
||||
rng_seed(drng, &seed);
|
||||
unshuffle(drng, data);
|
||||
|
||||
float fmin = (float)DBL_MAX;
|
||||
float fmax = (float)-DBL_MAX;
|
||||
ar_reinterpret(data, sizeof(float));
|
||||
minmax_floats(&fmin, &fmax, data);
|
||||
double fseed[2];
|
||||
fseed[0] = fmin;
|
||||
fseed[1] = fmax;
|
||||
rng_seed(frng, fseed);
|
||||
unshuffle(frng, data);
|
||||
|
||||
int64_t min = INT64_MAX;
|
||||
int64_t max = INT64_MIN;
|
||||
ar_reinterpret(data, sizeof(int64_t));
|
||||
minmax_int64_ts(&min, &max, data);
|
||||
uint64_t xseed[2];
|
||||
xseed[0] = min;
|
||||
xseed[1] = max;
|
||||
rng_seed(xoro, xseed);
|
||||
unshuffle(xoro, data);
|
||||
}
|
||||
void shuffle3(S_LEXENV, array_t data)
|
||||
{
|
||||
if(ar_type(data)!=sizeof(int64_t))
|
||||
ar_reinterpret(data, sizeof(int64_t));
|
||||
RNG xoro = S_LEXCALL(rng_object, rng_new(RNG_ALGO(xoroshiro128plus)));
|
||||
RNG frng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(frng)));
|
||||
RNG drng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(drng)));
|
||||
|
||||
int64_t min = INT64_MAX;
|
||||
int64_t max = INT64_MIN;
|
||||
|
||||
float fmin = (float)DBL_MAX;
|
||||
float fmax = (float)-DBL_MAX;
|
||||
|
||||
int8_t bmax = INT8_MIN;
|
||||
int8_t bmin = INT8_MAX;
|
||||
|
||||
minmax_int64_ts(&min, &max, data);
|
||||
uint64_t xseed[2];
|
||||
xseed[0] = min;
|
||||
xseed[1] = max;
|
||||
|
||||
rng_seed(xoro, xseed);
|
||||
shuffle(xoro, data);
|
||||
|
||||
ar_reinterpret(data, sizeof(float));
|
||||
minmax_floats(&fmin, &fmax, data);
|
||||
|
||||
double fseed[2];
|
||||
fseed[0] = fmin;
|
||||
fseed[1] = fmax;
|
||||
rng_seed(frng, fseed);
|
||||
shuffle(frng, data);
|
||||
|
||||
ar_reinterpret(data,sizeof(int8_t));
|
||||
minmax_sbytes(&bmin, &bmax, data);
|
||||
uint32_t seed = (0xfffa << 16) | (bmin<<8) | bmax;
|
||||
rng_seed(drng, &seed);
|
||||
shuffle(drng, data);
|
||||
}
|
||||
|
||||
void print_usage(char** argv)
|
||||
{
|
||||
printf("Usage: %s -[b]s <file> [<outfile>]\nUsage: %s -[b]u <file> [<outfile>]\n", argv[0], argv[0]);
|
||||
printf("Usage: %s -S <string>|-\nUsage: %s -U <string>|-\n", argv[0], argv[0]);
|
||||
printf(" <file>\t\tFile to shuffle\n");
|
||||
printf(" <outfile>\tOutput file (only valid for buffered mode)\n");
|
||||
printf(" <string>\tThe string to shuffle (for -(S|U)). If `-`, read from stdin\n");
|
||||
printf("\ts\tShuffle\n");
|
||||
printf("\tu\tReverse shuffle\n");
|
||||
printf("\tS\tShuffle string\n");
|
||||
printf("\tU\tReverse shuffle string\n");
|
||||
printf("\n\tb\tUse buffered mode instead of shuffling in place (must specify output file)\n");
|
||||
}
|
||||
|
||||
array_t read_whole_file(S_NAMED_LEXENV(base), FILE* fp)
|
||||
{
|
||||
fseek(fp,0,SEEK_END);
|
||||
int64_t sz = ftell(fp);
|
||||
fseek(fp,0,SEEK_SET);
|
||||
|
||||
array_t ar;
|
||||
MANAGED({
|
||||
void* buf = smalloc(sz);
|
||||
|
||||
register size_t read;
|
||||
register size_t full=0;
|
||||
while ((read=fread(buf, 1, sz-full, fp))>0)
|
||||
full+=read;
|
||||
|
||||
assert(full == sz);
|
||||
|
||||
ar = ar_create_memory_from(base, buf, 1, sz);
|
||||
});
|
||||
return ar;
|
||||
}
|
||||
|
||||
#define BUFSIZE 512
|
||||
#define BUFMIN 128
|
||||
|
||||
char* read_stdin()
|
||||
{
|
||||
char* input, *p;
|
||||
int len,remain,n,size;
|
||||
|
||||
size = BUFSIZE;
|
||||
input = malloc(size);
|
||||
memset(input,0,size);
|
||||
len=0;
|
||||
remain = size;
|
||||
|
||||
while(!feof(stdin))
|
||||
{
|
||||
if(remain<= BUFMIN)
|
||||
{
|
||||
remain+=size;
|
||||
size *=2;
|
||||
p = realloc(input, size);
|
||||
if(p==NULL) {
|
||||
free(input);
|
||||
return NULL;
|
||||
}
|
||||
input = p;
|
||||
}
|
||||
fgets(input+len, remain, stdin);
|
||||
n+=strlen(input+len);
|
||||
len+=n;
|
||||
remain-=n;
|
||||
}
|
||||
return input;
|
||||
}
|
||||
|
||||
int string_shuffle(char* string, int un)
|
||||
{
|
||||
if(!string)
|
||||
{
|
||||
char* ptr = read_stdin();
|
||||
if(!ptr) {
|
||||
printf("! read from stdin failed\n");
|
||||
return -1;
|
||||
}
|
||||
int ret = string_shuffle(ptr, un);
|
||||
free(ptr);
|
||||
return ret;
|
||||
}
|
||||
else if(strcmp(string, "-") == 0)
|
||||
{
|
||||
return string_shuffle(NULL, un);
|
||||
}
|
||||
else {
|
||||
surpress_out=1;
|
||||
MANAGED({
|
||||
array_t ar = ar_create_memory_from(LEXENV, string, 1, strlen(string));
|
||||
|
||||
if(un) unshuffle3(LEXENV, ar);
|
||||
else shuffle3(LEXENV, ar);
|
||||
|
||||
ar_reinterpret(ar, sizeof(char));
|
||||
|
||||
for(int i=0;i<ar_size(ar);i++)
|
||||
{
|
||||
printf("%c", ar_get_v(ar, char, i));
|
||||
}
|
||||
});
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
int main(int argc, char** argv)
|
||||
{
|
||||
int is_buffered=0;
|
||||
int i=1;
|
||||
int is_unshuffling =0;
|
||||
if(!argv[1] || !argv[2] || argv[1][0] != '-')
|
||||
{
|
||||
print_usage(argv);
|
||||
return 1;
|
||||
}
|
||||
else {
|
||||
do_switch:
|
||||
switch(argv[1][i])
|
||||
{
|
||||
case 'b':
|
||||
if(is_buffered)
|
||||
{
|
||||
print_usage(argv);
|
||||
return 1;
|
||||
}
|
||||
is_buffered = 1;
|
||||
i+=1;
|
||||
goto do_switch;
|
||||
case 'u':
|
||||
is_unshuffling=1;
|
||||
break;
|
||||
case 's':
|
||||
break;
|
||||
case 'S':
|
||||
return string_shuffle(argv[2], 0);
|
||||
case 'U':
|
||||
return string_shuffle(argv[2], 1);
|
||||
default:
|
||||
print_usage(argv);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
if(is_buffered && !argv[3])
|
||||
{
|
||||
printf("Buffered option requires an output file\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
MANAGED_RETURNABLE(int ,{
|
||||
array_t array;
|
||||
|
||||
if(is_buffered)
|
||||
{
|
||||
FILE* infile = fopen(argv[2], "rb");
|
||||
if(!infile)
|
||||
{
|
||||
printf("! could not open file for reading\n");
|
||||
MANAGED_RETURN(1);
|
||||
}
|
||||
array = read_whole_file(LEXENV, infile);
|
||||
fclose(infile);
|
||||
printf(" buffered file (%ld bytes)\n", ar_size(array));
|
||||
}
|
||||
else {
|
||||
FILE* infile = fopen(argv[2], "r+b");
|
||||
if(!infile)
|
||||
{
|
||||
printf("! could not open file for reading+writing\n");
|
||||
MANAGED_RETURN(1);
|
||||
}
|
||||
array = ar_create_file(LEXENV, infile, 1, 1);
|
||||
}
|
||||
|
||||
//print_array(array);
|
||||
|
||||
if(is_unshuffling) unshuffle3(LEXENV, array);
|
||||
else shuffle3(LEXENV, array);
|
||||
|
||||
if(is_buffered)
|
||||
{
|
||||
FILE* outfile = fopen(argv[3], "wb");
|
||||
if(!outfile)
|
||||
{
|
||||
printf("! could not open outfile for writing\n");
|
||||
MANAGED_RETURN(1);
|
||||
}
|
||||
void* wbuf = smalloc(ar_full_size(array));
|
||||
if(!ar_ndump(array, wbuf, ar_full_size(array), 0, ar_size(array)))
|
||||
printf("W memory dump failed, continuing anyway\n");
|
||||
fwrite(wbuf, 1, ar_full_size(array), outfile);
|
||||
fclose(outfile);
|
||||
printf(" write completed (%d bytes)\n", (int)ar_size(array));
|
||||
}
|
||||
});
|
||||
return 0;
|
||||
}
|
||||
|
After Width: | Height: | Size: 5.0 KiB |
|
After Width: | Height: | Size: 5.1 KiB |
|
After Width: | Height: | Size: 41 KiB |
|
After Width: | Height: | Size: 25 KiB |
|
After Width: | Height: | Size: 42 KiB |
|
After Width: | Height: | Size: 51 KiB |
@ -0,0 +1,567 @@
|
||||
desc: (none)
|
||||
cmd: ./shuffle3-release small-u
|
||||
time_unit: i
|
||||
#-----------
|
||||
snapshot=0
|
||||
#-----------
|
||||
time=0
|
||||
mem_heap_B=0
|
||||
mem_heap_extra_B=0
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=1
|
||||
#-----------
|
||||
time=4048133
|
||||
mem_heap_B=32816
|
||||
mem_heap_extra_B=8
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=2
|
||||
#-----------
|
||||
time=4050820
|
||||
mem_heap_B=8
|
||||
mem_heap_extra_B=16
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=3
|
||||
#-----------
|
||||
time=4057134
|
||||
mem_heap_B=72712
|
||||
mem_heap_extra_B=24
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=4
|
||||
#-----------
|
||||
time=4282565
|
||||
mem_heap_B=73736
|
||||
mem_heap_extra_B=32
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=5
|
||||
#-----------
|
||||
time=4289663
|
||||
mem_heap_B=74248
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=6
|
||||
#-----------
|
||||
time=4295729
|
||||
mem_heap_B=75784
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=7
|
||||
#-----------
|
||||
time=4303612
|
||||
mem_heap_B=77832
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=8
|
||||
#-----------
|
||||
time=4318766
|
||||
mem_heap_B=86024
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=9
|
||||
#-----------
|
||||
time=4322930
|
||||
mem_heap_B=81928
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=10
|
||||
#-----------
|
||||
time=4353188
|
||||
mem_heap_B=98312
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=11
|
||||
#-----------
|
||||
time=4361448
|
||||
mem_heap_B=90120
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=12
|
||||
#-----------
|
||||
time=4421914
|
||||
mem_heap_B=122888
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=13
|
||||
#-----------
|
||||
time=4438366
|
||||
mem_heap_B=122888
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=detailed
|
||||
n3: 122888 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x1: ???
|
||||
n1: 72704 0x1FFF000AC6: ???
|
||||
n0: 72704 0x1FFF000AD9: ???
|
||||
n1: 49152 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n1: 49152 0x109385: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n0: 49152 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||
n0: 1032 in 3 places, all below massif's threshold (1.00%)
|
||||
#-----------
|
||||
snapshot=14
|
||||
#-----------
|
||||
time=4438366
|
||||
mem_heap_B=106504
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=15
|
||||
#-----------
|
||||
time=4559248
|
||||
mem_heap_B=172040
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=16
|
||||
#-----------
|
||||
time=4592084
|
||||
mem_heap_B=172040
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=detailed
|
||||
n3: 172040 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||
n1: 98304 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n1: 98304 0x109385: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n0: 98304 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x1: ???
|
||||
n1: 72704 0x1FFF000AC6: ???
|
||||
n0: 72704 0x1FFF000AD9: ???
|
||||
n0: 1032 in 3 places, all below massif's threshold (1.00%)
|
||||
#-----------
|
||||
snapshot=17
|
||||
#-----------
|
||||
time=4592084
|
||||
mem_heap_B=139272
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=18
|
||||
#-----------
|
||||
time=4833798
|
||||
mem_heap_B=270344
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=19
|
||||
#-----------
|
||||
time=4899402
|
||||
mem_heap_B=270344
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=detailed
|
||||
n3: 270344 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||
n1: 196608 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n1: 196608 0x109385: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n0: 196608 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x1: ???
|
||||
n1: 72704 0x1FFF000AC6: ???
|
||||
n0: 72704 0x1FFF000AD9: ???
|
||||
n0: 1032 in 3 places, all below massif's threshold (1.00%)
|
||||
#-----------
|
||||
snapshot=20
|
||||
#-----------
|
||||
time=4899402
|
||||
mem_heap_B=204808
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=21
|
||||
#-----------
|
||||
time=5382780
|
||||
mem_heap_B=466952
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=22
|
||||
#-----------
|
||||
time=5513920
|
||||
mem_heap_B=466952
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=peak
|
||||
n3: 466952 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||
n1: 393216 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n1: 393216 0x109385: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n0: 393216 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x1: ???
|
||||
n1: 72704 0x1FFF000AC6: ???
|
||||
n0: 72704 0x1FFF000AD9: ???
|
||||
n0: 1032 in 3 places, all below massif's threshold (1.00%)
|
||||
#-----------
|
||||
snapshot=23
|
||||
#-----------
|
||||
time=5513920
|
||||
mem_heap_B=335880
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=24
|
||||
#-----------
|
||||
time=5892087
|
||||
mem_heap_B=73736
|
||||
mem_heap_extra_B=32
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=25
|
||||
#-----------
|
||||
time=5989421
|
||||
mem_heap_B=73744
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=26
|
||||
#-----------
|
||||
time=5996279
|
||||
mem_heap_B=73992
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=27
|
||||
#-----------
|
||||
time=6002698
|
||||
mem_heap_B=74248
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=28
|
||||
#-----------
|
||||
time=6015289
|
||||
mem_heap_B=75272
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=29
|
||||
#-----------
|
||||
time=6015414
|
||||
mem_heap_B=74760
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=30
|
||||
#-----------
|
||||
time=6040651
|
||||
mem_heap_B=76808
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=31
|
||||
#-----------
|
||||
time=6040828
|
||||
mem_heap_B=75784
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=32
|
||||
#-----------
|
||||
time=6091310
|
||||
mem_heap_B=79880
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=detailed
|
||||
n4: 79880 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x1: ???
|
||||
n1: 72704 0x1FFF000AC6: ???
|
||||
n0: 72704 0x1FFF000AD9: ???
|
||||
n2: 6144 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n1: 6144 0x1095C2: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n0: 6144 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||
n0: 0 in 1 place, below massif's threshold (1.00%)
|
||||
n1: 1024 0x4CB6E03: _IO_file_doallocate (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CC570F: _IO_doallocbuf (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CC48A7: _IO_file_overflow@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CC3955: _IO_file_xsputn@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CB8330: fwrite (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4877FD7: fmt::v7::vprint(_IO_FILE*, fmt::v7::basic_string_view<char>, fmt::v7::format_args) (in /usr/lib/libfmt.so.7.1.2)
|
||||
n1: 1024 0x10A258: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n1: 1024 0x10933E: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n0: 1024 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||
n0: 8 in 2 places, all below massif's threshold (1.00%)
|
||||
#-----------
|
||||
snapshot=33
|
||||
#-----------
|
||||
time=6091591
|
||||
mem_heap_B=77832
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=34
|
||||
#-----------
|
||||
time=6192490
|
||||
mem_heap_B=86024
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=35
|
||||
#-----------
|
||||
time=6196654
|
||||
mem_heap_B=81928
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=36
|
||||
#-----------
|
||||
time=6398233
|
||||
mem_heap_B=98312
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=37
|
||||
#-----------
|
||||
time=6406493
|
||||
mem_heap_B=90120
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=38
|
||||
#-----------
|
||||
time=6809904
|
||||
mem_heap_B=122888
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=39
|
||||
#-----------
|
||||
time=6826356
|
||||
mem_heap_B=106504
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=40
|
||||
#-----------
|
||||
time=7633039
|
||||
mem_heap_B=172040
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=41
|
||||
#-----------
|
||||
time=7665875
|
||||
mem_heap_B=139272
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=42
|
||||
#-----------
|
||||
time=7951930
|
||||
mem_heap_B=73736
|
||||
mem_heap_extra_B=32
|
||||
mem_stacks_B=0
|
||||
heap_tree=detailed
|
||||
n3: 73736 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x1: ???
|
||||
n1: 72704 0x1FFF000AC6: ???
|
||||
n0: 72704 0x1FFF000AD9: ???
|
||||
n1: 1024 0x4CB6E03: _IO_file_doallocate (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CC570F: _IO_doallocbuf (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CC48A7: _IO_file_overflow@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CC3955: _IO_file_xsputn@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CB8330: fwrite (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4877FD7: fmt::v7::vprint(_IO_FILE*, fmt::v7::basic_string_view<char>, fmt::v7::format_args) (in /usr/lib/libfmt.so.7.1.2)
|
||||
n1: 1024 0x10A258: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n1: 1024 0x10933E: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n0: 1024 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||
n0: 8 in 3 places, all below massif's threshold (1.00%)
|
||||
#-----------
|
||||
snapshot=43
|
||||
#-----------
|
||||
time=7971885
|
||||
mem_heap_B=73744
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=44
|
||||
#-----------
|
||||
time=7979276
|
||||
mem_heap_B=75784
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=45
|
||||
#-----------
|
||||
time=7985879
|
||||
mem_heap_B=77832
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=46
|
||||
#-----------
|
||||
time=7998473
|
||||
mem_heap_B=86024
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=detailed
|
||||
n4: 86024 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|
||||
n1: 72704 0x493F45A: pool (eh_alloc.cc:123)
|
||||
n1: 72704 0x493F45A: __static_initialization_and_destruction_0 (eh_alloc.cc:262)
|
||||
n1: 72704 0x493F45A: _GLOBAL__sub_I_eh_alloc.cc (eh_alloc.cc:338)
|
||||
n1: 72704 0x40112DD: call_init.part.0 (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40113C7: _dl_init (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x40020C9: ??? (in /usr/lib/ld-2.32.so)
|
||||
n1: 72704 0x1: ???
|
||||
n1: 72704 0x1FFF000AC6: ???
|
||||
n0: 72704 0x1FFF000AD9: ???
|
||||
n2: 12288 0x10A384: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n1: 12288 0x10973E: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n0: 12288 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||
n0: 0 in 2 places, all below massif's threshold (1.00%)
|
||||
n1: 1024 0x4CB6E03: _IO_file_doallocate (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CC570F: _IO_doallocbuf (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CC48A7: _IO_file_overflow@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CC3955: _IO_file_xsputn@@GLIBC_2.2.5 (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4CB8330: fwrite (in /usr/lib/libc-2.32.so)
|
||||
n1: 1024 0x4877FD7: fmt::v7::vprint(_IO_FILE*, fmt::v7::basic_string_view<char>, fmt::v7::format_args) (in /usr/lib/libfmt.so.7.1.2)
|
||||
n1: 1024 0x10A258: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n1: 1024 0x10933E: ??? (in /home/avril/work/shuffle3/lean/shuffle3-release)
|
||||
n0: 1024 0x4C6A151: (below main) (in /usr/lib/libc-2.32.so)
|
||||
n0: 8 in 2 places, all below massif's threshold (1.00%)
|
||||
#-----------
|
||||
snapshot=47
|
||||
#-----------
|
||||
time=8002637
|
||||
mem_heap_B=81928
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=48
|
||||
#-----------
|
||||
time=8027775
|
||||
mem_heap_B=98312
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=49
|
||||
#-----------
|
||||
time=8036035
|
||||
mem_heap_B=90120
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=50
|
||||
#-----------
|
||||
time=8086261
|
||||
mem_heap_B=122888
|
||||
mem_heap_extra_B=48
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=51
|
||||
#-----------
|
||||
time=8102713
|
||||
mem_heap_B=106504
|
||||
mem_heap_extra_B=40
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=52
|
||||
#-----------
|
||||
time=8147526
|
||||
mem_heap_B=73736
|
||||
mem_heap_extra_B=32
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=53
|
||||
#-----------
|
||||
time=8156172
|
||||
mem_heap_B=1032
|
||||
mem_heap_extra_B=24
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
#-----------
|
||||
snapshot=54
|
||||
#-----------
|
||||
time=8157131
|
||||
mem_heap_B=8
|
||||
mem_heap_extra_B=16
|
||||
mem_stacks_B=0
|
||||
heap_tree=empty
|
||||
|
After Width: | Height: | Size: 4.1 KiB |
|
After Width: | Height: | Size: 4.5 KiB |
|
After Width: | Height: | Size: 3.2 KiB |
|
After Width: | Height: | Size: 3.6 KiB |
|
After Width: | Height: | Size: 4.1 KiB |
|
After Width: | Height: | Size: 3.0 KiB |
|
After Width: | Height: | Size: 32 KiB |
|
After Width: | Height: | Size: 38 KiB |
|
After Width: | Height: | Size: 10 KiB |
|
After Width: | Height: | Size: 17 KiB |
|
After Width: | Height: | Size: 19 KiB |
|
After Width: | Height: | Size: 23 KiB |
@ -0,0 +1,87 @@
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
#include <stdarg.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
#include <string.h>
|
||||
|
||||
#include <debug.h>
|
||||
|
||||
#include <colours.h>
|
||||
|
||||
enum jt_comp {
|
||||
JT_COMP_UNIX,
|
||||
JT_COMP_DATE,
|
||||
JT_COMP_TIME,
|
||||
JT_COMP_DATETIME,
|
||||
};
|
||||
|
||||
#define CLONE(p) ({ __auto_type $__p = (p); \
|
||||
__auto_type $__r = __builtin_choose_expr(__builtin_classify_type($__p) == 5, *$__p, $__p); \
|
||||
/* ((__typeof__($__r) []) { $__r }); }) */ \
|
||||
__typeof__($__r) $__rp[1]; \
|
||||
$__rp[0] = $__r; \
|
||||
&$__rp[0]; })
|
||||
|
||||
__attribute__((returns_nonnull, nonnull(1)))
|
||||
static inline
|
||||
const char* jtime(const struct tm* restrict lt, enum jt_comp kind)
|
||||
{
|
||||
_Thread_local static char datebuf[1
|
||||
+ 4 + 2 + 4 + 1 // `dd/mm/yyyy`
|
||||
+ 6 + 3 // ` hh:MM:ss`
|
||||
];
|
||||
_Static_assert(sizeof(datebuf) != sizeof(char*), "Unexpected decay");
|
||||
|
||||
//memset(datebuf, '0', sizeof(datebuf)-1);
|
||||
if(!kind)
|
||||
snprintf(datebuf, sizeof(datebuf), "%lu", mktime(CLONE(lt)));
|
||||
else {
|
||||
size_t w = 0, sz = sizeof(datebuf);
|
||||
if(kind & JT_COMP_DATE)
|
||||
w += snprintf(datebuf, sz, "%02u/%02u/%4.4u"
|
||||
, (unsigned)lt->tm_mday
|
||||
, (unsigned)lt->tm_mon
|
||||
, (unsigned)(1900 + lt->tm_year));
|
||||
|
||||
if(kind & JT_COMP_TIME)
|
||||
w += snprintf(datebuf+w, sz - w, " %02u:%02u:%02u"
|
||||
, (unsigned)lt->tm_hour
|
||||
, (unsigned)lt->tm_min
|
||||
, (unsigned)lt->tm_sec);
|
||||
}
|
||||
return datebuf;
|
||||
}
|
||||
|
||||
void _do_dprintf(struct debuginfo info, const char* fmt, ...)
|
||||
{
|
||||
#ifdef DEBUG
|
||||
va_list li;
|
||||
|
||||
time_t utime;
|
||||
struct tm ltime;
|
||||
time(&utime);
|
||||
localtime_r(&utime, <ime);
|
||||
|
||||
va_start(li, fmt);
|
||||
|
||||
#ifdef D_TRACE
|
||||
#define FMT_FUNC "%s"
|
||||
#else
|
||||
#define FMT_FUNC "%s()"
|
||||
#endif
|
||||
|
||||
#define TTY_OUT "["FGRN("dbg")" (%s %s) " FYEL("@") BOLD("%s") "->" FWHT(FMT_FUNC) ":" FYEL("%d") "]: "
|
||||
#define NCOL(x) x
|
||||
#define FIL_OUT "["NCOL("dbg")" (%s %s) " NCOL("@") NCOL("%s") "->" NCOL(FMT_FUNC) ":" NCOL("%d") "]: "
|
||||
|
||||
fprintf(stderr, isatty(fileno(stderr)) ? (TTY_OUT) : (FIL_OUT)
|
||||
, jtime(<ime, JT_COMP_DATETIME), ltime.tm_zone
|
||||
, info.file
|
||||
, info.function
|
||||
, info.line);
|
||||
vfprintf(stderr, fmt, li);
|
||||
fprintf(stderr, "\n");
|
||||
va_end(li);
|
||||
#endif
|
||||
}
|
||||
@ -0,0 +1,67 @@
|
||||
#include <unistd.h>
|
||||
#include <fsvec.h>
|
||||
#include <panic.h>
|
||||
|
||||
int fvec_new(fvec_t* restrict obj, const char* path)
|
||||
{
|
||||
obj->backing = fopen(path, "w+b");
|
||||
if(!obj->backing) {
|
||||
perror("fvec_new(): Failed to open file");
|
||||
return 0;
|
||||
}
|
||||
obj->len=0;
|
||||
return 1;
|
||||
}
|
||||
|
||||
void fvec_close(fvec_t* restrict obj)
|
||||
{
|
||||
if(obj->backing) fclose(obj->backing);
|
||||
obj->backing=NULL;
|
||||
obj->len=0;
|
||||
}
|
||||
|
||||
int fvec_pop_end(fvec_t* restrict obj, size_t sz)
|
||||
{
|
||||
if(!obj->len) return 0;
|
||||
|
||||
if(ftruncate(fileno(obj->backing), (obj->len-=sz))<0) {
|
||||
perror("Failed to pop buffer");
|
||||
return 0;
|
||||
}
|
||||
return 1;
|
||||
}
|
||||
|
||||
void fvec_push_whole_buffer(fvec_t* restrict obj, const void* _buffer, size_t sz)
|
||||
{
|
||||
size_t w= 0;
|
||||
size_t c;
|
||||
const unsigned char* buffer = _buffer;
|
||||
while ( w<sz && (c=fwrite(buffer+w, 1, sz-w, obj->backing))>0 ) w+=c;
|
||||
if(w!=sz) {
|
||||
perror("Corrupted buffer state, aborting");
|
||||
panic("Cannot continue");
|
||||
}
|
||||
obj->len += sz;
|
||||
}
|
||||
int fvec_get_whole_buffer(const fvec_t* restrict obj, void* _buffer, size_t _sz)
|
||||
{
|
||||
ssize_t sz = (ssize_t)_sz;
|
||||
if(_sz != (size_t)sz) panic("Buffer size %lu too large", _sz);
|
||||
|
||||
if(obj->len<(size_t)sz) return 0;
|
||||
if(fseek(obj->backing, -sz, SEEK_END)<0) {
|
||||
perror("fvec_get_whole_buffer() failed to seek");
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t w=0;
|
||||
ssize_t c;
|
||||
unsigned char* buffer = _buffer;
|
||||
while ( w<_sz && (c=fread(buffer+w, 1, sz-w, obj->backing))>0 ) w+=c;
|
||||
if (w!=_sz) {
|
||||
perror("Corrupted buffer state, aborting");
|
||||
panic("Cannot continue on FD %d (%p)", fileno(obj->backing), obj->backing);
|
||||
}
|
||||
|
||||
return 1;
|
||||
}
|
||||
@ -0,0 +1,73 @@
|
||||
#include <filesystem>
|
||||
|
||||
#include <fsvec.hpp>
|
||||
#include <uuid.hpp>
|
||||
#include <fsvec.h>
|
||||
#include <debug.h>
|
||||
|
||||
#include <tempfile.hpp>
|
||||
|
||||
#define FB file_back_buffer
|
||||
|
||||
|
||||
struct FB::impl
|
||||
{
|
||||
size_t cap;
|
||||
temp_file file;
|
||||
|
||||
fvec_t backing;
|
||||
};
|
||||
|
||||
FB::FB(size_t cap) : inner(std::make_unique<FB::impl>())
|
||||
{
|
||||
// Set cap
|
||||
inner->cap = cap;
|
||||
// Create then open `inner->file`
|
||||
if(!fvec_new(&inner->backing, inner->file->c_str())) panic("Failed to open backing for temp file buffer");
|
||||
|
||||
}
|
||||
FB::FB(file_back_buffer&& m) : inner(std::move(m.inner)){}
|
||||
FB::FB() : FB(DEFAULT_CAP){}
|
||||
FB::~FB()
|
||||
{
|
||||
fvec_close(&inner->backing);
|
||||
}
|
||||
|
||||
void FB::push_buf(byte* buf, size_t len)
|
||||
{
|
||||
fvec_push_whole_buffer(&inner->backing, (void*)buf, len);
|
||||
}
|
||||
|
||||
bool FB::back(byte* buf, size_t len) const
|
||||
{
|
||||
return !!fvec_get_whole_buffer(&inner->backing, (void*)buf, len);
|
||||
}
|
||||
|
||||
bool FB::pop_n(size_t len)
|
||||
{
|
||||
return !!fvec_pop_end(&inner->backing, len);
|
||||
}
|
||||
|
||||
extern "C" void _fb_run_tests()
|
||||
{
|
||||
fixed_spill_vector<int, 8> test;
|
||||
int r0,r1=0;
|
||||
for(int i=0;i<10;i++) {
|
||||
D_dprintf("push: %d", (10-i));
|
||||
test.push_back(10-i);
|
||||
r1+= (10-i);
|
||||
}
|
||||
D_dprintf("r1: %d", r1);
|
||||
r0=0;
|
||||
while(test.size())
|
||||
{
|
||||
r0+=test.back();
|
||||
D_dprintf("back: %d", test.back());
|
||||
test.pop_back();
|
||||
}
|
||||
D_dprintf("r0: %d", r0);
|
||||
|
||||
if(r0!=r1) panic("fb failed test: %d, %d", r0, r1);
|
||||
|
||||
D_dprintf("test successful");
|
||||
}
|
||||
@ -1,365 +1,83 @@
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
#include <float.h>
|
||||
#include <stdint.h>
|
||||
#include <string.h>
|
||||
|
||||
#include <rng.h>
|
||||
#include <rng_algos.h>
|
||||
#include <ptr_store.h>
|
||||
#include <array.h>
|
||||
|
||||
_Static_assert(sizeof(float)==4, "Float must be 32 bits.");
|
||||
#include <string.h>
|
||||
|
||||
int surpress_out=0;
|
||||
|
||||
/*void test_destructor(object* ptr)
|
||||
{
|
||||
printf("die\n");
|
||||
}*/
|
||||
void _rng_object_destructor(object* obj)
|
||||
{
|
||||
rng_free(obj->state);
|
||||
}
|
||||
RNG rng_object(S_LEXENV, RNG rng)
|
||||
{
|
||||
object proto = OBJ_PROTO;
|
||||
proto.state = rng;
|
||||
proto.destructor = &_rng_object_destructor;
|
||||
|
||||
return snew_obj(proto)->state;
|
||||
}
|
||||
#include <shuffle3.h>
|
||||
#include <panic.h>
|
||||
#include <reinterpret.h>
|
||||
#include <map.h>
|
||||
#include <rng.h>
|
||||
#include <work.h>
|
||||
|
||||
void shuffle(RNG with, array_t data)
|
||||
{
|
||||
if(!surpress_out) printf(" -> shuffling %d objects...", (int)ar_size(data));
|
||||
for(int i=ar_size(data)-1;i>0;i--)
|
||||
{
|
||||
int j = rng_next_int(with, i);
|
||||
ar_swap(data, i, j);
|
||||
}
|
||||
if(!surpress_out) printf(" Okay\n");
|
||||
}
|
||||
#include <debug.h>
|
||||
|
||||
void unshuffle(RNG with, array_t data)
|
||||
{
|
||||
int rng_values[ar_size(data)-1];
|
||||
int k=0;
|
||||
if(!surpress_out) printf(" -> unshuffling %d objects...", (int)ar_size(data));
|
||||
for(int i=ar_size(data)-1;i>0;i--)
|
||||
rng_values[k++] = rng_next_int(with, i);
|
||||
#define noreturn __attribute__((noreturn)) void
|
||||
|
||||
for(int i=1;i<ar_size(data);i++)
|
||||
{
|
||||
ar_swap(data, i, rng_values[--k]);
|
||||
}
|
||||
if(!surpress_out) printf(" Okay\n");
|
||||
}
|
||||
_Static_assert(sizeof(float)==sizeof(uint32_t), "float is not 32 bits");
|
||||
|
||||
void minmax_int64_ts(int64_t* min, int64_t* max, const array_t data)
|
||||
{
|
||||
for(register int i=0;i<ar_size(data);i++)
|
||||
{
|
||||
if(ar_get_v(data, int64_t, i) > *max) *max = ar_get_v(data, int64_t, i);
|
||||
if(ar_get_v(data, int64_t, i) < *min) *min = ar_get_v(data, int64_t, i);
|
||||
}
|
||||
}
|
||||
int valid_float(float f)
|
||||
{
|
||||
return !( (f!=f) || (f< -FLT_MAX || f> FLT_MAX));
|
||||
const char* _prog_name;
|
||||
|
||||
}
|
||||
void minmax_floats(float* min, float* max, const array_t data)
|
||||
{
|
||||
for(register int i=0;i<ar_size(data);i++)
|
||||
{
|
||||
if(!valid_float(ar_get_v(data,float,i))) continue;
|
||||
if(ar_get_v(data, float, i) > *max) *max = ar_get_v(data, float, i);
|
||||
if(ar_get_v(data, float, i) < *min) *min = ar_get_v(data, float, i);
|
||||
}
|
||||
}
|
||||
extern void _fb_run_tests();
|
||||
|
||||
void minmax_sbytes(int8_t* min, int8_t* max, const array_t data)
|
||||
noreturn help_then_exit()
|
||||
{
|
||||
for(register int i=0;i<ar_size(data);i++)
|
||||
{
|
||||
if(ar_get_v(data, int8_t, i) > *max) *max = ar_get_v(data, int8_t, i);
|
||||
if(ar_get_v(data, int8_t, i) < *min) *min = ar_get_v(data, int8_t, i);
|
||||
}
|
||||
fprintf(stderr, "Try passing `-h`\n");
|
||||
exit(1);
|
||||
}
|
||||
|
||||
void print_array(const array_t data)
|
||||
void usage()
|
||||
{
|
||||
printf("---%d elements---\n", (int)ar_size(data));
|
||||
for(register int i=0;i<ar_size(data);i++)
|
||||
printf("%d ", (int)ar_get_v(data, unsigned char, i));
|
||||
printf("\n---\n");
|
||||
printf( "shuffle3-lean - improved 3 pass binary shuffler\n"
|
||||
" written by Avril <flanchan@cumallover.me> (flanchan.moe) with <3\n"
|
||||
" licensed with GPL v3.0 or later\n\n"
|
||||
"Usage: %s -s <file>\n"
|
||||
"Usage: %s -u <file>\n", _prog_name, _prog_name);
|
||||
printf("\nOPTIONS:\n"
|
||||
" -s\tShuffle file in place\n"
|
||||
" -u\tUnshuffle file in place\n");
|
||||
}
|
||||
|
||||
void unshuffle3(S_LEXENV, array_t data)
|
||||
int main(int argc, char** argv)
|
||||
{
|
||||
if(ar_type(data)!=sizeof(int8_t))
|
||||
ar_reinterpret(data, sizeof(int8_t));
|
||||
RNG xoro = S_LEXCALL(rng_object, rng_new(RNG_ALGO(xoroshiro128plus)));
|
||||
RNG frng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(frng)));
|
||||
RNG drng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(drng)));
|
||||
_prog_name = argv[0];
|
||||
|
||||
int8_t bmax = INT8_MIN;
|
||||
int8_t bmin = INT8_MAX;
|
||||
minmax_sbytes(&bmin, &bmax, data);
|
||||
uint32_t seed = (0xfffa << 16) | (bmin<<8) | bmax;
|
||||
rng_seed(drng, &seed);
|
||||
unshuffle(drng, data);
|
||||
//_fb_run_tests();
|
||||
work_args_t parsed;
|
||||
|
||||
float fmin = (float)DBL_MAX;
|
||||
float fmax = (float)-DBL_MAX;
|
||||
ar_reinterpret(data, sizeof(float));
|
||||
minmax_floats(&fmin, &fmax, data);
|
||||
double fseed[2];
|
||||
fseed[0] = fmin;
|
||||
fseed[1] = fmax;
|
||||
rng_seed(frng, fseed);
|
||||
unshuffle(frng, data);
|
||||
if( !argv[1] || *(argv[1]) != '-') help_then_exit();
|
||||
|
||||
int64_t min = INT64_MAX;
|
||||
int64_t max = INT64_MIN;
|
||||
ar_reinterpret(data, sizeof(int64_t));
|
||||
minmax_int64_ts(&min, &max, data);
|
||||
uint64_t xseed[2];
|
||||
xseed[0] = min;
|
||||
xseed[1] = max;
|
||||
rng_seed(xoro, xseed);
|
||||
unshuffle(xoro, data);
|
||||
}
|
||||
void shuffle3(S_LEXENV, array_t data)
|
||||
{
|
||||
if(ar_type(data)!=sizeof(int64_t))
|
||||
ar_reinterpret(data, sizeof(int64_t));
|
||||
RNG xoro = S_LEXCALL(rng_object, rng_new(RNG_ALGO(xoroshiro128plus)));
|
||||
RNG frng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(frng)));
|
||||
RNG drng = S_LEXCALL(rng_object, rng_new(RNG_ALGO(drng)));
|
||||
|
||||
int64_t min = INT64_MAX;
|
||||
int64_t max = INT64_MIN;
|
||||
|
||||
float fmin = (float)DBL_MAX;
|
||||
float fmax = (float)-DBL_MAX;
|
||||
|
||||
int8_t bmax = INT8_MIN;
|
||||
int8_t bmin = INT8_MAX;
|
||||
|
||||
minmax_int64_ts(&min, &max, data);
|
||||
uint64_t xseed[2];
|
||||
xseed[0] = min;
|
||||
xseed[1] = max;
|
||||
|
||||
rng_seed(xoro, xseed);
|
||||
shuffle(xoro, data);
|
||||
|
||||
ar_reinterpret(data, sizeof(float));
|
||||
minmax_floats(&fmin, &fmax, data);
|
||||
|
||||
double fseed[2];
|
||||
fseed[0] = fmin;
|
||||
fseed[1] = fmax;
|
||||
rng_seed(frng, fseed);
|
||||
shuffle(frng, data);
|
||||
|
||||
ar_reinterpret(data,sizeof(int8_t));
|
||||
minmax_sbytes(&bmin, &bmax, data);
|
||||
uint32_t seed = (0xfffa << 16) | (bmin<<8) | bmax;
|
||||
rng_seed(drng, &seed);
|
||||
shuffle(drng, data);
|
||||
}
|
||||
|
||||
void print_usage(char** argv)
|
||||
{
|
||||
printf("Usage: %s -[b]s <file> [<outfile>]\nUsage: %s -[b]u <file> [<outfile>]\n", argv[0], argv[0]);
|
||||
printf("Usage: %s -S <string>|-\nUsage: %s -U <string>|-\n", argv[0], argv[0]);
|
||||
printf(" <file>\t\tFile to shuffle\n");
|
||||
printf(" <outfile>\tOutput file (only valid for buffered mode)\n");
|
||||
printf(" <string>\tThe string to shuffle (for -(S|U)). If `-`, read from stdin\n");
|
||||
printf("\ts\tShuffle\n");
|
||||
printf("\tu\tReverse shuffle\n");
|
||||
printf("\tS\tShuffle string\n");
|
||||
printf("\tU\tReverse shuffle string\n");
|
||||
printf("\n\tb\tUse buffered mode instead of shuffling in place (must specify output file)\n");
|
||||
}
|
||||
|
||||
array_t read_whole_file(S_NAMED_LEXENV(base), FILE* fp)
|
||||
{
|
||||
fseek(fp,0,SEEK_END);
|
||||
int64_t sz = ftell(fp);
|
||||
fseek(fp,0,SEEK_SET);
|
||||
|
||||
array_t ar;
|
||||
MANAGED({
|
||||
void* buf = smalloc(sz);
|
||||
fread(buf, 1, sz, fp);
|
||||
ar = ar_create_memory_from(base, buf, 1, sz);
|
||||
});
|
||||
return ar;
|
||||
}
|
||||
|
||||
#define BUFSIZE 512
|
||||
#define BUFMIN 128
|
||||
|
||||
char* read_stdin()
|
||||
{
|
||||
char* input, *p;
|
||||
int len,remain,n,size;
|
||||
|
||||
size = BUFSIZE;
|
||||
input = malloc(size);
|
||||
memset(input,0,size);
|
||||
len=0;
|
||||
remain = size;
|
||||
|
||||
while(!feof(stdin))
|
||||
D_dprintf("Parsing `%c'", argv[1][1]);
|
||||
switch(argv[1][1])
|
||||
{
|
||||
if(remain<= BUFMIN)
|
||||
{
|
||||
remain+=size;
|
||||
size *=2;
|
||||
p = realloc(input, size);
|
||||
if(p==NULL) {
|
||||
free(input);
|
||||
return NULL;
|
||||
}
|
||||
input = p;
|
||||
}
|
||||
fgets(input+len, remain, stdin);
|
||||
n+=strlen(input+len);
|
||||
len+=n;
|
||||
remain-=n;
|
||||
}
|
||||
return input;
|
||||
}
|
||||
|
||||
int string_shuffle(char* string, int un)
|
||||
{
|
||||
if(!string)
|
||||
{
|
||||
char* ptr = read_stdin();
|
||||
if(!ptr) {
|
||||
printf("! read from stdin failed\n");
|
||||
return -1;
|
||||
}
|
||||
int ret = string_shuffle(ptr, un);
|
||||
free(ptr);
|
||||
return ret;
|
||||
}
|
||||
else if(strcmp(string, "-") == 0)
|
||||
{
|
||||
return string_shuffle(NULL, un);
|
||||
}
|
||||
else {
|
||||
surpress_out=1;
|
||||
MANAGED({
|
||||
array_t ar = ar_create_memory_from(LEXENV, string, 1, strlen(string));
|
||||
|
||||
if(un) unshuffle3(LEXENV, ar);
|
||||
else shuffle3(LEXENV, ar);
|
||||
|
||||
ar_reinterpret(ar, sizeof(char));
|
||||
|
||||
for(int i=0;i<ar_size(ar);i++)
|
||||
case 's':
|
||||
parsed.op = OP_SHUFFLE_IP;
|
||||
if(!(parsed.data.op_shuffle_ip.file = argv[2]))
|
||||
{
|
||||
printf("%c", ar_get_v(ar, char, i));
|
||||
}
|
||||
});
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
int main(int argc, char** argv)
|
||||
{
|
||||
int is_buffered=0;
|
||||
int i=1;
|
||||
int is_unshuffling =0;
|
||||
if(!argv[1] || !argv[2] || argv[1][0] != '-')
|
||||
{
|
||||
print_usage(argv);
|
||||
return 1;
|
||||
}
|
||||
else {
|
||||
do_switch:
|
||||
switch(argv[1][i])
|
||||
{
|
||||
case 'b':
|
||||
if(is_buffered)
|
||||
{
|
||||
print_usage(argv);
|
||||
return 1;
|
||||
}
|
||||
is_buffered = 1;
|
||||
i+=1;
|
||||
goto do_switch;
|
||||
case 'u':
|
||||
is_unshuffling=1;
|
||||
break;
|
||||
case 's':
|
||||
break;
|
||||
case 'S':
|
||||
return string_shuffle(argv[2], 0);
|
||||
case 'U':
|
||||
return string_shuffle(argv[2], 1);
|
||||
default:
|
||||
print_usage(argv);
|
||||
fprintf(stderr, "Error: -s expected file argument.\n");
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
if(is_buffered && !argv[3])
|
||||
{
|
||||
printf("Buffered option requires an output file\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
MANAGED_RETURNABLE(int ,{
|
||||
array_t array;
|
||||
|
||||
if(is_buffered)
|
||||
{
|
||||
FILE* infile = fopen(argv[2], "rb");
|
||||
if(!infile)
|
||||
{
|
||||
printf("! could not open file for reading\n");
|
||||
MANAGED_RETURN(1);
|
||||
}
|
||||
array = read_whole_file(LEXENV, infile);
|
||||
fclose(infile);
|
||||
printf(" buffered file (%ld bytes)\n", ar_size(array));
|
||||
}
|
||||
else {
|
||||
FILE* infile = fopen(argv[2], "r+b");
|
||||
if(!infile)
|
||||
D_dprintf("parsed.op = %d", OP_SHUFFLE_IP);
|
||||
break;
|
||||
case 'u':
|
||||
parsed.op = OP_UNSHUFFLE_IP;
|
||||
if(!(parsed.data.op_unshuffle_ip.file = argv[2]))
|
||||
{
|
||||
printf("! could not open file for reading+writing\n");
|
||||
MANAGED_RETURN(1);
|
||||
fprintf(stderr, "Error: -u expected file argument.\n");
|
||||
return 1;
|
||||
}
|
||||
array = ar_create_file(LEXENV, infile, 1, 1);
|
||||
}
|
||||
|
||||
//print_array(array);
|
||||
D_dprintf("parsed.op = %d", OP_UNSHUFFLE_IP);
|
||||
break;
|
||||
case 'h':
|
||||
usage();
|
||||
return 0;
|
||||
default:
|
||||
fprintf(stderr, "Error: unknown argument `%s'\n\n", argv[1]);
|
||||
help_then_exit();
|
||||
panic("Unreachable");
|
||||
}
|
||||
|
||||
if(is_unshuffling) unshuffle3(LEXENV, array);
|
||||
else shuffle3(LEXENV, array);
|
||||
|
||||
if(is_buffered)
|
||||
{
|
||||
FILE* outfile = fopen(argv[3], "wb");
|
||||
if(!outfile)
|
||||
{
|
||||
printf("! could not open outfile for writing\n");
|
||||
MANAGED_RETURN(1);
|
||||
}
|
||||
void* wbuf = smalloc(ar_full_size(array));
|
||||
if(!ar_ndump(array, wbuf, ar_full_size(array), 0, ar_size(array)))
|
||||
printf("W memory dump failed, continuing anyway\n");
|
||||
fwrite(wbuf, 1, ar_full_size(array), outfile);
|
||||
fclose(outfile);
|
||||
printf(" write completed (%d bytes)\n", (int)ar_size(array));
|
||||
}
|
||||
});
|
||||
return 0;
|
||||
return do_work(parsed);
|
||||
}
|
||||
|
||||
|
||||
@ -0,0 +1,171 @@
|
||||
#define _GNU_SOURCE
|
||||
#include<stdio.h>
|
||||
#include<stdlib.h>
|
||||
#include<string.h>
|
||||
#include<unistd.h>
|
||||
#include <sys/types.h>
|
||||
#include <sys/stat.h>
|
||||
#include<sys/mman.h>
|
||||
#include<fcntl.h>
|
||||
|
||||
#define FILEMODE S_IRWXU | S_IRGRP | S_IROTH
|
||||
|
||||
#include <map.h>
|
||||
|
||||
__attribute__((pure, nonnull(1)))
|
||||
static inline int _map(mmap_t* restrict map)
|
||||
{
|
||||
//TODO: Based on map->len, choose (or calculate) appropriate `MAP_HUGE_*` flag
|
||||
return (map->ptr = mmap(NULL, map->len, PROT_READ | PROT_WRITE, MAP_SHARED, map->fd, 0)) != MAP_FAILED;
|
||||
}
|
||||
//#define _map(...) _map(__VA_ARGS__, 0)
|
||||
|
||||
int map_advise_rand(mmap_t* restrict ptr, int need)
|
||||
{
|
||||
int flags = MADV_RANDOM | (need ? MADV_WILLNEED : 0);
|
||||
if(madvise(ptr->ptr, ptr->len, flags)) {
|
||||
perror("madvise() failed");
|
||||
return 0;
|
||||
}
|
||||
return 1;
|
||||
}
|
||||
|
||||
int map_raw_fd(int fd, mmap_t* restrict ptr, const size_t sz[static 1])
|
||||
{
|
||||
if(fd < 0) return 0;
|
||||
struct stat st;
|
||||
|
||||
|
||||
if(sz && ftruncate(fd, *sz)) {
|
||||
perror("Failed to allocate");
|
||||
return 0;
|
||||
} else if(fstat(fd, &st) < 0) {
|
||||
perror("Failed to stat file");
|
||||
return 0;
|
||||
}
|
||||
|
||||
struct mmap map = { .fd = fd, .ptr = NULL, .len = sz ? *sz : st.st_size };
|
||||
|
||||
if (!_map(&map)) {
|
||||
perror("mmap() failed");
|
||||
return 0;
|
||||
}
|
||||
|
||||
*ptr = map;
|
||||
|
||||
return 1;
|
||||
|
||||
}
|
||||
|
||||
int open_and_map(const char* file, mmap_t* restrict ptr)
|
||||
{
|
||||
int fd;
|
||||
struct stat st;
|
||||
if ((fd = open(file, O_RDWR, FILEMODE)) < 0) {
|
||||
perror("Failed to open file");
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (fstat(fd, &st) < 0) {
|
||||
perror("Failed to stat file");
|
||||
close(fd);
|
||||
return 0;
|
||||
}
|
||||
|
||||
struct mmap map = { .fd = fd, .ptr = NULL, .len = st.st_size };
|
||||
|
||||
if (!_map(&map)) {
|
||||
perror("mmap() failed");
|
||||
close(fd);
|
||||
return 0;
|
||||
}
|
||||
|
||||
*ptr = map;
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
int open_and_alloc(const char* file, mmap_t* restrict ptr, size_t sz)
|
||||
{
|
||||
int fd;
|
||||
if ((fd = open(file, O_CREAT | O_RDWR, FILEMODE)) < 0) {
|
||||
perror("Failed to open file");
|
||||
return 0;
|
||||
}
|
||||
|
||||
if(ftruncate(fd, sz))
|
||||
{
|
||||
perror("Failed to allocate");
|
||||
close(fd);
|
||||
return 0;
|
||||
}
|
||||
|
||||
struct mmap map = { .fd = fd, .ptr = NULL, .len = sz };
|
||||
|
||||
if (!_map(&map)) {
|
||||
perror("mmap() failed");
|
||||
close(fd);
|
||||
return 0;
|
||||
}
|
||||
|
||||
*ptr = map;
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
|
||||
inline
|
||||
int unmap_and_close_s(mmap_t map, int flags)
|
||||
{
|
||||
register int rval=1;
|
||||
if (map.fd && msync(map.ptr, map.len, flags))
|
||||
perror("msync() failed");
|
||||
if (munmap(map.ptr, map.len) < 0) {
|
||||
perror("munmap() failed");
|
||||
rval=0;
|
||||
}
|
||||
if (close(map.fd) <0) {
|
||||
perror("Failed to close fd");
|
||||
rval=0;
|
||||
}
|
||||
|
||||
return rval;
|
||||
}
|
||||
|
||||
|
||||
int unmap_and_close(mmap_t map)
|
||||
{
|
||||
return unmap_and_close_s(map, MS_SYNC /* | MS_INVALIDATE XXX: I don't think this is needed, we son't be using the mapping anymore*/);
|
||||
}
|
||||
|
||||
__attribute__((nonnull(1)))
|
||||
int dup_map(const mmap_t *in, mmap_t* restrict ptr, const size_t *new_size, enum map_flags flags)
|
||||
{
|
||||
int fd;
|
||||
if ( (fd = dup(in->fd)) < 0 )
|
||||
{
|
||||
perror("dup() failed");
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t sz;
|
||||
if(new_size) {
|
||||
if(ftruncate(fd, sz = *new_size))
|
||||
{
|
||||
perror("Failed to set cloned map size");
|
||||
close(fd);
|
||||
return 0;
|
||||
}
|
||||
} else sz = in->len;
|
||||
|
||||
struct mmap map = { .fd = fd, .ptr = NULL, .len = sz };
|
||||
|
||||
if ((map.ptr = mmap(NULL, map.len, PROT_READ | (flags & MMF_READONLY ? 0 : PROT_WRITE), (flags & MMF_PRIVATE) ? MAP_PRIVATE : MAP_SHARED,fd, 0)) == MAP_FAILED) {
|
||||
perror("mmap() failed");
|
||||
close(fd);
|
||||
return 0;
|
||||
}
|
||||
|
||||
*ptr = map;
|
||||
return 1;
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
|
||||
#include <sys/types.h>
|
||||
#include <sys/stat.h>
|
||||
#include <sys/mman.h>
|
||||
#include <fcntl.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <panic.h>
|
||||
#include <map.h>
|
||||
|
||||
|
||||
|
||||
namespace {
|
||||
constexpr inline auto MEMFD_DEFAULT_FLAGS = MFD_CLOEXEC;
|
||||
int memfd_alloc(const char* name, size_t size = 0, int flags = MEMFD_DEFAULT_FLAGS)
|
||||
{
|
||||
int fd = memfd_create(name, flags);
|
||||
if(fd < 0) panic("memfd_create() failed");
|
||||
if(size)
|
||||
if(ftruncate(fd, size)) { close(fd); panic("ftruncate() failed"); }
|
||||
return fd;
|
||||
}
|
||||
}
|
||||
|
||||
namespace mm {
|
||||
vmap::vmap()
|
||||
: mmap(create_raw_fd(memfd_alloc(__PRETTY_FUNCTION__))) {}
|
||||
vmap::vmap(size_t sz)
|
||||
: mmap(create_raw_fd(memfd_alloc(__PRETTY_FUNCTION__, sz))) {}
|
||||
|
||||
}
|
||||
@ -0,0 +1,17 @@
|
||||
|
||||
#include <shuffle3.h>
|
||||
#include <reinterpret.hpp>
|
||||
#include <map.h>
|
||||
|
||||
|
||||
extern "C" void* map_and_then(const char* file, map_cb callback, void* user)
|
||||
{
|
||||
mm::mmap map(file);
|
||||
return callback(std::as_const(map).as_raw(), user);
|
||||
}
|
||||
|
||||
extern "C" void* map_fd_and_then(int fd, map_cb callback, void* user)
|
||||
{
|
||||
auto map = mm::mmap::map_raw_fd(fd);
|
||||
return callback(std::as_const(map).as_raw(), user);
|
||||
}
|
||||
@ -0,0 +1,12 @@
|
||||
#include <stdint.h>
|
||||
#include <stddef.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
int _can_allocate(size_t bytes)
|
||||
{
|
||||
void* shim = malloc(bytes);
|
||||
if(shim) {
|
||||
free(shim);
|
||||
return 1;
|
||||
} else return 0;
|
||||
}
|
||||
@ -0,0 +1,41 @@
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
#include <unistd.h>
|
||||
#include <stdarg.h>
|
||||
|
||||
#include <panic.h>
|
||||
|
||||
#include <colours.h>
|
||||
|
||||
#ifndef DEFAULT_PANIC_HANDLER
|
||||
#if __cpp_exceptions || 1 /* XXX: Cannot detect this here */
|
||||
# define DEFAULT_PANIC_HANDLER "_panic__start_unwind_cxx"
|
||||
#else
|
||||
# define DEFAULT_PANIC_HANDLER "_panic__start_unwind_abort"
|
||||
#endif
|
||||
#endif
|
||||
|
||||
|
||||
extern void _panic__start_unwind_cxx(void);
|
||||
static void _panic__start_unwind_abort(void*)
|
||||
__attribute__((weakref("abort"), noreturn));
|
||||
|
||||
static void _panic__start_unwind(void* payload)
|
||||
__attribute__((noreturn, weakref(DEFAULT_PANIC_HANDLER)));
|
||||
|
||||
__attribute__((noreturn, weak)) void _do_panic(struct panicinfo info, const char* fmt, ...)
|
||||
{
|
||||
va_list li;
|
||||
va_start(li, fmt);
|
||||
#define FMT_TTY BOLD(UNDL(FRED("[!]"))) " (" BOLD("%s") "->" BOLD(FRED("%s")) ":" FYEL("%d") ") " BOLD(FRED("fatal error")) ": "
|
||||
#define FMT_FIL "[!]" " (" "%s" "->" "%s" ":" "%d" ") " "fatal error" ": "
|
||||
fprintf(stderr, isatty(fileno(stderr))
|
||||
? FMT_TTY
|
||||
: FMT_FIL , info.file, info.function, info.line);
|
||||
vfprintf(stderr, fmt, li);
|
||||
fprintf(stderr, "\n");
|
||||
va_end(li);
|
||||
|
||||
_panic__start_unwind(&info);
|
||||
__builtin_unreachable();
|
||||
}
|
||||
@ -0,0 +1,47 @@
|
||||
|
||||
#include <utility>
|
||||
#include <stdexcept>
|
||||
|
||||
#include <cstdlib>
|
||||
|
||||
extern "C" {
|
||||
struct ErrorPayload {};
|
||||
}
|
||||
|
||||
struct FatalError {
|
||||
FatalError(void* payload)
|
||||
: payload(payload) {}
|
||||
FatalError(FatalError&& m)
|
||||
: payload(std::exchange(m.payload, nullptr)) {}
|
||||
|
||||
FatalError& operator=(FatalError&& m)
|
||||
{
|
||||
if(this != &m) {
|
||||
if(payload) destroy();
|
||||
payload = std::exchange(m.payload, nullptr);
|
||||
} return *this;
|
||||
}
|
||||
virtual ~FatalError() noexcept {
|
||||
destroy();
|
||||
}
|
||||
protected:
|
||||
virtual void destroy() noexcept
|
||||
{
|
||||
/*if(auto* p = dynamic_cast<ErrorPayload*>(payload))
|
||||
delete p;*/
|
||||
}
|
||||
public:
|
||||
void* payload;
|
||||
};
|
||||
|
||||
extern "C" {
|
||||
[[noreturn, gnu::weak]]
|
||||
void _panic__start_unwind_cxx(void* payload)
|
||||
{
|
||||
#if __EXCEPTIONS
|
||||
throw FatalError(payload);
|
||||
#else
|
||||
::abort();
|
||||
#endif
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,135 @@
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include <fmt/format.h>
|
||||
|
||||
#include <sys/ioctl.h>
|
||||
|
||||
#include <perc.h>
|
||||
|
||||
using fract_t = pr::Progress::fract_t;
|
||||
|
||||
namespace {
|
||||
using namespace pr;
|
||||
size_t twidth(int fd = STDIN_FILENO, size_t orr = Progress::DEFAULT_WIDTH) noexcept
|
||||
{
|
||||
struct winsize w;
|
||||
if(ioctl(fd, TIOCGWINSZ, &w) == -1) return orr;
|
||||
return size_t(w.ws_col);
|
||||
}
|
||||
|
||||
size_t def_bar_width(size_t tw) noexcept
|
||||
{
|
||||
return size_t(std::round(Progress::DEFAULT_TERM_FRACT * fract_t(tw)));
|
||||
}
|
||||
}
|
||||
|
||||
namespace pr {
|
||||
|
||||
struct Progress::_impl {
|
||||
Progress::fract_t fract; // 0..=1
|
||||
FILE* output;
|
||||
size_t width;
|
||||
std::string aux;
|
||||
};
|
||||
|
||||
size_t& Progress::width() noexcept { return inner_->width; }
|
||||
size_t Progress::width() const noexcept { return inner_->width; }
|
||||
|
||||
fract_t& Progress::fraction() noexcept { return inner_->fract; }
|
||||
fract_t Progress::fraction() const noexcept { return inner_->fract; }
|
||||
|
||||
fract_t Progress::percentage() const noexcept { return inner_->fract * 100.0l; }
|
||||
void Progress::percentage(fract_t fract) noexcept { inner_->fract = fract / 100.0l; }
|
||||
|
||||
std::string& Progress::tag() noexcept { return inner_->aux; }
|
||||
const std::string& Progress::tag() const noexcept { return inner_->aux; }
|
||||
|
||||
FILE* Progress::output() const noexcept { return inner_->output; }
|
||||
FILE*& Progress::output() noexcept { return inner_->output; }
|
||||
|
||||
void Progress::render(bool flush) const noexcept
|
||||
{
|
||||
constinit thread_local static std::vector<char> buffer{};
|
||||
|
||||
const auto& inner = *inner_;
|
||||
FILE* out = inner.output;
|
||||
auto wf = fract_t(inner.width) * inner.fract;
|
||||
auto perc = percentage();
|
||||
size_t cf = size_t(std::floor(wf));
|
||||
|
||||
|
||||
//auto print = []<typename F> (F f, auto&&... args) {
|
||||
#define print(...) fmt::format_to(std::back_inserter(buffer), __VA_ARGS__);
|
||||
//};
|
||||
|
||||
buffer.clear();
|
||||
// Render bar
|
||||
buffer.push_back('[');
|
||||
for(size_t i=0;i<inner.width;i++)
|
||||
if(i<=cf) buffer.push_back( '#' );
|
||||
else buffer.push_back( ' ' );
|
||||
buffer.push_back( ']' );
|
||||
print(": {:00.2}%", perc);
|
||||
if(inner.aux.size())
|
||||
print(" ({})", std::string_view(inner.aux));
|
||||
// Print bar
|
||||
fprintf(out, "\r%.*s", int(buffer.size() & INT_MAX), static_cast<const char*>(buffer.data()));
|
||||
// Flush output stream
|
||||
if(flush) fflush(out);
|
||||
#undef print
|
||||
}
|
||||
void Progress::spin(increment_t by, bool r, bool f) noexcept
|
||||
{
|
||||
auto& inner = *inner_;
|
||||
inner.fract=by;
|
||||
if(inner.fract > 1.0l)
|
||||
inner.fract=1.0l;
|
||||
if(r) this->render(f);
|
||||
}
|
||||
|
||||
[[gnu::nonnull(1)]]
|
||||
Progress::Progress(FILE* p) noexcept
|
||||
: inner_(std::make_unique<_impl>(0, p, def_bar_width(twidth(fileno(p))) )) {}
|
||||
|
||||
Progress::Progress(const Progress& p) noexcept
|
||||
: inner_(p.inner_ ? std::make_unique<_impl>(*p.inner_) : nullptr) {}
|
||||
|
||||
Progress::Progress(Progress&& p) noexcept
|
||||
: inner_(std::move(p.inner_)) {}
|
||||
|
||||
Progress& Progress::operator=(Progress&& p) noexcept
|
||||
{
|
||||
if(this != &p) inner_ = std::move(p.inner_);
|
||||
return *this;
|
||||
}
|
||||
|
||||
Progress& Progress::operator=(Progress const& p) noexcept
|
||||
{
|
||||
if(this != &p) inner_ = p.inner_;
|
||||
return *this;
|
||||
}
|
||||
|
||||
Progress::~Progress() {
|
||||
if(inner_)
|
||||
fputc('\n', inner_->output);
|
||||
}
|
||||
|
||||
|
||||
#ifdef SPINNER
|
||||
Spinner::~Spinner() {
|
||||
//TODO: Add output: if(cur_) fputc('\n', output);
|
||||
}
|
||||
|
||||
void Spinner::render(bool flush) {
|
||||
/*char arr[] = {
|
||||
'\r',
|
||||
cur_,
|
||||
0,
|
||||
};
|
||||
fputs(arr, output);
|
||||
if(flush) fflush(output);*/
|
||||
}
|
||||
#endif
|
||||
}
|
||||
@ -0,0 +1,22 @@
|
||||
#include <shuffle3.h>
|
||||
#include <reinterpret.hpp>
|
||||
|
||||
template<typename T>
|
||||
static inline T* bytes_to_t(std::uint8_t* ptr, std::size_t ptr_sz, std::size_t* restrict nsize)
|
||||
{
|
||||
span<uint8_t> bytes(ptr, ptr_sz);
|
||||
auto tout = bytes.reinterpret<T>();
|
||||
*nsize = tout.size();
|
||||
return tout.as_ptr();
|
||||
}
|
||||
|
||||
extern "C" {
|
||||
uint64_t* bytes_to_long(uint8_t* ptr, size_t ptr_sz, size_t* restrict nsize)
|
||||
{
|
||||
return bytes_to_t<uint64_t>(ptr, ptr_sz, nsize);
|
||||
}
|
||||
float* bytes_to_float(uint8_t* ptr, size_t ptr_sz, size_t* restrict nsize)
|
||||
{
|
||||
return bytes_to_t<float>(ptr, ptr_sz, nsize);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,165 @@
|
||||
#include <stdexcept>
|
||||
|
||||
#include <iostream>
|
||||
#include <cmath>
|
||||
#include <sys/types.h>
|
||||
|
||||
#include <rng/impl.hpp>
|
||||
|
||||
#include <error.h>
|
||||
#include <panic.h>
|
||||
|
||||
|
||||
//#define IMPORT_ONE(NS, NAME) using NAME = NS :: NAME
|
||||
|
||||
//IMPORT_ONE(std, ssize_t);
|
||||
|
||||
template<typename To, typename T, typename Unit=unsigned char>
|
||||
static inline To* offset_ptr(T* ptr, ssize_t by) noexcept
|
||||
{
|
||||
return reinterpret_cast<To*>(reinterpret_cast<Unit*>(ptr)+by);
|
||||
}
|
||||
/*
|
||||
template<typename To, typename Unit>
|
||||
offset_ptr(auto* p, std::ssize_t by) -> offset_ptr<To, decltype(p), Unit>;
|
||||
template<typename To>
|
||||
offset_ptr(auto* p, std::ssize_t by) -> offset_ptr<To, decltype(p)>;
|
||||
*/
|
||||
#ifdef FT_PT_OPERATORS
|
||||
inline operator RNG*(rng_t ptr) noexcept
|
||||
{
|
||||
if(__builtin_expect(!ptr, false)) return nullptr;
|
||||
#ifdef DEBUG
|
||||
RNG* op =
|
||||
#else
|
||||
return
|
||||
#endif
|
||||
reinterpret_cast<RNG*>(offset_ptr<RNG>(ptr, -static_cast<ssize_t>(offsetof(RNG, _held))));
|
||||
#ifdef DEBUG
|
||||
if(__builtin_expect(!op, false)) return nullptr;
|
||||
if(__builtin_expect(op->_held, false)) panic("Invalid rng_t -> RNG conversion");
|
||||
return ptr;
|
||||
#endif
|
||||
}
|
||||
|
||||
RNG::operator rng_t() const noexcept
|
||||
{
|
||||
return reinterpret_cast<rng_t>(static_cast<RNG*>(this)->_held);
|
||||
}
|
||||
#endif
|
||||
|
||||
inline unsigned char RNG::byte()
|
||||
{
|
||||
return (unsigned char)next_int(255);
|
||||
}
|
||||
|
||||
void RNG::bytes(unsigned char* ptr, std::size_t len)
|
||||
{
|
||||
for(std::size_t i=0;i<len;i++)
|
||||
{
|
||||
ptr[i] = byte();
|
||||
}
|
||||
}
|
||||
|
||||
bool RNG::chance()
|
||||
{
|
||||
return chance(.5);
|
||||
}
|
||||
|
||||
bool RNG::chance(double d)
|
||||
{
|
||||
if (d<=0) return false;
|
||||
return sample() <= d;
|
||||
}
|
||||
|
||||
std::int32_t RNG::next_int(std::int32_t min, std::int32_t max)
|
||||
{
|
||||
return (std::int32_t)floor(sample() * (double)(max-min))+min;
|
||||
}
|
||||
|
||||
std::int32_t RNG::next_int()
|
||||
{
|
||||
return (chance() ? 1 : -1) * (std::int32_t)floor(sample() * (double)INT32_MAX);
|
||||
}
|
||||
|
||||
std::int64_t RNG::next_long(std::int64_t min, std::int64_t max)
|
||||
{
|
||||
return (std::int64_t)floor(sample() * (double)(max-min))+min;
|
||||
}
|
||||
|
||||
std::int64_t RNG::next_long()
|
||||
{
|
||||
return (chance() ? 1l : -1l) * (std::int64_t)floor(sample() * (double)INT64_MAX);
|
||||
}
|
||||
|
||||
#include <rng.h>
|
||||
namespace { // C interface
|
||||
using namespace std;
|
||||
[[gnu::always_inline, gnu::gnu_inline]]
|
||||
static inline RNG* extract_ptr(rng_t ptr)
|
||||
{ return reinterpret_cast<RNG*> (ptr); }
|
||||
static inline RNG& extract_ref(rng_t rng)
|
||||
{
|
||||
return *extract_ptr(rng);
|
||||
}
|
||||
template<std::derived_from<RNG> T>
|
||||
static inline rng_t wrap_ptr(T* ptr) noexcept
|
||||
{
|
||||
if(__builtin_expect(!ptr, false)) return nullptr;
|
||||
return reinterpret_cast<rng_t>(static_cast<RNG*>(ptr));
|
||||
}
|
||||
template<typename T>
|
||||
static inline T* extract_downcast_ptr(rng_t rng)
|
||||
{
|
||||
return dynamic_cast<T*>(extract_ptr(rng));
|
||||
}
|
||||
|
||||
extern "C"
|
||||
{
|
||||
rng_t rng_new(rng_init_opt opt)
|
||||
{
|
||||
switch(opt.kind)
|
||||
{
|
||||
case RNG_KIND_FRNG: return wrap_ptr( new rng::frng(opt.init.frng.state) );
|
||||
case RNG_KIND_DRNG: return wrap_ptr( new rng::drng(opt.init.drng.state) );
|
||||
case RNG_KIND_XORNG: return wrap_ptr( new rng::xoroshiro128plus(opt.init.xorng.state) );
|
||||
//case RNG_KIND_LORENX: return static_cast<rng_t>( new rng::lorenzAttractor(opt.init.lorenz); );
|
||||
default: panic("Unknown RNG init opt: %d", opt.kind);
|
||||
__builtin_unreachable();
|
||||
}
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
void rng_free(rng_t rng)
|
||||
{
|
||||
RNG* ptr = extract_ptr(rng);
|
||||
delete ptr;
|
||||
}
|
||||
|
||||
[[gnu::nonnull(1)]]
|
||||
void rng_test_spec(rng_t rng)
|
||||
{
|
||||
auto& ref = extract_ref(rng);
|
||||
cout << "rng_test_spec (" << typeid(ref).name() << ")..." << std::flush;
|
||||
if constexpr(!is_noexcept) {
|
||||
#if EXCEPT
|
||||
try {
|
||||
rng::test_algo(std::move(ref));
|
||||
}
|
||||
catch(const std::exception& except) {
|
||||
cout << "\r\r\r\tERROR: " << (except.what() ?: typeid(except).name()) << endl;
|
||||
rng_free(rng);
|
||||
throw;
|
||||
} catch(...) {
|
||||
cout << "\r\r\r\tERROR" << endl;
|
||||
rng_free(rng);
|
||||
throw;
|
||||
}
|
||||
#endif
|
||||
} else rng::test_algo(std::move(ref));
|
||||
|
||||
cout << "\r\r\r\tOK" << endl;
|
||||
rng_free(rng);
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,21 @@
|
||||
#include <cstdlib>
|
||||
#include <ctime>
|
||||
#include <iostream>
|
||||
|
||||
#include <rng/drng.hpp>
|
||||
|
||||
namespace rng
|
||||
{
|
||||
drng drng::from_time() { return drng(time(NULL)); }
|
||||
|
||||
int drng::rand()
|
||||
{
|
||||
return rand_r(&state);
|
||||
}
|
||||
|
||||
double drng::sample()
|
||||
{
|
||||
int val = rand_r(&state);
|
||||
return (double)val / (double)RAND_MAX;
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,3 @@
|
||||
#include <rng/impl.hpp>
|
||||
#include <rng/frng.hpp>
|
||||
#include <iostream>
|
||||
@ -0,0 +1,37 @@
|
||||
#include <iostream>
|
||||
#include <rng.h>
|
||||
|
||||
namespace rng {
|
||||
void test_algo(RNG&& rng)
|
||||
{
|
||||
using namespace std;
|
||||
|
||||
for(int i=0;i<10;i++) {
|
||||
double d = rng.next_double();
|
||||
long l = rng.next_long(-10, 10);
|
||||
|
||||
std::array<bool, 10> ar;
|
||||
for(auto& i : ar) i = rng.chance();
|
||||
|
||||
cout << "\t(Sampled: " << d;
|
||||
cout << ", Long: " << l;
|
||||
cout << ", Bools: [ ";
|
||||
for(const auto& i : ar) cout << i << " ";
|
||||
cout << "])" << endl;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
extern "C" void rng_test()
|
||||
{
|
||||
using namespace std;
|
||||
|
||||
cout << "frng:" << endl;
|
||||
rng::test_algo(rng::frng(1.0, 1.2));
|
||||
|
||||
cout << "drng:" << endl;
|
||||
rng::test_algo(rng::drng(10));
|
||||
|
||||
cout << "xoroshiro128+:" << endl;
|
||||
rng::test_algo(rng::xoroshiro128plus(100ul, 200ul));
|
||||
}
|
||||
@ -0,0 +1,86 @@
|
||||
#include <rng/xoroshiro128plus.hpp>
|
||||
#include <iostream>
|
||||
|
||||
using u64 = std::uint64_t;
|
||||
|
||||
#define XO xoroshiro128plus
|
||||
|
||||
static inline constexpr u64 rotl(u64 x, int k)
|
||||
{
|
||||
return (x << k) | (x >> (64 - k));
|
||||
}
|
||||
|
||||
namespace rng
|
||||
{
|
||||
inline constexpr u64 next(XO::State& s)
|
||||
{
|
||||
u64 s0 = s[0];
|
||||
u64 s1 = s[1];
|
||||
u64 result = s0 + s1;
|
||||
|
||||
s1 ^= s0;
|
||||
s[0] = rotl(s0, 24) ^ s1 ^ (s1 << 16);
|
||||
s[1] = rotl(s1, 37);
|
||||
|
||||
return result;
|
||||
}
|
||||
inline constexpr void xo_jump(XO::State& s)
|
||||
{
|
||||
constexpr const std::uint64_t JUMP[] = { 0xdf900294d8f554a5, 0x170865df4b3201fc };
|
||||
|
||||
std::uint64_t s0 = 0;
|
||||
std::uint64_t s1 = 0;
|
||||
for(u64 i = 0; i < sizeof JUMP / sizeof *JUMP; i++)
|
||||
for(int b = 0; b < 64; b++) {
|
||||
if (JUMP[i] & UINT64_C(1) << b) {
|
||||
s0 ^= s[0];
|
||||
s1 ^= s[1];
|
||||
}
|
||||
next(s);
|
||||
}
|
||||
|
||||
s[0] = s0;
|
||||
s[1] = s1;
|
||||
}
|
||||
|
||||
inline constexpr void xo_long_jump(XO::State& s)
|
||||
{
|
||||
constexpr const uint64_t LONG_JUMP[] = { 0xd2a98b26625eee7b, 0xdddf9b1090aa7ac1 };
|
||||
|
||||
std::uint64_t s0 = 0;
|
||||
std::uint64_t s1 = 0;
|
||||
for(u64 i = 0; i < sizeof LONG_JUMP / sizeof *LONG_JUMP; i++)
|
||||
for(int b = 0; b < 64; b++) {
|
||||
if (LONG_JUMP[i] & UINT64_C(1) << b) {
|
||||
s0 ^= s[0];
|
||||
s1 ^= s[1];
|
||||
}
|
||||
next(s);
|
||||
}
|
||||
|
||||
s[0] = s0;
|
||||
s[1] = s1;
|
||||
}
|
||||
|
||||
|
||||
void XO::jump() { xo_jump(state); }
|
||||
void XO::long_jump() { xo_long_jump(state); }
|
||||
|
||||
std::uint64_t XO::next_ulong()
|
||||
{
|
||||
return next(state);
|
||||
}
|
||||
std:: int64_t XO::next_long()
|
||||
{
|
||||
const u64 v = next_ulong();
|
||||
static_assert(sizeof(v) == sizeof(decltype(next_long())));
|
||||
|
||||
return *(const std::int64_t*)&v;
|
||||
}
|
||||
|
||||
|
||||
double XO::sample()
|
||||
{
|
||||
return (next(state) & ((INT64_C(1) << 53) - 1)) * (1.00 / (INT64_C(1) << 53));
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,142 @@
|
||||
#include <tuple>
|
||||
#include <functional>
|
||||
#include <bit>
|
||||
#include <cfloat>
|
||||
|
||||
#include <shuffle3.h>
|
||||
#include <panic.h>
|
||||
#include <map.h>
|
||||
#include <reinterpret.h>
|
||||
#include <shuffle.hpp>
|
||||
|
||||
#include <work.h>
|
||||
#include <debug.h>
|
||||
|
||||
template<typename T, typename Fn>
|
||||
std::tuple<T, T> minmax_t(const span<T>& array, Fn keep)
|
||||
{
|
||||
T highest
|
||||
#ifdef DEBUG
|
||||
{}
|
||||
#endif
|
||||
;
|
||||
T lowest
|
||||
#ifdef DEBUG
|
||||
{}
|
||||
#endif
|
||||
;
|
||||
bool init=false;
|
||||
D_dprintf("minmax_t: %p (%lu)", array.as_ptr(), array.size());
|
||||
for(std::size_t i=0;i<array.size();i++)
|
||||
{
|
||||
if(!keep(array[i])) continue;
|
||||
|
||||
auto item = array[i];
|
||||
if(!init) {
|
||||
init = true;
|
||||
lowest = highest = item;
|
||||
}
|
||||
else if(item < lowest) lowest = item;
|
||||
else if(item > highest) highest = item;
|
||||
}
|
||||
//fmt::print("MMX {}, {}\n", lowest, highest);
|
||||
return {lowest, highest};
|
||||
}
|
||||
|
||||
template<typename T>
|
||||
inline std::tuple<T, T> minmax_t(const span<T>& array)
|
||||
{
|
||||
return minmax_t(array, [](T _val) { return true; });
|
||||
}
|
||||
|
||||
namespace work
|
||||
{
|
||||
/// Shuffle or unshuffle in place
|
||||
template<bool unshuffle>
|
||||
int xshuffle_ip(const char* file)
|
||||
{
|
||||
mm::mmap map(file);
|
||||
map.access(mm::Access::Random, true);
|
||||
|
||||
if constexpr(unshuffle)
|
||||
{
|
||||
auto [byte_l, byte_h] = minmax_t(map.as_span().reinterpret<std::int8_t>());
|
||||
D_dprintf("MMX res (s8): %d -- %d", byte_l, byte_h);
|
||||
rng::drng drng((std::int32_t) ((0xfffa << 16) | (byte_l<<7) | byte_h ));
|
||||
rng::unshuffle(drng, map.as_span());
|
||||
|
||||
auto [float_l, float_h] = minmax_t(map.as_span().reinterpret<float>(), [](float f) -> bool { return !( (f!=f) || f < -FLT_MAX || f > FLT_MAX); });
|
||||
D_dprintf("MMX res (f32): %f -- %f", float_l, float_h);
|
||||
rng::frng frng(float_l, float_h);
|
||||
rng::unshuffle(frng, map.as_span().reinterpret<float>());
|
||||
|
||||
auto [long_l, long_h] = minmax_t(map.as_span().reinterpret<std::int64_t>());
|
||||
D_dprintf("MMX res (u64): %ld -- %ld", long_l, long_h);
|
||||
rng::xoroshiro128plus xorng(std::bit_cast<std::uint64_t>(long_l), std::bit_cast<std::uint64_t>(long_h));
|
||||
rng::unshuffle(xorng, map.as_span().reinterpret<std::int64_t>());
|
||||
} else {
|
||||
auto [long_l, long_h] = minmax_t(map.as_span().reinterpret<std::int64_t>());
|
||||
D_dprintf("MMX res (u64): %ld -- %ld", long_l, long_h);
|
||||
rng::xoroshiro128plus xorng(std::bit_cast<std::uint64_t>(long_l), std::bit_cast<std::uint64_t>(long_h));
|
||||
rng::shuffle(xorng, map.as_span().reinterpret<std::int64_t>());
|
||||
|
||||
auto [float_l, float_h] = minmax_t(map.as_span().reinterpret<float>(), [](float f) -> bool { return !( (f!=f) || f < -FLT_MAX || f > FLT_MAX); });
|
||||
D_dprintf("MMX res (f32): %f -- %f", float_l, float_h);
|
||||
rng::frng frng(float_l, float_h);
|
||||
rng::shuffle(frng, map.as_span().reinterpret<float>());
|
||||
|
||||
auto [byte_l, byte_h] = minmax_t(map.as_span().reinterpret<std::int8_t>());
|
||||
D_dprintf("MMX res (s8): %d -- %d", byte_l, byte_h);
|
||||
rng::drng drng((std::int32_t) ((0xfffa << 16) | (byte_l<<7) | byte_h ));
|
||||
rng::shuffle(drng, map.as_span());
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/// Shuffle or unshuffle out of place
|
||||
template<bool unshuffle>
|
||||
int xshuffle_op(const char* ifile, const char* ofile, bool is_buffered)
|
||||
{
|
||||
//TODO: Use libcow's `cow_create_fd()`/`Cow::Cow(fd, size)` to try to map `ifile`
|
||||
// Then, clone it, and use the `Fake` to do the shuffling on, and write the data to `ofile`.
|
||||
// If `ifile` cannot be mapped, create a shim `Cow(size(ifile))`, `sendfile()/copy_file_range()/splice()` `ifile` into the shim memory fd, and use that (do *not* clone it to a Fake,)
|
||||
// Then, perform the shuffling on the original `Cow`, and `sendfile()/copy_file_range()/splice()` the `ifile` file memory descriptor used for the `Cow` into `ofile`, and then destroy the `Cow`, **make sure to `msync()` the Cow'd range before the copy**.
|
||||
mm::vmap imap; //{ifile};
|
||||
|
||||
if constexpr(unshuffle)
|
||||
{
|
||||
|
||||
} else {
|
||||
|
||||
}
|
||||
panic("Unimplemented");
|
||||
return 0;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
int help()
|
||||
{
|
||||
//Print help then exit
|
||||
return 1;
|
||||
}
|
||||
|
||||
extern "C" int do_work(const work_args_t args)
|
||||
{
|
||||
using A = decltype(args.op);
|
||||
switch (args.op) {
|
||||
case A::OP_SHUFFLE_IP: return work::xshuffle_ip<false >(args.data.op_shuffle_ip.file);
|
||||
case A::OP_SHUFFLE_OP: return work::xshuffle_op<false >(args.data.op_shuffle_op.ifile,
|
||||
args.data.op_shuffle_op.ofile,
|
||||
args.data.op_shuffle_op.buffered == WORK_BO_BUFFERED);
|
||||
case A::OP_UNSHUFFLE_IP: return work::xshuffle_ip<true >(args.data.op_unshuffle_ip.file);
|
||||
case A::OP_UNSHUFFLE_OP: return work::xshuffle_op<true>(args.data.op_unshuffle_op.ifile,
|
||||
args.data.op_unshuffle_op.ofile,
|
||||
args.data.op_unshuffle_op.buffered == WORK_BO_BUFFERED);
|
||||
case A::OP_HELP: return help();
|
||||
|
||||
default: panic("Unknown op %d", (int)args.op);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
@ -0,0 +1,51 @@
|
||||
#!/bin/bash
|
||||
|
||||
TFILE=/tmp/shuffle3-test-$(uuidgen)
|
||||
TFILE2=$TFILE-cmp
|
||||
|
||||
FSIZE=${FSIZE:-$(( 1024 * 1024 * 10))}
|
||||
FCNT=${FCNT:-4}
|
||||
|
||||
TRUE_SIZE=$(( FSIZE * FCNT))
|
||||
|
||||
function cleanup {
|
||||
rm -f $TFILE $TFILE2
|
||||
}
|
||||
|
||||
trap cleanup EXIT
|
||||
|
||||
stest() {
|
||||
if fcmp $TFILE $TFILE2; then
|
||||
echo ": shuffling $TRUE_SIZE bytes"
|
||||
$1 -s $TFILE
|
||||
if fcmp $TFILE $TFILE2; then
|
||||
echo "Failed: Shuffle did nothing"
|
||||
exit 1
|
||||
else
|
||||
echo ": unshuffling $TRUE_SIZE bytes"
|
||||
$1 -u $TFILE
|
||||
if fcmp $TFILE $TFILE2; then
|
||||
echo "OK"
|
||||
else
|
||||
echo "Failed: Unshuffle didn't reproduce original file"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
else
|
||||
echo "Invalid init: Files weren't identical"
|
||||
exit -1
|
||||
fi
|
||||
}
|
||||
|
||||
echo ">>> Initialising"
|
||||
dd if=/dev/urandom of=$TFILE bs=$FSIZE count=$FCNT >> /dev/null 2>&1 || exit -1
|
||||
cp $TFILE $TFILE2 || exit -1
|
||||
|
||||
for ex in "$@"; do
|
||||
if [[ -f "${ex%% *}" ]] || command -v "${ex%% *}" >/dev/null 2>&1; then
|
||||
echo ">>> Testing $ex"
|
||||
stest "$ex" || exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Passed."
|
||||
Loading…
Reference in new issue