/// Best used for cold files, or files that are not accessed much, or when running low on memory and/or fds.
///
/// Corresponds to `DataCacheState::None`
None,
/// Open the file and cache the FD as std `File`
/// This can avoid the syscall overhead of needing to open the file next time it is accessed.
///
/// # Usage
/// Best used for frequently acessed files.
///
/// Corresponds to `DataCacheState::Open`
Low,
/// Open the file, cache the FD *and* map the file in memory.
/// This can provide efficient random-access of the file without the need to preload it.
///
/// # Usage
/// Best used for frequently read large files.
///
/// Corresponds to `DataCacheState::Mapped`
High,
/// Load the whole contents of the file into memory, without keeping it open.
/// This provides the most efficient acesss of the file, as well as not contributing to the process' fd limit, but at the cost of:
/// * Loading the whole file into a memory buffer, which may be a slow operation and/or take up massive memory space
/// * Allowing a potential desync if the file is changed on disk while the cache buffer is loaded.
///
/// ## Desync
/// While `mtfse` already assumes it has exclusive access to all files in its db root, generally a file being modified by an external program will cause an error to be returned within a `mtfse` operation eventually as file hashes are updated and/or files are encrypted/decrypted or even read (in the case of a file being deleted.)
///
/// When an item is cached at this level however, any of these kinds of changes made will not be visible. The store can then be put into an invalid state without realising it.
///
/// ## Tradeoffs
/// The caching operation itself is expensive, the memory requirements is expensive, and in some cases this can even *slow* reads compared to the other levels when used on large files, as we cannot take advantage of the kernel's internal file data caching for mapped random-acesss reads and we are bound to causing CPU cache misses.
///
/// # Usage
/// Best used for frequently read small files.
///
/// Corresponds to `DataCacheState::Memory`
Extreme,
}
/// Provides immutable caching of a file in a data entry.
/// Provides immutable caching of a file in a data entry.
#[derive(Debug)]
#[derive(Debug)]
pubenumDataCacheState
pub(super)enumDataCacheState
{
{
/// There is no file cache for this item.
/// There is no file cache for this item.
None,
None,
/// The file is open, we have an fd.
/// The file is open, we have an fd.
Open(File),
Open(File),
/// The file is not open, but its whole contents have been loaded into memory.
Memory(Bytes),// load from file via `BytesMut` buffer, then call `.freeze()`
/// The file is open and memory mapped.
/// The file is open and memory mapped.
Mapped(OpenMMap),
Mapped(OpenMMap),
/// The file is not open, but its whole contents have been loaded into memory.
Memory(Bytes),// load from file via `BytesMut` buffer, then call `.freeze()`.
}
}
implDefaultforDataCacheState
implDefaultforDataCacheState
@ -34,9 +121,159 @@ impl Default for DataCacheState
implDataCacheState
implDataCacheState
{
{
/// Read from the cache at `offset` into the provided buffer, and return the number of bytes read.
///
/// # Performance
///
/// When the cache has random access, this method completes without yielding. If not, it performs async seek & read operations to fill the buffer as much as possible from the offset.
///
/// # Returns
/// If `EOF` is encountered within the read, then it is terminated early and the number of bytes successfully read is returned (and will be less than the length of the buffer), otherwise, the full buffer was filled, and the full buffer's length will be returned.
///
/// ## Errors
///
/// If this cache is not active, it will return an `io::Error` with `io::ErrorKind::NotConnected`.
/// Any other error in I/O operations is propagated.
pubasyncfnread_at(&mutself,offset: usize,into: &mut[u8])-> io::Result<usize>// this takes `&mut self` only to ensure it cannot be called on different threads at the same time, as any file operations need to be atomic.
{
ifletSome(ar)=self.random_access()
{
returnOk(slice::copy_bytes(&ar[offset..],into));
}
ifletSome(file)=self.file()
{
usetokio::{
fs,prelude::*,
};
letmutfile=fs::File::from_std(file.try_clone()?);// this is what requires we take `&mut(ex) self`.
file.seek(io::SeekFrom::Start(u64::try_from(offset).expect("Arch size integer was out of bounds of u64 (this should never happen)"))).await?;
