apache/tsfile

Motivation C++ performance improvements such as #885 and #934 need a reusable bounded cache. Without a production-ready cache abstraction, individual components may introduce unbounded per-object caches or duplicate cache implementations, making memory usage and ownership harder to control. Current cache mechanisms The repository currently contains several cache-related implementations: cpp/src/common/cache/lru_cache.h provides a generic std::list + associative-map LRU skeleton with configurable maximum size and elasticity. TsFileIOReader::device_node_cache_ is an active, reader-specific cache protected by a mutex. It is unbounded and backed by a shared PageArena, so individual eviction cannot reclaim the cached allocations. The Java implementation has a working LRUCache based on LinkedHashMap, which can serve as a behavioral reference but cannot be reused directly by C++. Problems with the existing C++ LRU The generic C++ LRU is not currently used as a working production cache: get

Motivation There has been community interest in Go support since #73. A previous pure-Go implementation is no longer maintained and is incompatible with the current TsFile format. Now that the C++ implementation and C wrapper have become more mature, Go support can be provided without maintaining another independent implementation of the TsFile format: Go API → cgo → C API → C++ TsFile Core This would allow Go applications to read and write current TsFile files while sharing the format compatibility, bug fixes, and performance improvements of the C++ module. Proposed approach Add a Go module that: exposes idiomatic Go Reader, Writer, Schema, Tablet, and ResultSet APIs; keeps all cgo and unsafe code inside an internal native package; treats C/C++ objects as opaque handles and does not expose their internal representations; provides explicit Close methods with clearly defined memory ownership; maps C API error codes to Go errors; uses batch-oriented, columnar APIs as the primary r

Motivation The Java implementation can reopen a normally closed (complete) TsFile for appending through: RestorableTsFileIOWriter.getWriterForAppendingDataOnCompletedTsFile(File) This method detects a complete file, locates the start of FileMetadata from the footer, truncates the separator marker and all tail metadata, recovers the existing schema/chunk metadata, and then allows new data to be appended. Closing the writer generates a new metadata section and tail magic. The C++ implementation currently supports recovery and continued writing only when the file is incomplete or its tail is damaged. When the tail magic is valid, RestorableTsFileIOWriter::self_check() treats the file as complete, sets can_write_ = false, and closes the write handle. As a result, users of the C++ API and the C wrapper cannot append to a TsFile that was closed normally, even though the equivalent workflow is available in Java. Expected behavior Provide a supported C++ API equivalent to the Java completed-fi

Motivation The C++ TsFile reader currently accesses local files through storage::ReadFile. On POSIX platforms, ReadFile::read() uses pread to read data at a specified offset. The C API and Python reader ultimately share this read path. This proposal introduces mmap as an optional local-file read backend while retaining the existing pread implementation. Proposed behavior Support the following modes: AUTO: prefer mmap for supported local files and fall back to pread if mapping is unavailable or fails. MMAP: explicitly use memory-mapped I/O and return a clear error if it cannot be used. PREAD: preserve the current behavior. AUTO could be the default mode. The configuration should follow the existing C++ configuration style, for example: common::set_file_read_backend(FileReadBackend::AUTO); ReadFile can remain the common entry point and dispatch reads to the selected backend, avoiding changes to the existing metadata, chunk, and page read paths. Equivalent configuration should also be e

Currently, TsFile uses a fixed encoding algorithm for each data type. For example, floating-point data may use Gorilla encoding. However, data characteristics vary across datasets, and a fixed encoding algorithm may not always perform well. In some cases, it may even increase the encoded data size. We propose introducing an adaptive encoding mechanism that dynamically selects a more suitable encoding algorithm based on the actual data characteristics or encoding effectiveness. Falling back to PLAIN when the current encoding is unsuitable can be one adaptive option, but it should not be the only one. The selection strategy, evaluation criteria, and switching granularity can be discussed separately.