On my development VM, there were over 300,000 calls to malloc()
from EntryAttributeHandler::HandleAttribute() alone, which had
the most out of any AttributeHandler, but the others were still
significant (over another 10,000 at least.) On systems with more
packages and more attributes, there would be of course more calls
to malloc().
Since the Handlers are allocated and freed in a "stack"-like
configuration, we can use a simple "bump" allocation strategy
with the AttributeHandlerContext to avoid calling malloc() at all.
In my testing, the most memory that was used appeared to be around
2 KB or so (and the smallest was 216 bytes), so a single slab
should suffice for this.
AttributeHandlerContext seems to be created/destroyed around 530 times
during the boot process on my test machine; allocating and freeing the
allocator's slab page that many times should be negligible (allocations
that large still go through the block allocator.)
Performance-wise, the total time we spend with AttributeHandlerContext
objects "alive" goes from around ~172ms to ~156ms. So, not as much an
improvement as one might hope, but that just goes to show that our
kernel malloc() is pretty efficient. And this change will also keep
short-lived objects off the heap during a period when we are allocating
many long-lived objects, anyway.
Change-Id: I810888434aad788511f2af30143335009b34ee78
Reviewed-on: https://review.haiku-os.org/c/haiku/+/8230
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: waddlesplash <waddlesplash@gmail.com>
The kernel heap only uses object caches for objects up to size 8192.
Larger allocations have to go through the raw allocator. That can
get pretty expensive.
Adding instrumentation around the malloc/free calls in this function
showed that on my machine, some 596ms during boot were spent on
*malloc/free alone*, all else aside. After this change, we are at
around 110ms, or a >5x improvement. Running an fgrep -R on /system/
after boot increased the cumulative time in memory functions to over
5 seconds, while after this change it is "only" 1170ms.
Honestly, it seems like the object depots should be able to be faster
than that, even if this function is called thousands of times. But that
is a problem for a different investigation.
It would be even faster for every consumer of this data in
packagefs just allocated one set of buffers up front, or at least
for a single "read session", but plumbing that all the way
through the myriad abstractions of the Package Kit will
not be easy, and is left for another time, as well.
* It's safe to assume that if the file is shorter than
the provided header, things will go poorly.
* Avoids a random vauge ReadBuffer error.
* This doesn't fix #15230, but makes the issue clearer.
Change-Id: I3471e6de384a0c9be94049ad891c01be980f7846
Reviewed-on: https://review.haiku-os.org/c/1679
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>
Replace 'class' with 'struct', since they are already defined as 'struct'
in PackageInfoAttributeValue.h
Pointed by clang [-Wmismatched-tags]
Change-Id: I094d32c3444fe4299a3afe0872ade296f92debf9
Reviewed-on: https://review.haiku-os.org/471
Reviewed-by: waddlesplash <waddlesplash@gmail.com>
Besides that this is a nicer interface, it allows us to get a the HPKG
header as a side effect of initializing the reader, thus preventing
seeking backward in the file. This makes "package recompress - <file>"
work.
Until now we always declared in the HPKG header that the package file is
zlib compressed. For uncompressed files we would just store all
individual chunks uncompressed. Now we handle completely uncompressed
files slightly differently: We don't write the redundant chunk size
table anymore. The size savings are minor, but it makes the uncompressed
format read-streamable which may be handy.
* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.
* FDDataWriter and ZlibDataWriter weren't used anymore.
* AbstractDataWriter was implemented only by PackageFileHeapWriter,
which was only used by WriterImplBase.
* Add a PackageFileHeapWriter::AddDataThrows() which has semantics
equivalent to the previously inherited WriteDataThrows().
* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.
When removing a string attribute, decrement the referenced string's
usage count in the string cache. This fixes the potentially incorrect
usage counts in update mode. Not a serious problem, but it could lead
to only singly (or no longer) used strings to be written to the string
subsection instead of encoding them inline and thus to slightly greater
file sizes.
* Add NotifyDone() to all repository-attribute handlers and invoke that
to notify any listeners.
* Unify deletion to a single implementation of Delete() in the base
class. Before, the root handler for a repository didn't do that, but
just triggered the notification.
* Introduce new package attribute B_HPKG_ATTRIBUTE_ID_PACKAGE (valid
only in a repository file) to group the attributes belonging to a
package.
* BRepositoryContentHandler:
- No longer derive from BPackageContentHandler.
- Add hooks HandlePackage() and HandlePackageDone() that bracket the
attributes for a package. This is more explicit and robust than
handlers having to guess when one package ended and the next began.
* BRepositoryCache: Make use of BPackageInfoContentHandler. No need to
duplicate the code for reading a package info from package info
attributes.
* For all identifiers: Rename global settings file to global writable
file. We want to use the respective attribute also for other writable
files, not only settings files.
* User settings file/global writable file info/attribute: Add
isDirectory property/child attribute. This allows declaring global/
user settings directories associated with the package.
... the one expected for the respective attribute. Before it was
possible that e.g. a uint was read and then interpreted as a
const char*, if a string was expected for that attribute.
Global and user settings files can be declared. For global ones an
update policy can be specified. If not specified, the settings file is
not included in the package, but created by the program (or user) later.
If an update type is specified, it defines what to do with the settings
file when updating the package to a newer version.
User settings files are never included in the package; they are always
created by the program or the user. If the package contains a template/
default settings file, it can be declared, but for informative purposes
only.
* Add minor_version to hpkg_header and hpkg_repo_header and make
heap_compression uint16.
* If the minor version of a package/repository file is greater than the
current one unknown attributes are ignored without error. This allows
introducing new harmless attributes without making the resulting files
unreadable for older package kit versions.
* ReaderImplBase:
- Add virtual CreateCachedHeapReader() which can create a cached
reader based on the given heap reader.
- Rename HeapReader() to RawHeapReader() and add HeapReader() for the
cached heap reader.
- Add DetachHeapReader() to allow a clients to remove the heap
reader(s) after deleting the ReaderImplBase object.
* packagefs:
- Add CachedDataReader class, which wraps a given
BAbstractBufferedDataReader and provides caching for it using a
VMCache. The implementation is based on the IOCache implementation.
- Use CachedDataReader to wrap the heap reader. For file data that
means they are cached twice -- in the heap reader cache and in the
file cache -- but due to the heap reader using a VMCache as well,
the pages will be recycled automatically anyway. For attribute data
the cache should be very helpful, since they weren't cached at all
before.
* Add flags parameter to Init() of BPackageReader and friends.
* Introduce flag B_HPKG_READER_DONT_PRINT_VERSION_MISMATCH_MESSAGE and
don't print a version mismatch error when given.
* package extract/list: Use the new flag.
* Pull _UnwriteLastPartialChunk() out of Reinit() for reuse.
* _UnwriteLastPartialChunk(): fPendingDataSize wasn't set.
* _PushChunks(): Some simplifications for clarity.
* ChunkBuffer/RemoveDataRanges(): Use data reading and decompression
methods provided by our base class instead of duplicating the
implementation.
* RemoveDataRanges():
- _FlushPendingData() before starting, so we don't ignore the pending
data and _UnwriteLastPartialChunk() when done, so a partial chunk
is read back into the pending data buffer.
- fUncompressedHeapSize wasn't reset before the main processing loop,
thus resulting in an erroneous size later on.
* Introduce BPackageWriterParameters which comprises all parameters
for package creation, currently flags and compression level. Such an
object can be passed to BPackageWriter::Init() and is passed on to
PackageWriterImpl and WriterImplBase.
* PackageFileHeapWriter: Add compressionLevel property and pass the
value on to ZlibCompressor.
* package add/create: Add options -0 ... -9 to specify the compression
level to be used.
Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).
The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.
Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.
It uses sub-namespace BPackage::BHPKG::V1. Unlike the one for the
current format version, the V1 version of BPackageInfoContentHandler
lives in BHPKG(::V1) sub-namespace and is private.
* Use enums/constants/functions instead of preprocessor macros.
* Missing include in PackageInfoAttributeValue.h.
* PackageReaderImpl::Init(): Check version before header size and
return B_MISMATCHED_VALUES instead of B_BAD_DATA, if the version
doesn't match. This allows callers to determine the condition and
try a reader for a different version. A more flexible interface for
that case would be nice, but since we want to support the old package
version only temporarily, the current solution should be good enough.
The attribute is intended for simplifying package building. The
package's install path will be used for the package's .self package
symlink, allowing installation to a temporary directory when building
the package.
