5253 Commits

Author SHA1 Message Date
Ingo Weinhold
46122852f1 packagefs: Add caching for the package file heap reader
* ReaderImplBase:
  - Add virtual CreateCachedHeapReader() which can create a cached
    reader based on the given heap reader.
  - Rename HeapReader() to RawHeapReader() and add HeapReader() for the
    cached heap reader.
  - Add DetachHeapReader() to allow a clients to remove the heap
    reader(s) after deleting the ReaderImplBase object.
* packagefs:
  - Add CachedDataReader class, which wraps a given
    BAbstractBufferedDataReader and provides caching for it using a
    VMCache. The implementation is based on the IOCache implementation.
  - Use CachedDataReader to wrap the heap reader. For file data that
    means they are cached twice -- in the heap reader cache and in the
    file cache -- but due to the heap reader using a VMCache as well,
    the pages will be recycled automatically anyway. For attribute data
    the cache should be very helpful, since they weren't cached at all
    before.
2013-05-25 01:12:32 +02:00
Ingo Weinhold
d59e0feb59 package: Suppress version mismatch errors where V1 is supported
* Add flags parameter to Init() of BPackageReader and friends.
* Introduce flag B_HPKG_READER_DONT_PRINT_VERSION_MISMATCH_MESSAGE and
  don't print a version mismatch error when given.
* package extract/list: Use the new flag.
2013-05-25 01:12:31 +02:00
Ingo Weinhold
64bb883062 Support booting with a packaged boot loader
* Add new package haiku_loader.hpkg and move haiku_loader there. The
  package is built without compression, so that the stage 1 boot loader
  has a chance of loading it.
* Adjust the stage 1 boot loader to load the haiku_loader package and
  relocate the boot loader code accordingly.
2013-05-25 01:12:31 +02:00
Ingo Weinhold
5261923e87 Add dvprintf() 2013-05-25 01:12:29 +02:00
Ingo Weinhold
1f633814fa hpkg format: compress the whole heap
Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.
2013-05-25 01:12:25 +02:00
Ingo Weinhold
0ee1651856 Rename BBlockBufferCache and friends to *Pool*
Also move BBlockBufferPoolNoLock into BHPKG namespace with the other
classes. Not sure why it wasn't there before.
2013-05-25 01:12:25 +02:00
Ingo Weinhold
c663ca2171 Merge branch 'master' into package-management
Conflicts:
	build/jam/HaikuImage
	src/system/kernel/arch/x86/64/arch.S
2013-05-12 00:18:19 +02:00
Ingo Weinhold
1848fdc329 util/khash: Add hash_hash_string_part() 2013-05-11 23:17:55 +02:00
Axel Dörfler
4b723e3fb8 boot loader: Almost removed use of kernel_cpp.h.
* For some reason that is beyond me, fat/CachedBlock.cpp needs this header,
  or else building the boot loader fails.
2013-05-11 19:34:22 +02:00
Ingo Weinhold
0bfcf7312a Move deskbar menu directory to ~/config/settings/deskbar/menu
Also remove the "be" symlink in ~/config/settings.
2013-05-11 09:57:58 +02:00
Ingo Weinhold
f132bdfe9f boot loader: Add 256 KB additional heap
packagefs is using quite a bit additional memory for the MIME DB
entries and with all files identified.
2013-05-09 03:17:43 +02:00
Ingo Weinhold
00090d290d boot loader: print max heap usage info before entering kernel 2013-05-09 03:14:35 +02:00
Oliver Tappe
9abd3e6f53 Fix race condition during build of some file systems.
* make sure the zlib package is available before trying to use zlib.h
2013-05-06 20:01:40 +02:00
Ingo Weinhold
7c6bff1734 Complete outsourcing of zlib
* Add optional packages Zlib and Zlib-devel.
* Simplify the build feature section for zlib and also extract the
  source package.
* Replace all remaining references to the zlib instance in the tree and
  remove it.
2013-05-06 04:37:18 +02:00
Rene Gollent
1b17f7ab00 Fix x86-64 build. 2013-05-05 16:57:51 -04:00
Ingo Weinhold
5af6169bb2 x86[-64]: Fix arch_cpu_user_strlcpy() for small buffers
The case for small buffers was broken, since scasb operates on edi/rdi,
not esi/rsi.
2013-05-05 15:46:32 -04:00
Ingo Weinhold
bcbf22a316 x86[-64]: Fix arch_cpu_user_strlcpy() for small buffers
The case for small buffers was broken, since scasb operates on edi/rdi,
not esi/rsi.
2013-05-05 18:01:12 +02:00
Ingo Weinhold
25a7b01d15 Merge branch 'master' into package-management
Additional changes:
* Add src/system/kernel/lib/zlib, which builds a kernel version of zlib,
  needed by packagefs.
* BuildFeatures: Add a build feature "gcc2" to allow for easier checks.
* Referenceable.cpp: Include <OS.h> instead of <debugger.h>. The latter
  is not needed and prevents building for the build platform.
* zlib/zutil.h: Fix gcc 2 build. We really should use the external
  package instead.

Conflicts:
	.gitignore
	build/jam/BuildSetup
	build/jam/FileRules
	build/jam/FloppyBootImage
	build/jam/HaikuImage
	build/jam/ImageRules
	build/jam/KernelRules
	build/jam/NetBootArchive
	build/jam/OptionalBuildFeatures
	build/jam/OptionalLibPackages
	build/jam/OptionalPackageDependencies
	build/jam/OptionalPackages
	build/scripts/build_haiku_image
	configure
	data/bin/installoptionalpackage
	data/system/boot/Bootscript
	headers/os/app/Message.h
	headers/os/package/PackageInfo.h
	headers/os/package/PackageInfoAttributes.h
	headers/os/package/PackageInfoSet.h
	headers/os/package/PackageRoster.h
	headers/os/package/PackageVersion.h
	headers/os/package/hpkg/PackageInfoAttributeValue.h
	headers/os/storage/FindDirectory.h
	headers/os/storage/Node.h
	headers/os/support/StringList.h
	headers/private/system/directories.h
	src/add-ons/kernel/drivers/audio/ac97/es1370/Jamfile
	src/add-ons/kernel/file_systems/packagefs/AttributeIndex.cpp
	src/add-ons/kernel/file_systems/packagefs/Jamfile
	src/add-ons/kernel/file_systems/packagefs/Package.cpp
	src/add-ons/kernel/file_systems/packagefs/Package.h
	src/add-ons/kernel/file_systems/packagefs/PackageDomain.cpp
	src/add-ons/kernel/file_systems/packagefs/PackageDomain.h
	src/add-ons/kernel/file_systems/packagefs/PackageFSRoot.cpp
	src/add-ons/kernel/file_systems/packagefs/PackageLinkDirectory.cpp
	src/add-ons/kernel/file_systems/packagefs/PackageLinkDirectory.h
	src/add-ons/kernel/file_systems/packagefs/PackageLinkSymlink.cpp
	src/add-ons/kernel/file_systems/packagefs/PackageLinkSymlink.h
	src/add-ons/kernel/file_systems/packagefs/PackageLinksDirectory.cpp
	src/add-ons/kernel/file_systems/packagefs/PackageNode.h
	src/add-ons/kernel/file_systems/packagefs/ResolvableFamily.cpp
	src/add-ons/kernel/file_systems/packagefs/Version.cpp
	src/add-ons/kernel/file_systems/packagefs/Version.h
	src/add-ons/kernel/file_systems/packagefs/Volume.cpp
	src/add-ons/kernel/file_systems/packagefs/Volume.h
	src/add-ons/kernel/file_systems/packagefs/kernel_interface.cpp
	src/add-ons/kernel/file_systems/userlandfs/shared/driver_settings.c
	src/apps/deskbar/BarApp.cpp
	src/apps/deskbar/BarMenuBar.cpp
	src/apps/deskbar/BarMenuBar.h
	src/apps/deskbar/BarView.cpp
	src/apps/deskbar/BarView.h
	src/apps/deskbar/BarWindow.cpp
	src/apps/deskbar/BarWindow.h
	src/apps/deskbar/DeskbarMenu.cpp
	src/apps/deskbar/DeskbarMenu.h
	src/apps/deskbar/DeskbarUtils.cpp
	src/apps/deskbar/DeskbarUtils.h
	src/apps/deskbar/ExpandoMenuBar.cpp
	src/apps/deskbar/ExpandoMenuBar.h
	src/apps/deskbar/TeamMenu.cpp
	src/apps/processcontroller/ProcessController.cpp
	src/apps/remotedesktop/RemoteDesktop.cpp
	src/bin/bash/config-top.h
	src/bin/finddir.c
	src/bin/package/Jamfile
	src/bin/package/command_add.cpp
	src/bin/package/command_create.cpp
	src/bin/package/command_list.cpp
	src/bin/package_repo/command_list.cpp
	src/bin/pkgman/command_refresh.cpp
	src/build/libbe/support/Jamfile
	src/build/libpackage/Jamfile
	src/build/libroot/Jamfile
	src/build/libroot/fs.cpp
	src/build/libroot/remapped_functions.h
	src/kits/locale/MutableLocaleRoster.cpp
	src/kits/opengl/GLRendererRoster.cpp
	src/kits/package/PackageInfo.cpp
	src/kits/package/PackageInfoSet.cpp
	src/kits/package/PackageRoster.cpp
	src/kits/package/PackageVersion.cpp
	src/kits/package/RepositoryCache.cpp
	src/kits/package/hpkg/PackageWriterImpl.cpp
	src/kits/package/hpkg/ReaderImplBase.cpp
	src/kits/package/hpkg/WriterImplBase.cpp
	src/kits/print/PrintTransport.cpp
	src/kits/print/Printer.cpp
	src/kits/screensaver/ScreenSaverRunner.cpp
	src/kits/support/StringList.cpp
	src/kits/tracker/ContainerWindow.cpp
	src/kits/tracker/DeskWindow.cpp
	src/kits/tracker/PoseView.cpp
	src/libs/print/libprint/Transport.cpp
	src/preferences/printers/AddPrinterDialog.cpp
	src/preferences/screensaver/ScreenSaverWindow.cpp
	src/servers/debug/DebugServer.cpp
	src/servers/input/AddOnManager.cpp
	src/servers/media_addon/MediaAddonServer.cpp
	src/system/boot/Jamfile
	src/system/boot/loader/Jamfile
	src/system/boot/loader/loader.cpp
	src/system/boot/loader/vfs.cpp
	src/system/kernel/fs/vfs.cpp
	src/system/kernel/fs/vfs_boot.cpp
	src/system/libroot/os/find_directory.cpp
	src/system/runtime_loader/runtime_loader.cpp
	src/tools/package/Jamfile
2013-05-05 15:03:26 +02:00
Michael Lotz
1a5e87cc64 haiku_loader: Reuse BPlusTree implementation of the BFS add-on.
Instead of having an almost exact, albeit read only, duplicate of the
implementation.
2013-05-02 23:26:03 +02:00
Jérôme Duval
3756e0ac38 kernel: use uint64 instead of int64 to avoid narrowing conversion warnings.
* gdb_register is used only in arch_debug.cpp
2013-04-29 21:20:35 +02:00
Jérôme Duval
311e148798 vfs: fixed two warnings.
* warnings about comparison between signed and unsigned integer expressions.
2013-04-29 20:57:34 +02:00
Michael Lotz
a064168869 Fix truncation of comparison value introduced in 848acd67.
Casting the difference of the two off_t values to size_t may truncate
the result. Doing so before the comparison will therefore break it.
Instead cast the size to off_t to get around the signed versus unsigned
integer expression comparison and then cast the result of the comparison
to size_t again. Should fix #9714.
2013-04-27 18:42:05 +02:00
Jérôme Duval
32d7bcb470 VMAnonymousCache.cpp: use off_t instead of page_num_t
* use off_t instead of page_num_t to fit in swap_hash_key struct.
* this fixes narrowing conversion and signedness warnings on GCC 4.7.
2013-04-27 14:43:20 +02:00
Jerome Duval
dc0e22d800 Revert "VMAnonymousCache.cpp: changed page_index type to page_num_t"
This reverts commit f7176b0ee50d5367762d904a943a693b0a8e3e2f. Citing Ingo:
"off_t is the correct type to use for addressing pages in a cache/file,
which page_num_t should only be used for physical pages." I'll see how to
fix the GCC 4.7 warnings differently :)
2013-04-27 13:36:49 +02:00
Jérôme Duval
f7176b0ee5 VMAnonymousCache.cpp: changed page_index type to page_num_t
* consistently use page_num_t for page numbers and off_t for offsets and sizes.
2013-04-26 21:17:39 +02:00
Jérôme Duval
5f08555a59 GCC 4.7.x doesn't allow writes in a referenced packed struct.
* error: cannot bind packed field 'image->preloaded_elf64_image::elf_header'
    to 'ELFLoader<ELF64Class>::EhdrType& {aka Elf64_Ehdr&}'
2013-04-26 21:17:36 +02:00
Jérôme Duval
848acd672c devfs: casts as size_t before comparing.
* error: comparison between signed and unsigned integer expressions.
2013-04-26 21:17:34 +02:00
Jérôme Duval
d762df1df1 GCC 4.7.x doesn't allow setting a field in a referenced packed struct.
* error message: error: cannot bind packed field
'args->kernel_args::platform_args.platform_kernel_args::apm' to 'apm_info&'
* the reason would be that the reference doesn't have alignment information anymore.
* changed the reference to const for read access, and use the long form for setting a field.
2013-04-26 21:17:31 +02:00
Ingo Weinhold
3b5cb32533 find_directory(): Actually handle the new constants 2013-04-24 15:11:16 +02:00
Ingo Weinhold
3f85cfc132 Add a few missing directory_which constants 2013-04-24 14:55:03 +02:00
Pawel Dziepak
26b2423794 runtime_loader: Always map program image at random position 2013-04-24 14:06:05 +02:00
Pawel Dziepak
902d41fc75 util: Use real_time_clock() in secure PRNG 2013-04-24 14:05:28 +02:00
Pawel Dziepak
c36a000cb4 vm: Fix off-by-one mistakes in _InsertAreaSlot 2013-04-24 14:05:27 +02:00
Pawel Dziepak
0fef11f1a8 arch: some CPUID leaves may be not available 2013-04-23 02:43:26 +02:00
Rene Gollent
c24adb2950 Rework DefaultNotificationService registration.
- Instead of implicitly registering and unregistering a service
  instance on construction/destruction, DefaultNotificationService
  now exports explicit Register()/Unregister() calls, which subclasses
  are expected to call when they're ready.

- Adjust all implementing subclasses. Resolves an issue with deadlocks
  when booting a DEBUG=1 build.
2013-04-22 18:42:21 -04:00
Ingo Weinhold
237127fbe4 Fix _user_entry_ref_to_path() in chroot
* Add "bool kernel" parameter to vfs_entry_ref_to_path(), so it can be
  specified for which I/O context the entry ref shall be translated.
* _user_entry_ref_to_path(): Use the calling team's I/O context instead
  of the kernel's. Fixes the bug that in a chroot the syscall would
  return a path for outside the chroot.
2013-04-22 18:06:28 +02:00
Pawel Dziepak
be573dcd88 arch: make sure not to set NX-bit when it is reserved 2013-04-22 17:36:36 +02:00
Pawel Dziepak
103977d0a9 arch: NX is initialized too early on non-boot CPUs 2013-04-18 00:15:57 +02:00
Pawel Dziepak
84bb91df83 arch: use PRNGs from kernel utils for initializing stack pointer 2013-04-16 22:35:10 +02:00
Pawel Dziepak
8614737f71 elf: restore correct region protection after relocation 2013-04-16 03:44:38 +02:00
Pawel Dziepak
9f3bd49737 runtime_loader: explicitly randomize rld_heap and _rld_debug_ positions 2013-04-16 02:44:47 +02:00
Pawel Dziepak
db1ca60528 runtime_loader: randomize position of runtime_loader
* make runtime_loader a dynammically linked object
 * add kernel support for loading user images that need to be relocated
 * load runtime_loader at random address
2013-04-16 02:29:05 +02:00
Pawel Dziepak
cf35dcc5bc vm: make aslr use more secure PRNG 2013-04-11 17:16:49 +02:00
Pawel Dziepak
d0a8e6ef2b vm: remove unused static function log2() 2013-04-11 12:32:32 +02:00
Pawel Dziepak
87d1bdb87c util: add secure pseudorandom number generator 2013-04-11 12:31:58 +02:00
Pawel Dziepak
69042ecd1b util: style fixes 2013-04-11 12:15:47 +02:00
Pawel Dziepak
d9fa99bb60 scheduler: let schedulers use kernel utils for random numbers 2013-04-11 04:37:23 +02:00
Pawel Dziepak
5c455f803f vm: let aslr use kernel utils for random numbers 2013-04-11 04:35:11 +02:00
Pawel Dziepak
6003243ef3 util: introduce kernel utils for pseudorandom number generation
Currently there are two generators. The fast one is the same one the scheduler
is using. The standard one is the same algorithm libroot's rand() uses. Should
there be a need for more cryptographically PRNG MD4 or MD5 might be a good
candidates.
2013-04-11 04:34:59 +02:00
Pawel Dziepak
feae2b5a00 vm: force userland to use B_RANDOMIZED_* address specifications 2013-04-09 23:25:19 +02:00