summary refs log tree commit diff
path: root/qcow
Commit message (Collapse)AuthorAge
* sys_util: add write_zeroes_all() functionDaniel Verkamp2019-09-25
| | | | | | | | | | | | | | | | | | In the same spirit as write_all() for the standard io::Write::write() function, add a write_zeroes_all() function with a default implementation that calls write_zeroes() in a loop until the requested length is met. This will allow write_zeroes implementations that don't necessarily fulfill the entire requested length. BUG=None TEST=cargo test -p sys_util write_zeroes Change-Id: I0fc3a4b3fe8904946e253ab8a2687555b12657be Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1811466 Reviewed-by: Zach Reizner <zachr@chromium.org> Reviewed-by: Cody Schuffelen <schuffelen@google.com> Tested-by: kokoro <noreply+kokoro@google.com>
* use `SharedMemory::{named, anon}` to replace `::new`Zach Reizner2019-09-11
| | | | | | | | | | | | | | | | The new constructors are shorter and omit the bare `None` in the `anon` call sites which gave no clues to the reader what the effect of that `None` was. This should improve readability. TEST=./build_test BUG=None Change-Id: I2e34e7df9a4ccc5da50edf4e963a6a42e3d84b22 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1797188 Reviewed-by: Daniel Verkamp <dverkamp@chromium.org> Commit-Queue: Zach Reizner <zachr@chromium.org> Tested-by: Zach Reizner <zachr@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* Extract disk creation logic out of qcow and src.Cody Schuffelen2019-08-28
| | | | | | | | | Bug: b/133432409 Change-Id: Iba25d5f6bb5f60619bb2f5a3d72ddfd3a81650b4 Signed-off-by: Cody Schuffelen <schuffelen@google.com> Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1691460 Reviewed-by: Daniel Verkamp <dverkamp@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* qcow: replace byteorder with {to,from}_be_bytes()Daniel Verkamp2019-08-26
| | | | | | | | | | | | | | | | Use the standard byte order conversion functions rather than the byteorder crate. BUG=None TEST=./build_test TEST=cargo build -p qcow_utils TEST=cargo test -p qcow Change-Id: I9ff7368cc54c539db1996f81d4220cabf7e6e301 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1761151 Reviewed-by: Zach Reizner <zachr@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* qcow: bounds check the refcount table offset and sizeDylan Reid2019-07-31
| | | | | | | | | | | | | | | | | If the header puts the refcount table outside the file size or if it specifies a table much larger than needed, fail to open the file. These might not be hard qcow errors, but they are situations that crosvm will never encounter. BUG=986061 TEST=fuzzer with new test cases completes in less than 5 seconds. Signed-off-by: Dylan Reid <dgreid@chromium.org> Change-Id: If048c96f6255ca81740e20f3f4eb7669467dbb7b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1716365 Reviewed-by: Daniel Verkamp <dverkamp@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* qcow: Add a zero_cluster method to raw fileDylan Reid2019-07-19
| | | | | | | | | | | Zeroing a cluster will be done from more than one place in qcow.rs soon, add a helper to reduce duplication. Change-Id: Idb40539f8e4ed2338fc84c0d53b37c913f2d90fe Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1697122 Reviewed-by: Daniel Verkamp <dverkamp@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* qcow: limit the size of a qcow fileDylan Reid2019-07-09
| | | | | | | | | | | | | | There are many corner cases when handling sizes that approach u64::max. Limit the files to 16TB. BUG=979458 TEST=Added unittest to check large disks fail Signed-off-by: Dylan Reid <dgreid@chromium.org> Change-Id: I93a87c17267ae69102f8d46ced9dbea8c686d093 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1679892 Reviewed-by: Daniel Verkamp <dverkamp@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* qcow: Avoid overflow when taking ceiling of divisionDylan Reid2019-07-09
| | | | | | | | | | | | | | | The extra % operation will be slower, but none of these divisions are in hot paths. They are only used during setup. Many of these operations take untrusted input from the disk file, so need to be hardened. BUG=979458 TEST=unit tests still pass Signed-off-by: Dylan Reid <dgreid@chromium.org> Change-Id: I0e93c73b345faf643da53ea41bde3349d756bdc7 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1679891 Reviewed-by: Daniel Verkamp <dverkamp@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* qcow: disallow crazy l1 table sizesDylan Reid2019-06-21
| | | | | | | | | | | | | | | Before this change, a corrupt or malicious qcow file could cause crosvm to allocate absurd amounts of memory. The fuzzer found this case, limit the L1 table size so it can't cause issues. BUG=chromium:974123 TEST=run fuzzer locally, add unit test Change-Id: Ieb6db6c87f71df726b3cc9a98404581fe32fb1ce Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1660890 Reviewed-by: Daniel Verkamp <dverkamp@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* qcow: Fix invalid_cluster_bits testDylan Reid2019-06-21
| | | | | | | | | | | | | | | | Start with a valid header so the invalid cluster bits field is tested in isolation. Before this change the test would pass even if the cluster bits check was removed from the code because the header was invalid for other reasons. BUG=none TEST=this is a test Change-Id: I5c09417ae3f974522652a50cb0fdc5dc0e10dd44 Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1660889 Reviewed-by: Daniel Verkamp <dverkamp@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* qcow: Limit file setups that consume excessive RAMDylan Reid2019-06-12
| | | | | | | | | | | | | | | | | | | | | qcow currently makes assumptions about being able to fit the L1 and refcount tables in memory. This isn't necessarily true for very large files. The new limits to the size of in-memory buffers still allow 1TB files with default cluster sizes. Because there aren't any 1 TB chromebooks available yet, limit disks to that size. This works around issues found by clusterfuzz related to large files built with small clusters. BUG=972165,972175,972160,972172 TEST=fuzzer locally + new unit tests Change-Id: I15d5d8e3e61213780ff6aea5759b155c63d49ea3 Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1651460 Reviewed-by: Zach Reizner <zachr@chromium.org> Reviewed-by: Daniel Verkamp <dverkamp@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* qcow: Calculate the max refcounts as a u64Dylan Reid2019-06-12
| | | | | | | | | | | | u32's get multiplied together and can overflow. A usize was being returned, make everything a u64 to make sure it fits. Change-Id: I87071d294f4e62247c9ae72244db059a7b528b62 Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1651459 Reviewed-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-by: Zach Reizner <zachr@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* qcow: better limits on cluster sizeDylan Reid2019-06-12
| | | | | | | | | | | | | | | Add a lower limit because cases such as eight byte clusters aren't practical and aren't worth handling, tracking a cluster costs 16 bytes. Also put an upper limit on the cluster size, choose 21 bits to match qemu. Change-Id: Ifcab081d0e630b5d26b0eafa552bd7c695821686 Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1651458 Reviewed-by: Zach Reizner <zachr@chromium.org> Reviewed-by: Daniel Verkamp <dverkamp@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com>
* eliminate mut from non-mut referencesZach Reizner2019-06-04
| | | | | | | | | | | | | | | | | | | | | | | This manifested itself in a couple places that were turning shared memory buffers into slices for the purposes of passing these slices to `Read` and `Write` trait methods. However, this required the removal of the methods that took `Read` and `Write` instances. This was a convenient interface but impossible to implement safely because making slices from raw pointers without enforcing safety guarantees causes undefined behaviour in Rust. It turns out lots of code in crosvm was using these interfaces indirectly, which explains why this CL touches so much. TEST=crosvm run BUG=chromium:938767 Change-Id: I4ff40c98da6ed08a4a42f4c31f0717f81b1c5863 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1636685 Reviewed-by: Zach Reizner <zachr@chromium.org> Tested-by: Zach Reizner <zachr@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com> Commit-Queue: Zach Reizner <zachr@chromium.org>
* clippy: Iterate without calling .iter()David Tolnay2019-04-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | See: https://rust-lang.github.io/rust-clippy/master/index.html#explicit_iter_loop https://rust-lang.github.io/rust-clippy/master/index.html#explicit_into_iter_loop Before: for element in slice.iter() {...} After: for element in slice {...} TEST=grep -r '\.iter() {' TEST=grep -r '\.iter_mut() {' TEST=grep -r '\.into_iter() {' TEST=cargo check --all-features TEST=local kokoro Change-Id: I27f0df7cfa1064b2c8b162cba263513926a433a9 Reviewed-on: https://chromium-review.googlesource.com/1568525 Commit-Ready: David Tolnay <dtolnay@chromium.org> Tested-by: David Tolnay <dtolnay@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com> Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
* clippy: Resolve single_matchDavid Tolnay2019-04-17
| | | | | | | | | | | TEST=bin/clippy Change-Id: Iea0d3539b3ab587a2d97f676e1d9c7a239504308 Reviewed-on: https://chromium-review.googlesource.com/1566748 Commit-Ready: David Tolnay <dtolnay@chromium.org> Tested-by: David Tolnay <dtolnay@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com> Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
* edition: Remove extern crate linesDavid Tolnay2019-04-15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In Rust 2018 edition, `extern crate` is no longer required for importing from other crates. Instead of writing: extern crate dep; use dep::Thing; we write: use dep::Thing; In this approach, macros are imported individually from the declaring crate rather than through #[macro_use]. Before: #[macro_use] extern crate sys_util; After: use sys_util::{debug, error}; The only place that `extern crate` continues to be required is in importing the compiler's proc_macro API into a procedural macro crate. This will hopefully be fixed in a future Rust release. extern crate proc_macro; TEST=cargo check TEST=cargo check --all-features TEST=cargo check --target aarch64-unknown-linux-gnu TEST=local kokoro Change-Id: I0b43768c0d81f2a250b1959fb97ba35cbac56293 Reviewed-on: https://chromium-review.googlesource.com/1565302 Commit-Ready: David Tolnay <dtolnay@chromium.org> Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: David Tolnay <dtolnay@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com> Reviewed-by: David Tolnay <dtolnay@chromium.org>
* edition: Fill in macro importsDavid Tolnay2019-04-15
| | | | | | | | | | | | | | | | | | | | Macros were previously imported through `#[macro_use] extern crate`, which is basically a glob import of all macros from the crate. As of 2018 edition of Rust, `extern crate` is no longer required and macros are imported individually like any other item from a dependency. This CL fills in all the appropriate macro imports that will allow us to remove our use of `extern crate` in a subsequent CL. TEST=cargo check --all-features --tests TEST=kokoro Change-Id: If2ec08b06b743abf5f62677c6a9927c3d5d90a54 Reviewed-on: https://chromium-review.googlesource.com/1565546 Commit-Ready: David Tolnay <dtolnay@chromium.org> Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: David Tolnay <dtolnay@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com> Reviewed-by: David Tolnay <dtolnay@chromium.org>
* lints: Enforce sorted order for enum variantsDavid Tolnay2019-04-13
| | | | | | | | | | | | | | | | | | | | | | | | | To avoid wasting time re-sorting these things (CL:1492612). https://docs.rs/remain Disclaimer: I wrote the macro. This CL adds #[sorted] attributes to those Error enums that seemed to have made some effort to be in sorted order. TEST=cargo check TEST=cargo check --all-features TEST=cargo check --target aarch64-unknown-linux-gnu TEST=emerge-nami crosvm TEST=local kokoro CQ-DEPEND=CL:1524247 Change-Id: I89685ced05e2f149fa189ca509bc14c70aebb531 Reviewed-on: https://chromium-review.googlesource.com/1515998 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: David Tolnay <dtolnay@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com> Reviewed-by: David Tolnay <dtolnay@chromium.org>
* edition: Update qcow and qcow_util to 2018 editionDavid Tolnay2019-04-08
| | | | | | | | | | | | | | | | | | Separated out of CL:1513058 to make it possible to land parts individually while the affected crate has no other significant CLs pending. This avoids repeatedly introducing non-textual conflicts with new code that adds `use` statements. TEST=cargo check TEST=cargo check --all-features TEST=cargo check --target aarch64-unknown-linux-gnu Change-Id: Ief3fb967df340a99b8263ac185207e30e096105a Reviewed-on: https://chromium-review.googlesource.com/1519704 Commit-Ready: David Tolnay <dtolnay@chromium.org> Tested-by: David Tolnay <dtolnay@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com> Reviewed-by: David Tolnay <dtolnay@chromium.org>
* edition: Update absolute paths to 2018 styleDavid Tolnay2019-03-13
| | | | | | | | | | | | | | | | | | | | | | | | | | This is an easy step toward adopting 2018 edition eventually, and will make any future CL that sets `edition = "2018"` this much smaller. The module system changes in Rust 2018 are described here: https://doc.rust-lang.org/edition-guide/rust-2018/module-system/path-clarity.html Generated by running: cargo fix --edition --all in each workspace, followed by bin/fmt. TEST=cargo check TEST=cargo check --all-features TEST=cargo check --target aarch64-unknown-linux-gnu Change-Id: I000ab5e69d69aa222c272fae899464bbaf65f6d8 Reviewed-on: https://chromium-review.googlesource.com/1513054 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: David Tolnay <dtolnay@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com> Reviewed-by: David Tolnay <dtolnay@chromium.org>
* error: Print errors using Display implDavid Tolnay2019-02-20
| | | | | | | | | | | | | | | | | | | | | | | | | I have been running into Debug-printed error messages too often and needing to look up in the source code each level of nested errors to find out from the comment on the error variant what the short name of the variant means in human terms. Worse, many errors (like the one shown below) already had error strings written but were being printed from the calling code in the less helpful Debug representation anyway. Before: [ERROR:src/main.rs:705] The architecture failed to build the vm: NoVarEmpty After: [ERROR:src/main.rs:705] The architecture failed to build the vm: /var/empty doesn't exist, can't jail devices. TEST=cargo check --all-features TEST=FEATURES=test emerge-amd64-generic crosvm Change-Id: I77122c7d6861b2d610de2fff718896918ab21e10 Reviewed-on: https://chromium-review.googlesource.com/1469225 Commit-Ready: David Tolnay <dtolnay@chromium.org> Tested-by: David Tolnay <dtolnay@chromium.org> Tested-by: kokoro <noreply+kokoro@google.com> Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
* qcow: add no-op set_len() for QcowFileDaniel Verkamp2019-01-05
| | | | | | | | | | | | | | We won't support resizing a QcowFile for now, since it would require resizing refcount and L1 tables. BUG=chromium:858815 TEST=build_test Change-Id: Ia108e07be8b06b6fbe81838831a8ecbffdfb341c Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1394149 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Dylan Reid <dgreid@chromium.org>
* toolchain: Update to Rust 1.31.0David Tolnay2018-12-13
| | | | | | | | | | | | | | | | | | We updated the production toolchain from 1.30 to 1.31 in CL:1366446. This CL does the same upgrade for the local developer toolchain and Kokoro. The relevant changes are in rust-toolchain and kokoro/Dockerfile. The rest are from rustfmt. TEST=cargo fmt --all -- --check TEST=as described in kokoro/README.md Change-Id: I3b4913f3e237baa36c664b4953be360c09efffd4 Reviewed-on: https://chromium-review.googlesource.com/1374376 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: David Tolnay <dtolnay@chromium.org> Reviewed-by: Zach Reizner <zachr@chromium.org>
* qcow: scan for free clusters at startupDaniel Verkamp2018-12-04
| | | | | | | | | | | | | | | | | | | | During runtime, we track unreferenced clusters (via unref_clusters and avail_clusters) and reuse them before extending the disk image. However, across boots, we did not previously recover the list of unreferenced clusters, so the disk file could grow beyond the range that the reference table count represent. This patch adds a boot-time scan for all unreferenced clusters so that they get reused. BUG=chromium:899273 TEST=Boot with qcow2 image, fill the disk with dd, delete the dd'd file, refill with dd, and so on, repeatedly. Ensure that the disk image does not grow beyond the expected max size and that no clusters beyond the size of the refcount table are used. Change-Id: Idd21b08bb4c55b8244e7ecaccafc4ccc46b7b17a Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1327822 Reviewed-by: Dylan Reid <dgreid@chromium.org>
* lint: Resolve the easier clippy lintsDavid Tolnay2018-12-03
| | | | | | | | | | | | | | | | | | | | Hopefully the changes are self-explanatory and uncontroversial. This eliminates much of the noise from `cargo clippy` and, for my purposes, gives me a reasonable way to use it as a tool when writing and reviewing code. Here is the Clippy invocation I was using: cargo +nightly clippy -- -W clippy::correctness -A renamed_and_removed_lints -Aclippy::{blacklisted_name,borrowed_box,cast_lossless,cast_ptr_alignment,enum_variant_names,identity_op,if_same_then_else,mut_from_ref,needless_pass_by_value,new_without_default,new_without_default_derive,or_fun_call,ptr_arg,should_implement_trait,single_match,too_many_arguments,trivially_copy_pass_by_ref,unreadable_literal,unsafe_vector_initialization,useless_transmute} TEST=cargo check --features wl-dmabuf,gpu,usb-emulation TEST=boot linux Change-Id: I55eb1b4a72beb2f762480e3333a921909314a0a2 Reviewed-on: https://chromium-review.googlesource.com/1356911 Commit-Ready: David Tolnay <dtolnay@chromium.org> Tested-by: David Tolnay <dtolnay@chromium.org> Reviewed-by: Dylan Reid <dgreid@chromium.org>
* qcow: add support for rebuilding refcountsDaniel Verkamp2018-12-01
| | | | | | | | | | | | | | | This adds the ability to regenerate the reference counts by walking all of the L1/L2 tables and headers to find all reachable clusters. This is necessary for the next patch, which will use the reference count tables to find unused clusters to reuse. BUG=chromium:899273 TEST=cargo test -p cqow Change-Id: I93dd00d381d8d33010fddfc10aa18ca32586e1f4 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1327821 Reviewed-by: Dylan Reid <dgreid@chromium.org>
* qcow: calculate refcount table size correctlyDaniel Verkamp2018-11-16
| | | | | | | | | | | | | | | | The refcount table needs to include not only the data clusters and reftable clusters but also the L1 and L2 tables and main qcow2 header. Also add sanity checking to prevent allocating a cluster that cannot be indexed with the current reference count table size. BUG=chromium:899273 TEST=cargo test -p qcow Change-Id: I9da4515db3dccbabdeee4f60dc392b5b42d62cb2 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1308833 Reviewed-by: Dylan Reid <dgreid@chromium.org>
* sys_util: add trait to fsync File and QcowFileDaniel Verkamp2018-10-26
| | | | | | | | | | | | | | | | | | | | | | | | | File exposes sync_all() and sync_data() functions, which map to fsync() and fdatasync(), but these functions are not in a trait (they are just implemented directly on File), so they can't be implemented and used in a generic way for QcowFile. Add a new trait, FileSync, that exposes a fsync() function that may be used in the virtio block model. Previously, we were translating a block flush request into a call to File's flush() function, but this just flushes internal Rust library buffers to the file descriptor; it didn't actually result in a fsync() call. Using the new trait, we can cause an actual fsync() to occur for raw files, as intended. QcowFile was already safe, since its flush() function actually calls sync_all() under the hood. BUG=None TEST=sync with raw disk and verify fsync() in strace output Change-Id: I9bee2c0d2df3747aac1e7d9ec7d9b46a7862dc48 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1297839 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Zach Reizner <zachr@chromium.org>
* main: autodetect disk image typeDaniel Verkamp2018-10-19
| | | | | | | | | | | | | | | Make --disk and --rwdisk automatically distinguish between qcow2 and raw disk images. --qcow and --rwqcow are kept as aliases for compatibility. BUG=chromium:893380 TEST=Boot crosvm with both raw and qcow2 disk images. Change-Id: I5b572626ca5ab894c78454f59355f27d552cbf7d Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1275185 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Stephen Barber <smbarber@chromium.org> Reviewed-by: Zach Reizner <zachr@chromium.org>
* devices: virtio: block: ignore Discard failuresDaniel Verkamp2018-10-19
| | | | | | | | | | | | | | | | | | | | | Our branch of the 3.18 kernel has FALLOC_FL_PUNCH_HOLE disabled for the ext4 filesystem, which means that systems running that kernel always take the fallback path of writing buffers full of zeroes. This is not necessary for the Discard command, since it is just a hint and is not required to actually zero the blocks. Split the WriteZeroes trait up into a new PunchHole trait, which corresponds to fallocate() with FALLOC_FL_PUNCH_HOLE, and use the new trait to implement the virtio block Discard command. BUG=chromium:896314 TEST=`mkfs.btrfs /dev/vdb` and verify the desired fallocate() is used and no write() calls are issued when inducing a failure Change-Id: I67fd9a132758d8d766531ccca8358c7fe67b0460 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1286224 Reviewed-by: Dylan Reid <dgreid@chromium.org>
* qcow: add convert API and export it in qcow_utilsDaniel Verkamp2018-10-18
| | | | | | | | | | | | | | | | | | | | | This will be used in vm_concierge's ExportDiskImage function in order to allow a minimal qcow2 image to be written on the fly (containing only the required clusters in a tightly-packed image file). It also allows future flexibility to change the underlying disk image file format while still exporting qcow2 images (e.g. via `vmc export`). For testing, add a qcow_img `convert` command, which can convert between raw and qcow2 as both source and destination. BUG=None TEST=Use qcow_img to convert a raw image to qcow2 and back and verify its contents are the same as the original. Change-Id: I74167aca9a9c857d892e24adf5ee17afc0f6e6b5 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1272060 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Dylan Reid <dgreid@chromium.org>
* sys_util: add SeekHole traitDaniel Verkamp2018-10-18
| | | | | | | | | | | | | Allow seeking to the next hole or data region in File and QcowFile. BUG=None TEST=None Change-Id: I16e77e4791aa85b4cc96f38327026cd93f02b7e1 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1274147 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Zach Reizner <zachr@chromium.org>
* qcow: create BufWriter with exact capacityDaniel Verkamp2018-10-13
| | | | | | | | | | | | | | Rather than using the default BufWriter capacity (currently 8K), calculate the exact size required to contain the table. This further consolidates the write() system calls to the minimum required. BUG=chromium:891335 TEST=Manually verify expected write() calls with strace Change-Id: I9ade77024e60a92d30b65beb07c2385e5dacc167 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1250035 Reviewed-by: Dylan Reid <dgreid@chromium.org>
* cargo fmt all source codeZach Reizner2018-10-09
| | | | | | | | | | | | | | Now that cargo fmt has landed, run it over everything at once to bring rust source to the standard formatting. TEST=cargo test BUG=None Change-Id: Ic95a48725e5a40dcbd33ba6d5aef2bd01e91865b Reviewed-on: https://chromium-review.googlesource.com/1259287 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Zach Reizner <zachr@chromium.org> Reviewed-by: Zach Reizner <zachr@chromium.org>
* qcow: track deallocated clusters as unreferencedDaniel Verkamp2018-10-05
| | | | | | | | | | | | | | | | | | | | | In deallocate_cluster(), we call set_cluster_refcount() to unref the cluster that is being deallocated, but we never actually added the deallocated cluster to the unref_clusters list. Add clusters whose refcounts reach 0 to the unref_clusters list as well. Also add mremap() to the seccomp whitelist for the block device, since this is being triggered by libc realloc() and other devices already include it in the whitelist. BUG=chromium:850998 TEST=cargo test -p qcow; test crosvm on nami and verify that qcow file size stays bounded when creating a 1 GB file and deleting it repeatedly Change-Id: I1bdd96b2176dc13069417e0ac77f0768f9f26012 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1259404 Reviewed-by: Dylan Reid <dgreid@chromium.org>
* qcow: optimize sync_caches to avoid extra writesDaniel Verkamp2018-09-27
| | | | | | | | | | | | | | Track the clean/dirty state of the L1 table and the refcount table to avoid writing them out and doing an extra fsyncdata() if nothing has changed. BUG=None TEST=Manually verify strace output contains only the expected fsyncs Change-Id: I20bdd250024039a5b4142605462a8977ced1efcc Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1247442 Reviewed-by: Dylan Reid <dgreid@chromium.org>
* qcow: buffer reads and writes in QcowRawFileDaniel Verkamp2018-09-27
| | | | | | | | | | | | | | | | When reading and writing refcount blocks and pointer tables, the QcowRawFile implementation was performing many individual read() and write() system calls (one per table entry), which is quite inefficient. Use the read_*_into functions from ReadBytesExt for reads and BufWriter for writes to buffer the I/O into larger chunks. BUG=None TEST=Manually verify larger reads/writes with strace Change-Id: I276963db0a4e91b22335c26c799ae8fb55bf6ad3 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1247441 Reviewed-by: Dylan Reid <dgreid@chromium.org>
* qcow: Add accessors for more file stateDylan Reid2018-09-19
| | | | | | | | | | | | Being able to access the state of the qcow file makes debugging easier. These functions will be used from a helper program in the following commit. Change-Id: I1db7ddaeaff1c83363513a2c55c44a1825833634 Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1207454 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
* qcow: Cache address and refcount tablesDylan Reid2018-09-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | Cache the address lookup and refcount tables in RAM. This removes an absurd number of system calls required for accessing the qcow image as previously each lookup required reading from at least three locations on disk. Now the disk will only be accessed when there is a cache miss or when the structure of the disk need to be updated; when a cluster is added or removed. The L1 address lookup table and the refcount table are both read at creation time and kept in memory. For now all caches are committed to disk only when the file is flushed. Later a timer will be added to periodically flush the caches to disk so there is less window for data loss. The L2 and refcount blocks are cached as full clusters. Those clusters are kept in the cache until a flush or they need to be evicted to make room for a new entry. The eviction is currently very simple, writing back the first entry in the cache in whatever order a hashmap iterator uses. This can be improved, but the speedup is already enough that it seems acceptable to start. Change-Id: Ifcc55f243961d54eb1c6255b975a1529e2e987af Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1207453 Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
* qcow: Add refcounting helperDylan Reid2018-09-15
| | | | | | | | | | The refcounting helper breaks out management of the refcounts and caching the refcount blocks. Change-Id: I6e75fbe0eb47277ccf7a93af026b5020089875db Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1207452 Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
* qcow: Add vec_cacheDylan Reid2018-09-15
| | | | | | | | | | The `VecCache` struct will be used to represent the file clusters in caches. It ties a vector to a state of dirty or clean. Change-Id: I474eb67d2ad9f086da638ecc385ccce74737d3b9 Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1207451 Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
* qcow: Add raw file structDylan Reid2018-09-13
| | | | | | | | | | | | The raw file struct will be used to hold enough state for basic operations. This will allow mutating the file without taking a mutable reference to an entire QcowFile. Change-Id: Ia0a86537915da039274923df2f85c22d191b9969 Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1207450 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
* qcow: deallocate clusters in write_zeroesDaniel Verkamp2018-09-05
| | | | | | | | | | | | | | | | | When a write_zeroes call covers a whole cluster, we can deallocate the storage for that cluster rather than writing zeroes. This is currently implemented by removing the cluster allocation from the mapping tables, then attempting to release the backing storage using fallocate() with FALLOC_FL_PUNCH_HOLE. BUG=chromium:850998 TEST=cargo test -p qcow Change-Id: Ie4edb2e02bfaa1df9a19919b77eeb3c58c112d1c Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1187019 Reviewed-by: Dylan Reid <dgreid@chromium.org>
* qcow: implement WriteZeroes for QcowFileDaniel Verkamp2018-09-05
| | | | | | | | | | | | | | | Add a simple implementation of WriteZeroes for QcowFile that just writes zeroes to allocated clusters and skips clusters that are already unallocated (since they already read back as zeroes). BUG=chromium:850998 TEST=cargo test -p qcow Change-Id: I8f26c8cc4016c129850aaf08c7188dfe08d6dacb Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1187018 Reviewed-by: Dylan Reid <dgreid@chromium.org> Reviewed-by: Zach Reizner <zachr@chromium.org>
* qcow: fix check for compressed cluster bitDaniel Verkamp2018-08-17
| | | | | | | | | | | | | | | | | | | | | | | The bit that indicates a compressed cluster in the qcow2 format is stored in the L2 table entry (cluster descriptor), but the code was checking for it in the L1 table entry. Spotted by inspection based on the qcow2 format documentation (QEMU docs/interop/qcow2.txt). This has no impact in normal crosvm use cases, since crosvm never writes compressed clusters; the only case where it could have any effect would be when using a QEMU-created qcow2 image with compressed clusters, which would have returned incorrect data from a read instead of failing the read request as intended. BUG=chromium:874699 Change-Id: Ic247490dcac5440fab8e4d34d24ffe6fe9ab3110 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1176733 Reviewed-by: Stephen Barber <smbarber@chromium.org> Reviewed-by: Dylan Reid <dgreid@chromium.org>
* qcow: Set refcounts for initial clusters.Dylan Reid2018-07-16
| | | | | | | | | | | | | | | | | | | | | All qcow clusters need to have their refcounts set. Add a `new` method to `Qcowfile` and use it instead of just headers from the library. The new method will loop over the initial clusters and initialize their refcounts. Add a `create_qcow2` option to the main executable so there is a way to test image creation that doesn't require DBUS and Concierge. BUG=none TEST='crosvm create_qcow2 /tmp/file.qcow2 1000000' 'qemu-img check /tmp/file.qcow2' no errors reported. Change-Id: I8798df5942fb23f79cc7ca86820d0783d1f2b608 Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1136900 Reviewed-by: Stephen Barber <smbarber@chromium.org>
* devices: block: sync after setting refcountDylan Reid2018-07-13
| | | | | | | | | | Make sure that the refcount update is commited to disk before we start to write data to the cluster. Change-Id: Ibdf5dde7a75c3582f87df46af1095c7dc0007d98 Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1136019 Reviewed-by: Stephen Barber <smbarber@chromium.org>
* qcow: Call fsync(2) when we want to flush to diskDylan Reid2018-05-18
| | | | | | | | | | Signal to the OS that we want these writes committed all the way to disk. Replace an existing call to flush as that's not sufficient. Change-Id: I9df9e55d2182e283e15eebc02a54c1ce08434f42 Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1060696 Reviewed-by: Zach Reizner <zachr@chromium.org>
* qcow: avoid truncation if usize is 32 bitsDylan Reid2018-04-27
| | | | | | | | | | | | | The u64 offsets could be truncated when running on a 32 bit machine. Do the math in 64 bit, limit to usize::MAX, then truncate. BUG=837453 TEST=run crosvm and read/write files Change-Id: If44ec94cf730ca7c1e580eeddd202e54e2de1081 Signed-off-by: Dylan Reid <dgreid@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1031301 Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-06-03 12:01:25 +0000