feat: reshuffle_apps #103

addrian-77 · 2025-09-20T22:13:11Z

Pull Request Overview

This pull request adds the reshuffle_apps feature.

TODO or Help Wanted

This pull request still needs an implementation for creating padding apps

Checks

Using Rust tooling

Ran cargo fmt
Ran cargo clippy
Ran cargo test
Ran cargo build

Features tested:

install using reshuffle_apps on microbit-v2

GitHub Issue

This pull request closes <Implement reorder apps function #102>

addrian-77 · 2025-09-20T23:25:38Z

I temporarily changed the install function to just call reshuffle_apps and a new TAB to the existing list of apps, find a valid permutation and then write to the board. This works fine for installing C apps.

When trying to install a Rust app, there are a few problems that occur:

the TBF is not being selected correctly, the arch string is not enough when using extract_binary, we also need to choose a TBF in which the app starts at least at flash start address
currently I can't insert a padding because I have not yet found out how to create a padding app binary from scratch

I tried choosing the correct TBF manually, yet somehow tockloader-rs found that the app should start at 0x40080, while the python version found the start address to be 0x40000. This is not an error, but maybe I'm missing an extra step that the python version is doing, because it also finds 0x40080 but then decides to start writing the binary at 0x40000.
@bradjc can you check on this?

eva-cosma · 2025-09-22T04:32:10Z

Is it always 0x80 bytes off? This, intuitively, feels like a header is not being skipped in the calculation, or maybe due to page-alignment? Do we do page-alignment for fixed apps?

Until i get back to the office, try looking at elf2tab, they might create a padding app.

Also, try setting header_size to 0 or 16 (the length of the base header). Maybe the parsing algorithm can help you: https://github.com/WyliodrinEmbeddedIoT/tockloader-rs/blob/main/tbf-parser%2Fsrc%2Fparse.rs#L129

Edit: I'm pretty sure if you create a base header with header length 16 (bytes), you will have just created a padding app. Note that this does mean padding apps cannot be less than 16 bytes long or "0x10" in size

tockloader-lib/src/command_impl/generalized.rs

addrian-77 · 2025-09-27T18:31:17Z

Installing via reshuffle_apps is now working, rust apps had to be aligned correctly (contrary to my previous beliefs).
I installed the same Rust app multiple times (different TBFs selected manually). Both were displayed in picocom.
I also installed C apps along with these. The padding worked correctly and all apps were working as expected.

TBF selection should be changed, as if file.filename.starts_with(arch) is not enough for rust apps, there are multiple variants that fit this. The TBF should be selected either manually through an inquire or there should be more conditions for flash_address and ram_address

eva-cosma

Now that we have the IO rewrite, can we merge getting app attributes form probe-rs and serial?

I still have to review reshuffle and install, otherwise LGTM

tbf-parser/src/parse.rs

tockloader-lib/src/attributes/app_attributes.rs

tockloader-lib/src/bootloader_serial.rs

tockloader-lib/src/command_impl/erase_apps.rs

tockloader-lib/src/tabs/tab.rs

tockloader-lib/src/command_impl/serial/io.rs

eva-cosma · 2025-10-20T15:10:17Z

@addrian-77 Rebase on top of the latest main (don't forget to git fetch 😉 )

Signed-off-by: addrian-77 <lunguadrian30@gmail.com>

Signed-off-by: Eva Cosma <eva.cosma.mail@gmail.com>

eva-cosma

Okay, so, I rewrote reshuffle apps in a way that makes it so we don't depend at all on where the apps come from or what is the deal with them. It also minimizes adding unnecessary/hyper-specific fields in other structures.

I haven't refactored everything: there are still plenty of errors. I just did the reshuffle_apps function, I'll leave the rest to you.

This pass has revealed some issues though:

We do not make use of the IO rewrite in the app_attributes. We no longer need to differentiate between probe_rs and serial getting of apps thanks the IO rewrite.
PAGE_SIZE is board dependant, and should be in board_settings. This does however spread to the write function needing to have a page_size passed to it. the write function itself I would only add as an argument the page_size, and above that I would add board_setting
The algorithm you made is great, but does not tackle the main problem - rust apps being able to be put in multiple places. Adding permutations to that won't be feasible. I suggest we either just implement what tockloader has already, and tackle mixed at a later date, or we go full-force with the constraint satisfaction algorithm.
I don't think we care to minimize for minimum padding, as long as everything fits. Can be an option in the future.
I suggest we split this PR in two again, one for the IO rewrite, one for reshuffle. We won't merge them in main but in a separate feature branch, where we stack tab rework + io rewrite + reshuffle and only after we merge. We could have tab be merged directly into main since it only adds features, while io rewrite removes install, so that is why it must go into a feature branch. Opinions?

eva-cosma · 2025-10-20T20:23:59Z

tockloader-lib/src/attributes/app_attributes.rs

 ///
 /// See also <https://book.tockos.org/doc/tock_binary_format>
-#[derive(Debug)]
+#[derive(Debug, Clone)]


Do we still need clone

eva-cosma · 2025-10-20T20:25:22Z

tockloader-lib/src/command_impl/reshuffle_apps.rs

+    pub fn from_app_attributes(_app_attributes: &AppAttributes) -> Self {
+        todo!("Implement TockApp::from_app_attributes")
+    }
+
+    pub fn from_tab(_tab: &Tab) -> Self {
+        todo!("Implement TockApp::from_tab")
+    }


Implement and use these

eva-cosma · 2025-10-20T20:26:09Z

tockloader-lib/src/command_impl/reshuffle_apps.rs

+    address - address % ALIGNMENT
+}
+
+pub fn create_pkt(configuration: Vec<AppAttributes>, mut app_binaries: Vec<Vec<u8>>) -> Vec<u8> {


This doesn't make sense to be in reshuffle, move it to install

eva-cosma · 2025-10-20T20:26:30Z

tockloader-lib/src/command_impl/reshuffle_apps.rs

+}
+
+// this function takes a rust app's fixed address and aligns it down to ALIGNMENT (1024 currently)
+fn align_down(address: u64) -> u64 {


Do we not need to also align_down in places in reshuffle itself?

no, align_down is used in reconstruct_app, for rust apps

eva-cosma · 2025-10-20T20:35:54Z

Also please don't squash for now. Let's do it at the end after we are sure everything is okay, since I deleted some of your old code, which you might need back at some point.

addrian-77 · 2025-10-30T15:32:27Z

In the last commit I mostly implemented all the changes requested. I tried installing a C app for now and it worked.
Rust apps can't currently be installed. The code needs to get merged and Rust support will be added later along with the TBF Filtering PR

This PR needs some cleanup and maybe the traits have to be moved somewhere else.

eva-cosma · 2025-10-30T16:02:37Z

tockloader-lib/src/command_impl/reshuffle_apps.rs

+            return TockApp::Fixed(FixedApp {
+                idx: None,
+                board_address: None,
+                candidate_addresses: vec![address as u64], // (adi): change this when tbf selector gets merged


No? This is just correct? Since if it is an app attribute, it means that it's already installed

eva-cosma · 2025-10-30T16:02:48Z

tockloader-lib/src/command_impl/reshuffle_apps.rs


-    fn as_intermediate_index(&self, install_address: u64) -> IntermediateIndex {
-        IntermediateIndex {
+    // useless?


Delete function(s) then

eva-cosma · 2025-10-30T16:04:29Z

tockloader-lib/src/command_impl/reshuffle_apps.rs

+        //    tab: Option<&Tab>, settings: &BoardSettings) -> Option<AppAttributes>
+        let arch = settings
+            .arch
+            .as_ref()
+            .ok_or(InternalError::MisconfiguredBoardSettings(
+                "architechture".to_owned(),
+            ))
+            .unwrap();


Just get the arch then. Unwraps baaad :( and we can return an error above from the calling code

eva-cosma · 2025-10-30T16:07:03Z

tockloader-lib/src/command_impl/reshuffle_apps.rs

+
+        // extract the binary
+        // this should be changed to accomodate candidate_addresses
+        let binary = tab.extract_binary(arch).expect("invalid arch");


The amount of unwraps/expects scares me

(message inspired from a great thinker)

eva-cosma · 2025-10-30T16:09:43Z

tockloader-lib/src/command_impl/reshuffle_apps.rs

-    pub fn from_tab(_tab: &Tab) -> Self {
-        todo!("Implement TockApp::from_tab")
+    /// This function reads the full binary of a given app
+    pub async fn read_binary(&mut self, conn: &mut dyn IO) -> Result<Vec<u8>, TockloaderError> {


Why does a Tock app need to know how to read a binary?

The whole point of it is that it's a source-erased version of an app, so that reshuffle can just play around with it, and the calling code can then just re-add the source to it when needed

eva-cosma · 2025-10-30T16:10:50Z

tockloader-lib/src/command_impl/reshuffle_apps.rs

+        let mut footer_offset = binary_end_offset;
+        let mut footer_number = 0;
+
+        while footer_offset < total_size {


I don't understand what we are doing with the footers here

eva-cosma · 2025-10-30T16:11:08Z

tockloader-lib/src/command_impl/reshuffle_apps.rs

+#[derive(Clone)]
+pub struct FlexibleApp {
    idx: Option<usize>,
+    board_address: Option<u64>, // None if not installed


This ideally should not be needed. We do not care for where an application is installed. The calling code can deal with that. TockApp/FlexibleApp/FixedApp are just concepts that do not leave reshuffle_apps. These exist in a vacuum. The calling code is responsible to then understand the new order.

eva-cosma · 2025-10-30T16:13:27Z

tockloader-lib/src/command_impl/reshuffle_apps.rs

    }
 }

+// a vec of these is returned by reshuffleapps


What is this commend doing here? If you wanted to make a doc-comment, you missed a /

just notes for myself, my bad :))

eva-cosma · 2025-10-30T16:15:46Z

tockloader-lib/src/lib.rs

+        &mut self,
+        address: u64,
+        pkt: Vec<u8>,
+        settings: &BoardSettings,


This is geniuenly unfortunate that we need to pass board settings to this. We could keep it like this for now, but I'm thinking of maybe attaching the board settings to a connection, like inside the struct. Thoughts?

sounds good

Signed-off-by: addrian-77 <lunguadrian30@gmail.com>

eva-cosma · 2025-11-03T14:20:43Z

tockloader-lib/src/command_impl/reshuffle_apps.rs

+        }
+    }
+
+    Some(saved_configuration)


Add a check if we didn't find a permutation that works in the first 100_000 to return None

Signed-off-by: Adrian Lungu <lunguadrian30@gmail.com>

addrian-77 requested a review from eva-cosma September 20, 2025 22:13

addrian-77 self-assigned this Sep 20, 2025

addrian-77 linked an issue Sep 20, 2025 that may be closed by this pull request

Implement reorder apps function #102

Open

9 tasks

eva-cosma reviewed Sep 22, 2025

View reviewed changes

tockloader-lib/src/command_impl/generalized.rs Outdated Show resolved Hide resolved

addrian-77 force-pushed the add/reorder-apps branch 3 times, most recently from b017563 to dcd9db0 Compare October 13, 2025 13:50

eva-cosma requested changes Oct 15, 2025

View reviewed changes

addrian-77 requested a review from eva-cosma October 20, 2025 10:26

addrian-77 added 2 commits October 20, 2025 21:14

feat: reshuffle apps + generalizaiton

ed89811

Signed-off-by: addrian-77 <lunguadrian30@gmail.com>

refactor: separated IO Commands

d9c0012

Signed-off-by: addrian-77 <lunguadrian30@gmail.com>

eva-cosma force-pushed the add/reorder-apps branch from a9798b9 to d9c0012 Compare October 20, 2025 18:14

wip: restructure reshuffle_apps

a289e61

Signed-off-by: Eva Cosma <eva.cosma.mail@gmail.com>

eva-cosma requested changes Oct 20, 2025

View reviewed changes

addrian-77 requested a review from eva-cosma October 30, 2025 15:33

eva-cosma requested changes Oct 30, 2025

View reviewed changes

addrian-77 force-pushed the add/reorder-apps branch from 1e03e51 to ee1b2c6 Compare October 30, 2025 21:44

addrian-77 added 3 commits October 31, 2025 10:39

wip: refactored reshuffle

66965d1

Signed-off-by: addrian-77 <lunguadrian30@gmail.com>

refactor: remove read_binary

285bd61

Signed-off-by: addrian-77 <lunguadrian30@gmail.com>

refactor: move settings inside connection

26e5438

Signed-off-by: addrian-77 <lunguadrian30@gmail.com>

addrian-77 force-pushed the add/reorder-apps branch from c019c79 to 26e5438 Compare October 31, 2025 08:39

refactor: change vec to slice inside write function

42fbaf8

Signed-off-by: addrian-77 <lunguadrian30@gmail.com>

addrian-77 force-pushed the add/reorder-apps branch from f6cd01f to 42fbaf8 Compare October 31, 2025 09:46

eva-cosma reviewed Nov 3, 2025

View reviewed changes

wip

8b608af

Signed-off-by: Adrian Lungu <lunguadrian30@gmail.com>

feat: reshuffle_apps #103

Are you sure you want to change the base?

feat: reshuffle_apps #103

Uh oh!

Conversation

addrian-77 commented Sep 20, 2025

Pull Request Overview

TODO or Help Wanted

Checks

Using Rust tooling

Features tested:

GitHub Issue

Uh oh!

addrian-77 commented Sep 20, 2025

Uh oh!

eva-cosma commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

addrian-77 commented Sep 27, 2025

Uh oh!

eva-cosma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eva-cosma commented Oct 20, 2025

Uh oh!

eva-cosma left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eva-cosma commented Oct 20, 2025

Uh oh!

addrian-77 commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eva-cosma commented Sep 22, 2025 •

edited

Loading

addrian-77 commented Oct 30, 2025 •

edited

Loading