[RAUC] Bootloader updates

[RAUC] Bootloader updates

[Martin Hollingsworth]

[-- Attachment #1.1: Type: text/plain, Size: 1153 bytes --]

Hello RAUC Team,
I am currently implementing software update on my embedded target and evaluating the usage of RAUC. But I have trouble finding a solution for the following scenario, which needs a little explanation, but can be summerized as: "How does Rauc handle Bootloader Updates?"

Here is what I want to achieve:

A software release is a combination of multiple software components in specific versions, that have been tested and defined fit for usage together on an embedded target. Lets say a simple example would be a bootloader and a rootFS (which includes linux and some custom software pre-installed).

When I update my system, I secure my linux operating system using a symmetric A/B setup. If the update fails, I can always jump back to the last slot and continue regular device operation.

But when I update my bootloader AND my linux system together, after exchanging the bootloader I cannot jump back to the last running linux, because now the bootloader configuration might not match the configuration expected from the previous bootloader.

How does RAUC approach this issue?

Thanks in advance and greetings,
Martin


[-- Attachment #1.2: Type: text/html, Size: 3894 bytes --]

[-- Attachment #2: Type: text/plain, Size: 65 bytes --]

_______________________________________________
RAUC mailing list

Re: [RAUC] Bootloader updates

[Enrico Joerns]

Hello Martin,

On 05/18/2017 11:45 AM, Martin Hollingsworth wrote:
> Hello RAUC Team,
>
> I am currently implementing software update on my embedded target and
> evaluating the usage of RAUC. But I have trouble finding a solution for
> the following scenario, which needs a little explanation, but can be
> summerized as: „How does Rauc handle Bootloader Updates?”
>
> Here is what I want to achieve:
>
> A software release is a combination of multiple software components in
> specific versions, that have been tested and defined fit for usage
> together on an embedded target. Lets say a simple example would be a
> bootloader and a rootFS (which includes linux and some custom software
> pre-installed).

that sounds rational to me (despite the fact that you might not want to 
update you bootloader with each rootFS update).

> When I update my system, I secure my linux operating system using a
> symmetric A/B setup. If the update fails, I can always jump back to the
> last slot and continue regular device operation.

Yes, that is one of the common Scenarios RAUC will be able to handle for 
you.

> But when I update my bootloader AND my linux system together, after
> exchanging the bootloader I cannot jump back to the last running linux,
> because now the bootloader configuration might not match the
> configuration expected from the previous bootloader.
>
> How does RAUC approach this issue?

this is not something that really touches RAUC and I am not sure if I 
got you problem right.

You intend to exchange the bootloader, but this should not have any 
consequence for the stored update configuration. The information about 
which slot has which priority, remaining attempts, etc. will normally be 
persisted in a different storage location than your bootloader itself. 
E.g. a different partition on your FLASH or a distinct EEPROM location, etc.

Thus, after having updated your bootloader, it will read your previous 
configuration and know in which state you are!

The only scenario where you will get into troubles is when the new 
bootloader is unable to read the storage format your old one used. But 
1) this should not happen or is an issue that is really out of scope of 
what an update tool can do for you.

If I did not really get what your worries are, please let me know.

May I ask which bootloader you are using?


Best regards,

Enrico

-- 
Pengutronix e.K.                           | Enrico Jörns                |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-5080 |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |


_______________________________________________
RAUC mailing list

Re: [RAUC] Bootloader updates

[Martin Hollingsworth]

Hi Enrico,
thanks for the quick answer. I am using barebox with bootchoser and PTXdist for my linux.

> despite the fact that you might not want to update you bootloader with each rootFS update.
Indeed! I have to figure out a way to compare the installed bootloader with the shipped one and only update if necessary.

> Thus, after having updated your bootloader, it will read your previous configuration and know in which state you are!
Good to know, let's keep that in mind for the next section.
 
Ok, so let me rephrase my worries. So during an update, that includes a new bootloader, the following steps are done in the following order:
1) Update Bootloader by overwriting it (RAUC config stays intact).
2) Install RootFS onto second slot (say Slot B).

When 1) succeeds and 2) fails / is interrupted (power cut, etc.), the bootchoser configuration will decide to boot from Slot A again (the old linux system). 

Hence now we have a newer version of the bootloader, running with the previous (older) linux system. Hence we have a software combination, that was not intended to run together, probably not tested during development, but potentially running on a field device.

This is the main concern of this question. Could this lead to trouble? Is there a mechanism in RAUC to prevent falling into this situation? (Maybe my question is more a conceptual question than a RAUC question).

Hope this made my thoughts more clear. Thanks for the discussion and cheers,
Martin
_______________________________________________
RAUC mailing list

Re: [RAUC] Bootloader updates

[Jan Lübbe]

Hi Martin,

thanks for your interesting question!

On Do, 2017-05-18 at 11:29 +0000, Martin Hollingsworth wrote:
> Ok, so let me rephrase my worries. So during an update, that includes
> a new bootloader, the following steps are done in the following order:
> 1) Update Bootloader by overwriting it (RAUC config stays intact).
> 2) Install RootFS onto second slot (say Slot B).
> 
> When 1) succeeds and 2) fails / is interrupted (power cut, etc.), the
> bootchoser configuration will decide to boot from Slot A again (the
> old linux system).
> 
> Hence now we have a newer version of the bootloader, running with the
> previous (older) linux system. Hence we have a software combination,
> that was not intended to run together, probably not tested during
> development, but potentially running on a field device.
As long as you have only a single bootloader (which is consequently the
component which decides whether to boot A or B), I see only two possible
options:
- ensure and test that newer bootloaders can boot older kernels
or
- make sure that the new bootloader never boots the old system

> This is the main concern of this question. Could this lead to trouble?
> Is there a mechanism in RAUC to prevent falling into this situation?
> (Maybe my question is more a conceptual question than a RAUC
> question).
Which option is acceptable for you depends on your use-case and whether
you can test the fallback combinations sufficiently.

For the second option I can think of several variants:
- Deactivate the current slot before updating the bootloader.
  This basically means you don't have fallback support at all.
- Check the installed bootloader version matches the one tested with 
  the rootfs before launching the main application. On error, you can
  at least notify the user.

The one I'd suggest is:
- Store a copy of the bootloader in the RootFS.
- Use RAUC only to update the only the RootFS. The combinations to test 
  can limited by restringing which old versions are supported by an 
  update.
- Reboot into the new system.
- On boot, before starting the application, check that the current slot 
  is 'sane'. Then check if the installed bootloader is older than the 
  version shipped in the (new) rootfs. In that case:
  - Disable the old rootfs slot and update the bootloader (directly or 
    via barebox_update).
  - Reboot
- Start the application.

This way you still have fallback support for the rootfs upgrade and need
to test only:
- sanity check and bootloader installation when started from old 
  bootloader and new rootfs
- normal operation when started from new bootloader and new rootfs

The case of new bootloader with old rootfs can never happen, because you
disable the old one from the new before installing a new bootloader.

This also means that the actual bootloader update is handled outside of
the RAUC update installation during bootup (perhaps as a systemd
service).


If your system boots from an eMMC, you can usually also make the
bootloader update atomic (but not with fallback) by using the special
boot partitions.

Regards,
Jan Lübbe
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |


_______________________________________________
RAUC mailing list

AW: [RAUC] Bootloader updates

[Martin Hollingsworth]

Hallo Jan,
thanks for your detailed suggestions. I would like to add my thoughts to your proposal.

>As long as you have only a single bootloader (which is consequently the component which decides whether to boot A or B), I see only two possible
>options:
>- ensure and test that newer bootloaders can boot older kernels or
>- make sure that the new bootloader never boots the old system

I totally agree with you, that these are the only two options I have. The first one implies a lot of testing, especially when the bootloader changes often. 

The second one can be simply achieved by disabling fallback support, but ONLY when the update includes a new bootloader. In this case I would do all update steps regularly, and only at very last step disabling fallback and exchanging the bootloader. This would give me fallback support about most of the time during update, only the last step and the reboot into new system is unprotected. (No risk no fun ^^).

> For the second option I can think of several variants:
> The one I'd suggest is:
>- Store a copy of the bootloader in the RootFS.
>- Use RAUC only to update the only the RootFS.
>- Reboot into the new system.
>- On boot, before starting the application, check that the current slotis 'sane'. 
>- Disable the old rootfs slot and update the bootloader (directly or via barebox_update).
>- Reboot
>- Start the application.

I see one major downside to this. The new kernel must be booted by the old bootloader. What if the new kernel needs major differences in the kernel bootargs? How likely is it, that such a change then cannot be booted with the old bootloader anymore and hence my whole update system may be unfit for the desired kernel changes? (Actually how often do kernel bootargs change in real life...?).

I have one other idea:
- Make a raw memory copy of the current bootloader sections into a designated file.
- Update the bootloader.
- Update the RootFS, while keeping the old slot still active as fallback (regular RootFS update using RAUC).
- If we successfully reboot into new system, all is fine, disable fallback slot, run new system.
- If the update fails, we fallback to the old linux slot, with the new bootloader active. 
The new bootloader should then detect the designated bootloader backup file, stop the regular startup process and overwrite itself with the backup of the bootloader.
After a reboot the old bootloader is running again with the old linux slot.

However I am not sure yet if this is a feasible solution, and technical possible. Furthermore this introduces the risk, that the bootloader overwrites itself with a file from filesystem. This could be dangerous.

Seems like there is no simple solution to this. Cheers and best regards,
Martin