Re: [RAUC] [NEWSLETTER]Re: Robust u-boot environment with RAUC

["Jan Lübbe" <jlu@pengutronix.de>]

Hi,

On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote:
> > > Hi,
> > > 
> > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot
> > > > for
> > > > booting. For some systems in the field we have the u-boot environment on
> > > > the
> > > > FAT boot partition and we mount that in fstab so that RAUC can access it
> > > > with
> > > > the fw_print/setenv commands.
> > > > 
> > > > One issue we have seen is that the env-file gets corrupted every now and
> > > > then.
> > > > After corruption we can't RAUC update. The only solution we have to this
> > > > problem now is to delete the corrupted env-file and reboot, then we can
> > > > perform the upgrade.
> > > > 
> > > > I have no idea how to track down whatever corrupts the file and I was
> > > > wondering if anyone has any input.
> > > 
> > > You could try placing the environment on a separate partition to avoid any
> > > potential issues in the FAT implementation. Also, I think U-Boot has a way
> > > to
> > > support redundant environments.
> 
> I have just done this for our newer systems. I moved the GPT partitions back
> 4MB and placed two redundant environments between the GPT and the first GPT
> partition.
> 
> It is my understanding though that redundant environments are not supported
> when storing the env on FAT?

That's probably a question for the U-Boot mailing list. :)

> > Exactly. This should also be documented in the U-Boot integration guideline
> > for eMMC:
> > 
> >  
> > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card
> > 
> > When writing to the FAT very short before hard rebooting, I could imagine
> > this
> > can lead to failures. Do you see the corruption only after updates, or also
> > suddenly after n boots?
> 
> Yes, this is something we have been able to test. If we cut the power
> precisely when the env is written to FAT we can corrupt the entire boot
> partition.
> Super scary but this is not the problem we're seeing in the field. That
> problem is more subtle.

It should be possible to mount fat with the 'sync' option, but I'm not sure if
that would help in this case. I'd recommend avoiding mounting FAT filesystems
R/W if possible.

> > How does the system report the corruption?
> 
> fw_printenv and fw_setenv stops working and says that the env is corrupted.
> That also means that RAUC update fails, that is usually when we notice it.
> 
> Is there a way to watch a file and record any process that modifies it?

There is blktrace, but you don't see the contents that way. It still may be
enough detail to understand what's happening here.

Regards,
Jan
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |


_______________________________________________
RAUC mailing list