Re: [RAUC] [NEWSLETTER]Re: Robust u-boot environment with RAUC

[Einar Vading <Einar.Vading@RHIMagnesita.com>]

[-- Attachment #1: Type: text/plain, Size: 3198 bytes --]

> Hi,
>
> On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote:
> > > > Hi,
> > > >
> > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> > > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot
> > > > > for
> > > > > booting. For some systems in the field we have the u-boot environment on
> > > > > the
> > > > > FAT boot partition and we mount that in fstab so that RAUC can access it
> > > > > with
> > > > > the fw_print/setenv commands.
> > > > >
> > > > > One issue we have seen is that the env-file gets corrupted every now and
> > > > > then.
> > > > > After corruption we can't RAUC update. The only solution we have to this
> > > > > problem now is to delete the corrupted env-file and reboot, then we can
> > > > > perform the upgrade.
> > > > >
> > > > > I have no idea how to track down whatever corrupts the file and I was
> > > > > wondering if anyone has any input.
> > > >
> > > > You could try placing the environment on a separate partition to avoid any
> > > > potential issues in the FAT implementation. Also, I think U-Boot has a way
> > > > to
> > > > support redundant environments.
> >
> > I have just done this for our newer systems. I moved the GPT partitions back
> > 4MB and placed two redundant environments between the GPT and the first GPT
> > partition.
> >
> > It is my understanding though that redundant environments are not supported
> > when storing the env on FAT?
>
> That's probably a question for the U-Boot mailing list. :)
>
> > > Exactly. This should also be documented in the U-Boot integration guideline
> > > for eMMC:
> > >
> > >
> > > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card
> > >
> > > When writing to the FAT very short before hard rebooting, I could imagine
> > > this
> > > can lead to failures. Do you see the corruption only after updates, or also
> > > suddenly after n boots?
> >
> > Yes, this is something we have been able to test. If we cut the power
> > precisely when the env is written to FAT we can corrupt the entire boot
> > partition.
> > Super scary but this is not the problem we're seeing in the field. That
> > problem is more subtle.
>
> It should be possible to mount fat with the 'sync' option, but I'm not sure if
> that would help in this case. I'd recommend avoiding mounting FAT filesystems
> R/W if possible.

Maybe it could help with the problem I'm investigating. Don't think it would help with
the total corruption on powerloss when writing u-boot env, since that is in u-boot and
the fs is not "mounted" yet.

> > > How does the system report the corruption?
> >
> > fw_printenv and fw_setenv stops working and says that the env is corrupted.
> > That also means that RAUC update fails, that is usually when we notice it.
> >
> > Is there a way to watch a file and record any process that modifies it?
>
> There is blktrace, but you don't see the contents that way. It still may be
> enough detail to understand what's happening here.

Great, I'll check that out.

> Regards,
> Jan

Thanks for all the help.

Regards,
Einar


[-- Attachment #2: Type: text/html, Size: 4946 bytes --]