From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: From: Einar Vading Date: Sun, 28 Mar 2021 10:11:01 +0000 Message-ID: References: <7a2fc0a9cb6bb54455d4cb69403a469e2fe832d8.camel@pengutronix.de> , , In-Reply-To: MIME-Version: 1.0 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_AM0PR08MB45809490AB5920AE54AB4540E57F9AM0PR08MB4580eurp_" Subject: Re: [RAUC] [NEWSLETTER]Re: Robust u-boot environment with RAUC To: =?iso-8859-1?Q?Enrico_J=F6rns?= , "rauc@pengutronix.de" , "jlu@pengutronix.de" List-ID: --_000_AM0PR08MB45809490AB5920AE54AB4540E57F9AM0PR08MB4580eurp_ Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable > Hi, > > On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote: > > > > Hi, > > > > > > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote: > > > > > We have a Raspberry Pi 4 system set up using RAUC for updates and= u-boot > > > > > for > > > > > booting. For some systems in the field we have the u-boot environ= ment on > > > > > the > > > > > FAT boot partition and we mount that in fstab so that RAUC can ac= cess it > > > > > with > > > > > the fw_print/setenv commands. > > > > > > > > > > One issue we have seen is that the env-file gets corrupted every = now and > > > > > then. > > > > > After corruption we can't RAUC update. The only solution we have = to this > > > > > problem now is to delete the corrupted env-file and reboot, then = we can > > > > > perform the upgrade. > > > > > > > > > > I have no idea how to track down whatever corrupts the file and I= was > > > > > wondering if anyone has any input. > > > > > > > > You could try placing the environment on a separate partition to av= oid any > > > > potential issues in the FAT implementation. Also, I think U-Boot ha= s a way > > > > to > > > > support redundant environments. > > > > I have just done this for our newer systems. I moved the GPT partitions= back > > 4MB and placed two redundant environments between the GPT and the first= GPT > > partition. > > > > It is my understanding though that redundant environments are not suppo= rted > > when storing the env on FAT? > > That's probably a question for the U-Boot mailing list. :) > > > > Exactly. This should also be documented in the U-Boot integration gui= deline > > > for eMMC: > > > > > > > > > https://rauc.readthedocs.io/en/latest/integration.html#example-settin= g-up-u-boot-environment-on-emmc-sd-card > > > > > > When writing to the FAT very short before hard rebooting, I could ima= gine > > > this > > > can lead to failures. Do you see the corruption only after updates, o= r also > > > suddenly after n boots? > > > > Yes, this is something we have been able to test. If we cut the power > > precisely when the env is written to FAT we can corrupt the entire boot > > partition. > > Super scary but this is not the problem we're seeing in the field. That > > problem is more subtle. > > It should be possible to mount fat with the 'sync' option, but I'm not su= re if > that would help in this case. I'd recommend avoiding mounting FAT filesys= tems > R/W if possible. Maybe it could help with the problem I'm investigating. Don't think it woul= d help with the total corruption on powerloss when writing u-boot env, since that is in= u-boot and the fs is not "mounted" yet. > > > How does the system report the corruption? > > > > fw_printenv and fw_setenv stops working and says that the env is corrup= ted. > > That also means that RAUC update fails, that is usually when we notice = it. > > > > Is there a way to watch a file and record any process that modifies it? > > There is blktrace, but you don't see the contents that way. It still may = be > enough detail to understand what's happening here. Great, I'll check that out. > Regards, > Jan Thanks for all the help. Regards, Einar --_000_AM0PR08MB45809490AB5920AE54AB4540E57F9AM0PR08MB4580eurp_ Content-Type: text/html; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable > Hi,
>
> On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote:
> > > > Hi,
> > > >
> > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wr= ote:
> > > > > We have a Raspberry Pi 4 system set up using = RAUC for updates and u-boot
> > > > > for
> > > > > booting. For some systems in the field we hav= e the u-boot environment on
> > > > > the
> > > > > FAT boot partition and we mount that in fstab= so that RAUC can access it
> > > > > with
> > > > > the fw_print/setenv commands.
> > > > >
> > > > > One issue we have seen is that the env-file g= ets corrupted every now and
> > > > > then.
> > > > > After corruption we can't RAUC update. The on= ly solution we have to this
> > > > > problem now is to delete the corrupted env-fi= le and reboot, then we can
> > > > > perform the upgrade.
> > > > >
> > > > > I have no idea how to track down whatever cor= rupts the file and I was
> > > > > wondering if anyone has any input.
> > > >
> > > > You could try placing the environment on a separat= e partition to avoid any
> > > > potential issues in the FAT implementation. Also, = I think U-Boot has a way
> > > > to
> > > > support redundant environments.
> >
> > I have just done this for our newer systems. I moved the GPT= partitions back
> > 4MB and placed two redundant environments between the GPT an= d the first GPT
> > partition.
> >
> > It is my understanding though that redundant environments ar= e not supported
> > when storing the env on FAT?
>
> That's probably a question for the U-Boot mailing list. :)
>
> > > Exactly. This should also be documented in the U-Boot i= ntegration guideline
> > > for eMMC:
> > >
> > >
> > > https://rauc.readthedocs.io/en/latest/integration.html#= example-setting-up-u-boot-environment-on-emmc-sd-card
> > >
> > > When writing to the FAT very short before hard rebootin= g, I could imagine
> > > this
> > > can lead to failures. Do you see the corruption only af= ter updates, or also
> > > suddenly after n boots?
> >
> > Yes, this is something we have been able to test. If we cut = the power
> > precisely when the env is written to FAT we can corrupt the = entire boot
> > partition.
> > Super scary but this is not the problem we're seeing in the = field. That
> > problem is more subtle.
>
> It should be possible to mount fat with the 'sync' option, but I'= m not sure if
> that would help in this case. I'd recommend avoiding mounting FAT= filesystems
> R/W if possible.

Maybe it could help with the problem I'm investigating. Don't think it= would help with
the total corruption on powerloss when writing u-boot env, since that = is in u-boot and
the fs is not "mounted" yet.

> > > How does the system report the corruption?
> >
> > fw_printenv and fw_setenv stops working and says that the en= v is corrupted.
> > That also means that RAUC update fails, that is usually when= we notice it.
> >
> > Is there a way to watch a file and record any process that m= odifies it?
>
> There is blktrace, but you don't see the contents that way. It st= ill may be
> enough detail to understand what's happening here.

Great, I'll check that out.

> Regards,
> Jan

Thanks for all the help.

Regards,
Einar

--_000_AM0PR08MB45809490AB5920AE54AB4540E57F9AM0PR08MB4580eurp_--