From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from de-smtp-delivery-102.mimecast.com ([62.140.7.102]) by metis.ext.pengutronix.de with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lPfL5-0008Pt-Ph for rauc@pengutronix.de; Fri, 26 Mar 2021 06:48:57 +0100 From: Einar Vading Date: Fri, 26 Mar 2021 05:48:50 +0000 Message-ID: References: <7a2fc0a9cb6bb54455d4cb69403a469e2fe832d8.camel@pengutronix.de>, In-Reply-To: MIME-Version: 1.0 Content-Language: en-US Subject: Re: [RAUC] [NEWSLETTER]Re: Robust u-boot environment with RAUC List-Id: RAUC Project - Discussion List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0014993540==" Errors-To: rauc-bounces@pengutronix.de Sender: "RAUC" To: =?iso-8859-1?Q?Enrico_J=F6rns?= , =?iso-8859-1?Q?Jan_L=FCbbe?= , "rauc@pengutronix.de" --===============0014993540== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_AM0PR08MB4580CB2D01A5F69F0BD8112AE5619AM0PR08MB4580eurp_" --_000_AM0PR08MB4580CB2D01A5F69F0BD8112AE5619AM0PR08MB4580eurp_ Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable > > Hi, > > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote: > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-b= oot for > > > booting. For some systems in the field we have the u-boot environment= on the > > > FAT boot partition and we mount that in fstab so that RAUC can access= it with > > > the fw_print/setenv commands. > > > > > > One issue we have seen is that the env-file gets corrupted every now = and then. > > > After corruption we can't RAUC update. The only solution we have to t= his > > > problem now is to delete the corrupted env-file and reboot, then we c= an > > > perform the upgrade. > > > > > > I have no idea how to track down whatever corrupts the file and I was > > > wondering if anyone has any input. > > > > You could try placing the environment on a separate partition to avoid = any > > potential issues in the FAT implementation. Also, I think U-Boot has a = way to > > support redundant environments. I have just done this for our newer systems. I moved the GPT partitions bac= k 4MB and placed two redundant environments between the GPT and the first GPT par= tition. It is my understanding though that redundant environments are not supported= when storing the env on FAT? > Exactly. This should also be documented in the U-Boot integration guideli= ne for > eMMC: > > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up= -u-boot-environment-on-emmc-sd-card > > When writing to the FAT very short before hard rebooting, I could imagine= this > can lead to failures. Do you see the corruption only after updates, or al= so > suddenly after n boots? Yes, this is something we have been able to test. If we cut the power preci= sely when the env is written to FAT we can corrupt the entire boot partition. Super scary but this is not the problem we're seeing in the field. That pro= blem is more subtle. > How does the system report the corruption? fw_printenv and fw_setenv stops working and says that the env is corrupted.= That also means that RAUC update fails, that is usually when we notice it. Is there a way to watch a file and record any process that modifies it? > > Regards, Enrico > > > Regards, > > Jan > Best Regards, Einar --_000_AM0PR08MB4580CB2D01A5F69F0BD8112AE5619AM0PR08MB4580eurp_ Content-Type: text/html; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable > > Hi,
> >
> > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> > > We have a Raspberry Pi 4 system set up using RAUC for u= pdates and u-boot for
> > > booting. For some systems in the field we have the u-bo= ot environment on the
> > > FAT boot partition and we mount that in fstab so that R= AUC can access it with
> > > the fw_print/setenv commands.
> > >
> > > One issue we have seen is that the env-file gets corrup= ted every now and then.
> > > After corruption we can't RAUC update. The only solutio= n we have to this
> > > problem now is to delete the corrupted env-file and reb= oot, then we can
> > > perform the upgrade.
> > >
> > > I have no idea how to track down whatever corrupts the = file and I was
> > > wondering if anyone has any input.
> >
> > You could try placing the environment on a separate partitio= n to avoid any
> > potential issues in the FAT implementation. Also, I think U-= Boot has a way to
> > support redundant environments.

I have just done this for our newer systems. I moved the GPT partition= s back 4MB
and placed two redundant environments between the GPT and the first GP= T partition.

It is my understanding though that redundant environments are not supp= orted when
storing the env on FAT?

> Exactly. This should also be documented in the U-Boot integration= guideline for
> eMMC:
>
> https://rauc.readthedocs.io/en/latest/integration.html#example-se= tting-up-u-boot-environment-on-emmc-sd-card
>
> When writing to the FAT very short before hard rebooting, I could= imagine this
> can lead to failures. Do you see the corruption only after update= s, or also
> suddenly after n boots?

Yes, this is something we have been able to test. If we cut the power = precisely
when the env is written to FAT we can corrupt the entire boot partitio= n.
Super scary but this is not the problem we're seeing in the field. Tha= t problem
is more subtle.

> How does the system report the corruption?

fw_printenv and fw_setenv stops working and says that the env is corru= pted. That
also means that RAUC update fails, that is usually when we notice it.<= /div>

Is there a way to watch a file and record any process that modifies it= ?

>
> Regards, Enrico
>
> > Regards,
> > Jan
>

Best Regards,
Einar
--_000_AM0PR08MB4580CB2D01A5F69F0BD8112AE5619AM0PR08MB4580eurp_-- --===============0014993540== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ RAUC mailing list --===============0014993540==--