mail archive of the rauc mailing list
 help / color / mirror / Atom feed
* [RAUC] Robust u-boot environment with RAUC
@ 2021-03-25 15:22 Einar Vading
  2021-03-25 15:52 ` Jan Lübbe
  0 siblings, 1 reply; 9+ messages in thread
From: Einar Vading @ 2021-03-25 15:22 UTC (permalink / raw)
  To: rauc


[-- Attachment #1.1.1: Type: text/plain, Size: 948 bytes --]

Hi,

We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot for booting. For some systems in the field we have the u-boot environment on the FAT boot partition and we mount that in fstab so that RAUC can access it with the fw_print/setenv commands.

One issue we have seen is that the env-file gets corrupted every now and then. After corruption we can't RAUC update. The only solution we have to this problem now is to delete the corrupted env-file and reboot, then we can perform the upgrade.

I have no idea how to track down whatever corrupts the file and I was wondering if anyone has any input.


Vänliga hälsningar / Best Regards



Einar Vading

Software Developer AGELLIS AB


[cid:08b60f59-61a0-41f4-8a02-48d7f47c2730]


T +46 73 420 76 81

Agellis Group AB
Tellusgatan 15,
SE-224 57, Lund, Sweden

www.agellis.com<http://www.agellis.com/>; www.rhimagnesita.com<http://www.rhimagnesita.com/>

[-- Attachment #1.1.2: Type: text/html, Size: 4543 bytes --]

[-- Attachment #1.2: Outlook-ggtizjsb.png --]
[-- Type: image/png, Size: 10716 bytes --]

[-- Attachment #2: Type: text/plain, Size: 66 bytes --]

_______________________________________________
RAUC mailing list

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [RAUC] Robust u-boot environment with RAUC
  2021-03-25 15:22 [RAUC] Robust u-boot environment with RAUC Einar Vading
@ 2021-03-25 15:52 ` Jan Lübbe
  2021-03-25 18:54   ` Enrico Jörns
  0 siblings, 1 reply; 9+ messages in thread
From: Jan Lübbe @ 2021-03-25 15:52 UTC (permalink / raw)
  To: Einar Vading, rauc

Hi,

On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot for
> booting. For some systems in the field we have the u-boot environment on the
> FAT boot partition and we mount that in fstab so that RAUC can access it with
> the fw_print/setenv commands.
> 
> One issue we have seen is that the env-file gets corrupted every now and then.
> After corruption we can't RAUC update. The only solution we have to this
> problem now is to delete the corrupted env-file and reboot, then we can
> perform the upgrade.
> 
> I have no idea how to track down whatever corrupts the file and I was
> wondering if anyone has any input.

You could try placing the environment on a separate partition to avoid any
potential issues in the FAT implementation. Also, I think U-Boot has a way to
support redundant environments.

Regards,
Jan
-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |


_______________________________________________
RAUC mailing list

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [RAUC] Robust u-boot environment with RAUC
  2021-03-25 15:52 ` Jan Lübbe
@ 2021-03-25 18:54   ` Enrico Jörns
  2021-03-26  5:48     ` [RAUC] [NEWSLETTER]Re: " Einar Vading
  0 siblings, 1 reply; 9+ messages in thread
From: Enrico Jörns @ 2021-03-25 18:54 UTC (permalink / raw)
  To: Jan Lübbe, Einar Vading, rauc

Am Donnerstag, dem 25.03.2021 um 16:52 +0100 schrieb Jan Lübbe:
> Hi,
> 
> On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot for
> > booting. For some systems in the field we have the u-boot environment on the
> > FAT boot partition and we mount that in fstab so that RAUC can access it with
> > the fw_print/setenv commands.
> > 
> > One issue we have seen is that the env-file gets corrupted every now and then.
> > After corruption we can't RAUC update. The only solution we have to this
> > problem now is to delete the corrupted env-file and reboot, then we can
> > perform the upgrade.
> > 
> > I have no idea how to track down whatever corrupts the file and I was
> > wondering if anyone has any input.
> 
> You could try placing the environment on a separate partition to avoid any
> potential issues in the FAT implementation. Also, I think U-Boot has a way to
> support redundant environments.

Exactly. This should also be documented in the U-Boot integration guideline for
eMMC:

https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card

When writing to the FAT very short before hard rebooting, I could imagine this
can lead to failures. Do you see the corruption only after updates, or also
suddenly after n boots?

How does the system report the corruption?


Regards, Enrico

> Regards,
> Jan

-- 
Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |


_______________________________________________
RAUC mailing list

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [RAUC] [NEWSLETTER]Re:  Robust u-boot environment with RAUC
  2021-03-25 18:54   ` Enrico Jörns
@ 2021-03-26  5:48     ` Einar Vading
  2021-03-26  9:13       ` Jan Lübbe
  0 siblings, 1 reply; 9+ messages in thread
From: Einar Vading @ 2021-03-26  5:48 UTC (permalink / raw)
  To: Enrico Jörns, Jan Lübbe, rauc


[-- Attachment #1.1: Type: text/plain, Size: 2262 bytes --]

> > Hi,
> >
> > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot for
> > > booting. For some systems in the field we have the u-boot environment on the
> > > FAT boot partition and we mount that in fstab so that RAUC can access it with
> > > the fw_print/setenv commands.
> > >
> > > One issue we have seen is that the env-file gets corrupted every now and then.
> > > After corruption we can't RAUC update. The only solution we have to this
> > > problem now is to delete the corrupted env-file and reboot, then we can
> > > perform the upgrade.
> > >
> > > I have no idea how to track down whatever corrupts the file and I was
> > > wondering if anyone has any input.
> >
> > You could try placing the environment on a separate partition to avoid any
> > potential issues in the FAT implementation. Also, I think U-Boot has a way to
> > support redundant environments.

I have just done this for our newer systems. I moved the GPT partitions back 4MB
and placed two redundant environments between the GPT and the first GPT partition.

It is my understanding though that redundant environments are not supported when
storing the env on FAT?

> Exactly. This should also be documented in the U-Boot integration guideline for
> eMMC:
>
> https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card
>
> When writing to the FAT very short before hard rebooting, I could imagine this
> can lead to failures. Do you see the corruption only after updates, or also
> suddenly after n boots?

Yes, this is something we have been able to test. If we cut the power precisely
when the env is written to FAT we can corrupt the entire boot partition.
Super scary but this is not the problem we're seeing in the field. That problem
is more subtle.

> How does the system report the corruption?

fw_printenv and fw_setenv stops working and says that the env is corrupted. That
also means that RAUC update fails, that is usually when we notice it.

Is there a way to watch a file and record any process that modifies it?

>
> Regards, Enrico
>
> > Regards,
> > Jan
>

Best Regards,
Einar

[-- Attachment #1.2: Type: text/html, Size: 3352 bytes --]

[-- Attachment #2: Type: text/plain, Size: 66 bytes --]

_______________________________________________
RAUC mailing list

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [RAUC] [NEWSLETTER]Re:  Robust u-boot environment with RAUC
  2021-03-26  5:48     ` [RAUC] [NEWSLETTER]Re: " Einar Vading
@ 2021-03-26  9:13       ` Jan Lübbe
  2021-03-28 10:11         ` Einar Vading
  0 siblings, 1 reply; 9+ messages in thread
From: Jan Lübbe @ 2021-03-26  9:13 UTC (permalink / raw)
  To: Einar Vading, Enrico Jörns, rauc

Hi,

On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote:
> > > Hi,
> > > 
> > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot
> > > > for
> > > > booting. For some systems in the field we have the u-boot environment on
> > > > the
> > > > FAT boot partition and we mount that in fstab so that RAUC can access it
> > > > with
> > > > the fw_print/setenv commands.
> > > > 
> > > > One issue we have seen is that the env-file gets corrupted every now and
> > > > then.
> > > > After corruption we can't RAUC update. The only solution we have to this
> > > > problem now is to delete the corrupted env-file and reboot, then we can
> > > > perform the upgrade.
> > > > 
> > > > I have no idea how to track down whatever corrupts the file and I was
> > > > wondering if anyone has any input.
> > > 
> > > You could try placing the environment on a separate partition to avoid any
> > > potential issues in the FAT implementation. Also, I think U-Boot has a way
> > > to
> > > support redundant environments.
> 
> I have just done this for our newer systems. I moved the GPT partitions back
> 4MB and placed two redundant environments between the GPT and the first GPT
> partition.
> 
> It is my understanding though that redundant environments are not supported
> when storing the env on FAT?

That's probably a question for the U-Boot mailing list. :)

> > Exactly. This should also be documented in the U-Boot integration guideline
> > for eMMC:
> > 
> >  
> > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card
> > 
> > When writing to the FAT very short before hard rebooting, I could imagine
> > this
> > can lead to failures. Do you see the corruption only after updates, or also
> > suddenly after n boots?
> 
> Yes, this is something we have been able to test. If we cut the power
> precisely when the env is written to FAT we can corrupt the entire boot
> partition.
> Super scary but this is not the problem we're seeing in the field. That
> problem is more subtle.

It should be possible to mount fat with the 'sync' option, but I'm not sure if
that would help in this case. I'd recommend avoiding mounting FAT filesystems
R/W if possible.

> > How does the system report the corruption?
> 
> fw_printenv and fw_setenv stops working and says that the env is corrupted.
> That also means that RAUC update fails, that is usually when we notice it.
> 
> Is there a way to watch a file and record any process that modifies it?

There is blktrace, but you don't see the contents that way. It still may be
enough detail to understand what's happening here.

Regards,
Jan
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |


_______________________________________________
RAUC mailing list

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [RAUC] [NEWSLETTER]Re:  Robust u-boot environment with RAUC
  2021-03-26  9:13       ` Jan Lübbe
@ 2021-03-28 10:11         ` Einar Vading
  2021-03-28 14:11           ` Matt Campbell
  0 siblings, 1 reply; 9+ messages in thread
From: Einar Vading @ 2021-03-28 10:11 UTC (permalink / raw)
  To: Enrico Jörns, rauc, jlu

[-- Attachment #1: Type: text/plain, Size: 3198 bytes --]

> Hi,
>
> On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote:
> > > > Hi,
> > > >
> > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> > > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot
> > > > > for
> > > > > booting. For some systems in the field we have the u-boot environment on
> > > > > the
> > > > > FAT boot partition and we mount that in fstab so that RAUC can access it
> > > > > with
> > > > > the fw_print/setenv commands.
> > > > >
> > > > > One issue we have seen is that the env-file gets corrupted every now and
> > > > > then.
> > > > > After corruption we can't RAUC update. The only solution we have to this
> > > > > problem now is to delete the corrupted env-file and reboot, then we can
> > > > > perform the upgrade.
> > > > >
> > > > > I have no idea how to track down whatever corrupts the file and I was
> > > > > wondering if anyone has any input.
> > > >
> > > > You could try placing the environment on a separate partition to avoid any
> > > > potential issues in the FAT implementation. Also, I think U-Boot has a way
> > > > to
> > > > support redundant environments.
> >
> > I have just done this for our newer systems. I moved the GPT partitions back
> > 4MB and placed two redundant environments between the GPT and the first GPT
> > partition.
> >
> > It is my understanding though that redundant environments are not supported
> > when storing the env on FAT?
>
> That's probably a question for the U-Boot mailing list. :)
>
> > > Exactly. This should also be documented in the U-Boot integration guideline
> > > for eMMC:
> > >
> > >
> > > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card
> > >
> > > When writing to the FAT very short before hard rebooting, I could imagine
> > > this
> > > can lead to failures. Do you see the corruption only after updates, or also
> > > suddenly after n boots?
> >
> > Yes, this is something we have been able to test. If we cut the power
> > precisely when the env is written to FAT we can corrupt the entire boot
> > partition.
> > Super scary but this is not the problem we're seeing in the field. That
> > problem is more subtle.
>
> It should be possible to mount fat with the 'sync' option, but I'm not sure if
> that would help in this case. I'd recommend avoiding mounting FAT filesystems
> R/W if possible.

Maybe it could help with the problem I'm investigating. Don't think it would help with
the total corruption on powerloss when writing u-boot env, since that is in u-boot and
the fs is not "mounted" yet.

> > > How does the system report the corruption?
> >
> > fw_printenv and fw_setenv stops working and says that the env is corrupted.
> > That also means that RAUC update fails, that is usually when we notice it.
> >
> > Is there a way to watch a file and record any process that modifies it?
>
> There is blktrace, but you don't see the contents that way. It still may be
> enough detail to understand what's happening here.

Great, I'll check that out.

> Regards,
> Jan

Thanks for all the help.

Regards,
Einar


[-- Attachment #2: Type: text/html, Size: 4946 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [RAUC] [NEWSLETTER]Re: Robust u-boot environment with RAUC
  2021-03-28 10:11         ` Einar Vading
@ 2021-03-28 14:11           ` Matt Campbell
  2021-03-28 20:23             ` [RAUC] [NEWSLETTER]Re: " Einar Vading
  2021-03-29  5:14             ` Einar Vading
  0 siblings, 2 replies; 9+ messages in thread
From: Matt Campbell @ 2021-03-28 14:11 UTC (permalink / raw)
  To: Einar Vading; +Cc: Enrico Jörns, rauc, jlu

[-- Attachment #1: Type: text/plain, Size: 4794 bytes --]

Not a solution per say, but I can give you some info on how we solve the
reliability issue in our product that uses RAUC.

* We store the env at a raw offset in the eMMC (this should work for SD as
well) rather than on a FAT partition as a file. You will need to set your
partition table up to leave room for this and modify the U-Boot config.
* We use redundant u-boot environments placed in different sectors of the
eMMC. This is a built-in feature of U-Boot that can be enabled in the
config. If one gets corrupted it will fall back on the previous gracefully.
* We have custom code both in U-Boot and in Linux that checks for
corrupt or inconsistent RAUC U-Boot environment vars. If they are totally
out of whack we will boot into our fail-safe recovery mode where the evn
vars are reset to a sane default and an update can be performed (no RMA
needed).

Over the past year we've had this setup. I haven't once seen or heard of
actually hitting a corrupt U-Boot env in any of our development units. We
unfortunately don't have analytics around this event in the field.

I know this isn't exactly an answer to your question, but hopefully some of
this helps you arrive at a robust solution for your setup.

Best,
~Matt

On Sun, Mar 28, 2021 at 6:11 AM Einar Vading <Einar.Vading@rhimagnesita.com>
wrote:

> > Hi,
> >
> > On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote:
> > > > > Hi,
> > > > >
> > > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> > > > > > We have a Raspberry Pi 4 system set up using RAUC for updates
> and u-boot
> > > > > > for
> > > > > > booting. For some systems in the field we have the u-boot
> environment on
> > > > > > the
> > > > > > FAT boot partition and we mount that in fstab so that RAUC can
> access it
> > > > > > with
> > > > > > the fw_print/setenv commands.
> > > > > >
> > > > > > One issue we have seen is that the env-file gets corrupted every
> now and
> > > > > > then.
> > > > > > After corruption we can't RAUC update. The only solution we have
> to this
> > > > > > problem now is to delete the corrupted env-file and reboot, then
> we can
> > > > > > perform the upgrade.
> > > > > >
> > > > > > I have no idea how to track down whatever corrupts the file and
> I was
> > > > > > wondering if anyone has any input.
> > > > >
> > > > > You could try placing the environment on a separate partition to
> avoid any
> > > > > potential issues in the FAT implementation. Also, I think U-Boot
> has a way
> > > > > to
> > > > > support redundant environments.
> > >
> > > I have just done this for our newer systems. I moved the GPT
> partitions back
> > > 4MB and placed two redundant environments between the GPT and the
> first GPT
> > > partition.
> > >
> > > It is my understanding though that redundant environments are not
> supported
> > > when storing the env on FAT?
> >
> > That's probably a question for the U-Boot mailing list. :)
> >
> > > > Exactly. This should also be documented in the U-Boot integration
> guideline
> > > > for eMMC:
> > > >
> > > >
> > > >
> https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card
> > > >
> > > > When writing to the FAT very short before hard rebooting, I could
> imagine
> > > > this
> > > > can lead to failures. Do you see the corruption only after updates,
> or also
> > > > suddenly after n boots?
> > >
> > > Yes, this is something we have been able to test. If we cut the power
> > > precisely when the env is written to FAT we can corrupt the entire boot
> > > partition.
> > > Super scary but this is not the problem we're seeing in the field. That
> > > problem is more subtle.
> >
> > It should be possible to mount fat with the 'sync' option, but I'm not
> sure if
> > that would help in this case. I'd recommend avoiding mounting FAT
> filesystems
> > R/W if possible.
>
> Maybe it could help with the problem I'm investigating. Don't think it
> would help with
> the total corruption on powerloss when writing u-boot env, since that is
> in u-boot and
> the fs is not "mounted" yet.
>
> > > > How does the system report the corruption?
> > >
> > > fw_printenv and fw_setenv stops working and says that the env is
> corrupted.
> > > That also means that RAUC update fails, that is usually when we notice
> it.
> > >
> > > Is there a way to watch a file and record any process that modifies it?
> >
> > There is blktrace, but you don't see the contents that way. It still may
> be
> > enough detail to understand what's happening here.
>
> Great, I'll check that out.
>
> > Regards,
> > Jan
>
> Thanks for all the help.
>
> Regards,
> Einar
>
> _______________________________________________
> RAUC mailing list
>


-- 
Matthew Campbell
Principal Engineer
mcampbell@izotope.com

iZotope, Inc.
www.izotope.com

[-- Attachment #2: Type: text/html, Size: 7151 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [RAUC] [NEWSLETTER]Re: [NEWSLETTER]Re: Robust u-boot environment with RAUC
  2021-03-28 14:11           ` Matt Campbell
@ 2021-03-28 20:23             ` Einar Vading
  2021-03-29  5:14             ` Einar Vading
  1 sibling, 0 replies; 9+ messages in thread
From: Einar Vading @ 2021-03-28 20:23 UTC (permalink / raw)
  To: Matt Campbell; +Cc: rauc, Enrico Jörns, jlu


[-- Attachment #1.1: Type: text/plain, Size: 5111 bytes --]

Great information!

With the exception of the custom code, that is exactly how we are planning to set it up for our future systems.
It’s reassuring to head that it seems to work well for you.

Regards,
Einar

Not a solution per say, but I can give you some info on how we solve the reliability issue in our product that uses RAUC.

* We store the env at a raw offset in the eMMC (this should work for SD as well) rather than on a FAT partition as a file. You will need to set your partition table up to leave room for this and modify the U-Boot config.
* We use redundant u-boot environments placed in different sectors of the eMMC. This is a built-in feature of U-Boot that can be enabled in the config. If one gets corrupted it will fall back on the previous gracefully.
* We have custom code both in U-Boot and in Linux that checks for corrupt or inconsistent RAUC U-Boot environment vars. If they are totally out of whack we will boot into our fail-safe recovery mode where the evn vars are reset to a sane default and an update can be performed (no RMA needed).

Over the past year we've had this setup. I haven't once seen or heard of actually hitting a corrupt U-Boot env in any of our development units. We unfortunately don't have analytics around this event in the field.

I know this isn't exactly an answer to your question, but hopefully some of this helps you arrive at a robust solution for your setup.

Best,
~Matt

On Sun, Mar 28, 2021 at 6:11 AM Einar Vading <Einar.Vading@rhimagnesita.com<mailto:Einar.Vading@rhimagnesita.com>> wrote:
> Hi,
>
> On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote:
> > > > Hi,
> > > >
> > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> > > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot
> > > > > for
> > > > > booting. For some systems in the field we have the u-boot environment on
> > > > > the
> > > > > FAT boot partition and we mount that in fstab so that RAUC can access it
> > > > > with
> > > > > the fw_print/setenv commands.
> > > > >
> > > > > One issue we have seen is that the env-file gets corrupted every now and
> > > > > then.
> > > > > After corruption we can't RAUC update. The only solution we have to this
> > > > > problem now is to delete the corrupted env-file and reboot, then we can
> > > > > perform the upgrade.
> > > > >
> > > > > I have no idea how to track down whatever corrupts the file and I was
> > > > > wondering if anyone has any input.
> > > >
> > > > You could try placing the environment on a separate partition to avoid any
> > > > potential issues in the FAT implementation. Also, I think U-Boot has a way
> > > > to
> > > > support redundant environments.
> >
> > I have just done this for our newer systems. I moved the GPT partitions back
> > 4MB and placed two redundant environments between the GPT and the first GPT
> > partition.
> >
> > It is my understanding though that redundant environments are not supported
> > when storing the env on FAT?
>
> That's probably a question for the U-Boot mailing list. :)
>
> > > Exactly. This should also be documented in the U-Boot integration guideline
> > > for eMMC:
> > >
> > >
> > > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card<https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card>
> > >
> > > When writing to the FAT very short before hard rebooting, I could imagine
> > > this
> > > can lead to failures. Do you see the corruption only after updates, or also
> > > suddenly after n boots?
> >
> > Yes, this is something we have been able to test. If we cut the power
> > precisely when the env is written to FAT we can corrupt the entire boot
> > partition.
> > Super scary but this is not the problem we're seeing in the field. That
> > problem is more subtle.
>
> It should be possible to mount fat with the 'sync' option, but I'm not sure if
> that would help in this case. I'd recommend avoiding mounting FAT filesystems
> R/W if possible.

Maybe it could help with the problem I'm investigating. Don't think it would help with
the total corruption on powerloss when writing u-boot env, since that is in u-boot and
the fs is not "mounted" yet.

> > > How does the system report the corruption?
> >
> > fw_printenv and fw_setenv stops working and says that the env is corrupted.
> > That also means that RAUC update fails, that is usually when we notice it.
> >
> > Is there a way to watch a file and record any process that modifies it?
>
> There is blktrace, but you don't see the contents that way. It still may be
> enough detail to understand what's happening here.

Great, I'll check that out.

> Regards,
> Jan

Thanks for all the help.

Regards,
Einar

_______________________________________________
RAUC mailing list


--
Matthew Campbell
Principal Engineer
mcampbell@izotope.com<mailto:mcampbell@izotope.com>

iZotope, Inc.
www.izotope.com<http://www.izotope.com>

[-- Attachment #1.2: Type: text/html, Size: 58828 bytes --]

[-- Attachment #2: Type: text/plain, Size: 66 bytes --]

_______________________________________________
RAUC mailing list

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [RAUC] [NEWSLETTER]Re: [NEWSLETTER]Re: Robust u-boot environment with RAUC
  2021-03-28 14:11           ` Matt Campbell
  2021-03-28 20:23             ` [RAUC] [NEWSLETTER]Re: " Einar Vading
@ 2021-03-29  5:14             ` Einar Vading
  1 sibling, 0 replies; 9+ messages in thread
From: Einar Vading @ 2021-03-29  5:14 UTC (permalink / raw)
  To: Matt Campbell; +Cc: rauc, Enrico Jörns, jlu


[-- Attachment #1.1: Type: text/plain, Size: 5109 bytes --]

Great information!

With the exception of the custom code, that is exactly how we are planning to set it up for our future systems.
It’s reassuring to head that it seems to work well for you.

Regards,
Einar

Not a solution per say, but I can give you some info on how we solve the reliability issue in our product that uses RAUC.

* We store the env at a raw offset in the eMMC (this should work for SD as well) rather than on a FAT partition as a file. You will need to set your partition table up to leave room for this and modify the U-Boot config.
* We use redundant u-boot environments placed in different sectors of the eMMC. This is a built-in feature of U-Boot that can be enabled in the config. If one gets corrupted it will fall back on the previous gracefully.
* We have custom code both in U-Boot and in Linux that checks for corrupt or inconsistent RAUC U-Boot environment vars. If they are totally out of whack we will boot into our fail-safe recovery mode where the evn vars are reset to a sane default and an update can be performed (no RMA needed).

Over the past year we've had this setup. I haven't once seen or heard of actually hitting a corrupt U-Boot env in any of our development units. We unfortunately don't have analytics around this event in the field.

I know this isn't exactly an answer to your question, but hopefully some of this helps you arrive at a robust solution for your setup.

Best,
~Matt

On Sun, Mar 28, 2021 at 6:11 AM Einar Vading <Einar.Vading@rhimagnesita.com<mailto:Einar.Vading@rhimagnesita.com>> wrote:
> Hi,
>
> On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote:
> > > > Hi,
> > > >
> > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> > > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot
> > > > > for
> > > > > booting. For some systems in the field we have the u-boot environment on
> > > > > the
> > > > > FAT boot partition and we mount that in fstab so that RAUC can access it
> > > > > with
> > > > > the fw_print/setenv commands.
> > > > >
> > > > > One issue we have seen is that the env-file gets corrupted every now and
> > > > > then.
> > > > > After corruption we can't RAUC update. The only solution we have to this
> > > > > problem now is to delete the corrupted env-file and reboot, then we can
> > > > > perform the upgrade.
> > > > >
> > > > > I have no idea how to track down whatever corrupts the file and I was
> > > > > wondering if anyone has any input.
> > > >
> > > > You could try placing the environment on a separate partition to avoid any
> > > > potential issues in the FAT implementation. Also, I think U-Boot has a way
> > > > to
> > > > support redundant environments.
> >
> > I have just done this for our newer systems. I moved the GPT partitions back
> > 4MB and placed two redundant environments between the GPT and the first GPT
> > partition.
> >
> > It is my understanding though that redundant environments are not supported
> > when storing the env on FAT?
>
> That's probably a question for the U-Boot mailing list. :)
>
> > > Exactly. This should also be documented in the U-Boot integration guideline
> > > for eMMC:
> > >
> > >
> > > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card<https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card>
> > >
> > > When writing to the FAT very short before hard rebooting, I could imagine
> > > this
> > > can lead to failures. Do you see the corruption only after updates, or also
> > > suddenly after n boots?
> >
> > Yes, this is something we have been able to test. If we cut the power
> > precisely when the env is written to FAT we can corrupt the entire boot
> > partition.
> > Super scary but this is not the problem we're seeing in the field. That
> > problem is more subtle.
>
> It should be possible to mount fat with the 'sync' option, but I'm not sure if
> that would help in this case. I'd recommend avoiding mounting FAT filesystems
> R/W if possible.

Maybe it could help with the problem I'm investigating. Don't think it would help with
the total corruption on powerloss when writing u-boot env, since that is in u-boot and
the fs is not "mounted" yet.

> > > How does the system report the corruption?
> >
> > fw_printenv and fw_setenv stops working and says that the env is corrupted.
> > That also means that RAUC update fails, that is usually when we notice it.
> >
> > Is there a way to watch a file and record any process that modifies it?
>
> There is blktrace, but you don't see the contents that way. It still may be
> enough detail to understand what's happening here.

Great, I'll check that out.

> Regards,
> Jan

Thanks for all the help.

Regards,
Einar

_______________________________________________
RAUC mailing list


--
Matthew Campbell
Principal Engineer
mcampbell@izotope.com<mailto:mcampbell@izotope.com>

iZotope, Inc.
www.izotope.com<http://www.izotope.com>

[-- Attachment #1.2: Type: text/html, Size: 31734 bytes --]

[-- Attachment #2: Type: text/plain, Size: 66 bytes --]

_______________________________________________
RAUC mailing list

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-03-29  5:14 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-25 15:22 [RAUC] Robust u-boot environment with RAUC Einar Vading
2021-03-25 15:52 ` Jan Lübbe
2021-03-25 18:54   ` Enrico Jörns
2021-03-26  5:48     ` [RAUC] [NEWSLETTER]Re: " Einar Vading
2021-03-26  9:13       ` Jan Lübbe
2021-03-28 10:11         ` Einar Vading
2021-03-28 14:11           ` Matt Campbell
2021-03-28 20:23             ` [RAUC] [NEWSLETTER]Re: " Einar Vading
2021-03-29  5:14             ` Einar Vading

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox