* [RAUC] Robust u-boot environment with RAUC @ 2021-03-25 15:22 Einar Vading 2021-03-25 15:52 ` Jan Lübbe 0 siblings, 1 reply; 9+ messages in thread From: Einar Vading @ 2021-03-25 15:22 UTC (permalink / raw) To: rauc [-- Attachment #1.1.1: Type: text/plain, Size: 948 bytes --] Hi, We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot for booting. For some systems in the field we have the u-boot environment on the FAT boot partition and we mount that in fstab so that RAUC can access it with the fw_print/setenv commands. One issue we have seen is that the env-file gets corrupted every now and then. After corruption we can't RAUC update. The only solution we have to this problem now is to delete the corrupted env-file and reboot, then we can perform the upgrade. I have no idea how to track down whatever corrupts the file and I was wondering if anyone has any input. Vänliga hälsningar / Best Regards Einar Vading Software Developer AGELLIS AB [cid:08b60f59-61a0-41f4-8a02-48d7f47c2730] T +46 73 420 76 81 Agellis Group AB Tellusgatan 15, SE-224 57, Lund, Sweden www.agellis.com<http://www.agellis.com/>; www.rhimagnesita.com<http://www.rhimagnesita.com/> [-- Attachment #1.1.2: Type: text/html, Size: 4543 bytes --] [-- Attachment #1.2: Outlook-ggtizjsb.png --] [-- Type: image/png, Size: 10716 bytes --] [-- Attachment #2: Type: text/plain, Size: 66 bytes --] _______________________________________________ RAUC mailing list ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RAUC] Robust u-boot environment with RAUC 2021-03-25 15:22 [RAUC] Robust u-boot environment with RAUC Einar Vading @ 2021-03-25 15:52 ` Jan Lübbe 2021-03-25 18:54 ` Enrico Jörns 0 siblings, 1 reply; 9+ messages in thread From: Jan Lübbe @ 2021-03-25 15:52 UTC (permalink / raw) To: Einar Vading, rauc Hi, On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote: > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot for > booting. For some systems in the field we have the u-boot environment on the > FAT boot partition and we mount that in fstab so that RAUC can access it with > the fw_print/setenv commands. > > One issue we have seen is that the env-file gets corrupted every now and then. > After corruption we can't RAUC update. The only solution we have to this > problem now is to delete the corrupted env-file and reboot, then we can > perform the upgrade. > > I have no idea how to track down whatever corrupts the file and I was > wondering if anyone has any input. You could try placing the environment on a separate partition to avoid any potential issues in the FAT implementation. Also, I think U-Boot has a way to support redundant environments. Regards, Jan -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | _______________________________________________ RAUC mailing list ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RAUC] Robust u-boot environment with RAUC 2021-03-25 15:52 ` Jan Lübbe @ 2021-03-25 18:54 ` Enrico Jörns 2021-03-26 5:48 ` [RAUC] [NEWSLETTER]Re: " Einar Vading 0 siblings, 1 reply; 9+ messages in thread From: Enrico Jörns @ 2021-03-25 18:54 UTC (permalink / raw) To: Jan Lübbe, Einar Vading, rauc Am Donnerstag, dem 25.03.2021 um 16:52 +0100 schrieb Jan Lübbe: > Hi, > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote: > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot for > > booting. For some systems in the field we have the u-boot environment on the > > FAT boot partition and we mount that in fstab so that RAUC can access it with > > the fw_print/setenv commands. > > > > One issue we have seen is that the env-file gets corrupted every now and then. > > After corruption we can't RAUC update. The only solution we have to this > > problem now is to delete the corrupted env-file and reboot, then we can > > perform the upgrade. > > > > I have no idea how to track down whatever corrupts the file and I was > > wondering if anyone has any input. > > You could try placing the environment on a separate partition to avoid any > potential issues in the FAT implementation. Also, I think U-Boot has a way to > support redundant environments. Exactly. This should also be documented in the U-Boot integration guideline for eMMC: https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card When writing to the FAT very short before hard rebooting, I could imagine this can lead to failures. Do you see the corruption only after updates, or also suddenly after n boots? How does the system report the corruption? Regards, Enrico > Regards, > Jan -- Pengutronix e.K. | Enrico Jörns | Embedded Linux Consulting & Support | https://www.pengutronix.de/ | Steuerwalder Str. 21 | Phone: +49-5121-206917-180 | 31137 Hildesheim, Germany | Fax: +49-5121-206917-9 | _______________________________________________ RAUC mailing list ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RAUC] [NEWSLETTER]Re: Robust u-boot environment with RAUC 2021-03-25 18:54 ` Enrico Jörns @ 2021-03-26 5:48 ` Einar Vading 2021-03-26 9:13 ` Jan Lübbe 0 siblings, 1 reply; 9+ messages in thread From: Einar Vading @ 2021-03-26 5:48 UTC (permalink / raw) To: Enrico Jörns, Jan Lübbe, rauc [-- Attachment #1.1: Type: text/plain, Size: 2262 bytes --] > > Hi, > > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote: > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot for > > > booting. For some systems in the field we have the u-boot environment on the > > > FAT boot partition and we mount that in fstab so that RAUC can access it with > > > the fw_print/setenv commands. > > > > > > One issue we have seen is that the env-file gets corrupted every now and then. > > > After corruption we can't RAUC update. The only solution we have to this > > > problem now is to delete the corrupted env-file and reboot, then we can > > > perform the upgrade. > > > > > > I have no idea how to track down whatever corrupts the file and I was > > > wondering if anyone has any input. > > > > You could try placing the environment on a separate partition to avoid any > > potential issues in the FAT implementation. Also, I think U-Boot has a way to > > support redundant environments. I have just done this for our newer systems. I moved the GPT partitions back 4MB and placed two redundant environments between the GPT and the first GPT partition. It is my understanding though that redundant environments are not supported when storing the env on FAT? > Exactly. This should also be documented in the U-Boot integration guideline for > eMMC: > > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card > > When writing to the FAT very short before hard rebooting, I could imagine this > can lead to failures. Do you see the corruption only after updates, or also > suddenly after n boots? Yes, this is something we have been able to test. If we cut the power precisely when the env is written to FAT we can corrupt the entire boot partition. Super scary but this is not the problem we're seeing in the field. That problem is more subtle. > How does the system report the corruption? fw_printenv and fw_setenv stops working and says that the env is corrupted. That also means that RAUC update fails, that is usually when we notice it. Is there a way to watch a file and record any process that modifies it? > > Regards, Enrico > > > Regards, > > Jan > Best Regards, Einar [-- Attachment #1.2: Type: text/html, Size: 3352 bytes --] [-- Attachment #2: Type: text/plain, Size: 66 bytes --] _______________________________________________ RAUC mailing list ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RAUC] [NEWSLETTER]Re: Robust u-boot environment with RAUC 2021-03-26 5:48 ` [RAUC] [NEWSLETTER]Re: " Einar Vading @ 2021-03-26 9:13 ` Jan Lübbe 2021-03-28 10:11 ` Einar Vading 0 siblings, 1 reply; 9+ messages in thread From: Jan Lübbe @ 2021-03-26 9:13 UTC (permalink / raw) To: Einar Vading, Enrico Jörns, rauc Hi, On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote: > > > Hi, > > > > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote: > > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot > > > > for > > > > booting. For some systems in the field we have the u-boot environment on > > > > the > > > > FAT boot partition and we mount that in fstab so that RAUC can access it > > > > with > > > > the fw_print/setenv commands. > > > > > > > > One issue we have seen is that the env-file gets corrupted every now and > > > > then. > > > > After corruption we can't RAUC update. The only solution we have to this > > > > problem now is to delete the corrupted env-file and reboot, then we can > > > > perform the upgrade. > > > > > > > > I have no idea how to track down whatever corrupts the file and I was > > > > wondering if anyone has any input. > > > > > > You could try placing the environment on a separate partition to avoid any > > > potential issues in the FAT implementation. Also, I think U-Boot has a way > > > to > > > support redundant environments. > > I have just done this for our newer systems. I moved the GPT partitions back > 4MB and placed two redundant environments between the GPT and the first GPT > partition. > > It is my understanding though that redundant environments are not supported > when storing the env on FAT? That's probably a question for the U-Boot mailing list. :) > > Exactly. This should also be documented in the U-Boot integration guideline > > for eMMC: > > > > > > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card > > > > When writing to the FAT very short before hard rebooting, I could imagine > > this > > can lead to failures. Do you see the corruption only after updates, or also > > suddenly after n boots? > > Yes, this is something we have been able to test. If we cut the power > precisely when the env is written to FAT we can corrupt the entire boot > partition. > Super scary but this is not the problem we're seeing in the field. That > problem is more subtle. It should be possible to mount fat with the 'sync' option, but I'm not sure if that would help in this case. I'd recommend avoiding mounting FAT filesystems R/W if possible. > > How does the system report the corruption? > > fw_printenv and fw_setenv stops working and says that the env is corrupted. > That also means that RAUC update fails, that is usually when we notice it. > > Is there a way to watch a file and record any process that modifies it? There is blktrace, but you don't see the contents that way. It still may be enough detail to understand what's happening here. Regards, Jan -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | _______________________________________________ RAUC mailing list ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RAUC] [NEWSLETTER]Re: Robust u-boot environment with RAUC 2021-03-26 9:13 ` Jan Lübbe @ 2021-03-28 10:11 ` Einar Vading 2021-03-28 14:11 ` Matt Campbell 0 siblings, 1 reply; 9+ messages in thread From: Einar Vading @ 2021-03-28 10:11 UTC (permalink / raw) To: Enrico Jörns, rauc, jlu [-- Attachment #1: Type: text/plain, Size: 3198 bytes --] > Hi, > > On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote: > > > > Hi, > > > > > > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote: > > > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot > > > > > for > > > > > booting. For some systems in the field we have the u-boot environment on > > > > > the > > > > > FAT boot partition and we mount that in fstab so that RAUC can access it > > > > > with > > > > > the fw_print/setenv commands. > > > > > > > > > > One issue we have seen is that the env-file gets corrupted every now and > > > > > then. > > > > > After corruption we can't RAUC update. The only solution we have to this > > > > > problem now is to delete the corrupted env-file and reboot, then we can > > > > > perform the upgrade. > > > > > > > > > > I have no idea how to track down whatever corrupts the file and I was > > > > > wondering if anyone has any input. > > > > > > > > You could try placing the environment on a separate partition to avoid any > > > > potential issues in the FAT implementation. Also, I think U-Boot has a way > > > > to > > > > support redundant environments. > > > > I have just done this for our newer systems. I moved the GPT partitions back > > 4MB and placed two redundant environments between the GPT and the first GPT > > partition. > > > > It is my understanding though that redundant environments are not supported > > when storing the env on FAT? > > That's probably a question for the U-Boot mailing list. :) > > > > Exactly. This should also be documented in the U-Boot integration guideline > > > for eMMC: > > > > > > > > > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card > > > > > > When writing to the FAT very short before hard rebooting, I could imagine > > > this > > > can lead to failures. Do you see the corruption only after updates, or also > > > suddenly after n boots? > > > > Yes, this is something we have been able to test. If we cut the power > > precisely when the env is written to FAT we can corrupt the entire boot > > partition. > > Super scary but this is not the problem we're seeing in the field. That > > problem is more subtle. > > It should be possible to mount fat with the 'sync' option, but I'm not sure if > that would help in this case. I'd recommend avoiding mounting FAT filesystems > R/W if possible. Maybe it could help with the problem I'm investigating. Don't think it would help with the total corruption on powerloss when writing u-boot env, since that is in u-boot and the fs is not "mounted" yet. > > > How does the system report the corruption? > > > > fw_printenv and fw_setenv stops working and says that the env is corrupted. > > That also means that RAUC update fails, that is usually when we notice it. > > > > Is there a way to watch a file and record any process that modifies it? > > There is blktrace, but you don't see the contents that way. It still may be > enough detail to understand what's happening here. Great, I'll check that out. > Regards, > Jan Thanks for all the help. Regards, Einar [-- Attachment #2: Type: text/html, Size: 4946 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RAUC] [NEWSLETTER]Re: Robust u-boot environment with RAUC 2021-03-28 10:11 ` Einar Vading @ 2021-03-28 14:11 ` Matt Campbell 2021-03-28 20:23 ` [RAUC] [NEWSLETTER]Re: " Einar Vading 2021-03-29 5:14 ` Einar Vading 0 siblings, 2 replies; 9+ messages in thread From: Matt Campbell @ 2021-03-28 14:11 UTC (permalink / raw) To: Einar Vading; +Cc: Enrico Jörns, rauc, jlu [-- Attachment #1: Type: text/plain, Size: 4794 bytes --] Not a solution per say, but I can give you some info on how we solve the reliability issue in our product that uses RAUC. * We store the env at a raw offset in the eMMC (this should work for SD as well) rather than on a FAT partition as a file. You will need to set your partition table up to leave room for this and modify the U-Boot config. * We use redundant u-boot environments placed in different sectors of the eMMC. This is a built-in feature of U-Boot that can be enabled in the config. If one gets corrupted it will fall back on the previous gracefully. * We have custom code both in U-Boot and in Linux that checks for corrupt or inconsistent RAUC U-Boot environment vars. If they are totally out of whack we will boot into our fail-safe recovery mode where the evn vars are reset to a sane default and an update can be performed (no RMA needed). Over the past year we've had this setup. I haven't once seen or heard of actually hitting a corrupt U-Boot env in any of our development units. We unfortunately don't have analytics around this event in the field. I know this isn't exactly an answer to your question, but hopefully some of this helps you arrive at a robust solution for your setup. Best, ~Matt On Sun, Mar 28, 2021 at 6:11 AM Einar Vading <Einar.Vading@rhimagnesita.com> wrote: > > Hi, > > > > On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote: > > > > > Hi, > > > > > > > > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote: > > > > > > We have a Raspberry Pi 4 system set up using RAUC for updates > and u-boot > > > > > > for > > > > > > booting. For some systems in the field we have the u-boot > environment on > > > > > > the > > > > > > FAT boot partition and we mount that in fstab so that RAUC can > access it > > > > > > with > > > > > > the fw_print/setenv commands. > > > > > > > > > > > > One issue we have seen is that the env-file gets corrupted every > now and > > > > > > then. > > > > > > After corruption we can't RAUC update. The only solution we have > to this > > > > > > problem now is to delete the corrupted env-file and reboot, then > we can > > > > > > perform the upgrade. > > > > > > > > > > > > I have no idea how to track down whatever corrupts the file and > I was > > > > > > wondering if anyone has any input. > > > > > > > > > > You could try placing the environment on a separate partition to > avoid any > > > > > potential issues in the FAT implementation. Also, I think U-Boot > has a way > > > > > to > > > > > support redundant environments. > > > > > > I have just done this for our newer systems. I moved the GPT > partitions back > > > 4MB and placed two redundant environments between the GPT and the > first GPT > > > partition. > > > > > > It is my understanding though that redundant environments are not > supported > > > when storing the env on FAT? > > > > That's probably a question for the U-Boot mailing list. :) > > > > > > Exactly. This should also be documented in the U-Boot integration > guideline > > > > for eMMC: > > > > > > > > > > > > > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card > > > > > > > > When writing to the FAT very short before hard rebooting, I could > imagine > > > > this > > > > can lead to failures. Do you see the corruption only after updates, > or also > > > > suddenly after n boots? > > > > > > Yes, this is something we have been able to test. If we cut the power > > > precisely when the env is written to FAT we can corrupt the entire boot > > > partition. > > > Super scary but this is not the problem we're seeing in the field. That > > > problem is more subtle. > > > > It should be possible to mount fat with the 'sync' option, but I'm not > sure if > > that would help in this case. I'd recommend avoiding mounting FAT > filesystems > > R/W if possible. > > Maybe it could help with the problem I'm investigating. Don't think it > would help with > the total corruption on powerloss when writing u-boot env, since that is > in u-boot and > the fs is not "mounted" yet. > > > > > How does the system report the corruption? > > > > > > fw_printenv and fw_setenv stops working and says that the env is > corrupted. > > > That also means that RAUC update fails, that is usually when we notice > it. > > > > > > Is there a way to watch a file and record any process that modifies it? > > > > There is blktrace, but you don't see the contents that way. It still may > be > > enough detail to understand what's happening here. > > Great, I'll check that out. > > > Regards, > > Jan > > Thanks for all the help. > > Regards, > Einar > > _______________________________________________ > RAUC mailing list > -- Matthew Campbell Principal Engineer mcampbell@izotope.com iZotope, Inc. www.izotope.com [-- Attachment #2: Type: text/html, Size: 7151 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RAUC] [NEWSLETTER]Re: [NEWSLETTER]Re: Robust u-boot environment with RAUC 2021-03-28 14:11 ` Matt Campbell @ 2021-03-28 20:23 ` Einar Vading 2021-03-29 5:14 ` Einar Vading 1 sibling, 0 replies; 9+ messages in thread From: Einar Vading @ 2021-03-28 20:23 UTC (permalink / raw) To: Matt Campbell; +Cc: rauc, Enrico Jörns, jlu [-- Attachment #1.1: Type: text/plain, Size: 5111 bytes --] Great information! With the exception of the custom code, that is exactly how we are planning to set it up for our future systems. It’s reassuring to head that it seems to work well for you. Regards, Einar Not a solution per say, but I can give you some info on how we solve the reliability issue in our product that uses RAUC. * We store the env at a raw offset in the eMMC (this should work for SD as well) rather than on a FAT partition as a file. You will need to set your partition table up to leave room for this and modify the U-Boot config. * We use redundant u-boot environments placed in different sectors of the eMMC. This is a built-in feature of U-Boot that can be enabled in the config. If one gets corrupted it will fall back on the previous gracefully. * We have custom code both in U-Boot and in Linux that checks for corrupt or inconsistent RAUC U-Boot environment vars. If they are totally out of whack we will boot into our fail-safe recovery mode where the evn vars are reset to a sane default and an update can be performed (no RMA needed). Over the past year we've had this setup. I haven't once seen or heard of actually hitting a corrupt U-Boot env in any of our development units. We unfortunately don't have analytics around this event in the field. I know this isn't exactly an answer to your question, but hopefully some of this helps you arrive at a robust solution for your setup. Best, ~Matt On Sun, Mar 28, 2021 at 6:11 AM Einar Vading <Einar.Vading@rhimagnesita.com<mailto:Einar.Vading@rhimagnesita.com>> wrote: > Hi, > > On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote: > > > > Hi, > > > > > > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote: > > > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot > > > > > for > > > > > booting. For some systems in the field we have the u-boot environment on > > > > > the > > > > > FAT boot partition and we mount that in fstab so that RAUC can access it > > > > > with > > > > > the fw_print/setenv commands. > > > > > > > > > > One issue we have seen is that the env-file gets corrupted every now and > > > > > then. > > > > > After corruption we can't RAUC update. The only solution we have to this > > > > > problem now is to delete the corrupted env-file and reboot, then we can > > > > > perform the upgrade. > > > > > > > > > > I have no idea how to track down whatever corrupts the file and I was > > > > > wondering if anyone has any input. > > > > > > > > You could try placing the environment on a separate partition to avoid any > > > > potential issues in the FAT implementation. Also, I think U-Boot has a way > > > > to > > > > support redundant environments. > > > > I have just done this for our newer systems. I moved the GPT partitions back > > 4MB and placed two redundant environments between the GPT and the first GPT > > partition. > > > > It is my understanding though that redundant environments are not supported > > when storing the env on FAT? > > That's probably a question for the U-Boot mailing list. :) > > > > Exactly. This should also be documented in the U-Boot integration guideline > > > for eMMC: > > > > > > > > > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card<https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card> > > > > > > When writing to the FAT very short before hard rebooting, I could imagine > > > this > > > can lead to failures. Do you see the corruption only after updates, or also > > > suddenly after n boots? > > > > Yes, this is something we have been able to test. If we cut the power > > precisely when the env is written to FAT we can corrupt the entire boot > > partition. > > Super scary but this is not the problem we're seeing in the field. That > > problem is more subtle. > > It should be possible to mount fat with the 'sync' option, but I'm not sure if > that would help in this case. I'd recommend avoiding mounting FAT filesystems > R/W if possible. Maybe it could help with the problem I'm investigating. Don't think it would help with the total corruption on powerloss when writing u-boot env, since that is in u-boot and the fs is not "mounted" yet. > > > How does the system report the corruption? > > > > fw_printenv and fw_setenv stops working and says that the env is corrupted. > > That also means that RAUC update fails, that is usually when we notice it. > > > > Is there a way to watch a file and record any process that modifies it? > > There is blktrace, but you don't see the contents that way. It still may be > enough detail to understand what's happening here. Great, I'll check that out. > Regards, > Jan Thanks for all the help. Regards, Einar _______________________________________________ RAUC mailing list -- Matthew Campbell Principal Engineer mcampbell@izotope.com<mailto:mcampbell@izotope.com> iZotope, Inc. www.izotope.com<http://www.izotope.com> [-- Attachment #1.2: Type: text/html, Size: 58828 bytes --] [-- Attachment #2: Type: text/plain, Size: 66 bytes --] _______________________________________________ RAUC mailing list ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RAUC] [NEWSLETTER]Re: [NEWSLETTER]Re: Robust u-boot environment with RAUC 2021-03-28 14:11 ` Matt Campbell 2021-03-28 20:23 ` [RAUC] [NEWSLETTER]Re: " Einar Vading @ 2021-03-29 5:14 ` Einar Vading 1 sibling, 0 replies; 9+ messages in thread From: Einar Vading @ 2021-03-29 5:14 UTC (permalink / raw) To: Matt Campbell; +Cc: rauc, Enrico Jörns, jlu [-- Attachment #1.1: Type: text/plain, Size: 5109 bytes --] Great information! With the exception of the custom code, that is exactly how we are planning to set it up for our future systems. It’s reassuring to head that it seems to work well for you. Regards, Einar Not a solution per say, but I can give you some info on how we solve the reliability issue in our product that uses RAUC. * We store the env at a raw offset in the eMMC (this should work for SD as well) rather than on a FAT partition as a file. You will need to set your partition table up to leave room for this and modify the U-Boot config. * We use redundant u-boot environments placed in different sectors of the eMMC. This is a built-in feature of U-Boot that can be enabled in the config. If one gets corrupted it will fall back on the previous gracefully. * We have custom code both in U-Boot and in Linux that checks for corrupt or inconsistent RAUC U-Boot environment vars. If they are totally out of whack we will boot into our fail-safe recovery mode where the evn vars are reset to a sane default and an update can be performed (no RMA needed). Over the past year we've had this setup. I haven't once seen or heard of actually hitting a corrupt U-Boot env in any of our development units. We unfortunately don't have analytics around this event in the field. I know this isn't exactly an answer to your question, but hopefully some of this helps you arrive at a robust solution for your setup. Best, ~Matt On Sun, Mar 28, 2021 at 6:11 AM Einar Vading <Einar.Vading@rhimagnesita.com<mailto:Einar.Vading@rhimagnesita.com>> wrote: > Hi, > > On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote: > > > > Hi, > > > > > > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote: > > > > > We have a Raspberry Pi 4 system set up using RAUC for updates and u-boot > > > > > for > > > > > booting. For some systems in the field we have the u-boot environment on > > > > > the > > > > > FAT boot partition and we mount that in fstab so that RAUC can access it > > > > > with > > > > > the fw_print/setenv commands. > > > > > > > > > > One issue we have seen is that the env-file gets corrupted every now and > > > > > then. > > > > > After corruption we can't RAUC update. The only solution we have to this > > > > > problem now is to delete the corrupted env-file and reboot, then we can > > > > > perform the upgrade. > > > > > > > > > > I have no idea how to track down whatever corrupts the file and I was > > > > > wondering if anyone has any input. > > > > > > > > You could try placing the environment on a separate partition to avoid any > > > > potential issues in the FAT implementation. Also, I think U-Boot has a way > > > > to > > > > support redundant environments. > > > > I have just done this for our newer systems. I moved the GPT partitions back > > 4MB and placed two redundant environments between the GPT and the first GPT > > partition. > > > > It is my understanding though that redundant environments are not supported > > when storing the env on FAT? > > That's probably a question for the U-Boot mailing list. :) > > > > Exactly. This should also be documented in the U-Boot integration guideline > > > for eMMC: > > > > > > > > > https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card<https://rauc.readthedocs.io/en/latest/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card> > > > > > > When writing to the FAT very short before hard rebooting, I could imagine > > > this > > > can lead to failures. Do you see the corruption only after updates, or also > > > suddenly after n boots? > > > > Yes, this is something we have been able to test. If we cut the power > > precisely when the env is written to FAT we can corrupt the entire boot > > partition. > > Super scary but this is not the problem we're seeing in the field. That > > problem is more subtle. > > It should be possible to mount fat with the 'sync' option, but I'm not sure if > that would help in this case. I'd recommend avoiding mounting FAT filesystems > R/W if possible. Maybe it could help with the problem I'm investigating. Don't think it would help with the total corruption on powerloss when writing u-boot env, since that is in u-boot and the fs is not "mounted" yet. > > > How does the system report the corruption? > > > > fw_printenv and fw_setenv stops working and says that the env is corrupted. > > That also means that RAUC update fails, that is usually when we notice it. > > > > Is there a way to watch a file and record any process that modifies it? > > There is blktrace, but you don't see the contents that way. It still may be > enough detail to understand what's happening here. Great, I'll check that out. > Regards, > Jan Thanks for all the help. Regards, Einar _______________________________________________ RAUC mailing list -- Matthew Campbell Principal Engineer mcampbell@izotope.com<mailto:mcampbell@izotope.com> iZotope, Inc. www.izotope.com<http://www.izotope.com> [-- Attachment #1.2: Type: text/html, Size: 31734 bytes --] [-- Attachment #2: Type: text/plain, Size: 66 bytes --] _______________________________________________ RAUC mailing list ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-03-29 5:14 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-03-25 15:22 [RAUC] Robust u-boot environment with RAUC Einar Vading 2021-03-25 15:52 ` Jan Lübbe 2021-03-25 18:54 ` Enrico Jörns 2021-03-26 5:48 ` [RAUC] [NEWSLETTER]Re: " Einar Vading 2021-03-26 9:13 ` Jan Lübbe 2021-03-28 10:11 ` Einar Vading 2021-03-28 14:11 ` Matt Campbell 2021-03-28 20:23 ` [RAUC] [NEWSLETTER]Re: " Einar Vading 2021-03-29 5:14 ` Einar Vading
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox