From mboxrd@z Thu Jan  1 00:00:00 1970
Return-path: <rauc-bounces@pengutronix.de>
Received: from de-smtp-delivery-102.mimecast.com ([62.140.7.102])
 by metis.ext.pengutronix.de with esmtps
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92)
 (envelope-from <einar.vading@rhimagnesita.com>) id 1lQkEO-0003mN-Cw
 for rauc@pengutronix.de; Mon, 29 Mar 2021 07:14:30 +0200
From: Einar Vading <Einar.Vading@RHIMagnesita.com>
Date: Mon, 29 Mar 2021 05:14:22 +0000
Message-ID: <AM0PR08MB4580282B727C41779681EB73E57F9@AM0PR08MB4580.eurprd08.prod.outlook.com>
References: <AM0PR08MB45802E04BF39F513E949C4D8E5629@AM0PR08MB4580.eurprd08.prod.outlook.com>
 <7a2fc0a9cb6bb54455d4cb69403a469e2fe832d8.camel@pengutronix.de>
 <fe91873602e932167c74ce59eef88c0c2b4ab0a1.camel@pengutronix.de>
 <AM0PR08MB4580CB2D01A5F69F0BD8112AE5619@AM0PR08MB4580.eurprd08.prod.outlook.com>
 <d0d058457830ced60923b7b79f80b13ff3c6189e.camel@pengutronix.de>
 <AM0PR08MB45809490AB5920AE54AB4540E57F9@AM0PR08MB4580.eurprd08.prod.outlook.com>
 <CAMDowVV00enD8Pc8NCJ1Ku=D8gZDbxojgaCUy_6jqBBxKtBYQQ@mail.gmail.com>
In-Reply-To: <CAMDowVV00enD8Pc8NCJ1Ku=D8gZDbxojgaCUy_6jqBBxKtBYQQ@mail.gmail.com>
MIME-Version: 1.0
Content-Language: en-US
Subject: Re: [RAUC] [NEWSLETTER]Re: [NEWSLETTER]Re: Robust u-boot
 environment with RAUC
List-Id: RAUC Project - Discussion List <rauc.pengutronix.de>
List-Unsubscribe: <http://metis.pengutronix.de/cgi-bin/mailman/options/rauc>, 
 <mailto:rauc-request@pengutronix.de?subject=unsubscribe>
List-Archive: <http://metis.pengutronix.de/cgi-bin/mailman/private/rauc/>
List-Post: <mailto:rauc@pengutronix.de>
List-Help: <mailto:rauc-request@pengutronix.de?subject=help>
List-Subscribe: <http://metis.pengutronix.de/cgi-bin/mailman/listinfo/rauc>,
 <mailto:rauc-request@pengutronix.de?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============0592601626=="
Errors-To: rauc-bounces@pengutronix.de
Sender: "RAUC" <rauc-bounces@pengutronix.de>
To: Matt Campbell <mcampbell@izotope.com>
Cc: "rauc@pengutronix.de" <rauc@pengutronix.de>, =?Windows-1252?Q?Enrico_J=F6rns?= <ejo@pengutronix.de>, "jlu@pengutronix.de" <jlu@pengutronix.de>

--===============0592601626==
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_AM0PR08MB4580282B727C41779681EB73E57F9AM0PR08MB4580eurp_"

--_000_AM0PR08MB4580282B727C41779681EB73E57F9AM0PR08MB4580eurp_
Content-Type: text/plain; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable

Great information!

With the exception of the custom code, that is exactly how we are planning =
to set it up for our future systems.
It=92s reassuring to head that it seems to work well for you.

Regards,
Einar

Not a solution per say, but I can give you some info on how we solve the re=
liability issue in our product that uses RAUC.

* We store the env at a raw offset in the eMMC (this should work for SD as =
well) rather than on a FAT partition as a file. You will need to set your p=
artition table up to leave room for this and modify the U-Boot config.
* We use redundant u-boot environments placed in different sectors of the e=
MMC. This is a built-in feature of U-Boot that can be enabled in the config=
. If one gets corrupted it will fall back on the previous gracefully.
* We have custom code both in U-Boot and in Linux that checks for corrupt o=
r inconsistent RAUC U-Boot environment vars. If they are totally out of wha=
ck we will boot into our fail-safe recovery mode where the evn vars are res=
et to a sane default and an update can be performed (no RMA needed).

Over the past year we've had this setup. I haven't once seen or heard of ac=
tually hitting a corrupt U-Boot env in any of our development units. We unf=
ortunately don't have analytics around this event in the field.

I know this isn't exactly an answer to your question, but hopefully some of=
 this helps you arrive at a robust solution for your setup.

Best,
~Matt

On Sun, Mar 28, 2021 at 6:11 AM Einar Vading <Einar.Vading@rhimagnesita.com=
<mailto:Einar.Vading@rhimagnesita.com>> wrote:
> Hi,
>
> On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote:
> > > > Hi,
> > > >
> > > > On Thu, 2021-03-25 at 15:22 +0000, Einar Vading wrote:
> > > > > We have a Raspberry Pi 4 system set up using RAUC for updates and=
 u-boot
> > > > > for
> > > > > booting. For some systems in the field we have the u-boot environ=
ment on
> > > > > the
> > > > > FAT boot partition and we mount that in fstab so that RAUC can ac=
cess it
> > > > > with
> > > > > the fw_print/setenv commands.
> > > > >
> > > > > One issue we have seen is that the env-file gets corrupted every =
now and
> > > > > then.
> > > > > After corruption we can't RAUC update. The only solution we have =
to this
> > > > > problem now is to delete the corrupted env-file and reboot, then =
we can
> > > > > perform the upgrade.
> > > > >
> > > > > I have no idea how to track down whatever corrupts the file and I=
 was
> > > > > wondering if anyone has any input.
> > > >
> > > > You could try placing the environment on a separate partition to av=
oid any
> > > > potential issues in the FAT implementation. Also, I think U-Boot ha=
s a way
> > > > to
> > > > support redundant environments.
> >
> > I have just done this for our newer systems. I moved the GPT partitions=
 back
> > 4MB and placed two redundant environments between the GPT and the first=
 GPT
> > partition.
> >
> > It is my understanding though that redundant environments are not suppo=
rted
> > when storing the env on FAT?
>
> That's probably a question for the U-Boot mailing list. :)
>
> > > Exactly. This should also be documented in the U-Boot integration gui=
deline
> > > for eMMC:
> > >
> > >
> > > https://rauc.readthedocs.io/en/latest/integration.html#example-settin=
g-up-u-boot-environment-on-emmc-sd-card<https://rauc.readthedocs.io/en/late=
st/integration.html#example-setting-up-u-boot-environment-on-emmc-sd-card>
> > >
> > > When writing to the FAT very short before hard rebooting, I could ima=
gine
> > > this
> > > can lead to failures. Do you see the corruption only after updates, o=
r also
> > > suddenly after n boots?
> >
> > Yes, this is something we have been able to test. If we cut the power
> > precisely when the env is written to FAT we can corrupt the entire boot
> > partition.
> > Super scary but this is not the problem we're seeing in the field. That
> > problem is more subtle.
>
> It should be possible to mount fat with the 'sync' option, but I'm not su=
re if
> that would help in this case. I'd recommend avoiding mounting FAT filesys=
tems
> R/W if possible.

Maybe it could help with the problem I'm investigating. Don't think it woul=
d help with
the total corruption on powerloss when writing u-boot env, since that is in=
 u-boot and
the fs is not "mounted" yet.

> > > How does the system report the corruption?
> >
> > fw_printenv and fw_setenv stops working and says that the env is corrup=
ted.
> > That also means that RAUC update fails, that is usually when we notice =
it.
> >
> > Is there a way to watch a file and record any process that modifies it?
>
> There is blktrace, but you don't see the contents that way. It still may =
be
> enough detail to understand what's happening here.

Great, I'll check that out.

> Regards,
> Jan

Thanks for all the help.

Regards,
Einar

_______________________________________________
RAUC mailing list


--
Matthew Campbell
Principal Engineer
mcampbell@izotope.com<mailto:mcampbell@izotope.com>

iZotope, Inc.
www.izotope.com<http://www.izotope.com>

--_000_AM0PR08MB4580282B727C41779681EB73E57F9AM0PR08MB4580eurp_
Content-Type: text/html; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1=
252">
<style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo=
ttom:0;} </style>
</head>
<body dir=3D"ltr">
<div class=3D"WordSection1">
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<font size=3D"2" face=3D"Calibri"><span style=3D"font-size:11.0pt">Great in=
formation!</span></font></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<font size=3D"2" face=3D"Calibri"><span style=3D"font-size:11.0pt">&nbsp;</=
span></font></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<font size=3D"2" face=3D"Calibri"><span style=3D"font-size:11.0pt">With the=
 exception of the custom code, that is exactly how we are planning to set i=
t up for our future systems.</span></font></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<font size=3D"2" face=3D"Calibri"><span style=3D"font-size:11.0pt">It=92s r=
eassuring to head that it seems to work well for you.</span></font></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<font size=3D"2" face=3D"Calibri"><span style=3D"font-size:11.0pt">&nbsp;</=
span></font></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<font size=3D"2" face=3D"Calibri"><span style=3D"font-size:11.0pt">Regards,=
</span></font></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<font size=3D"2" face=3D"Calibri"><span style=3D"font-size:11.0pt">Einar</s=
pan></font></p>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<font size=3D"2" face=3D"Calibri"><span style=3D"font-size:11.0pt">&nbsp;</=
span></font></p>
<div style=3D"border:none; border-left:solid blue 1.5pt; padding:0in 0in 0i=
n 4.0pt">
<div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<a name=3D"_MailOriginal"><font size=3D"2" face=3D"Calibri"><span style=3D"=
font-size:11.0pt">Not a solution per say, but I can give you some info on h=
ow we solve the reliability issue in our product that uses RAUC.</span></fo=
nt></a></p>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">* We store the env at a raw offset in the eMMC (this should work f=
or SD as well) rather than on a FAT partition as a file. You will need to s=
et your partition table up to leave room
 for this and modify the U-Boot config.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">* We use redundant&nbsp;u-boot environments placed in different se=
ctors of the eMMC. This is a built-in feature of U-Boot that can be enabled=
 in the config. If one gets corrupted it will
 fall back on the previous gracefully.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">* We have custom code both in U-Boot and in Linux that checks for =
corrupt&nbsp;or inconsistent&nbsp;RAUC U-Boot environment&nbsp;vars. If the=
y are totally out of whack we will boot into our fail-safe
 recovery mode where the evn vars are reset to a sane default and an update=
 can be performed&nbsp;(no RMA needed).</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">Over the past year we've had this setup. I haven't once seen or he=
ard of actually hitting&nbsp;a corrupt U-Boot env in any of our development=
&nbsp;units. We unfortunately don't have analytics
 around this event in the field.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">I know this isn't exactly an answer to your question, but hopefull=
y&nbsp;some of this helps you arrive at a robust solution for your setup.</=
span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">Best,</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">~Matt</span></font></span></p>
</div>
</div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
<div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">On Sun, Mar 28, 2021 at 6:11 AM Einar Vading &lt;</span></font></s=
pan><a href=3D"mailto:Einar.Vading@rhimagnesita.com"><span style=3D"">Einar=
.Vading@rhimagnesita.com</span><span style=3D""></span></a><span style=3D""=
>&gt;
 wrote:</span></p>
</div>
<blockquote style=3D"border:none; border-left:solid #CCCCCC 1.0pt; padding:=
0in 0in 0in 6.0pt; margin-left:4.8pt; margin-right:0in">
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; Hi,
</span></font></span></p>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; </span>
</font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; On Fri, 2021-03-26 at 05:48 +0000, Einar Vading wrote:</span>=
</font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; Hi,</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; On Thu, 2021-03-25 at 15:22 +0000, Einar Vadin=
g wrote:</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; We have a Raspberry Pi 4 system set up us=
ing RAUC for updates and u-boot</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; for</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; booting. For some systems in the field we=
 have the u-boot environment on</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; the</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; FAT boot partition and we mount that in f=
stab so that RAUC can access it</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; with</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; the fw_print/setenv commands.</span></fon=
t></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; One issue we have seen is that the env-fi=
le gets corrupted every now and</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; then.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; After corruption we can't RAUC update. Th=
e only solution we have to this</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; problem now is to delete the corrupted en=
v-file and reboot, then we can</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; perform the upgrade.</span></font></span>=
</p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; I have no idea how to track down whatever=
 corrupts the file and I was</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; &gt; wondering if anyone has any input.</span>=
</font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; You could try placing the environment on a sep=
arate partition to avoid any</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; potential issues in the FAT implementation. Al=
so, I think U-Boot has a way</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; to</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; &gt; support redundant environments.</span></font><=
/span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; I have just done this for our newer systems. I moved the=
 GPT partitions back</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; 4MB and placed two redundant environments between the GP=
T and the first GPT</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; partition.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; It is my understanding though that redundant environment=
s are not supported</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; when storing the env on FAT?</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; </span>
</font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; That's probably a question for the U-Boot mailing list. :)</s=
pan></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; </span>
</font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; Exactly. This should also be documented in the U-Bo=
ot integration guideline</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; for eMMC:</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt;
</span></font></span><a href=3D"https://rauc.readthedocs.io/en/latest/integ=
ration.html#example-setting-up-u-boot-environment-on-emmc-sd-card" target=
=3D"_blank"><span style=3D"">https://rauc.readthedocs.io/en/latest/integrat=
ion.html#example-setting-up-u-boot-environment-on-emmc-sd-card</span><span =
style=3D""></span></a><span style=3D""></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; When writing to the FAT very short before hard rebo=
oting, I could imagine</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; this</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; can lead to failures. Do you see the corruption onl=
y after updates, or also</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; suddenly after n boots?</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; Yes, this is something we have been able to test. If we =
cut the power</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; precisely when the env is written to FAT we can corrupt =
the entire boot</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; partition.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; Super scary but this is not the problem we're seeing in =
the field. That</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; problem is more subtle.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; </span>
</font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; It should be possible to mount fat with the 'sync' option, bu=
t I'm not sure if</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; that would help in this case. I'd recommend avoiding mounting=
 FAT filesystems</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; R/W if possible.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">Maybe it could help with the problem I'm investigating. Don't thin=
k it would help with</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">the total corruption on powerloss when writing u-boot env, since t=
hat is in u-boot and</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">the fs is not &quot;mounted&quot; yet.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; &gt; How does the system report the corruption?</span></=
font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; fw_printenv and fw_setenv stops working and says that th=
e env is corrupted.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; That also means that RAUC update fails, that is usually =
when we notice it.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; &gt; Is there a way to watch a file and record any process th=
at modifies it?</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; </span>
</font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; There is blktrace, but you don't see the contents that way. I=
t still may be</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; enough detail to understand what's happening here.</span></fo=
nt></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">Great, I'll check that out.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; Regards,</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&gt; Jan</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">Thanks for all the help.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">Regards,</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">Einar</span></font></span></p>
</div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">_______________________________________________<br>
RAUC mailing list</span></font></span></p>
</blockquote>
</div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt"><br style=3D"" clear=3D"all">
</span></font></span></p>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">-- </span>
</font></span></p>
<div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">Matthew Campbell</span></font></span></p>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">Principal Engineer</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""></span><a href=3D"mailto:mcampbell@izotope.com" target=3D"=
_blank"><span style=3D"">mcampbell@izotope.com</span><span style=3D""></spa=
n></a><span style=3D""></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">&nbsp;</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""><font size=3D"2" face=3D"Calibri"><span style=3D"font-size=
:11.0pt">iZotope, Inc.</span></font></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin: 0in; font-size: 11pt; font-family: =
&quot;Calibri&quot;, sans-serif;">
<span style=3D""></span><a href=3D"http://www.izotope.com" target=3D"_blank=
"><span style=3D"">www.izotope.com</span><span style=3D""></span></a><span =
style=3D""></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>

--_000_AM0PR08MB4580282B727C41779681EB73E57F9AM0PR08MB4580eurp_--



--===============0592601626==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
RAUC mailing list

--===============0592601626==--