gpart failing with no such geom after gpt corruption

Discussion:

(too old to reply)

Bartosz Stec

2010-04-01 13:23:11 UTC

Hello ZFS and GPT hackers :)

I'm sending this message to both freebsd-current and freebsd-fs because
it doesn't seems to be a CURRENT-specific issue.

Yesterday I tried to migrate my mixed UFS/RAIDZ config to clean RAIDZ
with GPT boot. I've following mostly this guide:
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1
I'm using CURRENT on 3x40GB HDDs (ad0-ad3) and additional 250GB HDD has
been used for data migration (ad4).

Data was copied form RAIDZ to 250GB HDD, GPT sheme was created on 40GB
HDDs, then new zpool on them, and finally data went back to RAIDZ.
Booting from RAIDZ was succesful, so far so good.

After a while I've noticed some SMART errors on ad1, so I've booted
machine with seatools for dos and made long test. One bad sector was
found and reallocated, nothing to worry about.
As I was in seatools already, I've decided to adjust LBA size on that
disk (seatools can do that), because it was about 30MB larger than the
other two, and because of that I had to adjust size of freebsd-zfs
partition on that disk to match exact size of others (otherwise 'zpool
create' will complain). So LBA was adjusted and system rebooted.

Yes, I was aware that changing disk size probably end with corrupted GPT
and data loss, but it doesn't seem to be a big deal for me as far as 2/3
of zpool is alive, because I can always recreate gpt and resilver ad1.

Unfortunately it wasn't so easy. First of all system booted, and as I
expected kernel message shows GPT error on ad1. Zpool was degraded but
alive and kicking. However, when I tried to execute any gpart command on
ad1, it return:

ad1: no such geom

ad1 was present under /dev, and it could be accessed by
sysinstall/fdisk, but no with gpart. I've created bsd slice with
sysinstall on ad1 and rebooted, with hope that after reboot I could
acces ad1 with gpart and recreate GPT scheme. Another surprise - system
didn't boot at all, rebooting after couple of seconds in loader
(changing boot device didn't make a difference).

Only way I could boot system at this moment was connecting 250GB HDD
which fortunately still had data from zpool migration and boot from it.
Another surprise - kernel was still complaining about GPT corruption on
ad1. I had no other ideas so I ran

dd if=/dev/zero of=/dev/ad1 bs=512 count=512

to clear beginning of the hdd. After that disk was still unaccesible
fromt gpart, so I tried sysinstall/fdisk againt to create standard BSD
partitioning scheme and rebooted system.
After that finally gpart started to talk with ad1 and GPT scheme and
zpool has been recreated and work as it supposed to.

Still, how can we clear broken GPT data after it got corrupted?
Why gpart has been showing "ad1: no such geom", and how can we deal with
this problem?
Finally, why gptzfsboot failed with GPT corrupted on other disk after
trying to fix it, while it booted at first place?

Or maybe changing LBA size of already partitioned HDD is extreme case,
and the only way these problems could be triggered ;)?

Cheers!

--
Bartosz Stec

Olivier Smedts

2010-04-01 16:36:43 UTC

Permalink

Post by Bartosz Stec
Hello ZFS and GPT hackers :)
I'm sending this message to both freebsd-current and freebsd-fs because it
doesn't seems to be a CURRENT-specific issue.
Yesterday I tried to migrate my mixed UFS/RAIDZ config to clean RAIDZ with
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1
I'm using CURRENT on 3x40GB HDDs (ad0-ad3) and additional 250GB HDD has been
used for data migration (ad4).
Data was copied form RAIDZ to 250GB HDD, GPT sheme was created on 40GB
HDDs, then new zpool on them, and finally data went back to RAIDZ. Booting
from RAIDZ was succesful, so far so good.
After a while I've noticed some SMART errors on ad1, so I've booted machine
with seatools for dos and made long test. One bad sector was found and
reallocated, nothing to worry about.
As I was in seatools already, I've decided to adjust LBA size on that disk
(seatools can do that), because it was about 30MB larger than the other two,
and because of that I had to adjust size of freebsd-zfs partition on that
disk to match exact size of others (otherwise 'zpool create' will complain).
So LBA was adjusted and system rebooted.
Yes, I was aware that changing disk size probably end with corrupted GPT and
data loss, but it doesn't seem to be a big deal for me as far as 2/3 of
zpool is alive, because I can always recreate gpt and resilver ad1.
Unfortunately it wasn't so easy. First of all system booted, and as I
expected kernel message shows GPT error on ad1. Zpool was degraded but alive
and kicking. However, when I tried to execute any gpart command on ad1, it
ad1: no such geom

Are you sure you created a partition scheme with gpart on ad1 before
issuing partition-related gpart commands ?

Post by Bartosz Stec
ad1 was present under /dev, and it could be accessed by sysinstall/fdisk,
but no with gpart. I've created bsd slice with sysinstall on ad1 and
rebooted, with hope that after reboot I could acces ad1 with gpart and
recreate GPT scheme. Another surprise - system didn't boot at all, rebooting
after couple of seconds in loader (changing boot device didn't make a
difference).
Only way I could boot system at this moment was connecting 250GB HDD which
fortunately still had data from zpool migration and boot from it. Another
surprise - kernel was still complaining about GPT corruption on ad1. I had
no other ideas so I ran
dd if=/dev/zero of=/dev/ad1 bs=512 count=512
to clear beginning of the hdd. After that disk was still unaccesible fromt
gpart, so I tried sysinstall/fdisk againt to create standard BSD
partitioning scheme and rebooted system.
After that finally gpart started to talk with ad1 and GPT scheme and zpool
has been recreated and work as it supposed to.
Still, how can we clear broken GPT data after it got corrupted?
Why gpart has been showing "ad1: no such geom", and how can we deal with
this problem?
Finally, why gptzfsboot failed with GPT corrupted on other disk after trying
to fix it, while it booted at first place?
Or maybe changing LBA size of already partitioned HDD is extreme case, and
the only way these problems could be triggered ;)?
Cheers!
--
Bartosz Stec
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-fs

--
Olivier Smedts _
ASCII ribbon campaign ( )
e-mail: ***@gid0.org - against HTML email & vCards X
www: http://www.gid0.org - against proprietary attachments / \

"Il y a seulement 10 sortes de gens dans le monde :
ceux qui comprennent le binaire,
et ceux qui ne le comprennent pas."

Olivier Smedts

2010-04-01 18:15:22 UTC

Permalink

I don't understand why you adjusted LBA. You're using GPT partitions,
so, couldn't you just make the zfs partition the same size on all
disks by adjusting it to the smallest disk, and let free space at the
end of the bigger ones ?

Cheers,
Olivier

Post by Bartosz Stec
Yes, I was aware that changing disk size probably end with corrupted GPT and
data loss, but it doesn't seem to be a big deal for me as far as 2/3 of
zpool is alive, because I can always recreate gpt and resilver ad1.
Unfortunately it wasn't so easy. First of all system booted, and as I
expected kernel message shows GPT error on ad1. Zpool was degraded but alive
and kicking. However, when I tried to execute any gpart command on ad1, it
ad1: no such geom
ad1 was present under /dev, and it could be accessed by sysinstall/fdisk,
but no with gpart. I've created bsd slice with sysinstall on ad1 and
rebooted, with hope that after reboot I could acces ad1 with gpart and
recreate GPT scheme. Another surprise - system didn't boot at all, rebooting
after couple of seconds in loader (changing boot device didn't make a
difference).
Only way I could boot system at this moment was connecting 250GB HDD which
fortunately still had data from zpool migration and boot from it. Another
surprise - kernel was still complaining about GPT corruption on ad1. I had
no other ideas so I ran
dd if=/dev/zero of=/dev/ad1 bs=512 count=512
to clear beginning of the hdd. After that disk was still unaccesible fromt
gpart, so I tried sysinstall/fdisk againt to create standard BSD
partitioning scheme and rebooted system.
After that finally gpart started to talk with ad1 and GPT scheme and zpool
has been recreated and work as it supposed to.
Still, how can we clear broken GPT data after it got corrupted?
Why gpart has been showing "ad1: no such geom", and how can we deal with
this problem?
Finally, why gptzfsboot failed with GPT corrupted on other disk after trying
to fix it, while it booted at first place?
Or maybe changing LBA size of already partitioned HDD is extreme case, and
the only way these problems could be triggered ;)?
Cheers!
--
Bartosz Stec
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-fs

Robert Noland

2010-04-01 19:02:33 UTC

Permalink

Post by Olivier Smedts

For that matter, my understanding is that ZFS just doesn't care. If you
have disks of different sized in a raidz, the pool size will be limited
by the size of the smallest device. If those devices are replaced with
larger ones, then the pool just grows to take advantage of the
additional available space.

robert.

Post by Olivier Smedts
Cheers,
Olivier

Robert Noland

2010-04-01 19:34:17 UTC

Permalink

Post by Robert Noland

Post by Olivier Smedts

balrog% gpart show
=> 34 2097085 md0 GPT (1.0G)
34 128 1 freebsd-boot (64K)
162 2096957 2 freebsd-zfs (1.0G)

=> 34 2097085 md1 GPT (1.0G)
34 128 1 freebsd-boot (64K)
162 2096957 2 freebsd-zfs (1.0G)

=> 34 4194237 md2 GPT (2.0G)
34 128 1 freebsd-boot (64K)
162 4194109 2 freebsd-zfs (2.0G)

balrog% zpool status
pool: test
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
raidz1 ONLINE 0 0 0
md0p2 ONLINE 0 0 0
md1p2 ONLINE 0 0 0
md2p2 ONLINE 0 0 0

errors: No known data errors

balrog% zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
test 2.98G 141K 2.98G 0% ONLINE -

robert.

Post by Robert Noland
robert.

Post by Olivier Smedts
Cheers,
Olivier

Post by Bartosz Stec
Yes, I was aware that changing disk size probably end with corrupted GPT and
data loss, but it doesn't seem to be a big deal for me as far as 2/3 of
zpool is alive, because I can always recreate gpt and resilver ad1.
Unfortunately it wasn't so easy. First of all system booted, and as I
expected kernel message shows GPT error on ad1. Zpool was degraded but alive
and kicking. However, when I tried to execute any gpart command on ad1, it
ad1: no such geom
ad1 was present under /dev, and it could be accessed by
sysinstall/fdisk,
but no with gpart. I've created bsd slice with sysinstall on ad1 and
rebooted, with hope that after reboot I could acces ad1 with gpart and
recreate GPT scheme. Another surprise - system didn't boot at all, rebooting
after couple of seconds in loader (changing boot device didn't make a
difference).
Only way I could boot system at this moment was connecting 250GB HDD which
fortunately still had data from zpool migration and boot from it. Another
surprise - kernel was still complaining about GPT corruption on ad1. I had
no other ideas so I ran
dd if=/dev/zero of=/dev/ad1 bs=512 count=512
to clear beginning of the hdd. After that disk was still unaccesible fromt
gpart, so I tried sysinstall/fdisk againt to create standard BSD
partitioning scheme and rebooted system.
After that finally gpart started to talk with ad1 and GPT scheme and zpool
has been recreated and work as it supposed to.
Still, how can we clear broken GPT data after it got corrupted?
Why gpart has been showing "ad1: no such geom", and how can we deal with
this problem?
Finally, why gptzfsboot failed with GPT corrupted on other disk after trying
to fix it, while it booted at first place?
Or maybe changing LBA size of already partitioned HDD is extreme case, and
the only way these problems could be triggered ;)?
Cheers!
--
Bartosz Stec
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-fs

_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current

Bartosz Stec

2010-04-01 19:42:21 UTC

Permalink

Post by Robert Noland

Post by Olivier Smedts

Post by Bartosz Stec
After a while I've noticed some SMART errors on ad1, so I've booted machine
with seatools for dos and made long test. One bad sector was found and
reallocated, nothing to worry about.
As I was in seatools already, I've decided to adjust LBA size on that disk
(seatools can do that), because it was about 30MB larger than the other two,
and because of that I had to adjust size of freebsd-zfs partition on that
disk to match exact size of others (otherwise 'zpool create' will complain).
So LBA was adjusted and system rebooted.

Well yes, I could indeed, and this is exactly what I did at the first
time (before LBA count adjusting). But while I was already using
software which could adjust LBA to make all HDD appear to be same size,
I've decided to do it to never have to remember about it while
partitioning ;) At least 'gpart show' isn't showing any unused (wasted)
space now ;) :

# gpart show
=> 34 78165293 ad0 GPT (37G)
34 128 1 freebsd-boot (64K)
162 2097152 2 freebsd-swap (1.0G)
2097314 76068013 3 freebsd-zfs (36G)

=> 34 78165293 ad1 GPT (37G)
34 128 1 freebsd-boot (64K)
162 2097152 2 freebsd-swap (1.0G)
2097314 76068013 3 freebsd-zfs (36G)

=> 34 78165293 ad2 GPT (37G)
34 128 1 freebsd-boot (64K)
162 2097152 2 freebsd-swap (1.0G)
2097314 76068013 3 freebsd-zfs (36G)

Post by Robert Noland
For that matter, my understanding is that ZFS just doesn't care. If
you have disks of different sized in a raidz, the pool size will be
limited by the size of the smallest device. If those devices are
replaced with larger ones, then the pool just grows to take advantage
of the additional available space.
robert.

Well, here's what man zpool says about zpool create:

"(...) The use of differently sized devices within a single raidz
or mirror group is also flagged as an error unless -f is specified."

I know I could force it, I just didn't know if I should.

After all it's just easier to type 3 times:

gpt add -t freebsd-zfs -l diskN

to use all free space on device than checking numbers on other disks and
type

gpt add -b 2097314 -s 76068013 -t freebsd-zfs -l diskN

and that's why all story begins :)

--
Bartosz Stec

Marcel Moolenaar

2010-04-01 17:34:23 UTC

Permalink

Post by Bartosz Stec
ad1: no such geom

If the GPT is rejected, no GEOM is created in the kernel. It's the GEOM
that gpart(8) talks to, so the error indicates that the GPT is not accepted
after you changed the disk size. For all practical purposes ad1 doesn't have
a partitioning.

The only gpart command you can use is:
gpart create -s gpt ad1

This creates a new GPT on the disk, wiping out whatever was there...

--
Marcel Moolenaar
***@mac.com

Bartosz Stec

2010-04-01 20:02:37 UTC

Permalink

Post by Marcel Moolenaar

Post by Bartosz Stec
ad1: no such geom

If the GPT is rejected, no GEOM is created in the kernel. It's the GEOM
that gpart(8) talks to, so the error indicates that the GPT is not accepted
after you changed the disk size. For all practical purposes ad1 doesn't have
a partitioning.
gpart create -s gpt ad1
This creates a new GPT on the disk, wiping out whatever was there...

It was a middle of night when I played with that, so I'm not remember
clearly if I tried it simpliest way like above or not. Maybe not, and
all noise I made was pointless and caused by my inexperience with gpart.
Probably I just expected output from "gpart show ad1" be more like "no
valid partitioning scheme on device" or "error in GPT table, please
recreate partition scheme" in place of mystic geom message ;)

Sorry for noise, thanks for a hint and happy easter everyone :)

--
Bartosz Stec

Paul Wootton

2010-04-01 19:10:05 UTC

Permalink

Bartosz,

One thing to remember is that GPT stores it's header and entry tables at
both the start and end of the disk for redundancy.
As far as I understand it, by making the disk physically smaller, the
GPT primary header and entry data would have become invalid as the last
partition would now be ending past the end of the disk

Partition table format

Offset Length Content
0 8 Signature ("EFI PART", 45 46 49 20 50 41 52 54)
...
24 8 Current LBA (location of this header copy)
32 8 Backup LBA (location of the other header copy)
40 8 First usable LBA for partitions (primary partition table last
LBA + 1)
48 8 Last usable LBA (secondary partition table first LBA - 1)
...

GUID partition entry format

Offset Length Content
0 16 Partition type GUID
...
32 8 First LBA (little-endian)
40 8 Last LBA (inclusive, usually odd)
...

See http://en.wikipedia.org/wiki/GUID_Partition_Table