A tool for remapping bad sectors in CURRENT?

[Eugeny N Dzhurinsky]

Hello, all!

Recently I've started to see the following logs in messages:

Mar 8 12:00:24 localhost smartd[795]: Device: /dev/ad4, 2 Currently unreadable (pending) sectors
Mar 8 12:00:24 localhost smartd[795]: Device: /dev/ad4, 2 Offline uncorrectable sectors

smartctl did really show that something is wrong with my HDD, but still no
remaps - just read errors.

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 60% 1198 222342559
# 2 Extended offline Completed: read failure 60% 1187 222342557
# 3 Extended offline Completed: read failure 60% 1180 222342559
# 4 Short offline Completed without error 00% 1178 -
# 5 Extended offline Aborted by host 90% 1178 -

and

ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
...
Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
...

Now can I find out which file owns the LBAs 222342557 and 222342559 ? How do I
force remapping of these sectors? I assume that I have to write something
directly to the sectors?

Thank you all in advance!

[Alexander Motin]

Post by Eugeny N Dzhurinsky
Mar 8 12:00:24 localhost smartd[795]: Device: /dev/ad4, 2 Currently unreadable (pending) sectors
Mar 8 12:00:24 localhost smartd[795]: Device: /dev/ad4, 2 Offline uncorrectable sectors
smartctl did really show that something is wrong with my HDD, but still no
remaps - just read errors.
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 60% 1198 222342559
# 2 Extended offline Completed: read failure 60% 1187 222342557
# 3 Extended offline Completed: read failure 60% 1180 222342559
# 4 Short offline Completed without error 00% 1178 -
# 5 Extended offline Aborted by host 90% 1178 -
and
ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
...
Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
...
Now can I find out which file owns the LBAs 222342557 and 222342559 ? How do I
force remapping of these sectors? I assume that I have to write something
directly to the sectors?

You may try to overwrite these sectors with dd. It should trigger sector
reallocation. To be sure, you may read them before and after the write.

[Eugene Dzhurinsky]

Post by Alexander Motin
You may try to overwrite these sectors with dd. It should trigger sector
reallocation. To be sure, you may read them before and after the write.

dd if=/dev/ad4 of=/dev/null skip=222342559 bs=512 count=1
dd: /dev/ad4: Input/output error
0+0 records in
0+0 records out
0 bytes transferred in 2.351940 secs (0 bytes/sec)

dd if=/dev/zero of=/dev/ad4 seek=222342559 bs=512 count=1
dd: /dev/ad4: Operation not permitted

Should I do it in single mode?

[Eugene Dzhurinsky]

Post by Eugene Dzhurinsky
dd if=/dev/ad4 of=/dev/null skip=222342559 bs=512 count=1
dd: /dev/ad4: Input/output error
0+0 records in
0+0 records out
0 bytes transferred in 2.351940 secs (0 bytes/sec)
dd if=/dev/zero of=/dev/ad4 seek=222342559 bs=512 count=1
dd: /dev/ad4: Operation not permitted
Should I do it in single mode?

sysctl kern.geom.debugflags=0x10

Did the trick, I was able to write directly to the sector, and now it seems to
work well. No remaps recorded thus, but no errors so far.

Thanks a lot!

[Ulrich Spörlein]

Post by Eugene Dzhurinsky

Post by Eugene Dzhurinsky
dd if=/dev/ad4 of=/dev/null skip=222342559 bs=512 count=1
dd: /dev/ad4: Input/output error
0+0 records in
0+0 records out
0 bytes transferred in 2.351940 secs (0 bytes/sec)
dd if=/dev/zero of=/dev/ad4 seek=222342559 bs=512 count=1
dd: /dev/ad4: Operation not permitted
Should I do it in single mode?

sysctl kern.geom.debugflags=0x10
Did the trick, I was able to write directly to the sector, and now it seems to
work well. No remaps recorded thus, but no errors so far.

It's too late now, but you really should have gone with something like
# recoverdisk /dev/ad4 /dev/ad4

this will re-write all blocks on the disk, it may fail at reading block
222342559, but there's a chance that the disk error correction gets it
right after a couple of times.

Regards,
Uli

[Poul-Henning Kamp]

Post by Eugeny N Dzhurinsky
Now can I find out which file owns the LBAs 222342557 and 222342559 ?
How do I force remapping of these sectors? I assume that I have to write
something directly to the sectors?

I would suggest you boot single-user and run

mdmfs -s 1m md /tmp
recoverdisk -w /tmp/_.wl /dev/ad4 /dev/ad4

That will find out how many bad sectors you have and try to recover
the contents of them if possible, leave it running as long as you
care for.

If you interrupt it, the /tmp/_.wl file will contain a list of areas
not yet successfully read/written.

[Eugene Dzhurinsky]

Post by Poul-Henning Kamp
I would suggest you boot single-user and run
mdmfs -s 1m md /tmp
recoverdisk -w /tmp/_.wl /dev/ad4 /dev/ad4
That will find out how many bad sectors you have and try to recover
the contents of them if possible, leave it running as long as you
care for.
If you interrupt it, the /tmp/_.wl file will contain a list of areas
not yet successfully read/written.

Well, I just want to force IDE drive to remap things :)

[Miroslav Lachman]

Post by Eugeny N Dzhurinsky
Hello, all!
Mar 8 12:00:24 localhost smartd[795]: Device: /dev/ad4, 2 Currently unreadable (pending) sectors
Mar 8 12:00:24 localhost smartd[795]: Device: /dev/ad4, 2 Offline uncorrectable sectors
smartctl did really show that something is wrong with my HDD, but still no
remaps - just read errors.
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 60% 1198 222342559
# 2 Extended offline Completed: read failure 60% 1187 222342557
# 3 Extended offline Completed: read failure 60% 1180 222342559
# 4 Short offline Completed without error 00% 1178 -
# 5 Extended offline Aborted by host 90% 1178 -
and
ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
...
Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
...
Now can I find out which file owns the LBAs 222342557 and 222342559 ? How do I
force remapping of these sectors? I assume that I have to write something
directly to the sectors?

We have this problem from time to time on bunch of machines. As we are
using gmirror, the easiest way is to force re-synchronization (rewrite)
of the whole drive. The problem is when there are Pending unreadable
sectors on both drives - it ends up with read error and some file(s) are
corrupted, but there is no easy way (on FreeBSD) to find what file.

I tried it in the past with fsdb / findblk, but it does not work as I
expect or I do not fully understand the needed calculations with slices
+ partitions offsets / LBAs and right meaning of the term "block". It
seems there are several meaning in different contexts.

It would be nice if somebody with enough FS / GEOM knowledge can write
some HowTo or shell script to do the calculations and operations to find
file containing bad sector(s) and put it in FAQ, Handbook, or Wiki.

Miroslav Lachman

[Wes Morgan]

Post by Eugeny N Dzhurinsky
Hello, all!
Mar 8 12:00:24 localhost smartd[795]: Device: /dev/ad4, 2 Currently
unreadable (pending) sectors
Mar 8 12:00:24 localhost smartd[795]: Device: /dev/ad4, 2 Offline
uncorrectable sectors
smartctl did really show that something is wrong with my HDD, but still no
remaps - just read errors.
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours)
LBA_of_first_error
# 1 Extended offline Completed: read failure 60% 1198
222342559
# 2 Extended offline Completed: read failure 60% 1187
222342557
# 3 Extended offline Completed: read failure 60% 1180
222342559
# 4 Short offline Completed without error 00% 1178
-
# 5 Extended offline Aborted by host 90% 1178
-
and
ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
...
Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always -
0
...
Now can I find out which file owns the LBAs 222342557 and 222342559 ? How do I
force remapping of these sectors? I assume that I have to write something
directly to the sectors?

We have this problem from time to time on bunch of machines. As we are using
gmirror, the easiest way is to force re-synchronization (rewrite) of the whole
drive. The problem is when there are Pending unreadable sectors on both drives
- it ends up with read error and some file(s) are corrupted, but there is no
easy way (on FreeBSD) to find what file.

*cough* zfs *cough*

I believe this kind of silent corruption is precisely what zfs was
designed to prevent. Even though you do have a mirror, how do you know
which copy is the correct one? If one drive re-allocates the sector
silently, what is the recovery method? If gmirror synchronizes, how do you
make sure that the *good* copy is the one synchronized? You'll notice it
eventually if you see it in a garbled file, but how does the filesystem
handle it?

I tried it in the past with fsdb / findblk, but it does not work as I expect
or I do not fully understand the needed calculations with slices + partitions
offsets / LBAs and right meaning of the term "block". It seems there are
several meaning in different contexts.
It would be nice if somebody with enough FS / GEOM knowledge can write some
HowTo or shell script to do the calculations and operations to find file
containing bad sector(s) and put it in FAQ, Handbook, or Wiki.

[Eugene Dzhurinsky]

Post by Miroslav Lachman
We have this problem from time to time on bunch of machines. As we are
using gmirror, the easiest way is to force re-synchronization (rewrite)
of the whole drive. The problem is when there are Pending unreadable
sectors on both drives - it ends up with read error and some file(s) are
corrupted, but there is no easy way (on FreeBSD) to find what file.
I tried it in the past with fsdb / findblk, but it does not work as I
expect or I do not fully understand the needed calculations with slices
+ partitions offsets / LBAs and right meaning of the term "block". It
seems there are several meaning in different contexts.
It would be nice if somebody with enough FS / GEOM knowledge can write
some HowTo or shell script to do the calculations and operations to find
file containing bad sector(s) and put it in FAQ, Handbook, or Wiki.

Miroslav, thank you for the suggestion - but I am not using gmirror, that HDD
is the one on my laptop. However suggestions about using dd to write something
into bad block to force IDE controller do it's service stuff about remapping
seems did the trick. And I was able to not calculate LBA but use it as block
offset, which seemed to be correct way :)

[Miroslav Lachman]

Post by Eugene Dzhurinsky

Post by Miroslav Lachman
We have this problem from time to time on bunch of machines. As we are
using gmirror, the easiest way is to force re-synchronization (rewrite)
of the whole drive. The problem is when there are Pending unreadable
sectors on both drives - it ends up with read error and some file(s) are
corrupted, but there is no easy way (on FreeBSD) to find what file.
I tried it in the past with fsdb / findblk, but it does not work as I
expect or I do not fully understand the needed calculations with slices
+ partitions offsets / LBAs and right meaning of the term "block". It
seems there are several meaning in different contexts.
It would be nice if somebody with enough FS / GEOM knowledge can write
some HowTo or shell script to do the calculations and operations to find
file containing bad sector(s) and put it in FAQ, Handbook, or Wiki.

Miroslav, thank you for the suggestion - but I am not using gmirror, that HDD
is the one on my laptop. However suggestions about using dd to write something
into bad block to force IDE controller do it's service stuff about remapping
seems did the trick. And I was able to not calculate LBA but use it as block
offset, which seemed to be correct way :)

Yes, rewriting by dd or any other way works for reallocating or clearing
pending sectors counter, but in server environment I need to know the
affected file, as it can be for example database file and then it is a
big problem! Rewriting the sector inside InnoDB ib_data file can cause
DB crash, data loss etc.

Miroslav Lachman

[Svein Skogen (Listmail Account)]

Post by Miroslav Lachman
Yes, rewriting by dd or any other way works for reallocating or
clearing pending sectors counter, but in server environment

In a server environment, you'd be a fool not to have some sort of
redundancy set up.

Post by Miroslav Lachman
I need to know the affected file, as it can be for example database
file and then it is a big problem! Rewriting the sector inside InnoDB
ib_data file can cause DB crash, data loss etc.

How is that different from *not* rewriting the sector? If there's a bad
sector somewhere in your data, your database is still going to crash.

Only if he hasn't listened to your first advice and set it up on a
non-redundant IO solution. If he's set it up on proper hardware, he'll
just get a friendly mail about replacing said disk next time he's in the
serverroom with a new fresh hostpare.

//Svein

- --
- --------+-------------------+-------------------------------
/"\ |Svein Skogen | ***@d80.iso100.no
\ / |Solberg Østli 9 | PGP Key: 0xE5E76831
X |2020 Skedsmokorset | ***@jernhuset.no
/ \ |Norway | PGP Key: 0xCE96CE13
| | ***@stillbilde.net
ascii | | PGP Key: 0x58CD33B6
ribbon |System Admin | svein-***@stillbilde.net
Campaign|stillbilde.net | PGP Key: 0x22D494A4
+-------------------+-------------------------------
|msn messenger: | Mobile Phone: [phone]
|***@jernhuset.no | RIPE handle: SS16503-RIPE
- --------+-------------------+-------------------------------
If you really are in a hurry, mail me at
svein-***@stillbilde.net
This mailbox goes directly to my cellphone and is checked
even when I'm not in front of my computer.
- ------------------------------------------------------------
Picture Gallery:
https://gallery.stillbilde.net/v/svein/
- ------------------------------------------------------------

[Dag-Erling Smørgrav]

Post by Miroslav Lachman
Yes, rewriting by dd or any other way works for reallocating or
clearing pending sectors counter, but in server environment

In a server environment, you'd be a fool not to have some sort of
redundancy set up.

Post by Miroslav Lachman
I need to know the affected file, as it can be for example database
file and then it is a big problem! Rewriting the sector inside InnoDB
ib_data file can cause DB crash, data loss etc.

How is that different from *not* rewriting the sector? If there's a bad
sector somewhere in your data, your database is still going to crash.

DES

[Miroslav Lachman]

Post by Miroslav Lachman
Yes, rewriting by dd or any other way works for reallocating or
clearing pending sectors counter, but in server environment

In a server environment, you'd be a fool not to have some sort of
redundancy set up.

I am using gmirror on low-end servers, so rewriting some sectors on one
disk drive is useless and in this case I prefer resync of the whole
gmirror (but it is log run - about 10 hours on busy server)

Post by Miroslav Lachman
I need to know the affected file, as it can be for example database
file and then it is a big problem! Rewriting the sector inside InnoDB
ib_data file can cause DB crash, data loss etc.

How is that different from *not* rewriting the sector? If there's a bad
sector somewhere in your data, your database is still going to crash.

It is not about "different", it is about "I need to know the affected
file" to know what actions should be taken.
If it is some logfile, I can delete it and then rewrite the sector. If
it is some "normal" unchanged file, I can restore it from backup, if it
is database file, I must take some special action. For example, stop DB
engine, try to repair/fix the DB file, dump & restore etc.
So the first step is to find "what file is affected", then take some
action AND rewrite the sector by dd to reallocate the sector. (or
replace the drive)

So... can somebody with enough knowledge write some docs / script how to
find the affected file based on LBA read error from messages / SMART log?

Miroslav Lachman

[Dag-Erling Smørgrav]

Post by Miroslav Lachman
So... can somebody with enough knowledge write some docs / script how
to find the affected file based on LBA read error from messages /
SMART log?

ZFS will tell you straight away, but I guess if you used ZFS, you
wouldn't be asking :)

For FFS, you can unmount the file system (boot from a CD or memory stick
or whatever if that file system is / or /usr), run fsdb on the failing
disk, use findblk to look up the inode number for the file that contains
the bad sector. Note that you have to convert the LBA to an offset
relative to the start of the partition.

Unfortunately, you can't easily go from inode to file name; you have to
mount the file system and use something like find -inum.

DES

[Miroslav Lachman]

Post by Dag-Erling Smørgrav

Post by Miroslav Lachman
So... can somebody with enough knowledge write some docs / script how
to find the affected file based on LBA read error from messages /
SMART log?

ZFS will tell you straight away, but I guess if you used ZFS, you
wouldn't be asking :)

Yes, but we have ZFS only on two servers, others are using UFS2 (some
with gmirror, some with gjournal)

Post by Dag-Erling Smørgrav
For FFS, you can unmount the file system (boot from a CD or memory stick
or whatever if that file system is / or /usr), run fsdb on the failing
disk, use findblk to look up the inode number for the file that contains
the bad sector. Note that you have to convert the LBA to an offset
relative to the start of the partition.

As I write in my first post to this thread, I already tried fsdb +
findblk, but without success. Findblk did not returned any inode. Maybe
the meaning of block is of different size or something else I can't
understand.

So can you please show me some real world example?


I have one from the past:

__________________
/var/log/messages:
Sep 23 23:58:00 edith kernel: ad4: FAILURE - READ_DMA
status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=79725056
Sep 23 23:58:00 edith kernel: GEOM_MIRROR: Request failed (error=5).
ad4[READ(offset=40819228672, length=131072)]

__________________
SMART log:
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 6f 82 c0 44 Error: UNC at LBA = 0x04c0826f = 79725167


The LBA of bad sector is *79725167*

__________________
Information about disk slices:

sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 209712447 (102398 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 1023/ head 254/ sector 63
The data for partition 2 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 209712510, size 1743807555 (851468 Meg), flag 0
beg: cyl 1023/ head 255/ sector 63;
end: cyl 1023/ head 254/ sector 63

__________________
According to LBA and size of s1, I thing the error is in s1

# /dev/mirror/gm0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 2097152 0 /
b: 25165824 2097152 swap
c: 209712447 0
d: 12582912 27262976 /var
e: 146800640 39845888 /var/db
f: 16777216 186646528 /usr
g: 6288703 203423744 /tmp


And LBA 79725056 is on */var/db* (between offset 39845888 and 186646528)

__________________
s1 starts 63 sectors from the beginning of the drive and /var/db has
offset 39845888. So am I right that I need to find block number
*39879105* by findblk command?

LBA err - s1 start - /var/db offset = findblk inside /dev/mirror/gm0s1e
79725056 - 63 - 39845888 = 39879105

__________________
/# fsdb -r /dev/mirror/gm0s1e
** /dev/mirror/gm0s1e (NO WRITE)
Examining file system '/dev/mirror/gm0s1e'
Last Mounted on /var/db
current inode: directory
I=2 MODE=40755 SIZE=512
BTIME=May 1 08:07:23 2009 [0 nsec]
MTIME=Sep 24 15:52:01 2009 [0 nsec]
CTIME=Sep 24 15:52:01 2009 [0 nsec]
ATIME=Sep 24 16:24:34 2009 [0 nsec]
OWNER=root GRP=wheel LINKCNT=11 FLAGS=0 BLKCNT=4 GEN=4ebc65fc

findblk 39879105
findblk 39879106
findblk 39879107
findblk 39879108
.
.

I tried more than 256 incrementing block numbers, but findblk didn't
found any inode! (length=131072 in error message means 256 sectors, right?)


So there must be some misunderstanding on my part and that's why I am
asking for some step-by-step documentation or script "how to find file
by LBA read error message"


I tried the fsdb + findblk on well known data, but again without success.

I created file /tmp/test.txt, it has inum 3, than I use fsdb on gm0s1f
(gm0s1f is mounted as /tmp). Command "inode 3" inside fsdb prompt
returned informations about this file, command "blocks" returned 3001 as
block number, but command "findblk 3001" returned nothing instead of
inum 3!
Where is the error? What I am doing wrong?

__________________
~/# echo test > /tmp/test.txt

~/# ls -i /tmp/test.txt
3 /tmp/test.txt

~/# fsdb -r /dev/mirror/gm0s1f
** /dev/mirror/gm0s1f (NO WRITE)
Examining file system '/dev/mirror/gm0s1f'
Last Mounted on /tmp
current inode: directory
I=2 MODE=41777 SIZE=512
BTIME=Feb 7 18:32:22 2008 [0 nsec]
MTIME=Mar 14 10:33:22 2010 [0 nsec]
CTIME=Mar 14 10:33:22 2010 [0 nsec]
ATIME=Mar 14 10:33:35 2010 [0 nsec]
OWNER=root GRP=wheel LINKCNT=7 FLAGS=0 BLKCNT=4 GEN=3f7c9384

fsdb (inum: 2)> inode 3
current inode: regular file
I=3 MODE=100644 SIZE=5
BTIME=Mar 14 10:33:22 2010 [0 nsec]
MTIME=Mar 14 10:33:22 2010 [0 nsec]
CTIME=Mar 14 10:33:22 2010 [0 nsec]
ATIME=Mar 14 10:33:22 2010 [0 nsec]
OWNER=root GRP=wheel LINKCNT=1 FLAGS=0 BLKCNT=4 GEN=45c26de1

fsdb (inum: 3)> blocks
Blocks for inode 3:
Direct blocks:
3001 (1 frag)

fsdb (inum: 3)> findblk 3001
fsdb (inum: 3)>

^^^^^^^^ findblk did not returned inode 3!

Post by Dag-Erling Smørgrav
Unfortunately, you can't easily go from inode to file name; you have to
mount the file system and use something like find -inum.

Yes, I know this.

Thanks in advance to help me understand and use fsdb + findblk commands.

Miroslav Lachman


PS: all above was tested on gmirror gm0, but I did the same tests on
single drive ad4 with the same "empty" result (info just for case if
fsdb can't be used on gmirror, but I don't think so)

[Gary Jennejohn]

On Sun, 14 Mar 2010 10:55:19 +0100
Miroslav Lachman <***@quip.cz> wrote:

[big snip]

Post by Miroslav Lachman
fsdb (inum: 3)> blocks
3001 (1 frag)
fsdb (inum: 3)> findblk 3001
fsdb (inum: 3)>
^^^^^^^^ findblk did not returned inode 3!

This is almost guaranteed to be a file system block and not
a disk block.

---
Gary Jennejohn

[Miroslav Lachman]

Post by Gary Jennejohn
On Sun, 14 Mar 2010 10:55:19 +0100
[big snip]

fsdb (inum: 3)> blocks
3001 (1 frag)
fsdb (inum: 3)> findblk 3001
fsdb (inum: 3)>
^^^^^^^^ findblk did not returned inode 3!

This is almost guaranteed to be a file system block and not
a disk block.

Do you mean the number 3001?
I am sorry for my ignorance, but it is not clear to me from fsdb manpage
what "blocks" means FS block and what disk block.

And how can I use (calculate with) this numbers?

How can I get the right number to pass to findlbk command (in the
example above) to give me back the inode 3?

If FS block is 16384 bytes, then it means 16384/512 = 32 disk blocks per
FS block.

If 3001 is FS block, then it means 3001*32 = 96032 disk block number. Am
I right?

fsdb (inum: 3)> findblk 96032
fsdb (inum: 3)>

Again - findblk did not returned inode 3.

So what is the exact formula to get the right findblk number and then
right inode number as result of findblk command?

I am still lost in terms (words) and numbers :(

Miroslav Lachman

[Gary Jennejohn]

On Sun, 14 Mar 2010 17:18:45 +0100

Post by Miroslav Lachman

Post by Gary Jennejohn
On Sun, 14 Mar 2010 10:55:19 +0100
[big snip]

fsdb (inum: 3)> blocks
3001 (1 frag)
fsdb (inum: 3)> findblk 3001
fsdb (inum: 3)>
^^^^^^^^ findblk did not returned inode 3!

This is almost guaranteed to be a file system block and not
a disk block.

Do you mean the number 3001?
I am sorry for my ignorance, but it is not clear to me from fsdb manpage
what "blocks" means FS block and what disk block.
And how can I use (calculate with) this numbers?
How can I get the right number to pass to findlbk command (in the
example above) to give me back the inode 3?
If FS block is 16384 bytes, then it means 16384/512 = 32 disk blocks per
FS block.
If 3001 is FS block, then it means 3001*32 = 96032 disk block number. Am
I right?
fsdb (inum: 3)> findblk 96032
fsdb (inum: 3)>
Again - findblk did not returned inode 3.
So what is the exact formula to get the right findblk number and then
right inode number as result of findblk command?
I am still lost in terms (words) and numbers :(

Well, it's pretty hairy.

Looking at findblk() it does this to go from disk block to file system
block (this is greatly simplified)

file_system_blockno = disk_blockno >> fs_fsbtodb;

So conversely, you'd do disk_blockno = file_system_blockno << fs_fsbtodb.

You can get this information using "ffsinfo -l 0x001 -o some_file
/dev/ataXY" (using ahci) and grep'ing for fsbtodb in some_file. The
0x001 means to only dump the first super block.

I looked at a file system which has default 16kB file system blocks and
fsbtodb is 2 ==> *multiply file_system_block by 4 not 32*. This is probably
because it's a multiple of a 4kB block, which is the smallest usable
file system block size AFAIK.

BTW looking at the code leads me to conclude that fsdb will not print out
anything if the disk block you're trying to find has bever been allocated
to an inode ==> unused disk block, safe to overwrite. This assumes that
you calculated the disk block correctly.

---
Gary Jennejohn

[Miroslav Lachman]

Post by Gary Jennejohn
On Sun, 14 Mar 2010 17:18:45 +0100

Post by Miroslav Lachman

Post by Gary Jennejohn
On Sun, 14 Mar 2010 10:55:19 +0100
[big snip]

fsdb (inum: 3)> blocks
3001 (1 frag)
fsdb (inum: 3)> findblk 3001
fsdb (inum: 3)>
^^^^^^^^ findblk did not returned inode 3!

This is almost guaranteed to be a file system block and not
a disk block.

Do you mean the number 3001?
I am sorry for my ignorance, but it is not clear to me from fsdb manpage
what "blocks" means FS block and what disk block.
And how can I use (calculate with) this numbers?
How can I get the right number to pass to findlbk command (in the
example above) to give me back the inode 3?
If FS block is 16384 bytes, then it means 16384/512 = 32 disk blocks per
FS block.
If 3001 is FS block, then it means 3001*32 = 96032 disk block number. Am
I right?
fsdb (inum: 3)> findblk 96032
fsdb (inum: 3)>
Again - findblk did not returned inode 3.
So what is the exact formula to get the right findblk number and then
right inode number as result of findblk command?
I am still lost in terms (words) and numbers :(

Well, it's pretty hairy.
Looking at findblk() it does this to go from disk block to file system
block (this is greatly simplified)
file_system_blockno = disk_blockno>> fs_fsbtodb;
So conversely, you'd do disk_blockno = file_system_blockno<< fs_fsbtodb.
You can get this information using "ffsinfo -l 0x001 -o some_file
/dev/ataXY" (using ahci) and grep'ing for fsbtodb in some_file. The
0x001 means to only dump the first super block.
I looked at a file system which has default 16kB file system blocks and
fsbtodb is 2 ==> *multiply file_system_block by 4 not 32*. This is probably
because it's a multiple of a 4kB block, which is the smallest usable
file system block size AFAIK.
BTW looking at the code leads me to conclude that fsdb will not print out
anything if the disk block you're trying to find has bever been allocated
to an inode ==> unused disk block, safe to overwrite. This assumes that
you calculated the disk block correctly.

I absolutely don't understand how you get the number 4 (it is some magic
for me :]) but it works!

fsdb (inum: 3)> blocks
Blocks for inode 3:
Direct blocks:
3001 (1 frag)

3001 * 4 = 12004

fsdb (inum: 3)> findblk 12004
12004: data block of inode 3

Thank you for this hint!

Miroslav Lachman

[Gary Jennejohn]

On Wed, 17 Mar 2010 12:41:33 +0100

Post by Miroslav Lachman
I absolutely don't understand how you get the number 4 (it is some magic
for me :]) but it works!
fsdb (inum: 3)> blocks
3001 (1 frag)
3001 * 4 = 12004
fsdb (inum: 3)> findblk 12004
12004: data block of inode 3
Thank you for this hint!

Umm, it's standard C code: 1 << 2 = 4. It's a power of 2, in this
case 2 squared.

---
Gary Jennejohn

[Miroslav Lachman]

Post by Gary Jennejohn
On Wed, 17 Mar 2010 12:41:33 +0100

Post by Miroslav Lachman
I absolutely don't understand how you get the number 4 (it is some magic
for me :]) but it works!

[...]

Post by Gary Jennejohn
Umm, it's standard C code: 1<< 2 = 4. It's a power of 2, in this
case 2 squared.

I am not a C programmer, so I didn't understand the syntax. Now it makes
sense.

Thank you again for the explanation.

Miroslav Lachman

[Dag-Erling Smørgrav]

Post by Miroslav Lachman
As I write in my first post to this thread, I already tried fsdb +
findblk, but without success. Findblk did not returned any inode.
Maybe the meaning of block is of different size or something else I
can't understand.

AFAICT, "block" is a disk block (i.e. 512-byte sector in most cases)
relative to the start of the partition.

Post by Miroslav Lachman
The LBA of bad sector is *79725167* [...] s1 starts 63 sectors from
the beginning of the drive and /var/db has offset 39845888. So am I
right that I need to find block number *39879105* by findblk command?

Uh, 79725167 - 63 = 79725104 and 79725104 - 39845888 = 39879216. How
did you arrive at 39879105?

DES

[Miroslav Lachman]

Post by Dag-Erling Smørgrav

Post by Miroslav Lachman
The LBA of bad sector is *79725167* [...] s1 starts 63 sectors from
the beginning of the drive and /var/db has offset 39845888. So am I
right that I need to find block number *39879105* by findblk command?

Uh, 79725167 - 63 = 79725104 and 79725104 - 39845888 = 39879216. How
did you arrive at 39879105?

I am sorry, it was my confusion.
My calculation was for *LBA=79725056* reported in messages:

ad4: FAILURE - READ_DMA status=51<READY,DSC,ERROR>
error=40<UNCORRECTABLE> LBA=79725056

79725056 - 63 - 39845888 = *39879105*

Your calculation is for LBA reported by SMART log

40 51 00 6f 82 c0 44 Error: UNC at LBA = 0x04c0826f = *79725167*

That's why I get different result ;) I must pay more attention to the
numbers next time!

It is interesting that there are two different LBAs for "same" error
(appeared at the same time)

Miroslav Lachman

[Dag-Erling Smørgrav]

Post by Miroslav Lachman

Post by Dag-Erling Smørgrav
Uh, 79725167 - 63 = 79725104 and 79725104 - 39845888 = 39879216. How
did you arrive at 39879105?

I am sorry, it was my confusion.
ad4: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=79725056

off-by-111...

Are you sure 'smartctl -l error' reports only one error?

DES

[Miroslav Lachman]

Post by Dag-Erling Smørgrav

Post by Miroslav Lachman

Post by Dag-Erling Smørgrav
Uh, 79725167 - 63 = 79725104 and 79725104 - 39845888 = 39879216. How
did you arrive at 39879105?

I am sorry, it was my confusion.
ad4: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=79725056

off-by-111...
Are you sure 'smartctl -l error' reports only one error?

There is really only one error. The example from my e-mail is half a
year old, but the disk is running fine from this time. The error occured
at the initial gmirror sync. No more errors shown after rewriting the
disk with zeros.

As you can see, there are really two different numbers
LBA=79725056 in messages and LBA = 0x04c0826f = 79725167 in SMART log.


***@edith ~/# zcat /var/log/messages.3.bz2 | grep LBA

Sep 23 23:58:00 edith kernel: ad4: FAILURE - READ_DMA
status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=79725056

-----------------------------

***@edith ~/# smartctl -l error /dev/ad4

smartctl version 5.38 [amd64-portbld-freebsd7.2] Copyright (C) 2002-8
Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
ATA Error Count: 1
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was
active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 6f 82 c0 44 Error: UNC at LBA = 0x04c0826f = 79725167

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 00 00 82 c0 44 00 25d+23:23:36.710 READ DMA
c8 00 00 00 81 c0 44 00 25d+23:23:36.710 READ DMA
c8 00 00 00 80 c0 44 00 25d+23:23:36.710 READ DMA
c8 00 00 00 7f c0 44 00 25d+23:23:36.710 READ DMA
c8 00 00 00 7e c0 44 00 25d+23:23:36.710 READ DMA

Miroslav Lachman

[Dag-Erling Smørgrav]

As you can see, there are really two different numbers LBA=79725056 in
messages and LBA = 0x04c0826f = 79725167 in SMART log.

I don't know how comfortable you are reading kernel code, but I would
suggest looking through the atadisk driver to see why the numbers are
different.

And if you're comfortable *writing* kernel code, I would suggest
implementing WORF in geom_mirror :)

DES

[Pieter de Goeje]

Post by Dag-Erling Smørgrav
And if you're comfortable *writing* kernel code, I would suggest
implementing WORF in geom_mirror :)

I am intrigued, what is this WORF you speak of?
Google says it's a certain character from a popular sci-fi show...

- Pieter

[Dag-Erling Smørgrav]

Post by Pieter de Goeje

Post by Dag-Erling Smørgrav
And if you're comfortable *writing* kernel code, I would suggest
implementing WORF in geom_mirror :)

I am intrigued, what is this WORF you speak of?

Write On Read Failure. It means that if you can't read a sector but you
have (or can recreate) a copy of the data that's supposed to be on it,
you rewrite that data to force the disk to reallocate the sector. I've
done this manually several times (dd'ed a sector from the other disk in
a mirror). I believe I even posted the procedure at some point; I'll
check my archive.

DES

[Miroslav Lachman]

Post by Dag-Erling Smørgrav

As you can see, there are really two different numbers LBA=79725056 in
messages and LBA = 0x04c0826f = 79725167 in SMART log.

I don't know how comfortable you are reading kernel code, but I would
suggest looking through the atadisk driver to see why the numbers are
different.
And if you're comfortable *writing* kernel code, I would suggest
implementing WORF in geom_mirror :)

As I sent to Pieter, I am not a C programmer, so I cannot read kernel
code. I was poor webdeveloper before I turned in to sysadmin about 5
years ago. My programming knowledge ends with PHP / SQL / JS and SH
coding :)

Miroslav Lachman

[Alex Keda]

Post by Eugeny N Dzhurinsky
Hello, all!
Mar 8 12:00:24 localhost smartd[795]: Device: /dev/ad4, 2 Currently unreadable (pending) sectors
Mar 8 12:00:24 localhost smartd[795]: Device: /dev/ad4, 2 Offline uncorrectable sectors
smartctl did really show that something is wrong with my HDD, but still no
remaps - just read errors.
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 60% 1198 222342559
# 2 Extended offline Completed: read failure 60% 1187 222342557
# 3 Extended offline Completed: read failure 60% 1180 222342559
# 4 Short offline Completed without error 00% 1178 -
# 5 Extended offline Aborted by host 90% 1178 -
and
ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
...
Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
...
Now can I find out which file owns the LBAs 222342557 and 222342559 ? How do I
force remapping of these sectors? I assume that I have to write something
directly to the sectors?

use mhdd