[Hamara-devel] flash drives, SSD, wear-levelling and low-level instructions

[shirish]

Hi all,
Just had a bit of long conversation with aparna and this is the gist of it.

The thing is I had not slept for the last 2 days while working yesterday 
and had been trusting debian-documentation for the right set of 
instructions to share forward.

https://www.debian.org/releases/stable/amd64/ch04s03.html.en

While some of it is good some of it is somewhat suspect.

For instance if I was in the right frame of mind the 'sync' command 
would have alerted me that probably some things are not right here.

Some older and newer documentation links need to be shared :-

http://lkml.iu.edu/hypermail/linux/kernel/0505.1/1249.html - While the 
article is about 10 years old, at least in India the quality of 
pen-drives that we get are very suspect so the issue shared there is 
probably relevant today as well.

Part of the issue is also wear-levelling which has been discussed to 
death but without letting the joe on the street know about 
flash-controllers and actually knowing what he paid for, it's a lottery 
if it works.

A slightly more newer lwn.net article also throws light on this specific 
problem.

https://lwn.net/Articles/428584/ (it is a bit of a long read, probably 
for the weekend :) )

And as far as I know there aren't any consumer-facing tools which tells 
about the quality of a flash drive/thumbdrive/pen-drive and which are 
good methods and which are not. Most apps. use ExFAT or FAT32 and leave 
the intelligence to the drive controller and that's about it.

In Debian though AFAIK there isn't a single app. which does this till 
date. In Ubuntu there is something called 'Startupdiskcreator' which I 
have no idea about. I have used unetbootin in the past and had good as 
well as bad experiences with it and because of the variety of usb disks 
that is out there didn't care or even had time to actually explore 
reasons for failure when it failed.

For fixing corrupted media there is testdisk which I shared with her :-

[$] aptitude show testdisk

Package: testdisk
State: installed
Automatically installed: no
Version: 6.14-3+b4
Priority: optional
Section: admin
Maintainer: Jean-Michel Kelbert <kelbert at debian.org>
Architecture: amd64
Uncompressed Size: 1,242 k
Depends: e2fslibs (>= 1.41.0), libc6 (>= 2.14), libcomerr2 (>= 1.01), 
libjpeg62-turbo (>= 1.3.1), libncursesw5 (>= 5.6+20070908), 
libntfs-3g853,libtinfo5, libuuid1 (>= 2.16), zlib1g (>= 1:1.1.4), ntfs-3g

Description: Partition scanner and disk recovery tool, and PhotoRec file 
recovery tool
  TestDisk checks the partition and boot sectors of your disks. It is 
very useful in forensics, recovering lost partitions. It works with :
  * DOS/Windows FAT12, FAT16 and FAT32
  * NTFS ( Windows NT/2K/XP )
  * Linux Ext2 and Ext3
  * BeFS ( BeOS )
  * BSD disklabel ( FreeBSD/OpenBSD/NetBSD )
  * CramFS (Compressed File System)
  * HFS and HFS+, Hierarchical File System
  * JFS, IBM's Journaled File System
  * Linux Raid
  * Linux Swap (versions 1 and 2)
  * LVM and LVM2, Linux Logical Volume Manager
  * Netware NSS
  * ReiserFS 3.5 and 3.6
  * Sun Solaris i386 disklabel
  * UFS and UFS2 (Sun/BSD/...)
  * XFS, SGI's Journaled File System

  PhotoRec is file data recovery software designed to recover lost 
pictures from digital camera memory or even Hard Disks. It has been 
extended to search also for non audio/video headers. It searches for 
following files and is able to undelete them:
  * Sun/NeXT audio data (.au)
  * RIFF audio/video (.avi/.wav)
  * BMP bitmap (.bmp)
  * bzip2 compressed data (.bz2)
  * Source code written in C (.c)
  * Canon Raw picture (.crw)
  * Canon catalog (.ctg)
  * FAT subdirectory
  * Microsoft Office Document (.doc)
  * Nikon dsc (.dsc)
  * HTML page (.html)
  * JPEG picture (.jpg)
  * MOV video (.mov)
  * MP3 audio (MPEG ADTS, layer III, v1) (.mp3)
  * Moving Picture Experts Group video (.mpg)
  * Minolta Raw picture (.mrw)
  * Olympus Raw Format picture (.orf)
  * Portable Document Format (.pdf)
  * Perl script (.pl)
  * Portable Network Graphics (.png)
  * Raw Fujifilm picture (.raf)
  * Contax picture (.raw)
  * Rollei picture (.rdc)
  * Rich Text Format (.rtf)
  * Shell script (.sh)
  * Tar archive (.tar )
  * Tag Image File Format (.tiff)
  * Microsoft ASF (.wma)
  * Sigma/Foveon X3 raw picture (.x3f)
  * zip archive (.zip)

The tool is good but is primarily a CLI application. I don't know of any 
GUI applications for fixing corrupted media. It is a good tool provided 
we have a good machine, quite a bit of memory, good speed of USB bus + 
media that responds.

The not so good thing is that the documentation for the tool is very 
sparse. Most of the man-page given/shared is the long description which 
is shared up, so that would have to be documented as well.

Lastly, also came across this :-

https://bugs.debian.org/785512

and the following is very pertinent to our discussion :-

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785512#30

"Hmm, I didn't find *one* that leaves the sstick useable for other things,
and allows to boot into a variety of other rescue iso images etc
at the same time. Please let me know which one?

(Fat16 - was that a joke on the current jessie installer images about
needing fat16 on the usb stick? The options explained on
https://www.debian.org/releases/stable/i386/ch04s03.html.en
are all either destructive of previous data, or don't play well as
carry on sticks for the emergency case, nor nor nor. ....)" - Norbert 
Preining

I know the gentleman for sometime so know that he knows what he's 
talking about.

If I am/were a customer then I wouldn't want just a solution just for 
installating the OS but also use that disk for everyday data backup and 
other stuff as well.

So while we can give the unetbootin as a probable solution for now and 
just document that, for ourselves we would need to understand and also 
give some recommendations about good flash memory and whatever else we 
come to know in a way which is easy for him/her to work with.

That's about it on the flash drive part.
-- 
Regards,
Shirish Agarwal,
Community Lead,
Hamaralinux.org