Bugzilla Bug 111551

I'd like to propose a new document to be written to detail how extra storage
can
be used on gentoo.

Suggested sections:

HAL/pmount
A description of what HAL is and how it can automate much of the other stuff in
the document.

Creating, formatting, and using more partitions
A reminder of fdisk/mkfs and how to manipulate fstab

Using external usb storage devices
State the required kernel config, talk about how they appear as scsi disks, and
give an example fstab entry.

Using external firewire storage devices
I have no idea how these work.

Mounting your windows partitions
Also talk about the state of ntfs write support, and how to make vfat writable
by normal users (umask stuff)

Gentoo and your ipod
Talk about the extra software and filesystems needed to manipulate your ipod
music library

"I started writing this guide. I will post something soon"

Created an attachment (id=86657) [details]
external-devices.xml

I just included the fist version of external-devices.xml
this is my first time writing such thing so be free to comment/edit

Dimitry <<diox>> Bradt

Created an attachment (id=86658) [details]
external-devices.xml

external-devices.xml

Created an attachment (id=86660) [details]
external-devices.xml

external-devices.xml

Created an attachment (id=86661) [details]
external-devices.xml

I did a spell-check and it is now completely valid :))

Created an attachment (id=86662) [details]
external-devices.xml

# <i>foo</i> in stead of #<i>foo</i>

Created an attachment (id=86663) [details]
external-devices.xml

fixed some line-breaks (for reading) // spelling errors

Created an attachment (id=86664) [details]
external-devices.xml

renice the code listings

Created an attachment (id=86665) [details]
external-devices.xml

fixed :
*linewidth=79

*<p>foo</p> 
    to
<p>
foo
</p>

Created an attachment (id=86671) [details]
external-devices.xml

Created an attachment (id=86672) [details]
external-devices.xml

more information about hal and supported packages
should be well-formatted now

Created an attachment (id=86673) [details]
external-devices.xml

I went through the guide fixing the coding style, some misc typos and
capitalization. (Everything discussed with the author in #gentoo-doc). Now I
strongly recommend one of our native speakers (who we love so much :p) read the
guide and checked if everything in it is correct as far as english is
concerned. Big thanks for writing the guide, diox, let's now cooperate to have
it published soon. :)

thx rane for your help.
It's now being spellchecked :))

Thanks for producing this. Here are some comments. I've been thinking about
this document for some time but haven't had time to actually write it, so my
ideas are quite refined. Please don't take any of this negatively, I'm
extremely grateful that someone has started to put this together and I welcome
any disagreements :)

"<abstract>
This howto will show you how to use external (usb/firewire) storage devices.
</abstract>"

While thats probably the most important part of the document, I'd also like to
see it covering the other items I mentioned in comment #0: mounting windows
partitions, and ipod support.

"One example of the functionality provided
by HAL is when you plug in a USB storage device. HAL can automatically create
an appropriate entry in the /etc/fstab file and create the corresponding mount
point in <path>/mnt</path>."

While HAL can probably do this, I'm pretty sure that we don't use this kind of
setup in Gentoo anymore. Also you mentioned pmount in the <title> but didn't
mention it at all in the text. Plus I'm not sure if pmount is used any longer.
We need to figure this out.

Also, I don't think theres much point linking to the outdated HAL design page.

"Before emerging the needed programs you should enable <c>KOBJECT_UEVENT</c>
and
<c>CONFIG_HOTPLUG</c> in your kernel."

Don't mention CONFIG_HOTPLUG - the option is extremely hidden (nobody will
realistically disable it), and on older kernels, udev will be completely
non-functional without it. Just assume it is present.

Are you sure that UEVENT is involved with the whole HAL thing? It's relatively
new, and HAL worked just fine before it appeared. Can you confirm what role it
plays? (Not in the doc, just convince me.. :))

At this point the document is written completely KDE-centric, and could easily
be interpreted that HAL is KDE-specific. Also, if you have USE=hal and KDE
installed, do you really need to manually emerge hal, dbus, hotplug, and ivman?

The whole KDE thing looks really scary and complicated - to get decent
automounting you have to go through the above kerfuffle, and create fstab
entries, and create udev rules, and configure ivman? Ouch. On GNOME it is dead
simple out of the box - just plug a device in, and it gets mounted and an icon
appears on your desktop.

I just asked on IRC and people suspect that KDE 3.4's HAL integration is broken
which is why the more complicated process is needed. If this is the case, I
suggest that we wait for KDE 3.5 to be stable (apparently happening soon)
before documenting KDE+HAL in the report. Comments?

I don't have time to go over the rest in detail now, but at a glance: Some of
the kernel configuration stuff there is covered (or will be covered) in
separate documents. You do not mention the limitations of the in-kernel ntfs
driver or why captive is needed. As captive is not in the stable tree, I don't
think it should be detailed here.

Thanks!!

Progress external-devices.xml

changes made to external-devices.xml (patch)
-------------------------------------------
1)
<abstract>
This howto will show you how to use external (usb/firewire) storage devices,
and how to mount and use windows partitions
</abstract>

2)
removed the link to the design page

3)
mention about udev and link to the Gentoo udev Guide (/doc/en/udev-guide.xml)

currently busy on;
-----------------
i'm now doing research on pmount / hal in general => i got my hal perfectly
working with hal and pmount (but that was a few months back).
i'm currently emerging gnome to test hal again (i'm a fluxbox-user)

i'm now making a big difference between gnome and kde (a whole new chapter) 
because the configuration that is needed is completely different => gnome has 
some build-in programms : the only thing needed would be udev so i'll put a 
link to the gentoo udev guide.

also; i've just put the CONFIG_HOTPLUG in the howto to make sure udev
was present and working => i'll now just mention to have udev installed

Anyways i got these options from
http://gentoo-wiki.com/HOWTO_D-BUS,_HAL,_KDE_media:/ => i've tested that howto
and seems to work on my system => i didn't wonder if it was really needed but
i'll find if it's really needed. I'll search around on irc for ppl that know 
what these options do.

i heard from irc-folks kde *maybe* periodicallychecking checks the system for
new devices with the kernel .. (and that's why these options are needed ..)

anyways back to research
any help would be very apreciated :))

greetings
diox

changes made to external-devices.xml (patch)
-------------------------------------------

1) added troubleshooting part

KOBJECT_UEVENT needed for kernels < 2.6.17
after => now you just need hotplug and net enabled (but CONFIG_HOTPLUG and
        CONFIG_NET should be enabled for netlink socket to be enabled)
        (requirements of udev)

<note>
These basic options should be turned on by default. If someone turns them off
they should already know the consequences

 udev >= 079 need for kernels 2.6.16 and earlier CONFIG_NET, CONFIG_HOTPLUG and
CONFIG_UEVEN
T and for kernels 2.6.17 and later, it just need CONFIG_NET, CONFIG_HOTPLUG as
CONFIG-UEVENT
 have been folded into CONFIG_NET
</note>

currently busy on
-----------------

* chmod +x rights on hal ?

* Testing hal-related programs => gnome-extra/hal-device-manager,
update-notifier, gnome-vol
ume-manager

Created an attachment (id=91810) [details]
external-devices.diff

external-devices.diff

i finished the guide but i'm currently working out troubleshooting; tough, as i
have no input on it (by users), i basicly can't add it.

i've read that a common problem is that some files must have chmod+x rights (in
ubuntu)(but if this the same under gentoo, i must find out) Here Inserting an
external device just works (usb/scsi).

Can this guide maybe already put into the cvs and await user comments ?

greetings
diox

Created an attachment (id=91823) [details]
external-devices

external-devices

Created an attachment (id=91834) [details]
Edited Edition

Edited diox's original XML file.

temp online at 
http://dev.gentoo.org/~fox2mike/doc/en/additional-storage.xml

Created an attachment (id=91841) [details]
external-devices.xml

Ok, checked through it, all is good :D

(In reply to comment #23)
> Created an attachment (id=91841) [edit] [details]
> external-devices.xml
> 
> Ok, checked through it, all is good :D
> 

Thx, welp for your editing on this guide !
greetings
diox

Created an attachment (id=92348) [details]
external-devices.xml

Euh... I've tested the howto (apart from the firewire/SCSI section... I don't
have either), and it all works for me :)

okay.

Daniel, could you read and comment on the guide now when it's ready?

Some comments made on IRC. I only read this really quickly, so there many be
more points or I might have misread parts...:

Many points made in comment #15 have not been addressed

We should not recommend users to get packages out of the the testing tree in
the official documentation

If KDE really does require ivman for automount-like functionality, then we
should split into two sections for KDE and GNOME. GNOME provides ivman-like
functionality out of the box, kinda dumb to merge it there.

USB and firewire storage should be seperated into their own sections.

The doc talks about not using auto in fstab because that will mean it uses
msdos instead of vfat. Actually, you can change this in /etc/filesystems.

Then a little further down, the above point is repeated, this time it does talk
about /etc/filesystems!?

Does windows FAT really support UTF-8? I thought not, but I'll happily accept
otherwise if someone can convince me that I'm wrong.



I'm short of time right now, I am on holiday from 26th July until 11th August,
then I'm moving out to the US, etc. As usual, I'm the bottleneck. Although I
have fairly clear ideas what *I'd* like to see in such a document, I understand
completely if the GDP want to take this over. If that happens, please ensure
accuracy and please do make sure that defects such as the above do not get
through, we have high documentation standards to meet :)

(In reply to comment #27)
> If KDE really does require ivman for automount-like functionality, then we
> should split into two sections for KDE and GNOME. GNOME provides ivman-like
> functionality out of the box, kinda dumb to merge it there.

Depends on what you call "automount". If a dialog asking "what to do" with
option "do that next time as well" is enough, then it doesn't need ivman.

> I'm short of time right now, I am on holiday from 26th July until 11th August,
> then I'm moving out to the US, etc.

Enjoy :)

Created an attachment (id=92655) [details]
addional storage

hi;
thx for your input on this doc all. Hope to see it published soon:

Last Changes (on dsd's comment):
** i removed the line to use the experimental arch; 
** added an extra section (with a <note> where i tell them it's for kde-only.
(because in gnome that works without it. :) 
** separated USB and firewire storage into their own sections.
** separated IPOD part
** putted /etc/filesystem part together .. 

greetings
diox

Created an attachment (id=92656) [details]
external-devices.xml

external-devices.xml

deleted the utf-8 part; it actually made no sense :)
greetings
diox

Created an attachment (id=92708) [details]
external-devices.xml

Small cleanups (concerning only the beginning of the doc):

a) better coding style (still needs to get checked)
b) changed some occurences of "a USB" to "an USB"
c) stuff like "see this link: http://..." is ugly
d) no need to speak about syslog and virtual consoles when the `dmesg` is
enough
e) probably other fixes I just forgot about

It's "a USB" -- you wouldn't say "an universal"

Still some points in comment #15 to be addressed in full...

Created an attachment (id=92712) [details]
external-devices.xml - by jkt

(In reply to comment #32)
> It's "a USB" -- you wouldn't say "an universal"

Oops, fixed, thanks.

I got bored and tried to rewrite this guide; current state is attached,
rendered HTML is at http://gentoo.basa.flaska.net/doc/en/external-devices.xml
(work in progress).

*** Bug 122285 has been marked as a duplicate of this bug. ***

(In reply to comment #33)
Thanks all for helping with the guide. 

dsd; is there something that still needs editing ?
greetings
diox

Let's wait for Jan to finish the new version before we review it.

*poke*

iirc, when this doc was last updated, ivman was still ~ARCH keyworded - it's
marked stable now, maybe there should be a section added on it? Or is that what
Jan's working on now? Whatever, this doc needs a 'lil poke up the backside :)

So, continuing where I've stopped:

* .config blurb needs to be redone (touching .config by hand is evil, use
menuconfig)
* intra-document links using #doc_chapX stuff are bad
* why do I need sg3_utils?
* we do need the "auto" stuff for the USB sticks as it could contain more
different filesystems
* it's called "external-devices", why does it talk about windows filesystems?
(though I see that it was a part of the original request)
* '<p><c>umask</c>: Octal file permissions<br />...</p>' is evil
* I'm not sure if it's a good think to talk about Captive as long as it's
masked...

So, in short, I'd throw away the part about manual mounting etc and write a
guide about how to make things Just Work (tm) with a troubleshooting section.

(In reply to comment #38)
> * it's called "external-devices", why does it talk about windows filesystems?
> (though I see that it was a part of the original request)

The original request was for a document about *additional* storage, not
external. I still think that mounting windows partitions should be documented
as it is a common scenario and is not documented elsewhere. IMO this document
should lean away from the external-only approach that has apparently been
taken.

I think manual mounting should be documented, but certainly wouldn't object to
such a document being more geared towards the automatic systems e.g. hal which
realistically don't need special considerations at all, unless you are a KDE
user *cough* ;)

(In reply to comment #38)
> So, continuing where I've stopped:
Thx for taking the time for enchancing this guide.
> * why do I need sg3_utils?
Have you ever seen a working scsi device without sg3_utils?
> * it's called "external-devices", why does it talk about windows filesystems?
> (though I see that it was a part of the original request)
That was just a name i gave. I didn't tought of it to be the actual document
name back then.
> * '<p><c>umask</c>: Octal file permissions<br />...</p>' is evil
 what's so evil about octal file permissions? it's something i'd call basic
knowledge when mounting windows partitions. (as dsd requested). You don't want
to tell users to mount a windows partition with write access etc (what you get
when you don't tell them about octal file permissons). You could end up with a
b0rked windows installation. I think we should include those mount-options.
(And you can't do that without the essential information on octal file
permissions imho)


(In reply to comment #39)
> I think manual mounting should be documented, but certainly wouldn't object to
> such a document being more geared towards the automatic systems e.g. hal which
> realistically don't need special considerations at all, unless you are a KDE
> user *cough* ;)
> 

These things could indeed be dropped, however most users want this to go
automaticly.that's in the first place i included this into this doc.

When the discussion about what this guide should and shouldn't contain, i'd be
pleased to finish the doc.

Created a little fix for the Xfce section.

http://pastebin.ca/379573

re-assigning to docs-team.
Please give it some love :)

What is this guide actually about? It started one way and already contains info
on  what hal/pmount is and can be configured (which is somewhat more advanced)
yet sais that "sdb1" is cryptic... it asks to configure a kernel by editing
.config (this is not something I want in Gentoo's docs) and jumps from USB/iPod
(which is external hardware) to Windows partitions.

Imo, the document is looking for its own identity.

Mounting Windows partitions is a FAQ imo and should be put there. 

External hardware is a generic topic; if you start documenting USB, iPod then
the next thing is ${my digital camera}, ${my Mio GPS}, ${my bluetooth business
card exchanger} etc. Wouldn't it be better to talk about external/detacheable
devices in general and giving pointers on how to access them rather than start
a database on those devices?

(In reply to comment #43)
> What is this guide actually about? It started one way and already contains info
> on  what hal/pmount is and can be configured (which is somewhat more advanced)
> yet sais that "sdb1" is cryptic... it asks to configure a kernel by editing
> .config (this is not something I want in Gentoo's docs) and jumps from USB/iPod
> (which is external hardware) to Windows partitions.
> 
> Imo, the document is looking for its own identity.
> 
> Mounting Windows partitions is a FAQ imo and should be put there. 
> 
> External hardware is a generic topic; if you start documenting USB, iPod then
> the next thing is ${my digital camera}, ${my Mio GPS}, ${my bluetooth business
> card exchanger} etc. Wouldn't it be better to talk about external/detacheable
> devices in general and giving pointers on how to access them rather than start
> a database on those devices?

This doc has lost sight of what it's trying to be -- no, scratch that. No one
ever really agreed on what it should be; everyone submitted different content
and patches. There never seemed to be a unified goal; no one ever knew exactly
what the doc should be.

I agree that it should be about detachable devices in general, but really, I
see nothing in this doc that's clear and coherent enough to be worth adding to
our repo. Personally, I'd just mark as RESO INSUFFICIENT or something and close
it. That, or *yet another* entirely new fresh draft needs to be written.
Something a lot shorter that just focuses on external devices and mounting.

Okay then, for a rewrite...

1/ What is this guide about?
   -> detachable storage and technologies to mount those

2/ Available tools / technologies
   -> famd, hal, 
   -> not in great detail, but sufficient for the reader to know what the tools
do in case he encounters them on his system. I'm sure that configuring those is
beyond the scope of user guides as this should happen well behind the scenes

3/ Mounting media
   -> usb-storage, firewire, ...

4/ Pitfalls when accessing such media
   -> permissions

Again, this shouldn't become a 'how to configure hal to do funky stuff' or 'how
to fiddly with <insert rare file system>' but rather a guide on the
technologies, how they work and how you can attach/detach remote media. The
more specific something becomes, the less content it should get in the guide
(no "connect to a sony handycam dcr-sr52e through module soso with config
blabla and mount-option foobar").

One of my comments about earlier proposed document drafts was that they missed
my intentions in that I had in mind an "additional storage guide", _not_ an
"external storage guide". Just look at the title of the bug compared to the
filenames of most of the attachments.

Of course, "additional" encompasses the external part, but that's my point: I
was hoping for something to cover the broader topics.

The user feedback that inspired this documentation request was as follows:
"How to umask a vfat partition so that users can access it?"
i.e. the user was pointing out we have no documentation on that.

Things have changed since, maybe HAL handles that for us (can someone with
windows partitions confirm -- does it automount them accessible by users)? In
which case we just need a document mentioning HAL to solve the windows/FAT
problem, and then the other topics which I haven't yet reconsidered.

I'll hopefully find some time to reconsider those and post my new vision for
this document on this bug, please poke me on IRC if I forget.

A "how to umask a vfat partition" is a one-paragraph explanation that could be
placed in the FAQ easily...

I felt that it drew attention to a larger topic which we haven't documented,
that's all..

Does this need to be W3C valid btw? If so, what? ..

(In reply to comment #49)
> Does this need to be W3C valid btw? If so, what? ..

That has nothing to do with anything.

Well, much as I hate marking stuff anything but FIXED, since that's like
throwing it away and not really ever taking care of it, it's been so long that
the original goal has been either forgotten or made irrelevant, or ...
something.

If anyone can distill it into a FAQ to add to faq.xml, please reopen with your
proposal.