A userspace daemon required to take advantage of the new CacheFiles functionality (aka fscache) in the 2.6.30 kernel. Reproducible: Always Steps to Reproduce: 1. emerge sys-fs/cachefilesd 2. mkdir /var/fscache; /etc/init.d/cachefilesd start 3. mount -t nfs -o fsc server:/export /mnt/path
Created attachment 195418 [details] sys-fs/cachefilesd/cachefilesd-0.9.ebuild
Created attachment 195421 [details] sys-fs/cachefilesd/files/cachefilesd.conf
Created attachment 195422 [details] sys-fs/cachefilesd/files/cachefilesd.init
This works great for me, I am really looking forward to seeing this in the tree.
I had to modify cachefilesd.init to look in /sys/module/fscache (note the module vs modules). In cachefilsed.conf I had to make $PROG the absolute path. And my last suggestion, is to perhaps add a check if user_xattr is supported on the desired filesystem. Though I haven't performed any large tests on it, it seems to work well.
On http://en.gentoo-wiki.com/wiki/HOWTO_FS-Cache I've put some pointers. This has been derived and updated from http://www.gentoo-wiki.info/HOWTO_FS-Cache.
I am havin major issues with this - Aug 5 02:22:17 dnetwww1 cachefilesd[3258]: Unable to open /dev/cachefiles: errno 2 (No such file or directory) There is no proc entry for cachefiles, even though i is installed... dnetwww1 ~ # lsmod Module Size Used by cachefiles 26192 0 fuse 41152 0 nfs 208144 1 auth_rpcgss 31792 1 nfs fscache 21748 2 cachefiles,nfs Any thoughts, ideas?
My problem is that it crashes right after it starts. Haven't really started to investigate, but here's the part from dmesg. S-Cache: Cache "mycache" added (type cachefiles) CacheFiles: File cache on dm-2 registered Pid: 6228, comm: kslowd Tainted: P 2.6.30-gentoo-r4 #2 Call Trace: [<ffffffff8030d8b1>] ? cachefiles_read_waiter+0x55/0xf3 [<ffffffff80225ea2>] ? __wake_up_common+0x44/0x73 [<ffffffff802260bc>] ? __wake_up+0x30/0x44 [<ffffffff8023fb25>] ? __wake_up_bit+0x28/0x2d [<ffffffff802c987f>] ? ext3_truncate+0x4c5/0x8f8 [<ffffffff802688f0>] ? unmap_mapping_range+0x5f/0x218 [<ffffffff80268c37>] ? vmtruncate+0xc2/0xe2 [<ffffffff8028c3e7>] ? inode_setattr+0x22/0x10a [<ffffffff802c7493>] ? ext3_setattr+0x122/0x160 [<ffffffff8028c639>] ? notify_change+0x16a/0x27d [<ffffffff8030ae35>] ? cachefiles_attr_changed+0x104/0x173 [<ffffffff8030b417>] ? cachefiles_lookup_object+0x7f/0xca [<ffffffff802c0f29>] ? fscache_lookup_object+0x109/0x120 [<ffffffff802c13d8>] ? fscache_object_slow_work_execute+0x1fc/0x709 [<ffffffff80255f89>] ? slow_work_thread+0x268/0x41b [<ffffffff80225ea2>] ? __wake_up_common+0x44/0x73 [<ffffffff8023fb4c>] ? autoremove_wake_function+0x0/0x2e [<ffffffff80255d21>] ? slow_work_thread+0x0/0x41b [<ffffffff8023f7f4>] ? kthread+0x54/0x80 [<ffffffff8020bdba>] ? child_rip+0xa/0x20 [<ffffffff8023f7a0>] ? kthread+0x0/0x80 [<ffffffff8020bdb0>] ? child_rip+0x0/0x20 CacheFiles: I/O Error: Readpage failed on backing file 8000000000000810 FS-Cache: Cache cachefiles stopped due to I/O error CacheFiles: File cache on dm-2 unregistering FS-Cache: Withdrawing cache "mycache"
(In reply to comment #7) > Aug 5 02:22:17 dnetwww1 cachefilesd[3258]: Unable to open /dev/cachefiles: > errno 2 (No such file or directory) Did cachefiles register a device? # cat /sys/devices/virtual/misc/cachefiles/dev 10:57 The minor number may be different on your system, but if it is registered, (a) your udev is busted, and (b) a manual mknod should work. # mknod /dev/cachefiles c 10 57 Otherwise, you have a kernel bug of some sort...
(In reply to comment #8) > My problem is that it crashes right after it starts. Haven't really started to > investigate, but here's the part from dmesg. Looks like this already reported issue? http://lkml.org/lkml/2009/6/21/159 There doesn't seem to be a definitive fix for it yet.
I have it running now - working like a charm. Problem was udev wasnt set to start at sysinit. (In reply to comment #9) > (In reply to comment #7) > > Aug 5 02:22:17 dnetwww1 cachefilesd[3258]: Unable to open /dev/cachefiles: > > errno 2 (No such file or directory) > Did cachefiles register a device? > # cat /sys/devices/virtual/misc/cachefiles/dev > 10:57 > The minor number may be different on your system, but if it is registered, (a) > your udev is busted, and (b) a manual mknod should work. > # mknod /dev/cachefiles c 10 57 > Otherwise, you have a kernel bug of some sort...
Hi All, There is a kernel patch to fix the "crash on startup" of cachefilesd - http://lkml.org/lkml/diff/2009/8/4/284/1
(In reply to comment #12) > Hi All, > There is a kernel patch to fix the "crash on startup" of cachefilesd - > http://lkml.org/lkml/diff/2009/8/4/284/1 Apologies, you will also need this patch http://lkml.org/lkml/diff/2009/6/22/62/1
Still getting the same error even with these patches... Aug 11 00:34:38 dnetwww1 Pid: 2065, comm: kslowd Not tainted 2.6.30-gentoo-r4 #1 Aug 11 00:34:38 dnetwww1 Call Trace: Aug 11 00:34:38 dnetwww1 [<ffffffffa012620a>] 0xffffffffa012620a Aug 11 00:34:38 dnetwww1 [<ffffffff80231dbe>] 0xffffffff80231dbe Aug 11 00:34:38 dnetwww1 [<ffffffff80232037>] 0xffffffff80232037 Aug 11 00:34:38 dnetwww1 [<ffffffff8024e4fd>] 0xffffffff8024e4fd Aug 11 00:34:38 dnetwww1 [<ffffffff80277663>] 0xffffffff80277663 Aug 11 00:34:38 dnetwww1 [<ffffffff802f4a7d>] 0xffffffff802f4a7d Aug 11 00:34:38 dnetwww1 [<ffffffff8027ff91>] ? 0xffffffff8027ff91 Aug 11 00:34:38 dnetwww1 [<ffffffff8028a437>] ? 0xffffffff8028a437 Aug 11 00:34:38 dnetwww1 [<ffffffff80232042>] ? 0xffffffff80232042 Aug 11 00:34:38 dnetwww1 [<ffffffff8028a5dd>] 0xffffffff8028a5dd Aug 11 00:34:38 dnetwww1 [<ffffffff802b13cc>] 0xffffffff802b13cc Aug 11 00:34:38 dnetwww1 [<ffffffff802f25d8>] 0xffffffff802f25d8 Aug 11 00:34:38 dnetwww1 [<ffffffff802b1644>] 0xffffffff802b1644 Aug 11 00:34:38 dnetwww1 [<ffffffffa01234e8>] 0xffffffffa01234e8 Aug 11 00:34:38 dnetwww1 [<ffffffffa0124dd7>] ? 0xffffffffa0124dd7 Aug 11 00:34:38 dnetwww1 [<ffffffffa0123b33>] 0xffffffffa0123b33 Aug 11 00:34:38 dnetwww1 [<ffffffffa009b2dc>] 0xffffffffa009b2dc Aug 11 00:34:38 dnetwww1 [<ffffffff80227b74>] ? 0xffffffff80227b74 Aug 11 00:34:38 dnetwww1 [<ffffffffa009b804>] 0xffffffffa009b804 Aug 11 00:34:38 dnetwww1 [<ffffffff8027670e>] 0xffffffff8027670e Aug 11 00:34:38 dnetwww1 [<ffffffff80235923>] ? 0xffffffff80235923 Aug 11 00:34:38 dnetwww1 [<ffffffff8024e522>] ? 0xffffffff8024e522 Aug 11 00:34:38 dnetwww1 [<ffffffff80276493>] ? 0xffffffff80276493 Aug 11 00:34:38 dnetwww1 [<ffffffff80276493>] ? 0xffffffff80276493 Aug 11 00:34:38 dnetwww1 [<ffffffff8024e178>] 0xffffffff8024e178 Aug 11 00:34:38 dnetwww1 [<ffffffff80211b8a>] 0xffffffff80211b8a Aug 11 00:34:38 dnetwww1 [<ffffffff8024e122>] ? 0xffffffff8024e122 Aug 11 00:34:38 dnetwww1 [<ffffffff80211b80>] ? 0xffffffff80211b80 Aug 11 00:34:38 dnetwww1 CacheFiles: I/O Error: Readpage failed on backing file 4000000000000810 Aug 11 00:34:38 dnetwww1 FS-Cache: Cache cachefiles stopped due to I/O error
after i cleared my fscache directory completely this now seems to be working once again. I'll monitor for 48 hours. Aug 11 01:00:31 dnetwww1 CacheFiles: Loaded Aug 11 01:00:31 dnetwww1 FS-Cache: Cache "mycache" added (type cachefiles) Aug 11 01:00:31 dnetwww1 CacheFiles: File cache on sdb1 registered Aug 11 01:00:32 dnetwww1 rpcbind: cannot create socket for udp6 Aug 11 01:00:32 dnetwww1 rpcbind: cannot create socket for tcp6 Aug 11 01:00:32 dnetwww1 rpc.statd[3849]: Version 1.1.6 Starting Aug 11 01:00:32 dnetwww1 rpc.statd[3849]: Flags: Aug 11 01:00:32 dnetwww1 rpc.statd[3849]: Unable to read state from '/var/lib/nfs/state': errno 0 (Success) Aug 11 01:00:32 dnetwww1 cachefilesd[3777]: Scan complete Aug 11 01:00:32 dnetwww1 cachefilesd[3777]: Decant (all 423) A
Created attachment 209869 [details] another cachefilesd ebuild This is my version of the ebuild (didn't know if this when i created it. It is slightly different so may be the future mantainer can be intrested in merging those file BTW it should depend on RDEPEND="sys-apps/dnotify" as the README says
i forgotten a thing: sorry for the tar.gz but it is just more handy then attaching all of the files in there :D
(In reply to comment #16) > BTW it should depend on RDEPEND="sys-apps/dnotify" as the README says No. It needs a filesystem that supports dnotify, but does not require the user-land "dnotify" application for building nor running.
Oh that was the meaning of the readme then.... thanks :D
Created attachment 209874 [details] new init script with xattr check I've tryed a way to check for xattr and i also fixed a typo. This files make obsolete the one in my tar.gz
Created attachment 210436 [details] Improoved cachefilesd ebuild With the help of Diego "Flameeyes" Pettenò i've reviewed my ebuild and the init script too (attached soon)
Created attachment 210437 [details] cachefilesd init script The new init script
Created attachment 211597 [details] a small fix for the init script
Created attachment 213770 [details] cachefilesd ebuild another clean up on the ebuild. thanks to sedzimir
Created attachment 213774 [details] improved init script clean up and new features in the init script: it reads the cachefilesd.conf config file to get the cache dir
It doesn't work for me. I use ~x86 system, the latest ebuild (with ~x86 keyword added) and vanilla 2.6.32.3 kernel with the following cachefs setup: # # Caches # CONFIG_FSCACHE=m CONFIG_FSCACHE_STATS=y # CONFIG_FSCACHE_HISTOGRAM is not set # CONFIG_FSCACHE_DEBUG is not set # CONFIG_FSCACHE_OBJECT_LIST is not set CONFIG_CACHEFILES=m # CONFIG_CACHEFILES_DEBUG is not set # CONFIG_CACHEFILES_HISTOGRAM is not set cachefilesd starts successfully, I can mount nfs directory with -o fsc, but stats shows that cache is not used: from /etc/mtab: 172.19.0.1:/ /mnt/orionis nfs4 ro,user=andrew,fsc,addr=172.19.0.1,clientaddr=172.19.0.12 0 0 from /proc/mounts: 172.19.0.1:/ /mnt/orionis nfs4 ro,relatime,vers=4,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.19.0.12,fsc,addr=172.19.0.1 0 0 172.19.0.1:/1Anime /mnt/orionis/1Anime nfs4 ro,relatime,vers=4,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.19.0.12,fsc,addr=172.19.0.1 0 0 /proc/fs/nfs/volumes: NV SERVER PORT DEV FSID FSC v4 ac130001 801 0:19 0:0 no v4 ac130001 801 0:20 f105e6094ea34de7 no /var/fscache contains only base layout but not cached files: ls -R /var/fscache/ /var/fscache/: cache graveyard /var/fscache/cache: /var/fscache/graveyard: My cachefilesd config: dir /var/fscache tag mycache brun 10% bcull 7% bstop 5% frun 10% fcull 7% fstop 5% df shows 29% available, so it should be ok. Why cache is not engaged?
Additional info: cache modules are loaded and used: cachefiles 29901 1 fscache 26083 1 cachefiles /proc/sys/fs/nfs has no nfs_fscache* files. Is this due to disabled cache debug and histogram? And /proc/fs/fscache/stats: FS-Cache statistics Cookies: idx=0 dat=0 spc=0 Objects: alc=0 nal=0 avl=0 ded=1 ChkAux : non=0 ok=0 upd=0 obs=0 Pages : mrk=0 unc=0 Acquire: n=0 nul=0 noc=0 ok=0 nbf=0 oom=0 Lookups: n=0 neg=0 pos=0 crt=0 tmo=0 Updates: n=0 nul=0 run=0 Relinqs: n=0 nul=0 wcr=0 rtr=0 AttrChg: n=0 ok=0 nbf=0 oom=0 run=0 Allocs : n=0 ok=0 wt=0 nbf=0 int=0 Allocs : ops=0 owt=0 abt=0 Retrvls: n=0 ok=0 wt=0 nod=0 nbf=0 int=0 oom=0 Retrvls: ops=0 owt=0 abt=0 Stores : n=0 ok=0 agn=0 nbf=0 oom=0 Stores : ops=0 run=0 pgs=0 rxd=0 olm=0 VmScan : nos=0 gon=0 bsy=0 can=0 Ops : pend=0 run=0 enq=0 can=0 rej=0 Ops : dfr=0 rel=0 gc=0 CacheOp: alo=0 luo=0 luc=0 gro=0 CacheOp: upo=0 dro=0 pto=0 atc=0 syn=0 CacheOp: rap=0 ras=0 alp=0 als=0 wrp=0 ucp=0 dsp=0
Oh, it occurs that fscache must be compiled into the kernel in nfs client is. In my original setup nfs client was in the kernel, but fscache was as a modules. This way it doesn't work because: depends on NFS_FS=m && FSCACHE || NFS_FS=y && FSCACHE=y Now it works fine. Please add ~x86 keyword and I'm looking forward it in the portage.
Great work guys. I'm willing to add it to the tree soon, after i do some testing on my side. One of six nodes in my http cluster crashed soon after it started. Others works till now. BUG: unable to handle kernel NULL pointer dereference at 0000000000000040 IP: [<ffffffff811083c0>] __fscache_read_or_alloc_pages+0x1cd/0x2fc PGD 11872a067 PUD 11872b067 PMD 0 Oops: 0000 [#1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:18.3/temp3_input CPU 0 Modules linked in: 8021q garp stp llc Pid: 21733, comm: apache2 Not tainted 2.6.32-gentoo-r3 #2 S2865 RIP: 0010:[<ffffffff811083c0>] [<ffffffff811083c0>] __fscache_read_or_alloc_pages+0x1cd/0x2fc RSP: 0018:ffff880118739b48 EFLAGS: 00010246 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff8800c2cf9580 RDI: 0000000000000000 RBP: ffff8800c2cf9580 R08: ffff880118738000 R09: 0000000000000000 R10: 00000000000b23a6 R11: ffff88011a091958 R12: ffff880114043810 R13: ffff88011cbadc80 R14: ffff880118739be4 R15: ffff880118739cb8 FS: 00007fd21415f750(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000040 CR3: 0000000118729000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process apache2 (pid: 21733, threadinfo ffff880118738000, task ffff88011f192680) Stack: ffff88010a2a86d8 00000000ffffff97 ffff88010a2a85c0 0000000000000001 <0> ffff880118739be4 ffff880118739cb8 ffff88010a2a86d8 ffffffff8118318c <0> 00000000000200da ffffffff810876a6 ffff88011b0a3d40 00000000ffffff8c Call Trace: [<ffffffff8118318c>] ? __nfs_readpages_from_fscache+0x77/0x163 [<ffffffff810876a6>] ? __alloc_pages_nodemask+0x10f/0x5e1 [<ffffffff8116aee2>] ? nfs_readpages+0x105/0x1b4 [<ffffffff81088fe5>] ? __do_page_cache_readahead+0x117/0x1a4 [<ffffffff8108908e>] ? ra_submit+0x1c/0x20 [<ffffffff81082bec>] ? generic_file_aio_read+0x1ff/0x549 [<ffffffff810b79e9>] ? do_sync_read+0xce/0x113 [<ffffffff810bad64>] ? cp_new_stat+0xe9/0xfc [<ffffffff81050c2f>] ? autoremove_wake_function+0x0/0x2e [<ffffffff810b8414>] ? vfs_read+0xaa/0x146 [<ffffffff810b856c>] ? sys_read+0x45/0x6e [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b Code: 74 06 48 8b 7a 28 ff d0 48 c7 c1 24 9a 9d 81 48 c7 c2 2c 9a 9d 81 4c 89 ee 48 89 ef e8 0b fb ff ff 85 c0 89 c3 78 5d 48 8b 45 70 <f6> 40 40 04 74 2c f0 ff 05 47 17 8d 00 48 8b 45 68 8b 4c 24 40 RIP [<ffffffff811083c0>] __fscache_read_or_alloc_pages+0x1cd/0x2fc RSP <ffff880118739b48> CR2: 0000000000000040 ---[ end trace 99d50162e34cc877 ]--- CacheFiles: Error: Overlong wait for old active object to go away object: OBJ855 objstate=OBJECT_LOOKING_UP fl=0 swfl=a ev=0[7b] ops=0 inp=0 exc=0 parent=ffff8801166e2700 cookie=ffff880114043810 [pr=ffff88011e43e588 nd=ffff88010a2a8418 fl=7] key=[28] '01000601f73f3a1b62334853a71970d59c83247403222500f37ac00e' xobject: OBJ854 xobjstate=OBJECT_DYING fl=0 swfl=8 ev=20[5] xops=1 inp=1 exc=0 xparent=ffff8801166e2700 xcookie=NULL CacheFiles: Error: Overlong wait for old active object to go away object: OBJ855 objstate=OBJECT_LOOKING_UP fl=0 swfl=a ev=0[7b] ops=0 inp=0 exc=0 parent=ffff8801166e2700 cookie=ffff880114043810 [pr=ffff88011e43e588 nd=ffff88010a2a8418 fl=7] key=[28] '01000601f73f3a1b62334853a71970d59c83247403222500f37ac00e' xobject: OBJ854 xobjstate=OBJECT_DYING fl=0 swfl=8 ev=20[5] xops=1 inp=1 exc=0 xparent=ffff8801166e2700 xcookie=NULL CacheFiles: Error: Overlong wait for old active object to go away object: OBJ855 objstate=OBJECT_LOOKING_UP fl=0 swfl=a ev=0[7b] ops=0 inp=0 exc=0 parent=ffff8801166e2700 cookie=ffff880114043810 [pr=ffff88011e43e588 nd=ffff88010a2a8418 fl=7] key=[28] '01000601f73f3a1b62334853a71970d59c83247403222500f37ac00e' xobject: OBJ854 xobjstate=OBJECT_DYING fl=0 swfl=8 ev=20[5] xops=1 inp=1 exc=0 xparent=ffff8801166e2700 xcookie=NULL CacheFiles: Error: Overlong wait for old active object to go away object: OBJ855 objstate=OBJECT_LOOKING_UP fl=0 swfl=a ev=0[7b] ops=0 inp=0 exc=0 parent=ffff8801166e2700 cookie=ffff880114043810 [pr=ffff88011e43e588 nd=ffff88010a2a8418 fl=7] key=[28] '01000601f73f3a1b62334853a71970d59c83247403222500f37ac00e' xobject: OBJ854 xobjstate=OBJECT_DYING fl=0 swfl=8 ev=20[5] xops=1 inp=1 exc=0 xparent=ffff8801166e2700 xcookie=NULL CacheFiles: Error: Overlong wait for old active object to go away object: OBJ855 objstate=OBJECT_LOOKING_UP fl=0 swfl=a ev=0[7b] ops=0 inp=0 exc=0 parent=ffff8801166e2700 cookie=ffff880114043810 [pr=ffff88011e43e588 nd=ffff88010a2a8418 fl=7] key=[28] '01000601f73f3a1b62334853a71970d59c83247403222500f37ac00e' xobject: OBJ854 xobjstate=OBJECT_DYING fl=0 swfl=8 ev=20[5] xops=1 inp=1 exc=0 xparent=ffff8801166e2700 xcookie=NULL CacheFiles: File cache on md1 unregistering FS-Cache: Withdrawing cache "mycache" I'm going to find why. All my boxes have the same hardware, so i guess it must be software problem. Thank you all.
Created attachment 229209 [details] cachefilesd-0.10.ebuild
Created attachment 229491 [details] cachefilesd-0.10-r1.ebuild
Created attachment 231523 [details] cachefilesd-0.10.1.ebuild revision bump. fscache moved (upstream) under /var/cache if ! use selinux comments out selinux stuff from cachefilesd.conf (pervents error on not selinux kernels) needs someone knowing selinux to add selinux policies management in pkg_config()
Created attachment 231525 [details] cachefilesd.conf
Created attachment 231527 [details] cachefilesd.init new init script. cahefilesd should start before nfsmount.
version 10.1 available
(In reply to comment #35) > version 10.1 available > comment #32 already contains 10.1 ebuild
@cla Any status update here? I would like to bring it in. Testing over 2 month gave no error.
(In reply to comment #37) > @cla > Any status update here? > I would like to bring it in. Testing over 2 month gave no error. > Go ahead. I no longer use it.
(In reply to comment #37) > @cla > Any status update here? > I would like to bring it in. Testing over 2 month gave no error. > It's working fine on 2 workstations since april.
(In reply to comment #39) > (In reply to comment #37) > > @cla > > Any status update here? > > I would like to bring it in. Testing over 2 month gave no error. > > > > It's working fine on 2 workstations since april. > Just when I said, I will bring it in, it starts mucking around. But I will do it once I am back and we will see what the live test are carrying out.
Please test and report all bugs. +*cachefilesd-0.10.1 (19 Sep 2010) + + 19 Sep 2010; Justin Lecher <jlec@gentoo.org> +files/0.10.1-makefile.patch, + +cachefilesd-0.10.1.ebuild, +files/cachefilesd.conf, + +files/cachefilesd.init, +metadata.xml: + New addition, 275014; thanks Bruno Redondi and everyone else for the + contribution +
