Bugzilla Bug 139451

Apache 2.0.58 has problems displaying indexes. In my case, I tested different
indexes containing:

a) 31 files and folders, 
b) 95 files and folders, 
c) 161 folders, 
d) 581 folders

In case of a), apache was able to index it in a few seconds. 
In case of b) indexing took a little longer 5-10 seconds. 
In case of c) indexing took about 2-3 minutes.
In case of d) indexing is very slow, even after 5 minutes the page was not
displayed and apache used all the rest of available CPU.

In all cases, no swap memory has been used and free memory has been available
(512MB total). The setup is a gentoo headless system with the latest kernel
(Linux pluto 2.6.16-gentoo-r11) and ebuilds.

Workaround: 

downgrading to apache-2.0.55-r1
reemerge apache2 after masking version 2.0.58
( echo "=net-www/apache-2.0.58">> /etc/portage/package.mask )

I downgraded to apache 2.0.55-r1 and did not change anything else of the setup. 
While back on apache 2.0.55-r1 even in case of d) indexing takes only a few
seconds. Normal behaviour, even the high cpu usage has gone.

Feel free to contact me and help me to debug the bug.

Besides the default configuration, I tested with the following modules:

LoadModule cache_module                  modules/mod_cache.so
LoadModule disk_cache_module             modules/mod_disk_cache.so
LoadModule mem_cache_module              modules/mod_mem_cache.so
LoadModule file_cache_module             modules/mod_file_cache.so

Withh or without the above modules, no difference. Apache 2.0.58 still very
slow in indexing.

Besides the default module configuration: I tested only with the modules
enabled below: 

# The following Modules are not commonly loaded for Apache
#
LoadModule case_filter_module            modules/mod_case_filter.so
LoadModule case_filter_in_module         modules/mod_case_filter_in.so
LoadModule echo_module                   modules/mod_echo.so
LoadModule mime_magic_module             modules/mod_mime_magic.so
LoadModule speling_module                modules/mod_speling.so
LoadModule unique_id_module              modules/mod_unique_id.so
LoadModule vhost_alias_module            modules/mod_vhost_alias.so

I tried the modules cillybabe listed and didn't get the slowdown that she
found.
I loaded a page w/ 112 directories in less than 2 seconds but it took 110
seconds to index 1186 file in /usr/share/doc and was ~90% CPU.  Without the
caching stuff it took 2:15 seconds and 100% CPU.  The machine is doing other
things so these number aren't definative and they are odd in that while
tweaking things it took between 2:00 - 2:15  (min:sec) to load the 1186 items
regardless of the settings of the modules she listed above.  (Including
disabling all of them where my first run was 110 seconds).   ????   I know
nothing.  :(

I tested both versions of apache on 581 items to index:

apache 2.0.58     581 folders     > 5 minutes
apache 2.0.55-r1  581 folders     ~ 12 seconds

same setup, with all the modules enabled above.

I tested apache 2.0.58-r2 and it is slow as hell in indexing large indexes. The
index-option is not usuable for large indexes.

*** Bug 146966 has been marked as a duplicate of this bug. ***

Additional Info:

Use the following Index-Options:

IndexOptions FancyIndexing IgnoreCase ScanHTMLTitles NameWidth=80
DescriptionWidth=32 FoldersFirst VersionSort

Update:

This bug is reproduceable with apache 2.0.59, apache 2.0.58 AND apache
2.0.55-r2.

I was able to trace back:

The issue is NOT reproduceable with apache 2.0.55-r1.

This means the bug was introduced with apache 2.0.55-r2 and a look in the
changelog explains probably the reason why autoindex it kind of broken in
apache 2.0.55-r2 and later:

*apache-2.0.55-r2 (31 Mar 2006)

  31 Mar 2006; Michael Stewart <vericgar@gentoo.org>
  +apache-2.0.55-r2.ebuild:
  Replace the patch for AddDirectoryIndex with a rewritten one. This should
  fix the segfaults reported in bug 106808

Thx for the help of Maxi, pZYchO, dok and others.

Hello,

looks like AddDiretoryIndex is the problem. After removing it (and expand
DirectoryIndex) everything works fine.

Greets pZYchO

Digging through some code.....

mod_autoindex is running slower after the patch becuase it does a subrequest on
every item in that directory.

Every directory that apache goes through (from / down to wherever your
documents are), apache does a merge routine, to merge the configurations from
the parent directory to the child directory.

Before the patch was added, this merge was simple for the DirectoryIndex
directive:

if DirectoryIndex is set in child, then use that DirectoryIndex, otherwise use
DirectoryIndex from parent.

Now that AddDirectoryIndex exists, this logic gets a bit more complicated, if
the child doesn't set a DirectoryIndex, then the parent's DirectoryIndex,
parent's AddDirectoryIndex and child's AddDirectoryIndex are merged together to
create the DirectoryIndex for that directory.

If the child's DirectoryIndex is set, then the child's DirectoryIndex and
AddDirectoryIndex are merged together to create the DirectoryIndex for the
directory.

This is causing every subrequest to take a bit longer then it usually would,
and when there are many directories, it adds up to a long time.

Anyone whos knows C++ is free to take a look at the patch and offer
improvements on how to make this faster. I'm not an experience C++ programmer,
so I may have missed some obvious optimization. The current patch is at
http://sources.gentoo.org/viewcvs.py/*checkout*/apache/trunk/dist/2.0/patches/07_all_mod_dir_incremental.patch

Otherwise, if you need speed when generating indexes, you have a few options:
1. don't use AddDirectoryIndex (i.e. comment out any directives and put thier
contents on the end of DirectoryIndex)
2. write a custom script to do it

Workaround:
Replace the single DirectoryIndex line in httpd.conf with:

<IfDefine !PHP5>
DirectoryIndex index.html index.html.var index.htm index.htm.var
</IfDefine>
<IfDefine PHP5>
DirectoryIndex index.html index.html.var index.htm index.htm.var index.php
index.phtml
</IfDefine>

and comment the line AddDirectoryIndex in modules.d/70_mod_php5.conf.

Additional info to my last comment:

http://bugs.gentoo.org/show_bug.cgi?id=139451#c11

The "DirectoryIndex $args" has to be a single line.

After commenting the AddDirectoryIndex line in modules.d/70_mod_php5.conf and
restarting Apache, I'm noticing indexes loading in 1-2 seconds where they took
about 30-45 seconds previously.  Apache's CPU usage is greatly reduced as well.
 I'm using Apache 2.0.58.

Seconding results found above re. AddDirectoryIndex patch. Scans all subdirs
for indexes. The detail I'm adding is that it doesn't even seem to matter if
mod_autoindex is added or if autoindexes are enabled, it still does this scan.
takes _forever_ in dirs with many subdirs.

the problem seems to be in mod_dir (as this seems to be where the horrible
AddDirectoryIndex patch lives). Having the AddDirectoryIndex command anywhere
in apache config will cause this subdir scan. There seems to be absolutely no
need for this patch- it's a horrendous performance hit. Can't you guys do
things with IfDefines like all other distros?

Remove this patch! bleaaah!

(In reply to comment #14)
> Seconding results found above re. AddDirectoryIndex patch. Scans all subdirs
> for indexes. The detail I'm adding is that it doesn't even seem to matter if
> mod_autoindex is added or if autoindexes are enabled, it still does this scan.
> takes _forever_ in dirs with many subdirs.

You must be mistaken, the code doesn't work that way. The added patch does no
directory scanning at all. Some module you are using most be doing that if
mod_autoindex isn't. The patch only does a few API calls that merge arrays.

> Can't you guys do
> things with IfDefines like all other distros?

We are not other distros. We like to give you the choice as to which add-on
modules you want. Other distros package these add-on modules with apache, or
expect you to edit your configuration after installing if you want a new
DirectoryIndex. We like to have things Just Work after they are built and
installed. IfDefines require every package to modify a single file. In portage,
this file would now be "owned" by every module as well as apache. There are not
allowed in Gentoo as they cause problems when updating and unmerging.

This is the solution we are using to work around the need for modules to add
thier own DirectoryIndex and at this point in time it will not be changing. We
came to this solution as the best option back at the beginning of last year as
we were redesigning the apache configuration and creating eclasses to help
maintain modules.

Improvements to the AddDirectoryIndex patch are welcome. Demanding that we
remove that patch are not. A patch that fixes mod_autoindex so that it doesn't
do a subrequest on every directory would also be nice, though that may require
re-writing much of it.

*** Bug 150947 has been marked as a duplicate of this bug. ***

*** Bug 151410 has been marked as a duplicate of this bug. ***

I request to remove the patch, since it breaks more than it fixes.

> We are not other distros. We like to give you the choice as to which add-on
> modules you want. Other distros package these add-on modules with apache, or
> expect you to edit your configuration after installing if you want a new
> DirectoryIndex. We like to have things Just Work after they are built and
> installed. 

Noble sentiment, but in my case this patch caused things to Just Break after
they were built and installed.  My production server was sidelined for nearly
two months because of bug #151410.

> Improvements to the AddDirectoryIndex patch are welcome. Demanding that we
> remove that patch are not.

Is it possible to prevent this directive from being used in .htaccess files? 
Or at least, could a big warning be added to the ebuild?  Once again, please
see bug #151410 for details.  I believe that problem is unacceptable due to the
potential for a DoS attack, which I'd be happy to demonstrate to any willing
party with a Gentoo server running Apache. :-)

I tried apache 2.2.3 and guess what:

autoindex now skips those dirs, which means they aren't listed at all

Do not reopen this bug.

The workarounds are listed, and work.

The patch does absolutely *nothing* if you don't have an AddDirectoryIndex
directive anywhere in your config. If you didn't have the patch, then that
directive wouldn't exist, and so if you want it ripped out, I assume you know
how to make things work on your system without that directive, so just quit
using that directive. The patch is NOT going away.

*** Bug 169989 has been marked as a duplicate of this bug. ***

I think I've found a simple solution that will please everyone: limit the use
of the AddDirectoryIndex directive to the global server config only.  This
seems sufficient for the modular purpose the for which the directive is
intended.  All of the associated security and performance bugs I've seen seem
to be related to having the directive in a per-directory context instead.  And
an AddDirectoryIndex directive in a VirtualHost context is still applied
server-wide anyway, so there's no need for it to be there either.

I'd be happy to discuss any questions, concerns, or criticisms.  Feel free to
/msg depquid on freenode if you want to have a live conversation.

I'll be attaching a modified patch that implements my recommendation.

Created an attachment (id=115204) [edit]
Fixed AddDirectoryIndex patch

This is a tweaked version of 07_all_mod_dir_incremental.patch from the Gentoo
Apache patchset.  It is intended to replace the current patch.

*** Bug 151410 has been marked as a duplicate of this bug. ***

Reopen.

(In reply to comment #23)
> I think I've found a simple solution that will please everyone: limit the use
> of the AddDirectoryIndex directive to the global server config only.  This
> seems sufficient for the modular purpose the for which the directive is
> intended.  All of the associated security and performance bugs I've seen seem
> to be related to having the directive in a per-directory context instead.  And
> an AddDirectoryIndex directive in a VirtualHost context is still applied
> server-wide anyway, so there's no need for it to be there either.
> 
> I'd be happy to discuss any questions, concerns, or criticisms.  Feel free to
> /msg depquid on freenode if you want to have a live conversation.
> 
> I'll be attaching a modified patch that implements my recommendation.

Yup, and that's what I just added to our svn repository.

(In reply to comment #27)
> Yup, and that's what I just added to our svn repository.

And this went now into the tree with apache-2.2.4-r5.

Why is this marked as NEEDINFO?

*** Bug 183280 has been marked as a duplicate of this bug. ***

Reopen to resolve properly.

FIXED in >=apache-2.2.4-r5

Created an attachment (id=137458) [edit]
rewritten patch

Several months later... Nobody has noticed that the bug actually isn't fixed.

The the problem is in the following lines in fixup_dir():

       if (d->index_names_incr) {
               if (!d->index_names) {
                       d->index_names = d->index_names_incr;
               } else {
                       apr_array_cat(d->index_names, d->index_names_incr);
               }
       }

Changes made to d->index_names are persistent and on high-load servers the list
grows very fast.
And when index_names has several thousands elements its processing takes much
time.

I've attached rewritten 02_all_gentoo_mod_dir_incremental.patch. I hope it
doesn't change the semantics of AddDirectoryEntry.
It's tested it on my server and seems to work.

I've read history of the patch:
http://sources.gentoo.org/viewcvs.py/apache/trunk/dist/2.2/patches/02_all_gentoo_mod_dir_incremental.patch?rev=152&r1=128&view=log

The bug was introduced in revision 69, and in revision 73 was backported to
apache 2.0. I think that reverting to revision 68 can be the way to go.

(In reply to comment #33)
> Several months later... Nobody has noticed that the bug actually isn't fixed.
> 
> The the problem is in the following lines in fixup_dir():

Are you experiencing problems, or merely trying to propose a more ideal
solution?  IIRC, the growth of the d->index_names array only occurred if
AddDirectoryIndex was processed in non-global context (which it should never
have been allowed to in the first place, as it has a server-wide effect). 
Therefore, the problem seems to be fixed (both practically and ideologically)
now that the directive is not allowed in any other context.  (My server's
highly improved performance suggests the same.)  Therefore, I say don't fix
what is already working acceptably.

On the other hand, if you are actually experiencing performance issues related
to this, what version of Apache are you using?

(In reply to comment #35)
> IIRC, the growth of the d->index_names array only occurred if
> AddDirectoryIndex was processed in non-global context (which it should never
> have been allowed to in the first place, as it has a server-wide effect). 
It occurred with default gentoo configuration files (AddDirectoryIndex was in
modules.d/70_mod_suphp.conf).

> Therefore, the problem seems to be fixed (both practically and ideologically)
> now that the directive is not allowed in any other context.  (My server's
> highly improved performance suggests the same.)  Therefore, I say don't fix
> what is already working acceptably.
But I think that users should be able to use AddDirectoryIndex in .htaccess (as
they can use DirectoryIndex).

> On the other hand, if you are actually experiencing performance issues related
> to this, what version of Apache are you using?
I was experiencing it on 2.2.6. Commenting lines in modules.d/70_mod_suphp.conf
fixed the problem.

(In reply to comment #36)
> But I think that users should be able to use AddDirectoryIndex in .htaccess (as
> they can use DirectoryIndex).

Well, as long as the directory context is handled properly, and thoroughly
tested for regressions, I'm happy with that.


> I was experiencing it on 2.2.6. Commenting lines in modules.d/70_mod_suphp.conf
> fixed the problem.

Hmm, could this be in anyway related to mod_suphp?  I wouldn't think so, but
I'm not using it, and, as I said, I've had no problems since AddDirectoryIndex
was restricted to global context.

I took apache 2.2.6 with gentoo patches and added the following lines to
fixup_dir():

if (d->index_names) ap_log_rerror(__FILE__, __LINE__, APLOG_DEBUG, 0, r,
"index_names count: %d", d->index_names->nelts);
        else ap_log_rerror(__FILE__, __LINE__, APLOG_DEBUG, 0, r, "empty
index_names");

I have AddDirectoryIndex only global:

$ grep DirectoryIndex /etc/apache2 /var/www/localhost -R
/etc/apache2/modules.d/00_autoindex.conf:# a file matching those listed in the
DirectoryIndex directive.
/etc/apache2/modules.d/00_default_settings.conf:# DirectoryIndex: sets the file
that Apache will serve if a directory
/etc/apache2/modules.d/00_default_settings.conf:# To add files to that list use
AddDirectoryIndex in a custom config
/etc/apache2/modules.d/00_default_settings.conf:        DirectoryIndex
index.htm
/etc/apache2/modules.d/00_default_settings.conf:        AddDirectoryIndex
index.html

logs:
[Sat Dec 01 21:51:59 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 2
[Sat Dec 01 21:52:01 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 2
[Sat Dec 01 21:52:01 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 3
[Sat Dec 01 21:52:02 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 3
[Sat Dec 01 21:52:13 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 4
[Sat Dec 01 21:52:14 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 4
[Sat Dec 01 21:52:16 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 5
[Sat Dec 01 21:52:17 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 5
[Sat Dec 01 21:52:27 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 6
[Sat Dec 01 21:52:29 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 6
[Sat Dec 01 21:52:30 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 7
[Sat Dec 01 21:52:31 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 7
[Sat Dec 01 21:52:32 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 8
[Sat Dec 01 21:52:33 2007] [debug] mod_dir.c(205): [client 127.0.0.1]
index_names count: 8

(I think it increases every second time, because apache has two processes)

So you can see that when AddDirectoryIndex is in global scope the bug still
occurres.

(In reply to comment #38)
> I took apache 2.2.6 with gentoo patches and added the following lines to
> fixup_dir():
And this test system doesn't have anything like suphp or even php.

And about limiting AddDirectoryIndex to global scope:
Two DirectoryIndex entries can coexist in one scope, eg. when you have:

DirectoryIndex index.html
DirectoryIndex index.php

then either file can be index, not only index.php. Thus AddDirectoryIndex is
really useful only in sub-scopes (DirectoryIndex in sub-scope causes inherited
values to be discarded).

And because DirectoryIndex can be used in sub-scopes, I can't see ideological
reasons not to allow AddDirectoryIndex.

Bump...

Joshua, did you manage to reproduce my results ? Or maybe you can confirm that
the bug isn't present on your system ?

(In reply to comment #41)
> Joshua, did you manage to reproduce my results ? Or maybe you can confirm that
> the bug isn't present on your system ?
> 

Sorry, I've been pre-occupied with other projects (gotta keep the paying
clients), so I haven't been able to look at this again yet.  Unfortunately, I 
don't know how soon I'll be able to thoroughly test this.

In the mean time, could you double-check your assertion about two
DirectoryIndex directives in the same scope?  I was under the impression that
the latter declaration would overwrite the former.  After all, this is the
reason that (at least according to comment #15) the AddDirectoryIndex directive
was created: so that conf files for mods could add filenames to the directory
index list without having to change httpd.conf.

Therefore, the ability to use AddDirectoryIndex in non-global context is
apparently not needed for Gentoo purposes, so the Gentoo devs may not want to
try to extend it's functionality in that way.  After all, we're already having
enough trouble getting the bugs out of it as is.

I have compiled apache without the 02_all_gentoo_mod_dir_incremental.patch
patch and it works as I said:

mareklin /etc/apache2 # grep DirectoryIndex . -R

modules.d/00_default_settings.conf:     DirectoryIndex index.htm
modules.d/00_default_settings.conf:     DirectoryIndex index.html
modules.d/70_mod_php5.conf:     DirectoryIndex index.php index.phtml

And all these index files work.

(In reply to comment #43)
> And all these index files work.

I have confirmed this ... DirectoryIndex in the same scope will be merged,
that's all we need, i will remove the patch from 2.2.8-r1 as soon as all
packages using it have switched over (php and mod_suphp)

php herd: please replace AddDirectoryIndex with DirectoryIndex (no need for new
revision, replace it in all config files)

*** Bug 207946 has been marked as a duplicate of this bug. ***

(In reply to comment #44)
> I have confirmed this ... DirectoryIndex in the same scope will be merged,
> that's all we need, i will remove the patch from 2.2.8-r1 as soon as all
> packages using it have switched over (php and mod_suphp)
> 
> php herd: please replace AddDirectoryIndex with DirectoryIndex (no need for new
> revision, replace it in all config files)
Done for dev-lang/php. Seems like you already did mod_suphp.
Also, as pointed out on IRC already, docs[1] need an update regarding
AddDirectoryIndex.

[1] http://www.gentoo.org/doc/en/apache-developer.xml

(In reply to comment #46)
> Also, as pointed out on IRC already, docs[1] need an update regarding
> AddDirectoryIndex.
> 
> [1] http://www.gentoo.org/doc/en/apache-developer.xml

yep, this is already tracked in bug 200617

patch removed in 2.2.8-r1, closing ...

I just emerged 2.2.8-r1, fixed my PHP.conf and can confirm that the CPU-hogging
of browsing indexed directories is gone. THANK YOU Gentoo devs!

*** Bug 209033 has been marked as a duplicate of this bug. ***

Yes, thank you for finally fixing this, instead of stonewalling :)