Files per directory

Hi,

Seeming as the list is teeming with activity at the moment, and no,
in the UK we are not experiencing network issues, I thought I'd wade
in.

I am trying to design a files system that is based around database
records. Each record within the DB could have up to 9 files (possibly
more). The best design I can come up with would mean that a directory
would end up with 9,999 files in.

Is that going to cause performance issues? The current file system
ext3. Would anyone suggest a limit I should set for the maximum or
say if they think 10K files is acceptable?

TIA,
Dp.

--
To unsubscribe from this list: send the line "unsubscribe linux-admin" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Files per directory

This is a cryptographically signed message in MIME format.

--------------ms040503050502010909090102
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Beginner wrote:
>
> Is that going to cause performance issues? The current file system
> ext3. Would anyone suggest a limit I should set for the maximum or
> say if they think 10K files is acceptable?
>

I'm no expert but the answer is probably: "depends on the application."

As far as I know there's no limit to the number of files in a directory
currently in ext3. There IS a limit to the number of files (actually
inodes) in the whole filesystem, which is a completely different thing.
According to Wikipedia "If V is the volume size in bytes, then the
default number of inodes is given by V/2^13 (or the number of blocks,
whichever is less)." There's also a limit to the number of
sub-directories in a directory, currently 32000.

What I can tell you is I have seen applications that use hundreds of
thousands of files in one directory.

HTH

--
Yuri Csapo
Academic Computing & Networking
Colorado School of Mines
CT-256
Phone: (303) 273-3503
Fax: (303) 273-3475
Email: ycsapo@mines.edu

Please use the following link to open a service request:
http://helpdesk.mines.edu
===========================================
With a PC, I always felt limited
by the software available.
On Unix, I am limited only by my knowledge.
--Peter J. Schoenster

--------------ms040503050502010909090102
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEH AQAAoIIH9zCC
AlYwggG/oAMCAQICEGGEG+8vIuZyO0EmnyFFyvgwDQYJKoZIhvcNAQEFBQAw YjELMAkGA1UE
BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4x LDAqBgNVBAMT
I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA3MTEx NTIzNDE1OFoX
DTA4MTExNDIzNDE1OFowQjEfMB0GA1UEAxMWVGhhd3RlIEZyZWVtYWlsIE1l bWJlcjEfMB0G
CSqGSIb3DQEJARYQeWNzYXBvQG1pbmVzLmVkdTCBnzANBgkqhkiG9w0BAQEF AAOBjQAwgYkC
gYEA44EP73U7iwsNfLgFfAvF468YJ5d+11KpnVJZQY5OxUb0BKqP9tcWLfh7 Di/+s0cjBu83
TGl94zUQkA1is7PdQaPJ5aGh0/5l3wjavQR1UH+silctf/ijp44ofigm+WVc f8LpIRROVLle
edqsHlfkvIYeBdmk3X5soTn5KBsVwjcCAwEAAaMtMCswGwYDVR0RBBQwEoEQ eWNzYXBvQG1p
bmVzLmVkdTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBQUAA4GBAK1xzmzy xbq+0k4He4Z0
G5MRg0yxbqE3Ft9j5Sfkab9xhGdNLz/TQnJ7hmsi1kw+OsXH+A+Ki6M644XH 3hQErAiwRDUv
C9ImqeIuHoQLGlWn1gDXKc+fhbDlWxh6/fGEecqYyJwq9djxd/d2WQJeJV/3 cHOSn+Ei6dLQ
HXHj51ZgMIICVjCCAb+gAwIBAgIQYYQb7y8i5nI7QSafIUXK+DANBgkqhkiG 9w0BAQUFADBi
MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0 eSkgTHRkLjEs
MCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0Ew HhcNMDcxMTE1
MjM0MTU4WhcNMDgxMTE0MjM0MTU4WjBCMR8wHQYDVQQDExZUaGF3dGUgRnJl ZW1haWwgTWVt
YmVyMR8wHQYJKoZIhvcNAQkBFhB5Y3NhcG9AbWluZXMuZWR1MIGfMA0GCSqG SIb3DQEBAQUA
A4GNADCBiQKBgQDjgQ/vdTuLCw18uAV8C8Xjrxgnl37XUqmdUllBjk7FRvQE qo/21xYt+HsO
L/6zRyMG7zdMaX3jNRCQDWKzs91Bo8nloaHT/mXfCNq9BHVQf6yKVy1/+KOn jih+KCb5ZVx/
wukhFE5UuV552qweV+S8hh4F2aTdfmyhOfkoGxXCNwIDAQABoy0wKzAbBgNV HREEFDASgRB5
Y3NhcG9AbWluZXMuZWR1MAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQAD gYEArXHObPLF
ur7STgd7hnQbkxGDTLFuoTcW32PlJ+Rpv3GEZ00vP9NCcnuGayLWTD46xcf4 D4qLozrjhcfe
FASsCLBENS8L0iap4i4ehAsaVafWANcpz5+FsOVbGHr98YR5ypjInCr12PF3 93ZZAl4lX/dw
c5Kf4SLp0tAdcePnVmAwggM/MIICqKADAgECAgENMA0GCSqGSIb3DQEBBQUA MIHRMQswCQYD
VQQGEwJaQTEVMBMGA1UECBMMV2VzdGVybiBDYXBlMRIwEAYDVQQHEwlDYXBl IFRvd24xGjAY
BgNVBAoTEVRoYXd0ZSBDb25zdWx0aW5nMSgwJgYDVQQLEx9DZXJ0aWZpY2F0 aW9uIFNlcnZp
Y2VzIERpdmlzaW9uMSQwIgYDVQQDExtUaGF3dGUgUGVyc29uYWwgRnJlZW1h aWwgQ0ExKzAp
BgkqhkiG9w0BCQEWHHBlcnNvbmFsLWZyZWVtYWlsQHRoYXd0ZS5jb20wHhcN MDMwNzE3MDAw
MDAwWhcNMTMwNzE2MjM1OTU5WjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMc VGhhd3RlIENv
bnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFs IEZyZWVtYWls
IElzc3VpbmcgQ0EwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMSmPFVz VftOucqZWh5o
wHUEcJ3f6f+jHuy9zfVb8hp2vX8MOmHyv1HOAdTlUAow1wJjWiyJFXCO3cnw K4Vaqj9xVsuv
PAsH5/EfkTYkKhPPK9Xzgnc9A74r/rsYPge/QIACZNenprufZdHFKlSFD0gE f6e20TxhBEAe
ZBlyYLf7AgMBAAGjgZQwgZEwEgYDVR0TAQH/BAgwBgEB/wIBADBDBgNVHR8E PDA6MDigNqA0
hjJodHRwOi8vY3JsLnRoYXd0ZS5jb20vVGhhd3RlUGVyc29uYWxGcmVlbWFp bENBLmNybDAL
BgNVHQ8EBAMCAQYwKQYDVR0RBCIwIKQeMBwxGjAYBgNVBAMTEVByaXZhdGVM YWJlbDItMTM4
MA0GCSqGSIb3DQEBBQUAA4GBAEiM0VCD6gsuzA2jZqxnD3+vrL7CF6FDlpSd f0whuPg2H6ot
nzYvwPQcUCCTcDz9reFhYsPZOhl+hLGZGwDFGguCdJ4lUJRix9sncVcljd2p nDmOjCBPZV+V
2vf3h9bGCE6u9uo05RAaWzVNd+NWIXiC3CEZNd4ksdMdRv9dX2VPMYIC4zCC At8CAQEwdjBi
MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0 eSkgTHRkLjEs
MCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EC EGGEG+8vIuZy
O0EmnyFFyvgwCQYFKw4DAhoFAKCCAcMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkq
hkiG9w0BCQUxDxcNMDgwNzEwMTc1MjAzWjAjBgkqhkiG9w0BCQQxFgQUziq9 q86kwlsiTYjV
Xqsa8QTWSRIwUgYJKoZIhvcNAQkPMUUwQzAKBggqhkiG9w0DBzAOBggqhkiG 9w0DAgICAIAw
DQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgYUGCSsG AQQBgjcQBDF4
MHYwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5n IChQdHkpIEx0
ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5n IENBAhBhhBvv
LyLmcjtBJp8hRcr4MIGHBgsqhkiG9w0BCRACCzF4oHYwYjELMAkGA1UEBhMC WkExJTAjBgNV
BAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1Ro YXd0ZSBQZXJz
b25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAhBhhBvvLyLmcjtBJp8hRcr4MA0G CSqGSIb3DQEB
AQUABIGARjJiIOD7PRh+QvHvOoQ8rLXkTjwaYIe9W6Q6FOFTAel97jcEH1Xu eP6hJ00EPphK
dG0G099K5eJcCPbEsBzsgN8H/NgIz+aC+TM5Utm50U7T2xNABoqNKnSKW8qY 83KobxRf03SW
elmhjbqp0n4cEFwZgVA+bylMH/+SeaQhCLMAAAAAAAA=
--------------ms040503050502010909090102--
--
To unsubscribe from this list: send the line "unsubscribe linux-admin" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Files per directory

Yuri Csapo wrote:

> > Is that going to cause performance issues? The current file system
> > ext3. Would anyone suggest a limit I should set for the maximum or
> > say if they think 10K files is acceptable?
>
> I'm no expert but the answer is probably: "depends on the application."
>
> As far as I know there's no limit to the number of files in a directory
> currently in ext3. There IS a limit to the number of files (actually
> inodes) in the whole filesystem, which is a completely different thing.

ext3 also has a limit of 32000 hard links, which means that a
directory can't have more than 31998 subdirectories.

However, the original poster wasn't asking about hard limits, but
efficiency.

If the filesystem wasn't created with the dir_index option, then
having thousands of files in a directory will be a major performance
problem, as any lookups will scan the directory linearly.

Even with the dir_index option, large directories could be an issue. I
think that you would really need to conduct tests to see exactly how
much of an issue.

OTOH, even if you keep the directories small, a database consisting of
many small files will be much slower than e.g. BerkeleyDB or DBM.

--
Glynn Clements
--
To unsubscribe from this list: send the line "unsubscribe linux-admin" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html