[OT] Solaris + Apache 2.2 / PHP 5 cluster config help

Hi all,
I know that bits and pieces of high-load configuration questions have
been posted to this list (users@httpd) and am CC-ing the mod_perl folks
(since I know there are a bunch of knowledgeable people on the subject
lurking there, but please post responses to users@httpd), but I'd like
to put a bunch of configuration questions in one neat post. Apologies
if some people can think of a more appropriate place to have posted.

The short story is that I need to design a Apache/MySQL/PHP cluster
(dual [redundant] firewalls + switches + dual [redundant] LBs and/or
frontend caching proxies (likely squid) + switches + 4-6 webservers
(likely running Solaris) with read-only replicating mysqls - with the
hopes that we can scale these up just by adding more hardware and just
telling the LBs + dual [redundant] mysql replication masters + shared
storage - hopefully a NetApp :)), and am looking for the smartest design
for the webservers and application layer in general. I know plenty of
theory on the subject, but it's really the first time I've had to design
such a setup on my own.

My thoughts were to build httpd-2.2.x-worker, and mod_fastcgi + PHP with
the FastCGI SAPI (to keep apache threaded, but allow for non-threadsafe
PHP components - since even if I could be sure we weren't using
non-threadsafe components in PHP, some unsuspecting developer will
undoubtedly screw-up down the line if I build mod_php5). Also, do
people have concrete benchmarks of keeping a read-only replication mysql
on the webservers vs a single read/write shared mysql server?

I also thought to buffer common DB query results (like php's sessions)
in memcached on reads (though don't have any experience there and
looking for tips on how to set it up best, how to update on writes - I
assume update memcached + backend - and how to prune the old entries
from memcached ).

I'm also looking to pre-compile the PHP scripts - preferably in shared
memory on each webserver - for faster execution (think
ModPerl::Registry), and am a bit stuck there. I've previously used
eAccellerator/Turck mmcache, but have found that it doesn't seem to work
(for me, at least) with PHP objects and classes, and we use those
heavily (at the moment, anyway - if that's the only way to get
pre-compilation, then I may force them to stop using OO). We actually
already have Zend Guard, but I've been led to understand that that's not
really helpful performance-wise, unless you have the full Zend Server,
which I'm not sure I'd want to migrate to.

Any helpful pointers would be appreciated.

Issac

Re: [OT] Solaris + Apache 2.2 / PHP 5 cluster confighelp

> Hi all,
> I know that bits and pieces of high-load configuration questions have
> been posted to this list (users@httpd) and am CC-ing the mod_perl folks
> (since I know there are a bunch of knowledgeable people on the subject
> lurking there, but please post responses to users@httpd), but I'd like
> to put a bunch of configuration questions in one neat post. Apologies
> if some people can think of a more appropriate place to have posted.
>
> The short story is that I need to design a Apache/MySQL/PHP cluster
> (dual [redundant] firewalls + switches + dual [redundant] LBs and/or
> frontend caching proxies (likely squid) + switches + 4-6 webservers
> (likely running Solaris) with read-only replicating mysqls - with the
> hopes that we can scale these up just by adding more hardware and just
> telling the LBs + dual [redundant] mysql replication masters + shared
> storage - hopefully a NetApp :)), and am looking for the smartest design
> for the webservers and application layer in general. I know plenty of
> theory on the subject, but it's really the first time I've had to design
> such a setup on my own.
>
> My thoughts were to build httpd-2.2.x-worker, and mod_fastcgi + PHP with
> the FastCGI SAPI (to keep apache threaded, but allow for non-threadsafe
> PHP components - since even if I could be sure we weren't using
> non-threadsafe components in PHP, some unsuspecting developer will
> undoubtedly screw-up down the line if I build mod_php5). Also, do
> people have concrete benchmarks of keeping a read-only replication mysql
> on the webservers vs a single read/write shared mysql server?
>
> I also thought to buffer common DB query results (like php's sessions)
> in memcached on reads (though don't have any experience there and
> looking for tips on how to set it up best, how to update on writes - I
> assume update memcached + backend - and how to prune the old entries
> from memcached ).
>
> I'm also looking to pre-compile the PHP scripts - preferably in shared
> memory on each webserver - for faster execution (think
> ModPerl::Registry), and am a bit stuck there. I've previously used
> eAccellerator/Turck mmcache, but have found that it doesn't seem to work
> (for me, at least) with PHP objects and classes, and we use those
> heavily (at the moment, anyway - if that's the only way to get
> pre-compilation, then I may force them to stop using OO). We actually
> already have Zend Guard, but I've been led to understand that that's not
> really helpful performance-wise, unless you have the full Zend Server,
> which I'm not sure I'd want to migrate to.
>
> Any helpful pointers would be appreciated.

i can write you something about our server setup:
we've a single firewall + LB , 4 web, 2 MySQL and 1 mail server.
we use apache 2.0 with mod_php and eaccelerator. eaccelerator gives a
pretty good performance improvement.
we also use eaccelerator for shared memory access to cache some MySQL queries
and other 'settings' which don't change often.

Both MySQL servers are not clustered, each one is a standalone server.
one for realtime/user data, and the other for statistic/backup.
i'm try to split up tables even more to spread out load.
so maybe we will buy 2 additional MySQL servers soon.

All MySQL tables are replicated to our local office where i can backup them
without downtime of the online DB.

we are currently testing some 'content delivery network' (CDN) providers
like cavecreek, level3 and pantherexpress to outhouse images and movies.
if you have mostly static content (images, css, js, movies, whatever)
CDN is really nice.
that's why we are ok with only 4 web servers.

i also read about the Zend Platform which sounds very interesting.
i had no time to test it yet :(


maybe you got some more ideas... ? :)

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [OT] Solaris + Apache 2.2 / PHP 5 cluster config help

On Sun, May 18, 2008 at 8:05 AM, Issac Goldstand wrote:
> Also, do people have concrete
> benchmarks of keeping a read-only replication mysql on the webservers vs a
> single read/write shared mysql server?

Any time you can spread the reads over multiple servers it will help.
This is particularly true for MyISAM tables where reads blocks
writers. (InnoDB tables behave like Oracle and Postgres, with a MVCC
model.)

> I also thought to buffer common DB query results (like php's sessions) in
> memcached on reads

I suspect that caching on the level of db queries will get annoying if
you try to do it everywhere. You'd be better off caching at a higher
"object" level if you can, or just caching specific high-value
queries.

- Perrin

Re: [OT] Solaris + Apache 2.2 / PHP 5 cluster config help

Perrin Harkins wrote:
> On Sun, May 18, 2008 at 8:05 AM, Issac Goldstand wrote:
>> Also, do people have concrete
>> benchmarks of keeping a read-only replication mysql on the webservers vs a
>> single read/write shared mysql server?
>
> Any time you can spread the reads over multiple servers it will help.
> This is particularly true for MyISAM tables where reads blocks
> writers. (InnoDB tables behave like Oracle and Postgres, with a MVCC
> model.)
>

Right - I know that in theory, but was worried about the disk/ram/cpu
overhead of replicating the writes to all of the slave servers
offsetting that benefit...

>> I also thought to buffer common DB query results (like php's sessions) in
>> memcached on reads
>
> I suspect that caching on the level of db queries will get annoying if
> you try to do it everywhere. You'd be better off caching at a higher
> "object" level if you can, or just caching specific high-value
> queries.
>

Well, yes, I meant the objects or data structures that result from the
DB query - not the query or the result itself (I'd use Mysql's query
cache for that anyway, not memcached).

Thanks for your input Perrin.

Issac

> - Perrin

Re: [OT] Solaris + Apache 2.2 / PHP 5 cluster config help

On Tue, May 20, 2008 at 4:00 PM, Issac Goldstand wrote:
> Right - I know that in theory, but was worried about the disk/ram/cpu
> overhead of replicating the writes to all of the slave servers offsetting
> that benefit...

Good point. I'd suggest you look at how much RAM you can spare and
see if you can fit the data you want to query (including indexes) into
that much space. If you can't, you're probably better off adding a
separate read-only server instead of running it on your web server
machines. The only real advantage to running MySQL on your web
servers is the use of local sockets for connecting, which does reduce
the overhead.

- Perrin

Re: [OT] Solaris + Apache 2.2 / PHP 5 cluster config help

On May 20, 2008, at 4:00 PM, Issac Goldstand wrote:
> Right - I know that in theory, but was worried about the disk/ram/
> cpu overhead of replicating the writes to all of the slave servers
> offsetting that benefit...

The explanation i gave to this on the Pylons list today was such:

when you're successful, you cluster to a couple of servers with
replication
when you're SUCCESSFUL!!!!!!, you cluster to a handful of servers
with replication and horizontal partitioning

you honestly shouldn't have an issue with writes/reads until you have
a hugely utilized service

you can also do psuedo-partitioning to make things work faster--
keep 1 DB + replication system for your app's business logic
( accounts, meta data, etc)
split out your sessions and logging facilities (if any) to a
seperate box(es). those write on every request - just give them
entirely different boxes & databases to save to, and free up the
write/read/replication structure for your data

personally, i think one should always architect applications with
separate DB handles for read/write/log/session at the outset. you
can essentially make them 'one' handle until necessary... but it
takes barely any time to program with that paradigm from the start...
vs trying to retrofit in clustering to an app when its too late.

my bigger questoin on your design would be this:

why are you using apache?

off your setup, you don't seem to suggest any need for apache. i
think you'd be MUCH better off using nginx/lighttpd with fastCGI .

Have you tried using APC ( http://pecl.php.net/package/APC ) ?

I know it used to suck ass. IIRC , APC used to be worthless and
eAccelerator was the only thing that worked - and was great. then
APC was working, and eAccelerator kicked its ass. but then
eAccelerator broke for like 10 months, and APC worked - so it picked
up a lot of steam.

looking at your specs, i'd toss as much ram on your http machines as
money allows, run nginx + fastcgi on the boxes. give fcgi some
generour ram and APC .5GB. then run the rest of the box on
memcached. its not worth running replicants on your http machines -
you're better off tossing the CPU to php and the RAM to memcached.


// Jonathan Vanasco

w. http://findmeon.com/user/jvanasco
e. jonathan@findmeon.com

| Founder/CEO - FindMeOn, Inc.
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Privacy Minded Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - -

Re: [OT] Solaris + Apache 2.2 / PHP 5 cluster config help

On Wed, May 21, 2008 at 7:32 PM, Jonathan Vanasco wrote:
> personally, i think one should always architect applications with separate
> DB handles for read/write/log/session at the outset. you can essentially
> make them 'one' handle until necessary... but it takes barely any time to
> program with that paradigm from the start... vs trying to retrofit in
> clustering to an app when its too late.

The module I'm presenting at YAPC::NA and OSCON this year,
DBIx::Router, will make it easier to add this kind of partitioning
after the fact. It lets you set up rules to choose which database to
send a query to. When I have a public prototype on CPAN, I'll
announce it on this list and ask for testing help.

- Perrin

Re: [OT] Solaris + Apache 2.2 / PHP 5 cluster config help

On May 22, 2008, at 1:14 PM, Perrin Harkins wrote:

> The module I'm presenting at YAPC::NA and OSCON this year,
> DBIx::Router, will make it easier to add this kind of partitioning
> after the fact. It lets you set up rules to choose which database to
> send a query to. When I have a public prototype on CPAN, I'll
> announce it on this list and ask for testing help.

Awesome!

Have you looked into what youtube have been presenting on, or the
skype partitioning tools ?
Both are python, but useful - as they're massively internet scale

If you haven't seen the skype stuff:
https://developer.skype.com/SkypeGarage/DbProjects/

The whitepaper is pretty good

The stuff on parition hashing is in PL/Proxy

https://developer.skype.com/SkypeGarage/DbProjects/PlProxy