Deployment strategies...

Get a fairly typical mod-perl setup.

We have two load-balanced servers acting as front-end. They run threaded
apache configured for lots of connections and KeepAlive on. They serve up as
much stuff statically as possible, everything else is sent through
mod_proxy_balancer to a number of backend servers.

The three back-end servers run a bunch of Catalyst apps under mod_perl -
each server is configured for all apps. We've got around 30 sites in total,
and currently have 6 distinct apps. We're building more new apps for
different sites.

We've now got a couple of problems:

1) We want to run different versions of the same app for different sites -
this means I have a namespace problem for MyApp v1 vs MyApp v2 etc.

2) The load times for each app are reasonable, however the load time for the
whole lot is now getting towards a couple of minutes. However whilst apache
is waiting for the perl code to load it accepts connections.


What are my deployment options for working around these problems?

Carl

Re: Deployment strategies...

On Thu, Feb 26, 2009 at 12:38 PM, Carl Johnstone
wrote:
> 1) We want to run different versions of the same app for different sites -
> this means I have a namespace problem for MyApp v1 vs MyApp v2 etc.

This has been discussed many times on the list. You can't run two
modules with the same name in the same perl interpreter, so you either
need to change the namespaces for different versions or run separate
servers.

> 2) The load times for each app are reasonable, however the load time for the
> whole lot is now getting towards a couple of minutes. However whilst apache
> is waiting for the perl code to load it accepts connections.

Two minutes? That's a very long load time. I've seen huge codebases
load in 10-20 seconds. I think something might be wrong there, like a
slow database connection or DNS lookup.

Is the real problem that apache is accepting connections during
startup? You could solve that by making your front-end smarter. Some
systems, like perlbal, will wait until apache responds to a HEAD
request before considering a server up. You could also rig your
frontend so that you can manually tell it when a server is on or off
using an external script, and use that to take restarting servers out
of rotation until you verify they're up.

- Perrin

Re: Deployment strategies...

Perrin Harkins wrote:
> You can't run two modules with the same name in the same perl
> interpreter, so you either need to change the namespaces for
> different versions or run separate servers.

Yeah it's a pity that the perchild mpm never moved out of experimental. In
theory something like loading different versions of modules in each child
after the fork could solve the main part of the problem.

The thing is that as I'm running the app under Catalyst, once I've started
splitting off into different server instances, there's not as much of an
advantage in using mod_perl - I can use FastCGI or HTTP::Prefork or even
just run catalyst_server.pl.

> Two minutes? That's a very long load time. I've seen huge codebases
> load in 10-20 seconds. I think something might be wrong there, like a
> slow database connection or DNS lookup.

I'll do some profiling.

Although we don't have a huge amount of code we have a lot of Controllers
inheriting from base classes - checking the 3 main culprit apps they contain
~500 Controllers between them. As it's built up gradually I've always just
presumed the overhead was from that.


> Is the real problem that apache is accepting connections during
> startup?

Yes. The actual restarting to load updated code is reasonably automated so
I'm not sitting waiting. I have had complaints that whilst it's happening
the sites seem to freeze.

> You could solve that by making your front-end smarter. Some
> systems, like perlbal, will wait until apache responds to a HEAD
> request before considering a server up.

Actually I think the hardware load-balancing we've got does that - so rather
than getting apache to balance, I should probably feed the requests back
through that.

As ever, thanks for your advice Perrin.

Carl

Re: Deployment strategies...

Hello,

Perrin Harkins wrote:
> On Thu, Feb 26, 2009 at 12:38 PM, Carl Johnstone
> wrote:
>
>> 1) We want to run different versions of the same app for different sites -
>> this means I have a namespace problem for MyApp v1 vs MyApp v2 etc.
>>
>
> This has been discussed many times on the list. You can't run two
> modules with the same name in the same perl interpreter, so you either
> need to change the namespaces for different versions or run separate
> servers.
>
>
Carl, can you solve your problem by having a global class and then each
site being an object of that class, inheriting or overriding methods as
necessary?

I'm sure I'm missing something in your question because it sounds so
much like what we do and the answer seems so clear that I feel stupid
for suggesting it since you probably already thought of it and refjected
it. :-)

Take care,

Kurt Hansen

Re: Deployment strategies...

Carl Johnstone wrote:

> 1) We want to run different versions of the same app for different sites -
> this means I have a namespace problem for MyApp v1 vs MyApp v2 etc.

Just my opinion, but it's sooo much easier to manage multiple sites if they are
all using the same version of the application. What are your reasons for wanting
to do that?

--
Michael Peters
Plus Three, LP

Re: Deployment strategies...

On Fri, Feb 27, 2009 at 5:30 AM, Carl Johnstone
wrote:
> The thing is that as I'm running the app under Catalyst, once I've started
> splitting off into different server instances, there's not as much of an
> advantage in using mod_perl - I can use FastCGI or HTTP::Prefork or even
> just run catalyst_server.pl.

Well, you probably could run it on FastCGI now. I don't think this
really changes much. You wouldn't want to make every single app use a
different FastCGI backend because it would waste memory, so either way
you'll want to group things together as much as possible.

Some people have suggested using the interpreter config options to
start different interpreters for different virtual hosts. I haven't
tried that and I'm not sure it works.

(Also, you'd be nuts to run anything serious on HTTP::Prefork.)

> Although we don't have a huge amount of code we have a lot of Controllers
> inheriting from base classes - checking the 3 main culprit apps they contain
> ~500 Controllers between them. As it's built up gradually I've always just
> presumed the overhead was from that.

1500 classes is not enough code to cause a 2 minute compile on modern
hardware. I've heard people say that DBIx::Class has a slow startup,
so if you use that you might look there first.

> Actually I think the hardware load-balancing we've got does that - so rather
> than getting apache to balance, I should probably feed the requests back
> through that.

You mean go directly to mod_perl? You'd still want a front-end proxy
for buffering. I'd suggest checking with the mod_proxy_balancer
people to see if they have a solution.

One trick I've used is to make my front end check a specific URL on
the backend for alive-ness. When I want to remove a backend from the
pool, I just move the file aside and put it back when the server has
restarted.

- Perrin

Re: Deployment strategies...

Perrin Harkins wrote:
> different FastCGI backend because it would waste memory, so either way
> you'll want to group things together as much as possible.

Yeah, that confirms my thinking.

> I've heard people say that DBIx::Class has a slow startup,
> so if you use that you might look there first.

We do, I'll look.

> You mean go directly to mod_perl? You'd still want a front-end proxy
> for buffering. I'd suggest checking with the mod_proxy_balancer
> people to see if they have a solution.

I mean balancers -> front-end -> balancers -> back-end.

Using different IPs on the balancers for front/back.

It was supposed to work that way originally, but we never got round to
implementing it once we discovered mod_proxy_balancer in Apache 2.2.

Thanks

Carl

Re: Deployment strategies...

On Fri, 27 Feb 2009, Carl Johnstone wrote:

> Perrin Harkins wrote:
>
> > I've heard people say that DBIx::Class has a slow startup,
> > so if you use that you might look there first.
>
> We do, I'll look.

If you're loading the schema from the database instead of
defining it from all the modules, it helps to do this in a
startup script so the schema loader class gets cloned to
kids after figuring out all the table relationships.

> On Thu, Feb 26, 2009 at 12:38 PM, Carl Johnstone
> wrote:
> > 1) We want to run different versions of the same app for
> > different sites - this means I have a namespace problem
> > for MyApp v1 vs MyApp v2 etc.
>
> This has been discussed many times on the list. You can't
> run two modules with the same name in the same perl
> interpreter, so you either need to change the namespaces
> for different versions or run separate servers.

What about PerlOptions +Parent?
http://perl.apache.org/docs/2.0/user/config/config.html#C_Pa rent_

Mark

Re: Deployment strategies...

On Fri, Feb 27, 2009 at 3:06 PM, Mark Hedges wrote:
> What about PerlOptions +Parent?
> http://perl.apache.org/docs/2.0/user/config/config.html#C_Pa rent_

That's the thing I was referring to that I haven't tried. Can anyone
confirm that this works?

- Perrin

Re: Deployment strategies...

Perrin Harkins wrote:
> On Fri, Feb 27, 2009 at 3:06 PM, Mark Hedges wrote:
>> What about PerlOptions +Parent?
>> http://perl.apache.org/docs/2.0/user/config/config.html#C_Pa rent_
>
> That's the thing I was referring to that I haven't tried. Can anyone
> confirm that this works?
>

There are some threads in the archives from people about it, mostly
related to people having trouble with one thing or another though iirc.

This post seems to indicate they were using it successfully.
http://marc.info/?l=apache-modperl&m=119500073425071&w=2

Adam

Re: Deployment strategies...

Perrin Harkins wrote:
> On Fri, Feb 27, 2009 at 3:06 PM, Mark Hedges wrote:
>
>> What about PerlOptions +Parent?
>> http://perl.apache.org/docs/2.0/user/config/config.html#C_Pa rent_
>>
>
> That's the thing I was referring to that I haven't tried. Can anyone
> confirm that this works?
>
I can now. After a few days of internal testing, I deployed it onto our
live servers yesterday. I've currently got two branches of the code-base
running side-by-side fine.

Carl