Friday, April 29, 2005

Non-obstrusive configuration

When writing an application the author has to think about wiring and configuration issues which often contradicts with fine-grained approach used during building it. These issues depends heavily both upon the design of the code itself (how much state does it have, how it is strutured) and the nature of the library (application) in question.

To clarify: an application consists of a number of functions/classes/modules each of them having (usually) a defined and well-understood scope and set of responsibilities. So far so simple - this is just good engineering practice. To get a coherent whole you need to wire it all together, configuring the parts along the way (network ports, passwords and other runtime options). And this is where it get messier.

How to achieve ease of wiring/configuration while not compromising on system's design on the same time?

I have run into these problems myself a number of times and almost every time has to come with some novel approach. Partly this is to be expected as different requirements calls for different solutions. Still, my own naiveness (experience level) has to play some role as well, as I may have missed some patterns obvious to a more experience programmer.

One of the biggest problems I had is that most of these solutions are obtrusive: once installed they tend to spread thru the code like viruses which hurts maintainability (if you ever change your mind and try a different approach) and code reuse.

The rule of thumb I've learnt so far is to try to split the application into two parts: core (a "library" in a traditional sense) and, um, application itself. While this does not solve the problem, at least it puts it within well-defined boundaries. OTOH, when this "application" gets incorporated into something still bigger you have to overhaul it again which is no good. Even without incorporating, the "library" part sometimes became smaller than the "application" part, which spoils the whole idea.

To put at least some meat into this post, I'd like to mention PyContainer library of Rafal Sniezynski, which is a Python implementation of "Inversion of Control" (IoC for short) idea. IoC in general and various implementations (called "containers") is a hot topic in Java community.

On the surface, this is exactly what I'm looking for: container manages wiring and configuration of its object according to given set of rules. But I can't tell for sure until I try. What I like most is the perceived level of obtrusiveness (lack of): components that go within container have no idea they are within "something" and thus can be developed, unit-tested and reused freely. Only part of the application that uses the container has to deal with it - and my only concern is to make it as small as possible.

P.S.: Re-reading my post I realized that problems I'm talking about may not be so obvious and/or so painful to the readers as they were to me. What was your experience? Do you share my frustration?

8 Comments:

Anonymous Reinout van Rees said...

One problem/challenge is finding good defaults and finding out when to stop. Some configuration is good, too much configurability hurts. If something is configurable, are your defaults good enough or common enough?

Separating an application into a "real" application and a library? Perhaps. Fun thing about python is that you can also use an application as a library. Subclass the parts you want to modify and you've got your own. (At least, that's something that seems quite common in zope/plone).

Reinout

3:13 PM  
Blogger Anand said...

This comment has been removed by a blog administrator.

10:42 AM  
Blogger Anand said...

This comment has been removed by a blog administrator.

10:43 AM  
Blogger Anand said...

This comment has been removed by a blog administrator.

10:45 AM  
Anonymous Anand said...

Oh well, I can relate to your problem very well! I faced a similar problem when I wrote the HarvestMan web crawler fully in Python. HarvestMan has a pretty modular design with functionality cleanly split into separate modules. However, the problem was to share state of the application and its configuration variables between these modules...

I solved the problem by using a novel approach - I wrote a config module by myself, deriving the config class from the 'dict' type. Then I created a registry module which allows one to sort of 'register' all live objects with a unique name. The config object is also registered here along with other objects inside the HarvestMan application. When a module wants to obtain information about a config variable or state variable, it just looks up the global registry (there is a global function for this) with the key name and gets the config object. Once he gets it it is just a simple dictionary lookup to obtain the value of the particular configuration variable.

The system works pretty well and is scalable also, at least in my experience so far. But I have to say that I went through lots of iterations and rewrites before I arrived at this architecture.

I have to take a look at the "IoC" approach and see how it contrasts with my approach.

Btw, HarvestMan is available at
http://harvestman.freezope.org

Regards

-Anand

10:46 AM  
Anonymous Anand said...

Oh well, I can relate to your problem very well! I faced a similar problem when I wrote the HarvestMan web crawler fully in Python. HarvestMan has a pretty modular design with functionality cleanly split into separate modules. However, the problem was to share state of the application and its configuration variables between these modules...

I solved the problem by using a novel approach - I wrote a config module by myself, deriving the config class from the 'dict' type. Then I created a registry module which allows one to sort of 'register' all live objects with a unique name. The config object is also registered here along with other objects inside the HarvestMan application. When a module wants to obtain information about a config variable or state variable, it just looks up the global registry (there is a global function for this) with the key name and gets the config object. Once he gets it it is just a simple dictionary lookup to obtain the value of the particular configuration variable.

The system works pretty well and is scalable also, at least in my experience so far. But I have to say that I went through lots of iterations and rewrites before I arrived at this architecture.

I have to take a look at the "IoC" approach and see how it contrasts with my approach.

Btw, HarvestMan is available at
http://harvestman.freezope.org

Regards

-Anand

10:47 AM  
Blogger Anand said...

Well, I have no idea how blogger posted my comment 5 times!

Sorry for that. Must be some problem with the browser.

-Anand

10:54 AM  
Anonymous Anonymous said...

Well, I'll unfortunately have to agree with you - it is a hard problem! :)

I've been developing software for over 20 years now, and the best approach that I've seen is the general approach of 'Aspect Oriented Programming' - where you 'weave' aspects together: therefore you could (in theory at least) weave your configuration into your logic-modules and hey presto..but that's mostly theory.

In practice, I've found that configuration is required at a minimal level. Whenever I've encountered code that endeavoured to be 'fully configurable' - it's been a nightmare to read and understand...personally I think it's never truly necessary. What's usually required is good clean interfaces; clean separation of concerns between modules; and a minimal amount of configuration applied at the deployment stage.

This doesn't answer any of your issues, but I'd say "there's no 'best' solution", and what you've been doing is what we all do in the end: do whatever fits at the time!..after all we're craftsmen/women..not engineers ;)

- Cheers, Tushar.

7:57 PM  

Post a Comment

<< Home