Max's blog v0.2.1

The blog has moved

2005-11-08T19:06:00.000+02:00

I decided to stop using Blogger and moved to a dedicated hosting for blogging. Please check it out on max.textdriven.com.

Python programmer for sale

2005-08-17T07:32:00.000+03:00

This's been a hot summer for me but now it ends, along with my contract work.

So if you need a (hopefully) good programmer, skilled in UNIX development, Python and Java platforms, agile methods and traditional engineering, please contact me. I'm interested in both short-term and long-term contract work.

pre-built mock classes?

2005-06-08T19:08:00.000+03:00

There is a (relatively) widely used technique in unit testing, called mock objects. There is even a pMock library which provides a Mock class for a Python environment.

Given the "duck typing" nature of the Python itself, it's pretty trivial to build mocks without using any pre-built libraries. What is less trivial and potentially more worthwhile is to have a library of “stock” mock objects.

For instance, I found myself re-implementing ConnectionStub class again and again for different projects. And there is no shortage of other good candidates as well: socket, web request/response, thread objects, etc. Someone on a Python mailing list asks whether anyone implemented a mock filesystem interface, for example.

Having a library of such mock classes, realizing widely-used and “heavy” interfaces may be a good idea.

The only real problem I can think of is whether it’s possible to make these mocks generic enough. Mocks often contain some (application-specific) hard-coded rules and data to realize desired behavior. For instance, socket mock may fail when user attempts to write a “foobar” string – and unit test will use this to check how code handle these kind of errors. These rules may be generalized but it may lead to mock classes becoming unwieldy.

Even keeping this problem in mind, it’s still a reasonable idea for a library, but obviously I’m biased.

What do you think?

Non-obstrusive configuration

2005-04-29T14:02:00.000+03:00

When writing an application the author has to think about wiring and configuration issues which often contradicts with fine-grained approach used during building it. These issues depends heavily both upon the design of the code itself (how much state does it have, how it is strutured) and the nature of the library (application) in question.

To clarify: an application consists of a number of functions/classes/modules each of them having (usually) a defined and well-understood scope and set of responsibilities. So far so simple - this is just good engineering practice. To get a coherent whole you need to wire it all together, configuring the parts along the way (network ports, passwords and other runtime options). And this is where it get messier.

How to achieve ease of wiring/configuration while not compromising on system's design on the same time?

I have run into these problems myself a number of times and almost every time has to come with some novel approach. Partly this is to be expected as different requirements calls for different solutions. Still, my own naiveness (experience level) has to play some role as well, as I may have missed some patterns obvious to a more experience programmer.

One of the biggest problems I had is that most of these solutions are obtrusive: once installed they tend to spread thru the code like viruses which hurts maintainability (if you ever change your mind and try a different approach) and code reuse.

The rule of thumb I've learnt so far is to try to split the application into two parts: core (a "library" in a traditional sense) and, um, application itself. While this does not solve the problem, at least it puts it within well-defined boundaries. OTOH, when this "application" gets incorporated into something still bigger you have to overhaul it again which is no good. Even without incorporating, the "library" part sometimes became smaller than the "application" part, which spoils the whole idea.

To put at least some meat into this post, I'd like to mention PyContainer library of Rafal Sniezynski, which is a Python implementation of "Inversion of Control" (IoC for short) idea. IoC in general and various implementations (called "containers") is a hot topic in Java community.

On the surface, this is exactly what I'm looking for: container manages wiring and configuration of its object according to given set of rules. But I can't tell for sure until I try. What I like most is the perceived level of obtrusiveness (lack of): components that go within container have no idea they are within "something" and thus can be developed, unit-tested and reused freely. Only part of the application that uses the container has to deal with it - and my only concern is to make it as small as possible.

P.S.: Re-reading my post I realized that problems I'm talking about may not be so obvious and/or so painful to the readers as they were to me. What was your experience? Do you share my frustration?

A problem with Python

2005-03-30T16:05:00.001+03:00

Ian Bicking wrote a passionate post where he encourages community to put more efforts into Python-for-web-application domain.

I have another complain about the Python (both for language itself and tools and libraries around it). Here it is: Python cannot fully utilize hardware resources. You see, I have several [web] programs running on my machine, some in Java and some in Python. Some of these are the apps I'm writing and debugging myself, some are third-party.

When I'm working on Java app, I have some serious stuff loaded: Eclipse IDE, Tomcat web server, database server, Firefox with API documentation opened, etc. Basically, you need 1Gb of RAM to have it all work smoothly. And even then it takes 12 seconds to redeploy an application. And we all know Java technology has sophisticated JIT compilers which means execution speed is much higher compared to Python.

What do we see in Python? Instance of Vim editor, WebKit/Apache server, may be a Firefox running and a database. That's all. How much resources does it take? Nothing. How long does it take to redeploy a WebKit application? Nothing again. In fact, you could develop Python app on some archaic Celeron-666/256 Mb box and notice little difference with modern Athlon 2800+/1Gb. At least, that was my experience.

What the heck?

If the language can't make a serious load out of modern hardware it can't be nothing else then a toy language, right? Programming is hard and it must feel that way.

Therefore, I ask the community to put more efforts to "grow" both the language and the tools. The day I see my python application deployed in a 12 seconds to a WSGI-backed WebKit I know: we have a serious, mature language. No, not a "language" - it'd be called a "platform" by then. Definitely.

A problem with Python

2005-03-30T16:05:00.000+03:00

Ian Bicking wrote a passionate post where he encourages community to put more efforts into Python-for-web-application domain.

I have another complain about the Python (both for language itself and tools and libraries around it). Here it is: Python cannot fully utilize hardware resources. You see, I have several [web] programs running on my machine, some in Java and some in Python. Some of these are the apps I'm writing and debugging myself, some are third-party.

When I'm working on Java app, I have some serious stuff loaded: Eclipse IDE, Tomcat web server, database server, Firefox with API documentation opened, etc. Basically, you need 1Gb of RAM to have it all work smoothly. And even then it takes 12 seconds to redeploy an application. And we all know Java technology has sophisticated JIT compilers which means execution speed is much higher compared to Python.

What do we see in Python? Instance of Vim editor, WebKit/Apache server, may be a Firefox running, a shell. That's all. How much resources does it take? Nothing. How long does it take to redeploy a WebKit application? Nothing again. In fact, you could develop Python app on some archaic Celeron-666/256 Mb box and notice little difference with modern Athlon 2800+/1Gb. At least, that was my experience.

What the heck?

If the language can't make a serious load out of modern hardware it can't be nothing else then a toy language, right? Programming is hard and it must feel that way.

Therefore, I ask the community to put more efforts to "grow" both the language and the tools. The day I see my python application deployed in a 12 seconds to a WSGI-backed WebKit I know: we have a serious, mature language. No, not a "language" - it'd be called a "platform" by then. Definitely.

webfixture revised

2005-03-25T18:34:00.000+02:00

There was some interested in my webfixture project so I decided to bite the bullet and pulled it online. See repository and project page.

Software, process and people

2005-03-15T10:21:00.000+02:00

A recent article on FastCompany site tells a story about software engineers at NASA that write software for space modules and other such stuff. The story is entertaining and instructive but what I find naive is the idea that engineering approach used for this kind of projects at NASA could (and should!) be transferred to other kind of projects.

First, who writes it?

That's the culture: the on-board shuttle group produces grown-up software, and the way they do it is by being grown-ups. It may not be sexy, it may not be a coding ego- trip -- but it is the future of software.

It's a bit of a stretch, but I mostly agree here. To create a good software you need professionals. Software production is a complex activity and experience is a key. It doesn't really matter whether you look "sexy" or not and whether you have a spouse and kids. Of course, experience takes time and thus experienced programmers tend to be older and as such, may no longer care much about the "cool outfit" but care more about the family.

Next question: "how do they write the right stuff"?

The answer is, it's the process. The group's most important creation is not the perfect software they write -- it's the process they invented that writes the perfect software.

Yeah, I agree on this but I suspect for a different reason than original authors. Here is another quote which clarifies their reasoning:

The most important things the shuttle group does -- carefully planning the software in advance, writing no code until the design is complete, making no changes without supporting blueprints, keeping a completely accurate record of the code -- are not expensive. The process isn't even rocket science. Its standard practice in almost every engineering discipline except software engineering.

I think they are putting the cart before the horse. They succeed not because of the process per se but because the process was a good fit and probably was invented to match the kind of projects they are working on. And of course, no process would help if the execution is flawed. Attempting to copy the process to different project may lead to Cargo Cult Engineering, masterfully described by Steve McConnell in his Professional Software Development book.

Even more important and fundamental point - "expensiveness" of this kind of approach. The hard truth is that process above is not viable for a mainstream world of the software development. For a one, you can't really fix the requirements. And those who think they can often find themselves out of business. And another one - the process must be cost-effective which means finite resources and trade-offs and compromises.

I do believe we (industry) have a long way toward skillful software development and I'm looking for improvements that advances SWEBOK, but thinking that following NASA processes may give that to us is lame.

Well, this post is probably somewhat lame itself, as I'm not an expert either. ;-)

Wikiphobia

2005-02-28T11:28:00.000+02:00

Mr. Ed published another article on his hacknot.info site. This one is called Wikiphilia and tries to present Wikis as an inadequate tool for most (all?) application it has been put to. Or at least that is how I understood him. He writes controversial articles and (IMO, again) wrote nothing particularly convincing since a couple of earlier good ones.

The "wikiphilia illness" goes too far as well. While I agree that Wikis (like any other tool) can be misused or used to procrastinate I don't buy his general criticism. To me, it mostly sounds like a projection of his own (or made-up) phobias and behavioral models onto Wiki's users. A quote:

Whilst anyone can add their opinion to a Wiki page, anyone can come along and remove it. Therefore it is vacuous to claim that a Wiki affords equal opportunity for all users to express their opinion, when it allows any user to suppress that opinion through simple deletion. To ensure that their contribution persists, a user has to watch over their content, re-inserting it after anyone else has deleted it. Some users resort to writing bots for this purpose.

I'd say this is highly atypical (and unethical) pattern that the user constantly reinserts his own statement which were deleted by the community (for a reason, I suppose). Or even write bots (sic!) for this purpose. Of course, a malicious one may do so but that's a different matter.

I don't find it compelling to argue with the rest of the article, interested readers may read it themselves and made their own conclusions.

Python Grimoire Wiki

2005-02-28T11:25:00.000+02:00

Just stumbled upon an impressive Python Grimoire document. Woudn't it be great to have this content available as a Wiki, to ease maintaining it and expanding into some kind of uber-knowledge base for pythonprogrammers?

I've been looking for this kind of repository since I've appreciated the analogue in a Tcl world. It's just great and help tremendously.

Yes, I know about ActiveState's cookbook but this imho a different thing.

update: Fix formatting issues and add missing link
update: There is an effort-in-progress to convert Grimoire to a Wiki. See this.

The good, the bad and the ugly

2005-02-25T16:55:00.000+02:00

disclaimer: I am living in a developing country and I am working for oversees customers so keep this in mind while reading the following. This post is not about advocating offshore development model either.

A bit too often for my taste, I encounter on the Web the statement that "third-world" developers are inferior to that of USA or, say, Germany. Just to illustrate my point, here is a quote from a recent thread on JoS business forum:

I find (on average) that off-shore coders know one or two programming languages and how to implement things in them specifically. I find (on average) that on-shore coders [ that get disgusted with and leave RAC] are the kind that are multi-talented and flexible due to this.

I see little reason trying to fight this legend and it's sort of natural to see this point of view emerging. It's convenient for those who feel resentful by this trend and gives some kind of moral satisfaction. People in general are lazy and our mind usually tries to absorb ideas in a way that doesn't alter an existing picture of the world much. And so it sticks.

Plus, there are always plenty of people who tell the horrors about how bad the source code was that these "code monkeys" produce. And if it's not bad then it must stolen from some GPL product googled off the web. Rendering the developers not simply unprofessional but practically unlawful.

And yes, there there are indeed plenty of supportive cases against the "code monkeys" so calling this a legend is probably a stretch. But a simplification it is. And yet, to throw in some anecdotic kind of proof, the worst code base I've dealt with was handled to us from "on-shore" and the best programmers I have the luck to work with were sitting next to me.

To repeat, I'm not to defend offshore model in general. It is inefficient and currently viable only because of the peculiarities of the world's economic and drastic changes that were made in the last 50 years. But people are just that - people. There are cultural, historical, economical and other differences, but they are minor and has little inherent impact on one's professional abilities. Of course, these differences do impact the opportunities one may have to realize himself.

Amen.

Blog of the week

2005-02-24T16:28:00.000+02:00

Added Mike Spille to my blogroll. A couple of highlighting posts: Don't Let Yourself Get Unitized and Prevayler: There really must be one born every minute. Amazing reading.

Unfortunately, his full-text feed is badly formatted, so end up subscribing to excerpts.

announcing ua-devtalk

2005-02-21T15:59:00.000+02:00

Just in case I've been reading by someone from Ukraine...

I invite you to join a ua-devtalk google group. It's an attempt to gather and organize ukrainian IT professionals community.

Project postmortem

2005-02-21T15:43:00.000+02:00

As my project reached 1.0 milestone I decided to run a postmortem. This blog post contains some parts of the lengthy postmortem document I actually wrote. I excluded project details/conclusions to protect my IP (so to say ;-)) from potential competitors but kept sections which casual reader may find useful.

Btw, if you're interested in postmortems in general, here are some links I found myself useful:

Conducting a Project Postmortem by Steve Pavlina. Concise and up-to-the-point. I used it as a table of contents for my own postmortem.
Postmortems: Looking Back, Looking Ahead. More coupled to games development but still a very interesting reading.
Nice repository of all kind of project templates and guidelines, including post project review questionnaire.

And a couple of seemingly interesting books (haven't had a chance to read them

I'm still not sure whether I should have published this, but I'm driven mainly by curiosity - we'll see if this will lead to anything.

So, let's end with the foreword, here it goes.

List of known problems

Overall, there were a few bugs discovered during and after the deployment. Partly this is because the system is new, small, freshly developed and thus completely understood. Partly (hope so) because I adhered to test-heavy development process.

Database connection stuck. This was most severe one: site was dysfunctional for several days before I noticed the problem. Luckily, quick fix was trivial - just have to restart an AppServer. The problem was with SQLObject's handling of db connections - if the db was restarted any further attempts to call a db operation lead to OperationalError exception. Unfortunately, I didn't (yet) find a solution to fix the root cause.
HTML mangling in XSL transformations. Another problem (kind of annoyance, really) is related to two-step XSL transformation algorithm employed while rendering web pages. Second step was folding a pair of empty tags into single empty one, e.g. <p> <p> was folded into <p/>. This turned to be a big problem for some HTML tags as it caused the page to rendered incorrectly. The problematic tags I had to "fix" were TEXTAREA, TABLE, TD. I failed to fix the
root cause again, and ended with inserting non-breakable space symbol to prevent XSL for doing the fold.

That's it for a list of known problems.

What went right

Here is a list of thing that I think I done right or had the luck to have it right:

Development platform/tools. I was already a rather experienced Python programmer and had used Apache/WebKit in several projects before. It didn't reveal any bad surprises during development and I feel this was a solid choice. The only new library I used for this project was SQLObject to handle object's persistence task. While I had some minor problems and one major one with it certainly was a net win for me and saved considerable amount of time.
Software quality. There were very few bugs discovered during and after the deployment. I attribute this (partly) to a test-heavy development process: the unit-test ratio coverage was at constantly at about 95%.
Good architecture. The system is organized in a simple, solid and straightforward way, providing a good starting point for further evolution. All major interfaces with third-party code, such as db persistence, web application server, XML toolkits
are properly abstracted. Of course, there are some skeletons buried here and there but overall I'm quite satisfied with it.
Project management tool. Following Joel advice, I decided on Excel as my project management tool. It worked out nice. Not that it went without any friction, but good enough and with minimal overhead. I explored some other alternatives, such as: Microsoft Project, plain-text file with outline mode (in Vim) or some Palm-based project tool. But in retrospection, Excel seems like a right choice.
Project management process. Being a business head / programming team in a single person meant I could judge very accurately what should be done next. In previous projects, lacking some important details from business context, we (a development team) often found ourselves spent a lot of time and efforts on things that were not,
strictly speaking, very important or sometimes even needed at all. The ability to guide development from the business perspective was very satisfying. I didn't understood this at first but in the end really appreciated it. The trick is to not getting charmed by technical problems but to concentrate on users and their needs.
Hosting provider. Being a web-project newbie, I had to figure out things like hosting, domain name registrations, etc. Luckily, my choice of hosting provider was solid and I have no problems with it.
Credit card processing. Certainly it made little sense to shot for a full card-processing routing on-site so I had to select a third-party vendor that would do this for me. I selected 2CheckOut and it was OK for me. At least, it had successfully processed a number of sales for me with relatively easy setup.

What went wrong

Business opportunity evaluation. It's quite possible that I shouldn't blame my lack of marketing skills for too much. There is a valid chance that the potential market is so small that no amount of marketing efforts would lead to a meaningful results. I had to think on project's feasibility harder.
Grossly overengineered solution. An awful lot of time (initially) was spent on things I didn't really need (yes, that YAGNI again). A remarkable example been a sophisticated, XSL-based, multilingual site generator, which produced all site's static pages in all supported languages. I ended up with a fully dynamic site and moreover, uni-language. Hence the effort was an (almost) complete waste of time. In a retrospect, I could have reduced dev time in about 5 times and still deliver a functional solution.
Marketing strategy. Well, I probably should say it was absent. Mind you, I have an idea but after it failed to bring expected results I
felt short.
Site design. Being a one-man project and lacking good design skills I ended up with a poorly-designed (even for my taste) site. Moreover, without proficiency in web frontpage technologies (like CSS) countless number of times I found myself in a situation where I knew what I'd like the page to look like but don't know how to implement this.
Credit card processing. I did mention this in a what went right section and now I mention it here. While 2CheckOut worked OK for me it is not that great choice for several reasons, esp. compared with another one, like iKobo (which I didn't know about at the time). The weak points of 2CheckOut for me were: up-front setup fee, relatively big per-transaction fee and very cumbersome and expensive way to transfer money from 2CheckOut account if you're outside USA/Canada (like me).

That's it.

While I intended to list at least ten things I don't know what else should I mention. The above items were a real problems and other difficulties I had are minor in comparison.

Concurrent Java books

2005-02-14T12:05:00.000+02:00

Recently I has been involved in a Java project and decided to refresh my multithread-programming skills and, more importantly, to grok how it gets done in Java.

I'm currently reading Concurrent and Real-Time Programming in Java by Andy Wellings. While the crux of the book is real-time system programming, it devotes about a third to the generally applicable concurrency topics and how they are exploited in Java. Plus, it even covers concurrency-related changes introduced in Java 1.5.

This is nice, especially compared to the latest edition of the classic Concurrent Programming in Java: Design Principles and Patterns, published in ... 1999! There are rumours that a third edition is underway, but until then this book could serve as a good, modern and concise introduction to concurrent programming in Java.

A pressure for concurrency

2005-02-14T12:04:00.000+02:00

Herb Sutter published an article called The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software. It's a fascinating reading in which he argues we (the programmers) are now facing the next major paradigm shift since introduction of object-oriented programming. The reason?

Applications will increasingly need to be concurrent if they want to fully exploit CPU throughput gains.

And this is caused by the new hardware trend towards multicore systems, in contrast with CPU clock speed wars we had till recently.

IMO, a need for concurrent progaramming would impact mostly the "high" end of the programmers spectrum: system programming (OS, RDBMS, Web servers, game engines, etc.) and library authors. For many (most?) end-user application the execution speed is not crucial for a long time.

PS: It will be interesting to watch out how tooling vendors react to this trend. For instance, CPython has an infamous GIL which looks like a real obstacle for Python programs on multicore systems.

Java may not be that bad after all

2005-02-11T09:57:00.000+02:00

There is a popular belief (which I'm not going to argue with) that programmer's productivity in Java is "inferior" to that in more dynamic languages, like Python.

That's OK but Python seems "inferior" to Java when it comes to innovations (in the environment, not in the language itself). There are countless (ok, shall I say many?) python packages which started as a port of some Java library. Those that come to mind first: unittest and logging packages, which are now even part of the Python standard library. Other examples: PyContainer, PyFIT, Cheetah, you name it. To some degree, larger frameworks often borrow as well: WebKit (servlet, request/response architecture), pyworks (similar to webworks), probably others.

Sometimes this pale imitations evolve into more pythonic things, sometimes it got replaced by built-from-scratch better alternatives (like, say, py.test). Of course, there are genuine python packages which shine even compared with the brightest Java counterparts (Twisted comes to mind first), but these are comparatively few.

The ideas' circulation between languages/platforms is a good thing and there is nothing wrong with it. Still, why Python-Java relationship are so asymmetric? Is it just reflects the difference in size of the respective communities? Is it something else here? Or are my perception skewed?

Efficient XML processing in Python

2005-02-06T13:32:00.000+02:00

In a recent article Decomposition, Process, Recomposition on xml.com, Uche Ogbuji talks about strategies for processing large XML documents in Python. This led me to recall my own experience with processing of multimegabytes XML documents.

In general, there are at least three popular strategies to deal with XML in Python: SAX-based, DOM-based and, er, shall I say, pythonic. SAX realizes a speedy and memory-light approach but which shifts the burden of keeping a processing context onto programmer. DOM gives cross-language, standard, verbose and memory-hungry strategy which scale poorly for large documents. By pythonic I meant a range of libraries, like gnosis.xml.objectify, ElementTree, xmltramp which share a common attitude to provide a nice, Python-friendly API.

I used ElementTree (and recently, it's new, C-based re-incarnation) library. The task was as follows: given a large (~ 10Mb) XML document of some complex structure the program had to significantly "extend" it with new information and write to a new file. This new data was partially specified using the external sources and partially computed from the document itself, according to a bunch of arcane rules. Computation involved several traversals along the document structure to gather needed information.

The program (and transformation) itself was just a single link in a lengthy chain of transformations. There were other parts, written as XSLT procedures and DOM-based Javascripts that used to perform other

A decompose, process, recompose approach, outlined in the article was realized by the ElementTree itself, my task was only to provide necessarily processing steps. In contrast with SAX, ElementTree builds an accurate in-memory presentation of the entire XML document, but unlike DOM, it's memory footprint grows much more slowly with the size of document, thanks to efficient representation. This gives the best of both worlds: versatile and convenient presentation model with good scalability for large documents.

My only major complaint was the lack of XPath support. While ElementTree does offer some very limited support for basic XPath expression it goes a long way towards a full-blown implementation. Thus I was forced to write a bunch of custom finders where a single XPath would do the trick. Luckily, it's pretty straightforward to write them.

Exception-based code antipatterns

2005-01-24T19:15:00.000+02:00

To continue the topic of exception-based code, there are cases when bad code is easy to spot. For example, this snippet is almost always inappropriate:

  try:

   # do something

 except: # catch any possible exception

   pass  # and just ignore it

Semantically, this is equals to error-code-based code with no error checking. It has valid use cases but there are a few.

Another antipattern, more coupled specifically to Python:

  try:

   # do something

 except Exception, e:

   # handle it

Most of the time you should catch specific exception class(es) instead of base Exception class. The problem with this approach is that in Python, everything is reported through exception mechanism, including syntax errors, keyboard interrupts, program exit, memory errors, etc. And usually you don't want to catch these kind of exceptional conditions.

This catch-all is often a sign of a sloppy programming, where a programmer was too lazy to list actual exception classes to catch or the calling code is known to raise all kind of exception and there is no documented exception class to look for (which is often the case for poorly written/abstracted modules).

Unfortunately, advise to catch specific exception classes has a drawback as well. When the calling code is updated it could raise new types of exception and thus handling code must be updated as well. For third-party code, this risk often could be mitigated by abstracting external interface and re-warping (and documentin) use of exception classes.

Exception-based code

2005-01-24T19:11:00.000+02:00

In a recent post, Raymond Chen writes about an important difference between plain old code from pre-exceptions age, that used to return special error codes to indicate errors and modern exception-based code:

...it is extraordinarily difficult to see the difference between bad exception-based code and not-bad exception-based code.

And then he concludes:

Consequently, when I write code that is exception-based, I do not have the luxury of writing bad code first and then making it not-bad later. If I did that, I wouldn't be able to find the bad code again, since it looks almost identical to not-bad code.

Arguably, exception-based code is safer because it fails early, as soon as exception is raised at the point of failure. Contrast this with error-checking code where sloppy programming style (not checking for a error code explicitly) could result in an obscure failure in a seemingly unrelated part of the program much later. On the other hand, sloppy programming style for exception-based code is more damaging, as you'd have to review the whole codebase line-by-line to find all potential places where exception-handling code need to be added.

When I started practicing TDD I often wrote simplest possible implementation that was arguably too simple with regard to exceptions. As I discovered over time, usually it's more
efficient to write necessarily exception-handling code (if any) right from the start. If I'm in hurry or not sure how to handle a possible exception I at least put in an assertion guard.

But the killer feature of exception-based code is this: programmer is freed from error-checking most of the time. While you do have to be careful while writing code and every line of code is a potential source of an exception, typically there is a relatively few places where you add
exception-handling code: external resources' handling, inter-system interfaces, input/output with the outer world, etc.

Overall, exception seems to be a net benefit. Result is a more stable program (fails early) with less efforts (fewer places to do error handling). Yes, it's hard to write a good exception-based code, harder then write a good error-code-based code but who said it's going to be easy?

There is no such thing as a free launch, after all.

quote of the week

2004-12-29T14:51:00.000+02:00

Whatever works is good enough. If you can understand what the customer wants, you win. Of course, if your customer doesn't know what they want, then it is like arguing with your wife: no matter how right you are, you are still sleeping on the couch.

Source

Optional Static Typing in Python

2004-12-28T13:46:00.000+02:00

Guido van Rossum talks about Adding Optional Static Typing to Python. Here is my take on it.

about "Compile Time vs. Run Time".

I 'd be completely happy if the type check would be performed at run-time, not on compile time or import time. After all, that's how a Python developer expects the things to be.

You may argue that's this doesnt buy you much but I disagree. ;-)

Firstly, it's a first and logical evolution from the current state of affairs. Secondly, it does provide some tangible benefits.

Currently, many developers are used to write something like this:


def foo(g,a):

   assert isinstance(a, str), a

   assert callable(g)

   ...

Strictly speaking, these assertions are not required, but they do help to detect errors as early as possible which is a sound engineering practice.

With the help of type interferencing or manifest typing we could free programmer from this tedious work, if he wants to.

Duck typing

What I really wanted to see in Python is an explicit support of DuckTyping, which is an integral part of type system anyway. The problem is IMO with a reasonable and helpfule implementation.

For some, a type interferencing (a sketch from the Haskell type system) may help. For instance if the foo() takes parameter a and then invokes method bar() on that parameter, compiler (or VM) could check (at least, at run-time) that the argument passed to foo() does have a callable bar attribute.

This also can be extended on operator overloading, such as __add__ or __lshift__.

And my point is, even with such a limited form (run-time checking of automatically derived type information), it could be a big step forward.

Optional type annotation (embedded in the source code) could be a next step.

simple and extensible blogging tool in Python

2004-12-18T14:18:00.000+02:00

I wonder, can anyone recommend some simple and extensible blogging tool, written in Python? The idea is to retrofit site News section to use a some kind of [primitive] blogging system, instead of hand-written RSS 2.0 feed and hand-made modification to the site's front page.

To realize this idea I'm looking for some, preferably python-based tool, which could be easily plugged-in into my custom presentation, authentication and persistance modules (ideally) or at least I should be able to provide my own presentation. The API is minimalistic: add post (item), add comment, get item as XML, get RSS/Atom feed and that's basically all.

Any recommendations?

Palm may be not dead after all

2004-12-16T14:05:00.000+02:00

In the course of the recent events in the Palm world (like infamous T5 release or Sony decision to exit PDA market), many, myself included, start seeing the PalmOS platform as an obvious loser. It will surely take a lot of hardwork and luck for Palm to break this negative trend, but these insiders' comments on PalmOS for Linux project give me some hope. Not that I care a lot, but still.

It'd interesting to watch whether PalmSource succeed to repeat Apple's indisputable victory with it's Mac OS X. And I wonder what's happened with BeOS kernel that's already in the disposition of PalmSource's engineers? Has it been abandoned in preference with Linux?

strategy for acceptance tests

2004-12-06T10:56:00.000+02:00

With the version 1.0 of my project out there, I have a planning session to decided what to do next. Of course, I'm going to start working on release 1.1 now but in parallel with that, I intend to work on the acceptance (functional) test suite which currently drags behind due to the 1.0 release schedule pressure.

Recent Robert Martin's post in the FIT mailing list summarises my strategy for these kind of tests nicely:

...[I] INSISTED that the majority of automated acceptance tests be run JUST BELOW the GUI. (i.e. not THROUGH the GUI).

Some acceptance tests, those that test GUI functionality only, MUST be run through the GUI. But I don't want any business rules tested through the GUI.

Of course QA must also get their fingers and eyes on the system. I don't want them doing this with any scripts. Anything that can be scripted should be automated. Rather, I want the QA folks using their ingenuity and creativity to manually operate the GUI and figure out devious ways to break the system.

Blog of the week

2004-12-03T16:36:00.000+02:00

Found a remarkable blogger this week - Phillip J. Eby, a primary author of the ambitious python framework for enterprise applications. I doubt I would use his framework anytime soon, but his general thoughts on python and software development in general are much appreciated.

Deleting code

2004-11-30T18:07:00.000+02:00

Just encountered a nice post titled Deleting code by Ned Batchelder. I mostly share author's attitude toward the subject so there is no point to argue. Just have a small addition regarding the code you wrote as experimentation or in anticipation for something and then discover that YAGNI. In this situation I often found myself not simply delete this code but move it to some temporary file instead.

By convention, this file is often called scratch. or just x. Here are the various code snippets that were purged from the production code or even never managed to get there got accumulated over the time. I found it easier and simpler to look through this file than to rely on SCM.

Ready? Set. Ship!

2004-11-27T13:26:00.000+02:00

For several last months, I've been quitely working on my small sideline project which I'm now ~~ready~~ decided to get out of the door. This is my first take on the entrepreneurial-like venture and I have no idea how and where it would land and whether it would take off at all. But even if it won't succeed I glad I managed to drive the project to the point where it's shipped to the wild and not hung as a vaporware forever.

In spite of a lengthy feature sheet laying in front of me I decided to ship the most basic, bare-bones version that is (potentially) useful to my audience. There is a certain risk that such a primitive version would scare away prospective customers but the risk of developing for too long without being exposed to the real-life feedback is IMO much bigger. If there is any market at all for this kind of service it would shown itself even with such a limited implementation. Or at least, this is what I'm hoping for. ;-)

Oh, I just re-read the post and noticed I didn't even mentioned what the service is. ;-) It's located at the http://postman.com.ua and, specifically for google bots, allows you to send photos, letters and postcard to your relatives, friends and business partners in Ukraine. Actually, you could only send photos for now but the rest will follow (crossed my fingers).

Two step view pattern implementation with XSLT

2004-11-23T13:24:00.000+02:00

For the project I'm currently working on (in Python/WebKit) I decided to use an XSL technology to handle a presentation issue, instead of a more stock options like Cheetah or PSP. The main reason for this was to escape from using niche solutions and try to employ more mainstreamo ones instead, leveraging benefits that comes from this.

For the presentation logic itself, I used a Two-Step View pattern, publicized by Martin Fowler's excellent book on enterprise patterns. The original implementation actually performed two xsl transformation. First one with a page-specific template, which were passed a page-specific data as XML input string and/or parameters dict. It returned an xml with a "logical markup" which were then passed to another xsl template, called site.xsl which is global per site and transformed this xml-based markup into proper (X)HTML.

It worked OK, except the fact that I had to call to XSL engine twice per page, in order to get a resulting (X)HTML which taken a considerable amount of time (about 0.2-0.4 seconds in total).
But recently, due to a nice tip from 4Suite co-author, I managed to fold these two steps into a single xsl transformation. The trick is a node-set() function which allows you apply the XSL engine to a fragment of output tree already produced.

Here is except from my implementation. The algorithm is as follows:

Build a "logical markup" tree and save it to a variable.
Apply an imported templates to transform this to required output format.

<xsl:import href="site.xsl" />



<xsl:output method="xml"

doctype-public="-//W3C//DTD XHTML 1.0 Strict//EN"

doctype-system="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"

encoding="UTF-8"/>



<xsl:template match="/">

<xsl:variable name="view">

  <xsl:call-template name="logical" />

</xsl:variable>



<xsl:apply-templates select="exslt:node-set($view)" mode="view"/>

</xsl:template>



<xsl:template name="logical">

<view:standard-page title="Test page">



  <view:content>

    <p>This is a  test page</p>

  </view:content>

</view:standard-page>

</xsl:template>

And here is a greatly simplified version of the site.xsl page:

<?xml version="1.0" encoding="UTF-8" ?>

<xsl:stylesheet version="1.0"

 xmlns:xsl="http://www.w3.org/1999/XSL/Transform"



 xmlns="http://www.w3.org/1999/xhtml"

 xmlns:view="http://postman.com.ua/xslt/view10">

 <xsl:output method="xml" encoding="UTF-8"/>



 <xsl:template match="view:standard-page" mode="view">

   <html>

     <head>

       <title><xsl:value-of select="@title" /></title>



     </head>

     <body>

       <xsl:apply-templates select="view:content" mode="view" />

     </body>



   </html>

 </xsl:template>



 <xsl:template match="view:content" mode="view">

   <div style="min-height: 400px;">



     <xsl:apply-templates mode="view"/>

   </div>

 </xsl:template>



 <xsl:template match="*" mode="view"> <xsl:copy-of select="current()" /></xsl:template>



</xsl:stylesheet>

Being an XSLT newbie myself, the actual code is a bit awkward: I have to use a special namespace (view) and mode parameter of the xsl:template to escape possible infinite loops and to get namespaces right in the output document. Hopefully, an XSLT expert could provide a more streamlined implementation of the same idea.

The node-set() extension function support both by MSXML and 4Suite XSLT which I happened to use on this project. The only minor problem is that you have to declare the namespace a bit differently for these two engines.

My conclusions? I was able to simplify the presentation logic and get a considerable speed up (in a 2x-10x range) with the price of some boilerplate code to add to each xsl page and potential portability problem between other XSL engines. Looks truly like a win to me.

PS: Glad to have a NoPaste service at hand.

using other language to write unit tests

2004-11-12T16:21:00.000+02:00

Just encountered an article on IBM's devWorks which gives an example on how to use Groovy to write unit tests for Java code. And this reminded me of experience I once had with mixing languages in unit testing.

In a rather big project I once took part, we used an in-house Tcl-based framework to perform system tests via a network. Some other libraries developed for that project were coded in C++ and Lua. I decided to test this libraries as well using the same Tcl framework I've already built. I used SWIG to make C++ code accessible from within Tcl. I used the same trick to access C API of the Lua interpreter and feed it with Lua code. Just imaging this mess -- evaluate Lua code from within Tcl virtual machine (written in C) via generated dispatching code through the Lua virtual machine and down to C, at last!

It was kind of nice to be able to load a C++ .so library into Tcl shell, experiment with it, call various methods and play with results. But as for production usage, it didn't worked out.
There were two main reasons:

Generation step always was a bit brittle. Even though SWIG managed to handle all C++ constructs we used it required some manual setup and it sometimes was hard to figure out how to use the generated code.
Using different languages (and I'd add, so distinctively different languages) were putting a significant burden on a developer. It just felt awkward.

I don't even mention that instead of developer who is proficient with one language you need a developer who is proficient in both. Plus could master SWIG tool.

Groovy was designed to be as close to Java as possible and it's interoperability with Java is excellent, so unit tests for Java code written in Groovy won't suffer from the same problems I had. Still, I think it's better to stick to a single language for both, no matter how close Groovy is to Java. If you buy arguments for Groovy's simplicity - then just write both production code and unit tests in Groovy. That's my opinion.

profiling unit tests

2004-11-12T14:05:00.000+02:00

As were already noted elsewhere, the so-proud-of, "batteries included" Python standard library has a lot to be improved. And while Java tools often being critiqued as overly complex, many of python modules are arguably simplistic, not just "as simple as possible".

I run into this just yesterday, when tried to profile my unit tests.

Usually unit tests run in a fraction of a second. Sometimes though, most often because of some heavy-weight interface not being stubbed, they suddenly could take a several seconds. So I thought to myself: "It would be nice if I could a get report about which test methods ate most of the time".

After struggling for half an hour with a profiler (hotshot or the old one, whatever, they both use pstats module) I give up. Sure enough, I could get an overall program statstics, even filter it by name of the function with a regexp, but there were no way to trace from the actual function (like socket.connect) to a test method that called it (like testReportServerStatus). Or at least, I didn't find it. The latter is probably the case and the reason I was not able to find it is because pstats has a contrived API. I would say, almost a lack of it thereof. When called, it just prints to the screen what's been told but have no way to just return accumulated statistical data lump, so I could play with it by myself.

But then another idea came to my mind. Why do I have to struggle with a profiler in the first place? I already have a framework (unittest, that is) which handles the task of runing the test methods. It should be simple to extend it to allow time measurement and report results, shouldn't it? Well, it turned out more or less simple.

I had to extend both TestSuite class and TestRunner but it did worked. The only major problem I encountered is that TestSuite per se has no idea from which module a particular test has come. Luckily, I was already using my own function to gather test suites so it was simple to establish a module-to-suite mapping. Still, it prints only name of the modules not name of the actual test methods but at least it works. Any idea how to fix this is welcome, though.

A perspective on software methodologies

2004-11-09T14:09:00.000+02:00

A interesting quote I read the other day by Humphrey, father of the CMM:

What people really want is a high-quality system that implements everything they want, at no cost, right now. Everything else is a trade-off.

What's striking in this metaphor is the idea of compromise. You always have to sacrifice something, one way or another. And some elaborate models, like the iron triangle by Alister Cockburn, were probably inspired by this idea.

I wonder, can we differentiate various software development methodologies by the approach they preach to find this compromise?

Programmer's HowTo

2004-11-03T11:28:00.000+02:00

Several years ago, when trying to master Linux I had to read a lot of documentations, man pages and HOW-TOs in particular. The topics covered by these manuals/how-tos were rather routine (as it seems to me today) - things like modem setup, font setup, pppd configuration, etc. The distinguishing feature for most of the good ones is that they're tightly packed with useful information - often I had to re-read them in order to grok completely.

Recently I stumbled upon another HOW-TO: How to be a Programmer by Robert L Read. While he addresses a different topic - the topic of a professional growth as a programmer - he managed to preserve the spirit of these HOW-TOs: concise, clear and packed with insights.

That's one of the best texts I had read on a topic in the last few years and arguably the one with the best size/usefulness ratio. It's a pity I haven't encountered it earlier (the first version was released in 2000).

Best Software Essays of 2004

2004-10-22T18:13:00.000+03:00

An interesting discussion has opened on Joel On software forum - Joel asks readers to post essay which they consider were the best essay on software in 2004 (2003 is accepted as well). Great way to discover articles and/or authors you may have missed.

As it turns out, I've already read about a third of them and they were indeed excellent. Looking forward to consume another two-thirds though.

Check it out here: Best Software Essays of 2004.

efficience vs. simplicity

2004-10-21T13:03:00.000+03:00

I had to write a function that would generate a fixed-length sequence of symbols to be used as short, human-readable identifier for some database entities.

The first version I wrote rather quickly used a brute-force algorithm:



def idgenerator(length, prefix=None, seq=None):

 if prefix:

     length = length - len(prefix)

 seq = seq or (string.digits + string.ascii_lowercase)

 while 1:

     bits = []

     for _ in range(length):

         bits.append(random.choice(seq))

     value = ''.join(bits)

     if prefix:

         value = prefix + value

     yield value

It works okay, but I was concerned with it's efficiency: it calls random.choice() function N times, where N is length of identifier. And I assume random.choice() must be computationally expensive. So, after some googling and hacking, I come up with the following:



def str_base(seq, n, base=10):

 results = []

 while n:

     n, r = divmod(n, base)

     results.append(seq[r])

 results.reverse()

 return ''.join(results)



def idgenerator_fast(length, prefix=None, seq=None):

    if prefix:

        length = length - len(prefix)

    seq = seq or (string.digits + string.ascii_lowercase)

    N = len(seq) # a magic number

    minvalue = int(seq[0]*(length-1), N)

    maxvalue = int(seq[-1]*length, N)

    while 1:

        n = random.randint(minvalue, maxvalue)

        value = str_base(seq, n, N)

        if len(value) < length:

            pad = seq[0]*(length-len(value))

            value = pad + value

        if prefix:

            value = prefix + value

        yield value

I didn't profile this code, but IMO it should be much more efficient. But the problem is that it is twice as big and more difficult to comprehend - due to this base36 conversion trick.
And if you take into account that this function is called by the db-access code than you realize than it's costs are negligible compared to interprocess db calls.

Therefore I decided to stick with the simpler version. Nevertheless, it was a nice coding exercise, something like a mini-Kata.

sofware as a service

2004-10-20T13:57:00.000+03:00

disclaimer: I'm nor a business wise nor I pretend to be one, so the following thoughts may be just bogus or make you laugh. OK with me. But if you laugh, please spare a minute and enlighten me as well.

I regret I couldn’t attend Web 2.0 conference, as it turned out to be quite fruitful for ideas, both for business approaches and technical questions. I did listen to some of the talks given on the conference, thanks to ITconversations and The Web 2.0 weblog sites.

One of them was a Gillmor Gang from October 8, 2004 on which Kim Polese (CEO), presented her new company, SpikeSource. Their business model assumes building a "solid product offering" on top of open source components. For their launch, they're planning to offer a product stack that would include MySQL, Apache, PHP and JBoss. In the future, they would create new offering, driven by the customer's demands. And they have been developing a "deep computer science" code that would allow (as I understood) to build and maintain these product stacks in a (semi-)automatic fashion. They are going to earn money from the support, training and other service-type activities, contributing the code back to the open source community.

While the idea looks interesting and the talk is quite entertaining I wonder how good this approach would scale in terms of revenue. One of the nice things with software is it's "softness" - once you built a product it costs nothing to duplicate (and sell) any number of copies. With any service, costs to run it usually proportional to it's size. Plus, scaling it depends on people - which mean you need right stuff, with appropriate training, experience, etc, etc. So, basically, building a software product is (potentially) much more profitable than selling a service and it's much easier to sell.

The SpikeSource (again, IIUC) has two assumptions to address this issue. Firstly, in the brave new world that's coming software components (at least, at the infrastructure level) are a commodity and hence it's very hard to squeeze a revenue in this market niche. IOW, could you build and sell a web server? Probably only a very specialized one, because it hard to compete with Apache. Secondly, there would be a custom software in their offering, as an value-add for basic OSS components. So, the success would depend on whether they could add enough value to the commodities to attract customers.

P.S.: I had read a couple of months ago about a similar initiative to provide a solid OSS-based platform for enterprises on top of Debian but just can't find a link. Pity.

P.P.S: About a year ago, I was dreaming about starting a similar company here in my country (Ukraine) but not having had enough experience, money and courage I put it aside. So I'm a bit envious (again). ;-)

update: google search for "Lightweight Business Models" turns up quite a few interesting links.
update: fascinating post from Jason Fried about "human scaling" and some intrigue details about the Basecamp product.

Google Desktop

2004-10-15T14:42:00.000+03:00

In recent weeks/months Google literally flooded us with new products and services. Yesterday they released a Google Desktop -- windows application that indexes your hard drive and provides an integrated search: both for web and local files simultaneously.

I don't intend to describe it in details here or post screenshots how it looks on my computer. Just envy to those guys that happen to be in the right place to work for Google and have an itch to build something remarkable by myself.

update: Bloglines's Most Popular Links puts google desktop on the top, with +641 increase from yesterday! I guess it's more than all other delta combined.

python coverage tools

2004-10-15T13:45:00.000+03:00

Some time ago, browsing source code for SQLObject, I found a decent module for code coverage analisys for python. It was hooked to the unittests' runner and in effect, provided you with a statistics on how much of the codebase were actually covered by these tests.

I decided to try out this idea on my project's test code and results were rather impressive. One of the problem with dynamic languages like Python is that you have to run a particular line of code to be sure that it doesn't contain some syntax error or a typo. Yes, the interpreter is able to detect some classes of errors upon the complication but far fewer than, say, Java compiler. Code coverage, while it does not guarantee anything about the code's behaviour (whether it's correct or not) at least tells you what was and what wasn’t executed. And for Python that's of double importance.

Now I'm hooked to the idea and pondering about the option to start and keep a coverage database. This should be even more helpful, as I could track the metric over time.

PS: I've also found an Apycot package from Logilab, which contains a whole suite of testing tools, including coverage, lint and more. Unfortunately, it's not very well decoupled from the logilab's internal development process and hence hard to configure. And it's not Windows-aware (where I happen to develop now).

Apart from the rather old (2001) coverage.py and logilab's project I didn't find a lot of other, metric-oriented tools for python. Any suggestions on what I have missed are welcome.

software is my business

2004-10-11T14:31:00.000+03:00

Today's been a plentiful day. I've added several high-quality blogs about software business:

Steve Pavlina's blog
a software startup by Lachlan Gemmell
going from software developer to salesman

The linkblogs are dead, long live del.icio.us

2004-10-06T17:14:00.000+03:00

"Linkblog" is a blog which just links to entries in other blogs/sites.

With del.icio.us I can mix-and-match available links to create just a kind of linkblog I want. Or even a bunch of them.

The procedure is very simple:
1. Register with del.icio.us.
2. Browse available links select ones you want to track and subscribe to them.
3. Grab aggregated feed from your inbox.

Voila!

Open source platform for rich web application

2004-10-06T09:18:00.000+03:00

Laszlo announced that they are open sourcing their platform for developing rich internet applications (RIA).

More info here: http://www.openlaszlo.org/

Looks very promising, at lest for enterprise world. Will see how this will take off.

update: Laszlo uses XML-based language to specify user interfaces and it is object-oriented. Result looks rather ... er ... ~~awkward~~, a bit unusual, but easy to get used to.

Gmail new features - have you seen them?

2004-10-05T11:45:00.000+03:00

When I started Gmail today I noticed some changes in the interface: the "Contacts" link is move to the left and now opens without a popup and I found "forwarding" option in the settings menu, Plus, there is prominent link "New features!" at the top right corner.

That's nothing new here, some UI changes I notice regularly. The interesting part of the story is that my co-worker, sitting next table and also using Gmail does not have this features but see the old interface.

This is interesting, indeed. A reasonable explanation I may be that may be some Gmail users are switched to the new version to gather feedback. Or, another one, that Gmail's users that are actually report bugs got new versions first/exclusively while the rest using more-or-less stable beta.

So, my question is - have you seen these new features?

update: others notice this too. See, for example, comments on this post about Gmail's new features.

Hire me

2004-10-01T14:21:00.000+03:00

Hi there!

I'm willing to provide software development and consulting services both for short-term and long-term projects. The price tag typically lies within 15-30 USD/hour, but this is all negotiable, depending on the kind of work.

If you managed to reach this page, hopefully you already have an idea how I could be helpful for your particular situation. Detailed resume and references are available upon request.

You could contact me via e-mail or phone: +38 050 1660266.

code smell: tests/production LOC ratio

2004-09-27T16:16:00.000+03:00

disclaimer: this reasoning assumes the project is striving for 100% production code coverage by unit tests.

As noted elsewhere, optimal lines of code (LOC) ratio for production code vs unit tests is about 1:1. I wonder, does significant deviation on either part indicate a code smell?

First, consider the case when the ratio is about 5:1. This could mean two things:

Code coverage is poor (a good guess -- about 20-30%). No good, obviously.
If code coverage is in 80-95%% range than we most probably have some flavour of acceptance (functional) tests not a canonical unit tests. Which is no good either.

Second scenario - when the ratio is 1:3 or even 1:5. Though less common, I still got into this situation from time to time. From my experience, this is usually a sign of a code smell:

Elaborate setup code and/or a lot of stubs may indicate not well factored class hierarchy.
Lengthy test cases for particular production methods can indicate a heavyweight class waiting for the Extract class refactoring to be applied.

Usually this ratio for my own code varies between 2:1 and 1:2. If it grows more, I tend to pay attention to either tests or the code itself. My experience here is mostly Python-based, in more verbose language like Java these figures may be a bit more vague.

PS: May be a "design smell" is a more appropriate term here?

update: My post's arguments were completed thrashed at c2.com. I'm still trying to figure out how much I (dis-)agree with them.

Another argument for Google browser

2004-09-25T13:15:00.000+03:00

To support community buzz about Google making a browser, they have launched a version of the frontpage specifically for Mozilla browser. Things are getting interesting.

More about inheritance vs. composition

2004-09-17T18:51:00.000+03:00

I was reading excellent Effective Java by Joshua Bloch, on a topic about typical pitfalls when implementing Object.equals method. One of the least obvious is related to extending equals' implementation. To quote an author:

There is simply no way to extend an instantiable class and add an aspect while preserving the equals contract.

The author ended up with advice to use composition instead of inheritance with for such cases. I mention this story now because my reflections on unit tests for class hierarchies led me to the same conclusion.

Actually, I have in mind one specific case of class hierarchy: heavy-weight base and a bunch of subclasses that override/extend a particular aspect of it. An example may be a Finder superclass to define basic code to search the database and specific subclasses for specific domain objects, like Person.

The problem lies in unit tests: if you write them both for base class and for each of subclasses you'll find a sheer amount of duplication in you test code. Moreover, it could be very difficult to eliminate because the code would be a bit different for each of the subclasses' test case. A possible way out could be to replace inheritance with composition. There are two choices:

Aggregate former base class into attribute. This is usually a relatively simple and this allows you to "stub" base class for testing thus eliminating a duplication in test code. If the base class' interface was much richer than that of subclasses you may end up with a lot of delegate methods, though.
Pass an instance of a former subclass to the base class in order to configure it with specialized data and code. This would also mean that all former subclasses must implement the same protocol (interface) that would be used by the base class. That's could be harder or even just not feasible, though - the hierarchy was created in the first place because you saw no other way to factor out the similarities. Additionaly, if you succeed, this refactoring could make this already heavy base class simply unmanageable.

Nice example of how TDD actually drives a design, isn't it?

update: renamed post title and improved reasoning for the two choices.

FireFox 1.0 PR impressions

2004-09-15T13:23:00.000+03:00

First impressions on pre-1.0 release, in the order of their importantance to me:

Major flaw. Popup blocker is either broken or just fascist. It prevents windows from open when I click a link, requiring me to add each site I want to use to whitelist. That's just stupid. Moreover, even after I added gmail.google.com to that list I still can't open a "Contacts" pop-up. Sorry, it was my fault. The "Contacts" window now is working fine, so I guess this should have been a GMail bug, not FireFox.
Bug fix. Gmail interface for contacts management is now working (it was completely broken before). There is a possibility though, that this is a fix by Gmail team.
Bug fix. Individual feeds in bloglines can be opened (at last!).
Compatibility problem. Some of the installed extensions I happen to use were disabled as non-compatible with Firefox 0.10 (including ieview, livehttpheaders and text/plain). Hopefully their respective authors will update them soon so that I could continue to use them.
Minor irritation. The Find dialog is now appears as a toolbar at the bottom of the window. The problem is that there is no keyboard shortcut to close it. Admittedly, it's not really irritating because it's not clutter a browser window now. I changed my mind and I don't perceive this as an irritation anymore. Actually, it's a nice change because it allowed me to uncheck "Find as you type" option to allow Gmail's keyboard shortcuts and still have a handy search box. Nice tweak.
Nice tweak. Some (or may be most) windows that are shown by browser itself are now "inlined" at the top of the rendered page with a yellow substrate. I noticed this for popup blocker window, missing plugin notification and new find dialog.
Nice tweak. The address bar for secure sites is now colored in yellow which make them hard to overlook (compared to a small lock at the corner we have previously)
New feature. Live Bookmark feature is interesting, but I prefer bloglines service to read RSS channels. There is a new Feed Your Reader extension which is similar to Live Bookmark but allows for external RSS readers. Version 0.7 (due this week) is going to support bloglines as well.

My blog's PageRank

2004-09-15T13:09:00.000+03:00

Interesting, but my blog's site do not even appear on the first page of results when I search for max ischenko. Search for max ischenko blog gives it third position which is not great either.

Have to do something about that. Hey, microsoft.com guys, please put a link to my blog on your front page! Please do or I won't ever link back to your site, improving your PageRank.

update: not sure if someone at microsoft.com really linked to my site, but now both above mentioned searches yield profile page and index page at first two positions.

Spammers move on to blogs and forums

2004-09-15T11:58:00.000+03:00

Joel Spolsky writes:

If you think comment spam is not a big problem, you haven't moderated a discussion group lately... this is the number one priority for spammers these days, since email filters are starting to work pretty well and spamming a lot of discussion groups is perceived as a good way to trick Google into giving a site prominent placement.

I read about spam posting on one or two other sites but didn't think it was a serious problem until now. Actually, just never bothered to ponder over it. Of course, this is more of a problem for popular sites, like Joel's, with a high PageRank. I wonder, if any forum/blogging software already ships with filtering included or at least has a plug-in for this.

update: Comment Spam Manifesto

Testing web-based applications with FIT

2004-09-14T10:24:00.000+03:00

I have had a need to automate testing of our web-based applications because manual testing could never be completed in a reasonable timeframe.

For the last two weeks, I've been writing a fixture for FIT specifically to test web-based applications. The idea is to use very sophisticated HtmlUnit library for actual HTTP interactions and dealing with HTML (and Javascript) and to use FIT for specifing, running and reporting on test cases.

update: project is now hosted on public subversion repository. See http://opensvn.csie.org/webfixture.

For those who have missed it, here is a short description of the FIT framework. FIT is a web-based tool for specifying and executing acceptance tests for all kinds of application, with a particular accent on simplicity of the specs (they could be understood and may be even written by a non-programmer, i.e. customer). The central role in the FIT framework is devoted to a fixture. A fixture implements a vocabulary, which is used by user to describe things he wants FIT to perform. For example, to press a button or to follow a link. Or to evaluate 2+2 and to check that result is 4. Thus, user writes these checks using vocabulary and programmer implements this vocabulary with a fixture in a meaningful way. If you're intrigued, Michael Feathers has written a more elaborate intro in his Pitching a Fit post.

Originally, I tried to write a custom ActionFixture for each web page I had to handle but soon discovered that it was a bad idea. Firstly, it was too much work -- to write a Java class for each page I need to test. Secondly, and more important, each fixture would use different vocabulary for each page which means that test writer had to be familiar with an underlying Java class. Therefore I decided it would be better to write a single class (a fixture) that could be used to exercise any web page. Luckily web languages' expressiveness is rather limited so the resulting vocabulary to describe test cases should be manageable.

Shortly after I started to work on my WebFixture, I discovered a (semi?)-official HtmlFixture, created for the same purpose and which even uses the same HtmlUnit library! Nevertheless, I decided to continue to develop my own fixture, for these reasons:

To get experience with Java platform.
One of my goals with this mini-project has been to try out a Java platform. I've been on most C++, Python, Tcl camps these years and my Java programs never went beyond the "Hello, world!" exercise. This project seems like a perfect opportunity to learn a new platform.
To control fixture's development.
I want a control over this fixture to add things I want in a way I want. While the HtmlFixture is mature, its source code is undocumented and poorly written (to me). This probably a funny thing to say from a novice Java developer, but still, the source code seems like a mess to me and don't want to dive into it.
To design it differently.
If I decided to base my work on HtmlFixture there would be still no guarantee that my changes would ever go into original codebase. Why? Because I my approach is different and therefore there is a good chance that changes I propose won't "fit" author's concepts. Don't get me wrong, HtmlFixture is a fine work, I just want to implement the same concept a bit differently.

So how my fixture does look like?

The important design goal to me is to make fixture's vocabulary as simple and as high-level as possible. It should be similar in terms to those used by a web browser metaphor. It's better to sacrifice some underlying versatility of HtmlUnit to gain in simplicity and consistency for a user. It's better to overload verbs (where appropriate) than to invent new ones.

For example, I use verb click to follow links and submit buttons and use verb enter to enter data to input boxes, text areas and select boxes. This way, the same verb can performs technically different things as long as it means one thing to the user.

There are some other features I've added so far: site configuration, Java delegates, erroneous page detection.

The site command allows you to set some site options (or preferences) which include site root, timing information, error page patterns and more. These preferences are kept in a static storage so that you could set them in SetUp page and use on test pages.

The error page patterns used to detect erroneous web pages - normal HTML pages which are returned with HTTP response code 200 but which nevertheless are a sign of some internal error in the web application. For example, some application servers (like WebKit) show crash reports in this way and fixture could be configured to detect these automatically.

Java delegates are extension point to be able to hook custom checks that are beyond normal web processing, like verifying the database state. A delegate is passed a current HtmlElement and should return a boolean result - whether a test has passed or not.

And finally, some closing words on my first impressions on Java platform.

The IDE for a statically-typed language could do amazing things and Eclipse actually does them! Moreover, it is very capable in other aspects making handling of Java verbosity almost imperceptible. With advanced browsing support and context assist I managed to grasp an essence of both FIT and HtmlUnit library very quickly. And QuickFix assist helps greatly to correct my naive syntax errors. Overall, I must say, the development process goes much more smoothly and is much more fun that I have expected, thanks to IDE.

Mozilla FireFox extensions

2004-09-13T11:14:00.000+03:00

Flexbeta has posted a nice guide to Firefox extensions, including Target Alert for links, Adblock with a nice configuration file, Sage RSS reader and more. Definitely worth a read for anyone who uses FireFox as a primary browser.

Btw, Firefox 1.0 Preview Release is going to be release tomorrow.

Flickr services

2004-09-10T17:12:00.000+03:00

The Flickr.com is yet another photo service site. But the distinguishing feature (at least for me) is that it provides an API with array of supported access methods (REST, SOAP, XML-RPC) to get a full programming control of your account's data and settings. Quite impressive, really. I can't agree more with Jeremy Zawodny: [Flickr is a] next generation web services.

If we add to this Google, Blogger, A9 and others it surely looks like a trend. And you (myself in the first place) don't want to miss the idea while launching your next web-based service. Sure, this kind of access is helpful only for a fraction of web-based services out there, but it could gain them a lot.

Blogger

2004-09-01T16:49:00.000+03:00

While the Blogger web application is one of the most pleasing I have worked with, it seems a bit shaky. I have already seen several errors in the past two days, including NullPointerException from a Java application, a page saying "internal error has occured" and a misleading error message which said about an error deleting a comment while it was actually deleted.

Anyway, overall experience has been excellent so far. The only major thing I miss is ability to post into different categories.

The Python paradox and PIL

2004-08-31T17:14:00.000+03:00

In the recent remarkable talk and later clarification Paul Graham said:

Python programmers are smart. It's a lot of work to learn a new programming language. And people don't learn Python because it will get them a job; they learn it because they genuinely like to program and aren't satisfied with the languages they already know.

Today I had to perform a simple image transformation task in Python and decided to use well-known PIL library. While working with PIL I had to recall this Python Paradox. The reason?

The PIL's API is weird. Not only it contains stupid bugs but it is awkward to use. I can't judge the underlying C implementation, it may be just excellent, but the python wrapping code is far from perfect. It is simply mediocre at best. And who said the Python programmers are smart here? :-)

Just to support this rant, here are some issues I have discovered while trying to resize an image (a pretty typical use-case, isn't it?):

to create an image you'll need a file-like object, plain string won't work. Actually, this is very common in python libraries and can be seen quite often in standard lib as well. But this is still irritating. Why the library can't provide both methods instead of forcing the user to mess with StringIO buffer? From the implementation point of view one method could be easily done as a thin wrapper over the other. It is easy and facilitates unit testing. The only excuse could be the laziness of the library author.
you have to save the image just to get image data. To me, this feels awkward - why don't provide a method to get (or compute, who cares?) image data directly? The issue is emphasized by the save() method itself.
the save() method is just weird. Firstly, as you may have guessed, it requires a file-like object. Secondly, it requires you to specify a format in which you want to save it. It would be OK if this parameter is optional but it is mandatory! In my case I simply don't know original format nor I could care less. I just want to resize an image and get the result. As it is, I had to determine image type first, despite the fact that resizing routine accepts many different kind of image files. And the final stroke of untidiness -- format is specified as string and never explicitly checked for correctness. So if you pass JPG instead of JPEG you'd get a KeyError with very informative "JPG" explanation text.

On a positive side, the functionality is excellent - the resizing routine performed fine (after I specified more sophisticated resampling filter instead of NEAREST).

Blogger + Gmail

2004-08-30T10:25:00.000+03:00

In less than an hour after I opened a blogger account I received 5 invites in my Gmail account. A coincidence?

P.S.: Yes, I know that first gmail accounts were given to blogger people (among others)
P.P.S.: Anyone still need a Gmail invite?

Managing code repositories on two machines

2004-08-30T10:02:00.000+03:00

I have two machines on which I work and have to move complete source code repository back and forth to replicate changes. This is quite cumbersome as these repositories are rather large (hundreds of megabytes).

I have been pondering for a way to replicate only diffs but still haven't found a solution. My two candidates were (are) subversion (as I use it to host projects) and unison (file system synchronization utility). The problem is that I can't connect these machines via network and thus simple "svn up" is not possible.

I think there should be a subversion way to "export" changes and then "apply" it to another WC. But even if this is possible I need to replicate the subversion repository itself.

Internet-living Mastermind, are you hearing me?

update: Davin suggested Windows built-in 'briefcase' tool, that's what I'm going to try.