20091211: Chti'Jug on HTML 5

These are some notes from  yesterday's session of the Chti'Jug on HTML5 features. The session was in english and so are my notes. Speaker was Peter Lubbers from Kaazing, a California based company that develops software for real-time communication and provides training on it. By the way, Peter was in France to give training for Skill Matters through Zenika

Here is a list of some pointers on HTML 5 and Web sockets:

Current state

Background History

Standard Ajax applications shortcomings

Enters RIA & Ajax/Comet/Whatever: gives the user the illusion of real-time (fake time). Various techniques are used, most common being:  

Things get worse when we look at the architecture needed to support state-of-the-art Ajax applications communicating with "RT" data providers, this is Headache 2.0. Standard practice is:

HTTP has been designed for document-centric systems (hence the success of REST) transfering infrequently (on a human scale) medium to large chunk of data. Overhead of header is negligible in this case and the Web's infrastructure (caches, content provider, replications, If-Modified-Since, Encoding and chunks ...) allows for efficient transfer of data. But it is not fit for RT web applications.

As a an example, Peter evaluates the bandwidth requirement for a simple stock ticker application. 10000 clients polling every 60s needs 1.1 Mbps.

Web Sockets

Web Socket specification, currently in Draft mode at IETF and part of HTML 5, provides client code running in browser full-duplex communication to servers over vanilla TCP. Protocol principles are simple:

On the client side, WebSocket object provides a simple API:

WebSockets are interesting because they allow clients to talk  to remote servers using their native protocol (eg. AMPQ, STOMP, SMTP, HTTP :-), WTFP,...), instead of relying on remote adapters in the application server.  

The only browser to provide native support of WebSockets is Chromium. Support on the server side is growing (more easy to implement and change), Jetty for example provides support for WS.

Server-side events

Next part of the talk was about server-side events. Idea is to provide on the client-side a subscription API for events that will be sent by the server. Client provides listeners that react on events send by the server which are arbitrary strings. This feature is currently supported in Opera9 and latest Firefox snapshots.

XHR Level 2

XmlHttpRequest is the ubiquitous API behind Ajax. The level 2 XHR adds 2 features:

Cross-domain messaging is another extension that provides communication between pages (ie. between different iframes in the same page, or different pages in the same session ?). Security model is statically encoded by both ends of the communication channel which may or may not accept messages according to origin policy. Given the current intricacy of RPC in client-side applications, this is obviously a huge improvement!

The End

Talk ends with a presentation of Kaazing's offer based on these HTML5 extensions and some code examples of Web Socket development for a client speaking STOMP (Simple Text-Oriented Messaging Protocol), followed by the traditional couple questions, a short contest to win various prizes offered by the Chti'JUG organization and of course some beer.  

All in all, this session of the Chti'JUG was quite informative. I personally was not aware of all those developments on the Web infrastructure. Obviously, these are big improvements for developing very reactive and scalable web applications over what is currently state-of-the-art. It will surely remove a lot of fat from server-side and middleware code by allowing transparent relying of arbitrary TCP connections directly from client to server: no more need to implement complex mapping logic or protocol transformation to encapsulate streams of raw data into a stateless text-only request-driven protocol like HTTP.

I do not fully understand the consequences of all theses novelties on 2 aspects: security and maintenance. I am mostly interested with the second of those aspects of software development and given the current state of practice for developing and testing javascript code, I am rather concerned that this will lead to a rapid growth of the already large untested, ugly and bloated javascript codebase. Or maybe as some other blogger put it, this will push more people to embrace alternatives like GWT, Silverlight, Flex, whatever with strong IDE & library support.

20091211: Back from XP Day Benelux 2009

It's been more than a (busy) week since I came back from XP Day Benelux 2009 conference which took place the 23th and 24th of November, in Mechelen. I did not present any session this time, so was free to enjoy whatever session would suits my current mood (and needs!).

General feeling

What to keep

What to improve

Day one

It took us nearly 3 hours to drive the 120 km separating Lille from Mechelen. Hence, although we had some margin available starting at 7:30, we arrived very late on Monday morning and missed the introductory session where presenters introduce their session to the attendees.

Feature flow: From Chaos to Teamwork

This session was an experience report from a team who transitioned from plain Scrum to Lean/XP practices. The author started with a team which was implementing Scrum on a rewriting project (from MS to WOA) for a sensitive desktop application targeted at pharmacists. The team had troubles: low throughput, bad quality, yet people were working hard, with long overnights and much pressure.

First actions were targeted at the team's working environment and process, to reduce & regulate pressure on the team, which otherwise lead to  dysfunctional behavior, by installing a strong, visible and constraining process:

The author then talked about motivation: themes are used to group stories together, which are then planned and worked on together, giving a common goal to achieve. Pairing seems to be used as a way to reduce the bus-factor of the team, but not systematically (?).

The architecture was found to be counterproductive, getting in the path of implementing/adding features: It was needed to touch more than 20 files to implement a single feature, which is a sure sign of troubles. There seems to be a tension between conceptual integrity and flexibility which is not easily solved. The solution was to work on features horizontally (traversing all the layers at each increment), in the spirit of dimensional planning.

Testing was another bottleneck. By reversing the test & code phase (ie. automated test comes before coding, which is followed by smaller manual testing), they managed to make the test fix the cadence of the flow which in the end yielded better through put: Keep feedback very short to ensure knowledge flow through all the phases

Retrospectives were done just-in-time, when deviations from the expected process and figures were found.

Some key retrospective insights:

Quadrants of Effectiveness Game

This a session about a game designed by Gino Marckx on the GTD and Quadrants of Effectiveness time-management method by David Allen. Each player has 5 cards reprensenting actions in hand. Each card can have a urgency level and provides some points (its importance). The urgency is reflected by progress of each player on a track from 1 to 24, players can go back and forth on this track depending on the urgent cards they play. Tasks can be delegated to other players.  

The game is designed such that players' strategies are bend towards applying techniques advocated by GTD method and Eisenhower's time management rules:

The game is fun, well-designed and polished, and runs smoothly without too much complexity. All rules can be printed on the board which is quite a small set. I had mixed feelings about its underpinnings however, as it entails players to develop individualistic strategies without taking into account the aim of the group and its context.

Revving Up: Performance Engineering

This session was lead by Johan Peeters and is an introduction, using some example, to performance engineering of multi-tiered networked applications. Goal is to optimize time or more precisely increase throughput of some system.

The system is modelled as a multi-station closed queueing network and is expected to be in steady state. We start with some simple maths on QN theory to understand relationship with the various metrics we can extract from the system. First model is simple: There is no shared resource between work stations, hence no contention.

If Di is the service time for a station in the system, then the lower bound`````` for processing time when there is one request is \sum_i D_i and the upper bound for throughput is 1 / \sum_i D_i. When the system can process N requests in parallel (can hold N requests). Then the upper bound for throughput is N / \sum_i D_i

````````We then turn to model with contention on resources: Each station competes with other stations for a resource. Then the maximum throughput is simply 1/ max(D_i). In other words, throughput is bounded by the bottleneck in the system. According to Little's law, we have that L = \lambda W with \lambda being the average arrival rate, L the number of simultaneous requests in the system and W the average service time for one request. Substituting for values in the previous equation, we get that response time in system handling N requests with contention is bounded by N max(D_i)

The interesting conclusion of this analysis is: Always optimize the bottleneck before anything else!

After this light mathematical introduction, presentation moved to some practical aspects of performance engineering:

Some example system:

Some questions:

Day two

Evolutionary Design

A game about, well, evoluationary design.

Game is fun and highlights the necessity of keeping your design evolutive enough. There is maybe not enough constraint on what team can achieve, and how they can achieve it, so it is hard to optimize, choose between different options. see http://blogistherules.co.uk

Architecture & Agile

Subtitled: Mission Impossible.

A presentation of how architecture and agile can mingle to help reach better software quality, and what may be the role of an architect in agile teams. Not very conclusive, with lot of generalities about what is agile and what is architecture. Might need to be more practical and hands-on session, possibly with some case study or game like feature.  

Scaling Agility using Domain Inheritance

Thought-provoking and lively session by Koen Van Exem. I met Koen for the first time at Agile Open France 2008, in Itterswiller, and we had a passionnate discussion on programming languages and types. I really liked his Dimensional Planning method when I had the chance to get introduced to it and since then I reuse this concept very often to challenge assumptions about perfection and requirements we have developping software.

The question this technique addresses is simple to state: How to mix different domain concepts into shared elements? Canonical sample is the Person object: How to mix the Person object from a Banking POV with the Person object from a Sales POV? This is related to the overall goal of Domain Driven Design: Through Ubiquitous Language development and use, ensure that the system embodies the concepts of the business in a simple, consistent and unambiguous way (in particular, we don't want to maintain two different versions of a Person class...).

Koen goes through OO history to detail previous solutions and why they failed to offer this:

There is a presentation of the Allors framework which embodies theses concepts, plus adds some Naked Objects-like behavior to increase throughput for developing applications with minimal UI.

Functional Documentation & Testing

A presentation of Fitnesse, using some real-world use by expert testers and a sample system to model. Not much new but it was interesting to hear positive stories about putting Fitnesse to use on large and complex systems.

20091208: Catamorphisms

In a recent blog post, Tony Morris proposed to its readers an exercise in FP with catamorphisms in Scala. Roughly speaking, a catamorphism is the functional representation of patterns of constructor for some existentially quantified datatype.

For example, given the List type constructed with Nil :: List a and Cons :: a -> List a -> List a, a catamorphism for lists is a function whose signature is cata :: forall x . x -> a -> x -> x -> x. We can recognize the constructors in the arguments of cata: a nullary constructor (x) and a binary constructor taking an a, another x and yielding and x. So x here stands for some type (hence the notion of existential quantification) that can be used in that "ways".

Just like a datatype with recursive definition (eg. List) is actually stated as a recursive equation whose solution is a fixpoint (and the type in question), a catamorphism embodies in a single function this same datatype through the functions that can take it as arguments.

20091122: Polyglot Maven

Using content-type from HTTP protocol to define POMs format, and Content-accept to know how client can handle various formats. This implies that even for local repository some form of indirect access through HTTP protocol should be setup.

http://www.sonatype.com/people/2009/11/maven-3x-paving-the-desire-lines-part-two/

20091021: If any one of you is without sin...

When they kept on questioning him, he straightened up and said to them, "If any one of you is without sin, let him be the first to throw a stone at her." John, 8:1-11

As someone who spend a good deal of its time training people on test-driven development, shortening feedback loops, integrate continuously and in general way keep a wary eye on the code which is written, I might feel a bit ashamed to admit that I sinned today: I spent several hours refactoring code without executing tests, then when I tried to integrate my changes, I spent another couple of hours fixing failing tests and debugging - you know, that painful crawl through excruciatingly detailed and overwhelming stacks of code - before I could release the feature I was developing. And I must even confess I am not very confident about it...

But I find it more useful to reflect here on this small experience. Do not expect breaking news, this is only a recall of some old and well-known - at least in agile circles - practices and advices.  

The program I was working on is a small utility tool for testing (triple shame!) native C/C++ code, written in Java but relying on low-level native commands to do its job. It has about 3.3kloc of production code and 2kLoc of test code, a ratio of roughly 40% tests which is consistent with what other people found. The project produces a single executable jar from a single maven modul, and its architecture is rather straightforward:

This code has 85 unit tests and is quite clean for today's standards: lot of small functions and classes, no method more than 15 lines long, no class more than a couple of hundred lines long, meaningful test names...

I recently added a new feature which is actually a subset of the main feature, with some small variations, and made it work on Windows quite easily. It worked but I was not very happy with the way I implemented it: Its interface was a specific option on the command-line of the bigger tool, and it needed some properties file to be defined when what was really wanted was just processing the set of input files from the command-line, which had a different meaning in the first feature.  

I then switched to linux to make it work on this OS, and this where the trouble began. In hybris, I just started to do 2 things at the same time:

  1. implement the feature on linux
  2. and refactor code to have a cleaner separation of the 2 usages.

I had a hard time finding an angle of attack for doing 1 safely, and when I thought I found it, it started sprouting new objects from existing code, then new interfaces to stay consistent across OS, then glue code, then... Of course, refactoring with my IDE (Eclipse) was easy: extracting methods, adding parameters, lifting methods to super class then encapsulate in different class. Yet I lacked a clear roadmap of where I were heading, and above all I did not ran all the tests before the refactoring was considered over. And then I had 6 failing test cases which took me 2 hours to fix.

It's interesting to look at the pattern of commits I made in Git for this project (in reverse order of commit time):

null

Largest commit by far represents the result of this refactoring effort. We can see that more than 1500 LOCs changed, which is about 30% of the total codebase! And given there 81 files in the project, this also means that more than 50% of the files were touched by that commit while at the same time I barely added a handful of single unit tests.  

I can only conclude that the probability I introduced a (possibly severe) bug in this process is extremely high.  

How could it be so? This is a very common experience, and talking with a colleague at work, he confirmed that this is something he often faced, with the added pain that he had no unit tests and no sophisticated IDE to track the impact of its changes! Here are a couple of explanations, and some counter-measures I could thinkg of.  

  1. Doing 2 things at the same time: chasing 2 preys at once often means you catch neither.
  2. No clear plan of what I was doing and where I was heading. I started what I thought was "refactoring"  when it actually was mere hacking. Eclipse is good at refactoring but there are limits and some refactorings that could be safe are not done automatically, with the risk of introducing subtle errors.
  3. I did not run the tests often enough. I had long refactoring tunnels where my code wouldn't even compile, and when it compiled, I was sure tests would not pass so did not ran them.
  4. I did not commit changes into VCS enough, which prevented me to get back to stable state when things went awry,
  5. I changed too many different things at once. This is a sure sign that something might be wrong in my design that prevents small increment of changes,
  6. I worked alone. Having a single pair of eyes and a single brain looking at complex code and working on complex refactoring is not enough.  

Here are some standard advices from practice and theory of XP that I wish I had followed:

20090923: Commentaires d'Emmanuel Bernard

Quelques remarques d'Emmanuel Bernard sur mon commentaire de la soirée Chti'JUG

Salut, merci pour le resume de la soiree sur ton blog. Je voulais commenter mais tu n'as pas l'air d'ouvrir ton blog aux commentaires.

Pas assez de code / trop long

En fait, en general, je ne couvre qu'un seul sujet et pas deux et demi. Soit Bean Validation soit Hibernate Search. Dans cette configuration, je fais une demo de 15 minutes sur Bean Validation dans JBoss 5.x et j'ajoute une second demo sur Hibernate Search. Les Chti voulaient couvrir les deux sujets et je me suis dit que ca valait le coup d'essayer. Je ne le referai pas les prochaines fois :)

Bean Validation

Une annotation sur une annotation, on appelle ca meta annotation.

On n'utilise pas APT qui etait une API proprietaire de Sun mais AnnotationProcessor qui est la version standardisee d'APT.

Bean Validation la spec, s'integre avec JSF 2 et JPA 2, pas seulement Hibernate Validator

DTO: la réponse est de ne pas utiliser de DTO mais si on doit, il y a moyen d'utiliser l'API d'acces aux metadonnes pour effectuer le lien. Je ne l'ai pas dit Lundi soir, mais quand on fait des DTO et si on utilise un framework qui peuple les DTOs de maniere automatique, le lien peut etre construit naturellement dans le framework.

Hibernate Search

L'API de mapping programmatique (au lieu des annotations) est prevue pour la prochaine version. http://anonsvn.jboss.org/repos/hibernate/search/trunk/src/test/java/org/hibernate/search/test/configuration/ProgrammaticMappingTest.java Regarde la methode NotUseddefineMapping en bas pour avoir une idee de l'API.

20090922: Chti'JUG Hibernate Search & JSR303

Ma première soirée au Chti'JUG, dans les locaux de Norsys en plus! Je ne pouvais manquer cela. Il y avait du monde, probablement une petite centaine de personnes, la pompe à bière était prête, le soleil s'était mis de la partie ce qui n'était pas nécessairement une bonne idée compte-tenu de la disposition des locaux: la présentation se déroulait dans l'atrium, sous une verrière.

La présentation a commencé avec une demi-heure de retard: 19h au lieu des 18h30 prévues, ce qui est un peu regrettable, en particulier pour ceux qui ont la chance d'avoir une vie de famille et de travailler sur Paris. Denis Cassoret a fait une introduction rapide, mentionnant au passage l'Agile Tour à Lille puis a laissé la parole à Emmanuel Bernard, de JBoss, développeur d'Hibernate, venu nous parler de deux sujets: la mise en place de règles déclaratives de validation de contenus des JavaBeans (aka. JSR303 pour les intimes) et l'utilisation de la recherche plein texte avec Hibernate Search.

Quelques commentaires sur la forme:

JSR-303

La JSR303 offre un moyen de déclarer des contraintes sur les attributs d'un graphe d'objets au moyen d'annotations. Ces annotations sont récupérables dynamiquement (API java.lang.annotations.) ou statiquement (javax.lang.) au moyen d'apt, l'outil de traitement des annotations fourni dans le JDK. La JSR prévoit que divers frameworks et outils peuvent utiliser ces annotations pour déclencher la validation des attributs correspondant à l'exécution et ce dans différentes couches du système. L'objectif est bien sûr de pouvoir définir un seul modèle-métier susceptible d'être réutiliser de bout en bout, au besoin par génération de code, intercepteurs, ponts et autres adaptateurs...

Les annotations représentant des contraintes peuvent être composées par héritage et surannotation (une annotation sur une annotation), et il aussi possible de définir ses propres validateurs sur des types généraux ou spécifiques. L'ensemble utilise les possibilité nouvelles (!!) de Java 5: types génériques, APT, réflexion étendue... Une possibilité semble-t'il intéressante consiste à pouvoir grouper les contraintes voire à définir des séquences de groupes, l'idée étant que certaines validations seront pertinentes dans un certain contexte seulement, par exemple une pré-commande n'aura pas les mêmes contraintes qu'une commande ferme.  

Parmi les autres caractéristiques de cette norme, on trouve pêle-mêle la généralisation de l'utilisation d'interfaces fluides (fluent interface dans la langue de Britney Spears), les possibilités de paramétrage (i18n) des messages d'erreurs, une API offrant des méta-données sur les annotations de contraintes à l'usage d'outils divers et variés. Le moteur proposé par Hibernate (l'implémentation de référence semble-t'il) intègre directement le framework dans les Java Server Faces 2 et Java Persistence API 2. Enfin, les validateurs peuvent être injectés en Java EE 6 comme  des resources (@Resource).

Quelques questions posées:

Commentaires

En fait, ce dont on aurait vraiment envie, c'est de typer les attributs et objets au moyen de types métiers, sans nécessairement supporter le surcoût lié à l'encapsulation de types primitifs. Par exemple, au lieu de: 


  @NotNull
  @Range(min="5", max="5")
  private String zipCode;
ou même previous

  @ZipCode
  private String zipCode;
on aimerait mieux: previous

  private ZipCode zipCode;

Hibernate search

La deuxième partie de la présentation est consacrée à Hibernate Search, c'est-à-dire à l'utilisation d'une recherche plein texte basée sur Lucene par dessus une couche Hibernate. Le principe est assez simple: on annote (une fois de plus) des attributs d'entités Hibernate afin de définir des modalités d'indexation de ces attributs. On a alors à sa disposition une API permettant de construire des requêtes basées sur des techniques de similarité de documents (ie. TF-IDF et co.) et utilisant ces index par l'intermédiaire de Lucene. LA valeur ajoutée d'Hibernate consiste à fournir une API orientée-métier, c'est-à-dire permettant de produire des objets métiers typés au lieu de documents génériques à la Lucene: on utilise la couche ORM pour reconstruire les objets à partir des données brutes de tables produites par le moteur d'indexation.

De très nombreuses possibilités sont offertes pour paramétrer d'une part le processus d'indexation et d'autre part le processus de recherche:

Sur le plan technique, on a aussi la possibilité de distribuer les index, de les cacher dans les divers niveaux de cache d'Hibernate, de les éclater (sharding). Une solution utilisant une mise à jour asynchrone de l'index par file JMS est même évoquée.

Conclusion

Une soirée sympathique et bien organisée, si l'on fait abstraction d'un horaire un peu fantaisiste :-) Je ne suis ni un spécialiste, ni un grand fan d'Hibernate ou plus généralement des ORM (en reste-t'il d'autre ?), mais le talent d'Emmanuel Bernard emporterait presque le morceau s'il n'y avait cette malencontreuse prolifération de symboles cryptiques induite par l'abus d'annotations, dans un langage  qui n'est déjà pas réputé pour être très lisible ni laconique. La lourdeur des annotations ne doit pas nous faire regretter la lourdeur encore plus pénible du XML, mais dans certains cas tel que celui de l'annotation de la classe entité par différents indexeurs où l'annotation prend 20 lignes, il y a de l'abus.

En fait, j'aimerais bien que les concepteurs d'API de ce type laissent le choix aux développeurs et proposent soit une API programmatique avec un système de types et de classes complet, éventuellement basé sur un MOP spécifique; soit des annotations. Un peu comme il est de bon ton quand on écrit une application d'offrir à la fois une interface graphique et une interface ligne de commande, quand bien même cette dernière serait-elle plus simple.

20090921: Back from Citcon Europe 2009 in Paris

This year's (2009) Citcon Europe conference was held in Paris, so I had no excuse not to participate. I had missed previous conferences in Brussels and Amsterdam, and was eagerly waiting for this issue to collect, share and produce ideas and practices in these two domains: Continuous integration and Testing

General comments

The sessions

Pairing with PJ

Mock Objects

This session was proposed and lead by Steve Freeman who has some credentials to talk about Mock Objects! He is by the way in the process of publishing a book with Nat Pryce about OO design and the use of mock objects and TDD. Session started w/ some sort of planning about issues people were interested in: horror stories or how to avoid them (and mocking !), usefulness of expectations, best practices, mocks vs. doubles, frameworks and languages, mocking pojos...

Steve Freeman introduced the subject by recalling what (and what not) are mock objects:

  1. These are related to a particular school of software development, one that goes bottom up, or that tries to construct the software by exploring objects, their interactions, until some boundary is reached which marks the end of the system,  
  2. key to understanding of mocks are the concepts of roles (a modality of usage of some object from the point of view of some other object) and responsibilities, all summed up in the concept of a contract. So one mock roles, not objects or classes or (worse!) APIs, frameworks, filesystems, networks...
  3. Mocks scale to all levels, as long as one respects those basic principles. They are a design tool, like TDD and unit tests generally are.

Then David Gageot showed us some live code from its current project which uses Mockito framework. There was some heated discussion over this code, one issue raised being the legibility of mock specifications which in this case was not good (long chain of invocation of methods and complex names). Another issue was the duplication of mock preparations: I suggested it might be factored following standard DRY principle, some other people disagreed on the premise that it was best to make the tests standalone, self-explicits without obscure initialization/setup procedures. Factoring mock setups quickly would lead to tangled dependencies and unnecessary initiliazation code for each tests (ie. one big setup with lot of mock preparations covering different roles/responsibilities of the mock). I would rather agree, except that in this case duplication is accompanied by complexity and syntactic noise which even more clouds the intention behind the mock (ie. the role it plays). By the way, it is a tribute to the courage of David to show its code to such an assembly !

Some discussion ensued about the use of mock to mock databases or other complex external systems related to data and communication, which seem to miss the whole point of mock objects. Of course, we are not talking about mocking database connections but some DAO/Manager behavior, maybe with real data found to be problematic in production. This is somewhat irrelevant with the real purpose of mock objects: Help designing systems by making interfaces and their contracts explicit.

We finished the session with heated debates about whether or not it was sensible to mock real classes. Steve Freeman had a very strong position against it, saying that mocking concrete classes obscured the real abstraction (hence interface) laying in the relationship mocks were testing. It might even induce a violation of SRP by usign same implementation in different roles. Several other people, including David, Eric and me argued that it could be sensible and useful to do that when the mocked class only had one role.

This often occurs when one is in the process of developing some system: You have two classes that emerged independently, then you want to start making them collaborate, so you test the relationship using mocking but there is no interface to mock, only concrete class. To me, this is a tribute to the fact that the Java language has coined interface as a keyword, and being so widespread today, its concepts and terminology pervades any debate on OO design and development.

Interface is a concept, not a technology, it can be implemented in various languages by a lot of construct. Yet I understand Steve's point: By enforcing mocking  interfaces, one can induce quick appearance of interfaces thus abstraction of roles and responsibilities which may lay hidden while staying embedded in concrete classes.  

Citcon wiki link

Promoting Acceptance Testing

First session in the afternoon was about acceptance testing and how to promote it, make business people use it and generalize adoption of the practice and tools within software development projects. Gojko Adzic led the session which started by enumerating 5 (which transformed to 11) reasons why acceptance testing fails:

  1. No collaboration between the various entities/people responsible of the software development: Business Analysts, testers, developers, Product Owners/Managers... Tests are written by the developers  
  2. Focusing on how to write the tests vs. what tests to write
  3. Producing tests that are not documenting correctly the specification they represent and the software they accept
  4. Expecting too much from acceptance testing: Acceptance tests are no regression suite
  5. Focusing on tools instead of people and their needs  
  6. Not considering the added value of acceptance tests (ie. more throughput and less corrective cycles) but only their <costs
  7. Not maintaining acceptance tests with care and love
  8. Not aligning every people involved on the same objectives
  9. No management support
  10. Underestimating the skill required to write and maintain acceptance tests
  11. Using Fitnesse during specification workshops

Some of those points were debated, but there was a general consensus over them. A few people gave success stories about introducing acceptance testing in their organizations, yet I would have liked more detailed accounts of the processes involved. Session was somewhat crowded and it was difficult to really get anywhere beyond high-level statements and objectives.  

Citcon Wiki link

CI, reporting & metrics

Last session I attended (I had to leave earlier so I missed the last session slot) was much less crowded. Some of the questions that started the debates were about which metrics were useful for starting agile projects? Was it useful to track bugs and how? What was the point of view of management on CI & metrics?

We started by enumerating some metrics people were collecting with CI: code coverage, code analysis tools output (ie. the usual suspects Findbugs, PMD, Checkstyle). There was some experience report about configuring and using the latter, Checkstyle is too noisy to be really useful, need to be configured and tuned.

There was a question about the use/usefulness of monitoring other things than the code, eg. process data like time between 2 good builds, pairing metrics, commit frequencies...

We then moved to the usage patterns for those collected metrics. It was generally agreed that continuous monitoring of code related metrics was useless: Interests quickly fade away after initial installment. Metrics need to be monitored at regular review points for example, they also need to be correlated with defects found to better understand their patterns of appearance. There was the usual but always needed caveat about using metrics to manage people and performance, with the known risks of people gaming the metrics. Metrics can also be useful to introduce newcomers in a team, to help them monitor their progress integrating with the common habits of the team and to explain known issues and patterns.

At a higher-level, there is need (but no practical evidence...) of integrating CI and code-related metrics with business objectives, things like KPIs, KRIs, xabilities metrics that are important for managers. Metrics are also useful to build confidence for release management of the software.

Laurent Bossavit advised that using metrics to improve software development process is just like using a scales to lose weight: It is useful but not that often, and only if you take actions! Measures are just one element of the Plan-Do-Check-Act cycle (aka. Demings' wheel). Tom DeMarco is well known for having coined the phrase "You can't control what you can't measure", yet there is evidence that measurement alone is not useful, and that it may sometimes be useless. See  this article from July's issue of IEEE Software...

Some questions about test related metrics were asked. Relating defects to code locations and tests run can be useful as it is well known that bugs are nesting: Locations of defects are strongly correlated to one another.

Citcon wiki link

20090916: CfP AT2009 NL

Ook dit jaar zal de "Agile Tour" via Lille (Rijssel) passeren : haar beffroi, haar  mosselen-friet, de bieren en de sympathieke "agilistes".  Het vorige jaar, hebben wij ongeveer 80 deelnemers aangetrokken,  van alle achtergronden en van een breed scala van beroepen in de software  in de regio. Reacties op de conferentie waren unaniem  positief dank dus vooral om de kwaliteit van de inhoud  van de belanghebbenden. Dit jaar willen wij deze proef transformeren en duurzaamheid  in de software-ontwikkeling van Rijssel vaststellen.  

Agile Tour Lille 2009 wordt gehouden vanaf 30 oktober, van 13.30 uur tot/met 18.30 uur in het voormalige industriele gebied De Blan-Lafont, getransformeerd  door de komst van Euratechnologies (http://www.euratechnologies.com/).  

Vanaf 19.30 uur zal een OpenSpace in een minder formeel en meer interactief karakter plaatst nemen.  

Wij hebben u nodig! Dit is een oproep aan alle  "agilistes": extreme programmeurs,  frans of niet, ontwikkelaars, coaches, managers  of andere projectleiders. Kom en deel uw ervaringen, uw ideeën, uw  instrumenten, uw methoden van werken, uw wensen en uw teleurstellingen, met andere woorden, alles wat betrekking kan hebben met projecten. Bied een presentatie,  een workshop, uw ervaring ...  

Om dit te doen, twee oplossingen:  - Open een sessie online op Agile Tour 2009  (zie http://www.agiletour.org/fr/node/add/forum voor formulier  Online. Het is noodzakelijk om een account te hebben)  - Stuur een e-mail met uw voorstel na de organisatoren  Agile Tour Lille (agile-tour-lille@googlegroups.com)  

Lille Agile Tour 2009 wordt lokaal ondersteund door:  

20090914: On The Benefits of Dojo

Coding dojos are a great way to learn and deepen one's understanding of programming, especially TDD-guided programming. As TDD is becoming somewhat more mainstream, it might be interesting to reflect on the benefits one gets from a dojo and the mechanisms at stake in a dojo, to leverage this technique and possibly apply in other contexts. This idea was suggested by a colleague of mine when we talked about generalizing dojos company-wide.

Programming dojos are, well, about programming so is there something special about programming that would prevent applying the concept to other fields ? We can easily answer a no to this question by recalling that coding dojos are inspired by martial arts dojos an altogether entirely different field of practice.

Discipline

So what's at stake in a dojo ? First, there is practice of some gestures in a codified way. We are not throwing haphazardly random chunks of code, this is no hacking (in the negative connotation usually associated with this word in software development context, hacking happens to have positive connotations in other contexts), we are following test driven development, pair programming and other eXtreme Programming techniques introduced originally by other people like Kent Beck, Ward Cunningham, Ron Jeffries and a few others.

The notion of gesture is extremely important. We are not applying theoretical knowledge, nor sharing some practical knowledge and experience (well, not only), we are practicing some movements to write code that have been codified by others, forming a kind of tradition. The assumption is that practicing on the how shall give us deep insight on the what and probably also on the why. We are  shaping the way we write code, practicing until one day, maybe, this way becomes second nature to us. By incorporating these gestures into our habitus when dealing with programming problems, we gradually focus more and more on the underlying issues of our work, on the problems we are trying to solve and the interaction between people and machines.

By learning gestures we improve the way we acquire knowledge and skill. It is common knowledge that learning by doing is better than learning by passively listening, in the sense that:

  1. people learn faster,
  2. people learn more,
  3. what is learnt stays longer.

Even in very abstract fields like mathematics, some authors (René Guitart, in La Pulsation Mathématique, Harmattan, Paris, 1998) point at the crucial importance of learning the common gestures of the field to acquire a better understanding of the theories, in this case the ways of prooving things. Guitart even goes to the point of emphasizing the physical relationship with tools of the trade: blackboard and cray, paper and pen, layout of the proofs in space, the quirks of language used; its whole point being that (mathematical) knowledge is built on two legs: Rigor and Ambiguity.

This same idea pervades the concept of Software Craftmanship and more generally the acknowledgement that programming is more akin to craft than to engineering: knowledge is pointless without skill, acquiring skill requires time, practice and transmission.

"L'habileté de l'artisan, sa connaissance du métier se fondent sur une discipline dont l'essance est très simple, à savoir qu'à travers une pratique répétée, on s'améliore" (Richard Sennett, Economic Sociology, cited in Philosophie Magazine, May 2009)

Someone said the essence of pedagogy is repetition. By redoing the same gestures again and again, by reworking on the same problems with different eyes at different times, we slowly incorporate the movements that lead us to solve these problems and make them available for later use on other problems.

Safety

There used to be a recurring joke in the army, back when conscription was universal and mere mortals like you and me were enrolled for one year: The army had an unofficial authorized kill ratio of 5%, so people expected to be treated without mercy even during their training period.  

The movie "Kill Bill" by Quentin Tarantino depicts the brutal apprenticeship of two would-be assassins under chinese martial arts master Pai Mei. When one of the women insults the old master, she is punished by bare-hand (and live) enucleation.  

It is customary among cooks to treat their apprentices like slaves and star chefs are renowned to be worse than average in this respect: 18 hours work days, apprentices sleeping on benches, vexations are commonplace. This is not specific to cooking and is widespread among other crafts: Masonry, Carpentry, Medecine and a lot of others have traditions of sado-masochistic master-apprentice relationship. We can even see damped echoes of this behavior in the traditional coffee-making and photocopying tasks assigned to interns at bureaucratic organizations.

All of this is evidence of the widespread belief that learning is tied to suffering, that darwinism also applies at the individual and interpersonal level. To get the best from people, to improve their capacity, just breed competition among several one and select the survivors.

Then most people except the boldest (or dumbest !) refrain from experimenting with new techniques on the field for fear of consequences. This attitude does not foster innovation, creativity or adaptation, yet this is exactly the characteristics that are required for knowledge creation and transformation tasks.

While keeping the benefits of hands-on problem solving and pragmatic approach, dojos offer a safe place to learn and experiment. Here safe means:

  1. nobody shall judge dojos participants for their doings as everybody expects to learn something from other people's mistakes and endeavours,
  2. dojo participants have the assurance they won't hurt anybody or destroy anything during a session, which is another incentive for experimenting.

Group dynamics

Dojos offer a different learning experiment and are based on the premises that skills and problem solving techniques are best learnt in groups. Some controlled experiments have shown recurringly that groups are (on the average) better at solving problems than individuals (see Eduquer et Former, Ed. Sciences Humaines, 2003, pp. XXX). One experiment involved a group of sofware engineers and logical puzzles: While individually people got from 1 to 7 grades out of 10, they collectively reached 10. Another experiment from Lewitt et al. (op.cit, pp. YYY) demonstrated the effect of various kind of group dynamics and leadership (autocratic, democratic, competitive) within group of teenagers on their behavior and "performance", and it shown that democratic groups fared better on the average.

Other evidences abound that efficiency within a group is maximized by deep cooperation rather than authoritative leadership or individualistic competition (see "Scaling Agile & Lean Software Development", Larman & Vodde, Addison-Wesley, 2008; "The Toyota Production System", T.Ohno, Productivity Press, 1995; ...).

Within a dojo, equality is the norm. There of course may be differences in knowledge, skills, interest or energy, which may affect the dynamics of the group and bestow the burden of leadership for some lapse of time to one individual. The one who proposes a subject or organizes the dojo usually takes over this role, willingly or not, and attendants will turn to him as a temporary authority to help progress. Yet, this is a role not a specific person, and learning as said before is expected to come from the results and progress made by the group as a whole rather than from one individual's knowledge.

Groups may be as small as two people working together, and this is another important aspect of the dojo: Systematic use of Pair Programming

To ensure that group dynamics do not turn into chaos or boredom, they rest on the firm ground of discipline (common purpose and doings) and safety

Feedback

It is very easy when learning something to get trapped in bad habits, lured by some intuition to follow some path that happens ultimately to be a dead end but on which you become stuck, to incorporate the wrong gestures that will be very difficult to get rid of and correct. This is all the more easy when feedback is scarse, sparse or provided by one's brain, something which happens when learning alone through knowledge acquisition only.

Feedback is just the only to learn correctly gestures in crafts, cycling through: Seeing, doing, evaluating, fixing. Because external feedback is fundamental to learning and improving, it forms the core of most work on quality improvement techniques and models, whether related to Agile or not. E.Demming's Plan-Do-Check-Act model,  J.Weinberg system control model, XP's and Scrum's retrospectives and daily meetings, are some examples of feedback loops used for improving and correcting one's demeanor.

Coding dojos allows experimenting with two different kind of external feedback: The short-term feedback that is provided by unit tests while coding, which is of course the object of the coding dojo itself; the long-term feedback provided by the group during the session and the ensuing retrospective. In a sense, coding dojos are a way to learn how to look for and use feedback in software development.  

Summary

So we can come up with a short list of key characteristics of coding dojos: Discipline of practice, Safety to experiment, Group learning and Constant feedback. We can equate those features with the values of eXtreme Programming (Communication, Simplicity, Courage Feedback) and this is of course no surprise.

Yet the fact that we practice software programming and coding seems to be unrelated to the core benefits and characteristics outlined here, and it should be possible to apply those same techniques within different domains of knowledge and skill.  

References

20090902: Scrum Game

http://www.agilitrix.com/2009/08/scrum-simulation-xpgame/

20090831: DevOpsDays

French version

Certain l'appelle Administration Système Agile, d'autres devops ou devmin.

Que vous soyez un développeur intérssé par l'administration système, un adminsys intéressé par le développement, ou un testeur entre les deux, rejoignez-nous pour la première conférence cherchant à joindre le meilleur des deux mondes:

Devopsdays '09 , 30 & 31 Octobre, Gand, Belgique http://www.devopsdays.org

Deux jours de conférences pour réfléchir et s'amuser (http://www.devopsdays.org/programme

Avec, entre autre orateurs: Rachel Davies, Lindsay Holmwood, Matthias Skarin, Chris Read, Teyo Tyree, Matt Rechenburg (http://www.devopsdays.org/speakers

English version

Some have called this agile system administration <http://groups.google.com/group/agile-system-administration> others devops or devmin.

So if you are a /developer/ with a interest for /system administration/. or a /sysadmin/ interested in /development/. Or a /tester /somewhere in between.

Join us on the first conference that focuses on bringing the best of both worlds together,

/Devopsdays '09 , 30 & 31 October, Ghent, Belgium /http://www.devopsdays.org/ <http://www.devopsdays.org> / /the conference that tries to get the best of both dev and ops world. two days of fun and interesting talks. http://www.devopsdays.org/programme <http://www.devopsdays.org>

Speakers include: Rachel Davies, Lindsay Holmwood, Matthias Skarin, Chris Read, Teyo Tyree, Matt Rechenburg http://www.devopsdays.org/speakers <http://www.devopsdays.org>

20090724: Dîner Théorie des Centres

A l'invitation de Bernard Notarianni, j'ai participé en compagnie d'Antoine Contal, Marc Guiot, David Gageot, Emmanuel Gaillot et bien sûr Régis Médina à une dîner-groupe de réflexion sur la Théorie des Centres, dîner faisant suite à aux travaux de Régis et à sa présentation lors des XP Days Paris 2009

Ce fût une soirée animée et riche d'interactions, parfois de frictions, entre des personnes aux expériences et sensibilités très diverses mais animées par une passion commune: rendre le code et l'activité qui le produit plus vivants

J'en ai retiré:

20090719: Build system & User Mode Linux

Summary

We want to construct VM for build to be deployed either as VM or as real machine. We shall use KVM as virtualization technology.

VM will contain the necessary stuff to stand as a build appliance for development teams, which include at least the following:

A build appliance can be used in various modes according to expected usage:

It should be eays to migrate from one solution to the other according to scaling needs. VMs shall be available from the web as standard appliances to be installed as needed.

VMs will be versionned from a master appliance. We shall use Git to host the VMs version in a master.

The creating new VM should be as simple as: 

git clone /home/vm/master /home/vm/slave

Note: need to script creation of bootable image from bunch of files.

VM will host apache for virtualizing various services offered.  

Experiment

First try at installing debian lenny host with virt-install utility: 

sudo virt-install --connect qemu:///system -n mxbuild -r 512 -f /media/disk/home/nono/mxbuild.qcow2 -s 5 --location http://ftp.fr.debian.org/debian/dists/lenny/main/installer-i386/ --vnc --noautoconsole --os-type linux --os-variant debianLenny --accelerate --network=network:default

but does not work ! Fails with strange error message at script execution. 

Démarrage de l'installation ...
Récupération du fichier 100% |=========================| 1.5 kB    00:00     
Récupération du fichier 100% |=========================| 1.4 MB    00:07     
Récupération du fichier 100% |=========================| 5.6 MB    00:32     
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/__init__.py", line 773, in emit
    stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 24: ordinal not in range(128)

Try to download lenny install ISO image.

Try using ubuntu-vm-builder: 

sudo ubuntu-vm-builder kvm jaunty
2009-07-19 17:03:14,354 INFO     Creating disk image: /tmp/vmbuildernQG9Fq/disk0.img
2009-07-19 17:03:14,399 INFO     Adding partition table to disk image: /tmp/vmbuildernQG9Fq/disk0.img
2009-07-19 17:03:14,492 INFO     Adding type 1 partition to disk image: /tmp/vmbuildernQG9Fq/disk0.img
2009-07-19 17:03:14,515 INFO     Adding type 3 partition to disk image: /tmp/vmbuildernQG9Fq/disk0.img
2009-07-19 17:03:14,544 INFO     Creating loop devices corresponding to the created partitions
2009-07-19 17:03:14,569 INFO     Creating file systems
2009-07-19 17:03:14,619 INFO     mke2fs 1.41.4 (27-Jan-2009)
2009-07-19 17:03:16,102 INFO     Mounting target filesystems
2009-07-19 17:03:17,319 INFO     Installing guest operating system. This might take some time...
2009-07-19 17:15:01,740 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:15:01,925 INFO     Moving old data out of the way
2009-07-19 17:15:02,182 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:15:03,365 INFO     Searching for GRUB installation directory ... found: /boot/grub
2009-07-19 17:15:03,737 INFO     Searching for default file ... Generating /boot/grub/default file and setting the default boot entry to 0
2009-07-19 17:15:03,739 INFO     Searching for GRUB installation directory ... found: /boot/grub
2009-07-19 17:15:03,748 INFO     Testing for an existing GRUB menu.lst file ...
2009-07-19 17:15:03,749 INFO     
2009-07-19 17:15:03,750 INFO     Could not find /boot/grub/menu.lst file.
2009-07-19 17:15:03,751 INFO     Generating /boot/grub/menu.lst
2009-07-19 17:15:03,873 INFO     Searching for splash image ... none found, skipping ...
2009-07-19 17:15:04,198 INFO     grep: /boot/config*: No such file or directory
2009-07-19 17:15:04,345 INFO     Updating /boot/grub/menu.lst ... done
2009-07-19 17:15:04,346 INFO     
2009-07-19 17:15:04,730 INFO     Searching for GRUB installation directory ... found: /boot/grub
2009-07-19 17:15:04,805 INFO     Searching for default file ... found: /boot/grub/default
2009-07-19 17:15:04,815 INFO     Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
2009-07-19 17:15:04,980 INFO     Searching for splash image ... none found, skipping ...
2009-07-19 17:15:05,054 INFO     grep: /boot/config*: No such file or directory
2009-07-19 17:15:05,218 INFO     Updating /boot/grub/menu.lst ... done
2009-07-19 17:15:05,219 INFO     
2009-07-19 17:15:05,343 INFO     Searching for GRUB installation directory ... found: /boot/grub
2009-07-19 17:16:13,310 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:16:14,457 INFO     Done.
2009-07-19 17:16:18,154 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:16:18,334 INFO     Running depmod.
2009-07-19 17:16:18,525 INFO     update-initramfs: Generating /boot/initrd.img-2.6.28-13-server
2009-07-19 17:16:23,672 INFO     Running postinst hook script /usr/sbin/update-grub.
2009-07-19 17:16:23,884 INFO     Searching for GRUB installation directory ... found: /boot/grub
2009-07-19 17:16:24,024 INFO     Searching for default file ... found: /boot/grub/default
2009-07-19 17:16:24,035 INFO     Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
2009-07-19 17:16:24,333 INFO     Searching for splash image ... none found, skipping ...
2009-07-19 17:16:24,499 INFO     Found kernel: /boot/vmlinuz-2.6.28-13-server
2009-07-19 17:16:24,696 INFO     Replacing config file /var/run/grub/menu.lst with new version
2009-07-19 17:16:24,734 INFO     Updating /boot/grub/menu.lst ... done
2009-07-19 17:16:24,734 INFO     
2009-07-19 17:16:28,155 INFO     
2009-07-19 17:16:28,157 INFO     Current default timezone: 'Europe/Paris'
2009-07-19 17:16:28,163 INFO     Local time is now:      Sun Jul 19 17:16:28 CEST 2009.
2009-07-19 17:16:28,165 INFO     Universal Time is now:  Sun Jul 19 15:16:28 UTC 2009.
2009-07-19 17:16:28,165 INFO     
2009-07-19 17:16:35,175 INFO     update-initramfs: Generating /boot/initrd.img-2.6.28-13-server
2009-07-19 17:17:49,298 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:17:53,092 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:17:53,157 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:17:53,339 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:17:53,402 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:17:53,536 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:17:53,720 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:18:01,910 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:18:01,910 INFO     
2009-07-19 17:18:01,911 INFO     Current default timezone: 'Europe/Paris'
2009-07-19 17:18:01,912 INFO     Local time is now:      Sun Jul 19 17:17:55 CEST 2009.
2009-07-19 17:18:01,913 INFO     Universal Time is now:  Sun Jul 19 15:17:55 UTC 2009.
2009-07-19 17:18:01,914 INFO     Run 'dpkg-reconfigure tzdata' if you wish to change it.
2009-07-19 17:18:01,914 INFO     
2009-07-19 17:18:01,915 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:18:01,916 INFO     Can not write log, openpty() failed (/dev/pts not mounted?)
2009-07-19 17:18:01,916 INFO     grep: /proc/self/status: No such file or directory
2009-07-19 17:18:06,635 INFO     Copying to disk images
2009-07-19 17:19:36,807 INFO     Installing bootloader
2009-07-19 17:19:43,936 INFO     Unmounting target filesystem
2009-07-19 17:19:47,283 INFO     Converting /tmp/vmbuildernQG9Fq/disk0.img to qcow2, format ubuntu-kvm/disk0.qcow2
2009-07-19 17:21:40,068 INFO     Cleaning up

Install graphical manager: 

sudo apt-get install virt-manager virt-viewer

Host refuses to boot, stays locked at boot prompt...

Using debootstrap to create system

2009.07.19 18:18:32

First will install using deboostrap, then will try to convert to bootable appliance later on: 

mkdir ~∕ubuntu-kvm/mxbuild
cd ~/ubuntu-kvm/mxbuild
sudo debootstrap jaunty .

Installed apt-proxy which runs a proxy for installing packages on port 9999.

Ran into issues when installing br0 interface: there seems to be some conflict between NetworkManager and /etc/network/interfaces based installation. 

# create vm
# mount VM FS to /home/vm/master
cd /home/vm/master
git init
sudo cat > .gitignore <<EOF
*~
.ccache
/var/backups
/var/cache/
/var/log
/var/run
/var/spool
/var/tmp
/var/lib/mysql
/var/lib/apt
/var/lock
/sys/
/dev/
/tmp/
/proc
EOF

Don't forget to mount proc and other pseudoFS on target system (from host): 

mount -t proc proc proc/
mount devpts dev/pts -t devpts
mount sysfs sys -t sysfs

Get rid of annoying messages about locales (in guest): 

export LC_ALL=C
export LANG=C

Updating sources.list with universe, multiverse and restricted, plus security patches, and installing bunch of needed software: 

aptitude install sun-java6-jdk apache2 xvfb ssh mysql-server git-core

Then install manually hudson, nexus and sonar.

Links

20090718: inotify

http://inotify.aiken.cz/?sectioncommon&pagehome&lang=en has lot of information on inotify and provides C++ API for it. inotify-tools is a set of CLI tools for work w/ inotify. inotifywait in particular, allows for easy recursive scanning of complete directory tree and generates output on command-line about events affecting the tree, including adding/removal of files and directories.

20090707: TDD in C at the Coding Dojo

Yesterday at the Coding Dojo in Paris, we had a really fantastic session, at least as far as I am concerned ! And this is not because I was demoing a Kata in C ! During the roughly one and a half hour the dojo lasted, I felt the flow of the coding session. Things were running smoothly, we tackled problems one at a time and managed to overcome them all, even hard and dreaded problems of mysterious segfaults. Attendants had a distinctive awareness and vigilance about what was going on and I think we managed to get everybody through all of the session.

Christophe was pairing with me and its help, along with that of the audience, demonstrated the great benefits of pair programming in such grassroots languages like C: Quick detection of stupid mistakes, helpful hints at next step to keep things going, attention to even details like naming, YAGNI and DRY. Eric whose C experience was quite far behind proposed nice simplifications to the test code that exposed only the necessary logic:

20090702: Software Development RPG

Some ideas for a new session that could be run at an XP Day or another conference:

20090525: XP Day Paris - Day 1

Participé aujourd'hui à la première journée des XP Day Paris, qui sera aussi malheureusement la dernière, démarrage de mission oblige. J'y ai retrouvé des amis et connaissances, présenté deux sessions avec des résultats inégaux et assisté à une présentation par Régis Médina sur la Théorie des Centres de Christopher Alexander (celui qui est à l'origine du mouvement des patterns). J'ai aussi participé 2 heures au Xtreme Miles, un écriture collective d'un jeu de Mille Bornes

La Théorie des Centres

A partir d'une analyse des formes et des principes de la conception (en l'occurence de la conception d'objets, d'oeuvres d'art, de bâtiments, mais le transfert est aisé à la conception de logiciels), Alexander développe le concept de Centre pour analyser les affects que produisent ces objets/images/bâtiments sur nous, et plus particulièrement pour expliquer le sentiment de fluidité, d'évidence (traduction approximative du Flow anglais), de qualité qui nous saisit devant (ou dans) certaines réalisations du génie humain.

L'idée centrale est que nous percevons quelque chose qui peut s'exprimer comme une structure réticulée, dont les noeuds sont les fameux centres, et qui chacun ou ensemble exhibent des propriétés positives, c'est-à-dire dont la présence contribue à la qualité globale de la structure. Celles-ci sont réduites à 15, portent des noms plus ou moins exotiques et sont parfois difficiles à saisir, en tout cas à transporter dans l'univers logiciel. Il est question d'Echos, la réapparition d'un même motif à différents endroit sous des formes subtilement différentes (ie. des concepts transverses, ou qui le sont dans la structure actuelle mais pourrait ne pas l'être dans une autre structure en les identifiant en tant que centres); de Frontière, l'identification claire des transitions entre les choses; d'Echelles, la structuration des centres en hiérarchies...

Les centres eux-même peuvent être de différents types (??): forts, faibles, latents. Le processus de conception, puisqu'il s'agit bien d'un processus, tendra à produire de l'ordre en structurant les centres avec ces diverses propriétés, réduisant les centres faibles et renforcant les centres forts, faisant émerger des centres latents, un processus appelé Dépliage des centres. Toute la difficulté consiste bien sûr à identifier les bonnes évolutions, celles qui vont ajouter de nouvelles propriétés au système, améliorer sa structure, et non seulement localement mais aussi globalement.

La notion d'ordre est au centre - c'est le cas de la dire, de ce discours. D'ailleurs, l'ouvrage d'Alexander exposant ces théories s'appelle The Nature of Order ! La bonne conception serait celle qui ferait émerger un ordre dans lequel nous nous sentirions à l'aise, en harmonie, à partir d'un chaos indifférencié de centres équipotents et équivalents, tous reliés les uns aux autres. Le mérite de cette théorie semble d'être celui d'offrir un vocabulaire, une syntaxe, un cadre de discours, un langage donc, permettant d'exprimer des intentions, des impressions sur une conception.  

Mes sessions

20090513: Discours & développement logiciel

Je fais cette hypothèse que l'activité du développement de systèmes logiciels à l'usage d'êtres humains est production collective d'un discours incarné par le code source du logiciel produit. En tant que discours, que texte, ce code source en possède tous les attributs dont en particulier un ensemble de sens et de références, ainsi qu'une structure lexicale et syntactique, entendue comme différente de celle du langage hôte. De la même manière que les langues naturelles pour leurs locuteurs, le langage hôte constitue une contrainte, une donnée, pour le développeur mais en aucun cas une limite.  

Pour confirmer cette hypothèse, il doit être possible d'appliquer au code source des grilles d'analyse similaires, sinon identiques, à celles ayant cours dans le domaine de l'analyse textuelle. En particulier, il doit être possible de faire une analyse critique du discours qui mette en lumière sa pertinence et ses qualités selon des dimensions sociales et humaines: artistique, politique, économique, scientifique, philosophique...

20090421: Philosophie de l'informatique

Malgré mes recherches assidues, il semble bien qu'aucun auteur n'ait traité de l'informatique d'un point de vue philosophique. Les mathématiques ont bien sûr suscité une réflexion philosophique, ne serait-ce que parce que les deux disciplines sont nées à peu près simultanément et qu'elles ont longtemps été indissociables: mathématicien et philosophe pour les anciens grecs, c'était la même chose.  

Quelles questions philosophiques pose l'informatique ? Qu'est-ce qu'une question philosophique, d'ailleurs ? Qu'est ce que la philosophie a à proposer comme outils pour mieux comprendre cette discipline et ceux qui la pratiquent ?

L'informatique semble avoir pour objet le traitement de l'information : sa création, sa transformation, son codage et transcodage, son stockage. L'information fait-elle partie de la réalité, en est elle un attribut ou ne s'agit-il que d'une propriété qui n'est perceptible que des seuls êtres humains ? Cette question a-t'elle même un sens, puisque si aucun être humain n'existait, personne ne pourrait la poser. Quel est le lien entre l'information et les données sensibles qui nous parviennent ? Pourquoi des processus informatiques permettent de simuler des processus biologiques et cognitifs, au point d'en devenir indistinguables ?

Qui se pose ce genre de questions ? Il y a une communauté de chercheurs académiques sur la philosophie de l'informatique, ou plus précisèment la philosophie de la science calculatoire (computing science).  

20090421: Web Development and Haskell

20090420: About SPA2009

I have put online the summary I wrote about SPA 2009

20090420: Compiling latest version of Happstack

2009.04.20 09:40:21

20090415: Issues Running maven-shitty-plugin

``When one wants to use a different repository or settings file than the default built-in =HOME/.m2/repository= and =HOME/.m2/settings.xml=, it is necessary to add some flags definition to the plugin's execution, which can be done by defining a per-project (ie. per integration testign project launched by the plugin) flags file : 

 <build>
  <plugins>
   <plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>shitty-maven-plugin</artifactId>
    <executions>
     <execution>
      <goals>
       <goal>clean</goal>
       <goal>install</goal>
       <goal>test</goal>
      </goals>
     </execution>
    </executions>
    <configuration>
     <debug>true</debug>
     <flagsFile>flags.txt</flagsFile>
    </configuration>
   </plugin>   	
  </plugins>
 </build>

20090414: Manipulating Mercurial log history with MQs

I needed to change the username of log messages in Mercurial. I found this tip on the web. The principle is to use Mercurial Queues to:

  1. transform commits in patches,
  2. change what I want to change in the patches while popping the stack of patches,
  3. push back the stack of patches with the right information in,
  4. transform back the patches into "real" changesets.

20090403: Masters & Servants

There is a growing discussion in or around the agile community about Software Craftmanship and team organizations based on master-apprentice relationship, something which, as is denoted by Uncle Bob's first comment, is somewhat contradictory with a the notion of self-organizing teams and cross-functional team members often advocated in XP and Scrum. Here are two prominent links gathered recently:  

This idea was also expressed by Frederic Brooks in the form of the surgical team in The Mythical Man-Month. Here and there, the image of the commando or more generally of a military organization pops up.

20090331: Des Justes

Abraham dit : Que le Seigneur ne s'irrite point, et je parlerai. Peut-être s'y trouvera-t-il trente justes. Et l'Éternel dit : Je ne ferai rien, si j'y trouve trente justes. Abraham dit : Voici, j'ai osé parler au Seigneur. Peut-être s'y trouvera-t-il vingt justes. Et l'Éternel dit : Je ne la détruirai point, à cause de ces vingt. Abraham dit : Que le Seigneur ne s'irrite point, et je ne parlerai plus que cette fois. Peut-être s'y trouvera-t-il dix justes. Et l' Éternel dit : Je ne la détruirai point, à cause de ces dix justes. L'Éternel s'en alla lorsqu'il eut achevé de parler à Abraham. Et Abraham retourna dans sa demeure. (Génèse, 18, 30-33)

Emmanuel Gaillot n'est peut-être pas Dieu, mais il a trouvé ces dix justes: les dix personnes qui se sont passionnées pendant plus d'une heure trente sur l'utilisation de QuickCheck en Haskell au Dojo de Paris, le lundi 23 mars 2009.  

20090331: Contre la réunionite

http://sethgodin.typepad.com/seths_blog/2009/03/getting-serious-about-your-meeting-problem.html

20090316: Rapport sur la construction d'une poétique de la programmation

Nous pensons tout d'abord qu'il faut changer le monde. Nous voulons le changement le plus libérateur  de la société et de la vie où nous nous trouvons enfermés. Nous pensons que ce changement est possible par des actions appropriées.  

Notre affaire est précisèment l'emploi de certains moyens d'action, et la découverte de nouveaux, plus facilement reconnaissables dans le domaine de la culture et des moeurs, mais appliqués dans la perspective d'une interaction de tous les changements révolutionnaires.  

Ce que l'on appele la culture reflète, mais aussi préfigure, dans une société donnée les possibilités d'organisation de la vie. Notre époque est caractérisée fondamentalement par le retard de l'action politique révolutionnaire sur le développement des possibilités modernes de production, qui exigent une organisation supérieure du monde.  

Nous vivons une crise essentielle de l'histoire où chaque jour pose plus nettement le problème de la domination rationnelle des nouvelles forces productives, et de la formation d'une civilisation, à l'échelle mondiale. Le terme extrême de la rationalité apparait aujourd'hui de plus en plus nettement comme l'instauration d'une soumission généralisée dans laquelle la liberté du plus grand nombre, et même de ceux qui s'en croient protégés par leur fortune, leur naissance ou leur mérite, se trouve aliénée au fétichisme du spectacle. Le projet de l'homme maître et possesseur de la nature a enfanté d'une humanité esclave et possédée par la technique.

Il faut entreprendre maintenant un travail collectif organisé, tendant à un emploi unitaire de tous les moyens de bouleversement de la vie quotidienne. C'est-à-dire que nous devons d'abord reconnaître l'interdépendance de ces moyens, dans la perspective d'une plus grande domination de la nature, d'une plus grande liberté. Nous devons construire des logiciels nouveaux que soient à la fois le produit et l'instrument de comportements nouveaux.

Pour ce faire, il faut utiliser empiriquement, au départ, les démarches quotidiennes et les formes culturelles qui existent actuellement, en leur contestant toute valeur propre. Le critère même de nouveauté,  d'invention formelle, a perdu son sens dans le cadre traditionnel d'un art, c'est-à-dire d'un moyen fragmentaire insuffisant, dont les rénovations partielles sont périmées d'avance - donc impossibles.

Ceux qui, comme nous, se trouvent simultanément à l'avant-garde du de l'avancée de la technostructure et du rejet de la société spectaculaire marchande ont la possibilité unique, et peut-être le devoir historique, de contribuer à cette transformation de la vie même.  

Pour ce faire, nous proposons de multiplier les objets et les sujets poétiques, malheureusement si rares  actuellement que les plus minimes prennent une importance affective exagérée, et d'organiser les jeux de ces sujets poétiques parmi les objets poétiques. Et tout d'abord, de rendre à la construction de programmes sa portée esthétique, éthique et politique, qui s'est perdue au fil du temps dans l'utilitarisme médiocre et la stupidité béate de ce qu'il est convenu aujourd'hui d'appeler les technologies de l'information

Voila tout notre programme, qui est essentiellement transitoire. Nous n'avons pas  de recettes, ni de résultats définitifs. Nous proposons seulement une recherche expérimentale à mener collectivement dans quelques directions que nous définissons en ce moment, et dans d'autres qui restent à définir.

On a assez interprété les passions, il est s'agit maintenant d'en trouver d'autres.

20090325: Thinking inside out

Among the difficult thing I find writing code in Haskell is what I call thinking inside out particularly when dealing with monadic or functorial code (ie. code that is supposed to work on any instance of a functor, using higher-order types and functions).

The newtype construct together with the records' syntax for example, is a trap for OO programmers. We are accustomed to see record as an entity which is characterized by a set of named attributes which you can query given an instance of the record. In Haskell, when one writes something like:

previous

newtype Foo a = Foo { runFoo :: State Bar a }
    deriving (Monad, MonadState)

runFoo has actually the meaning of an observer of Foo instances, being a function of type Foo a -> State Bar a that is used implicitly in the wrapping/unwrapping  process taking place in deriving classes.

So in this particular case, one never needs to create explicitly an instnace of Foo a, it should be created only once, to start the ball rolling and interpret correctly (ie. in the right Monad) a monadic fragment. previous


doSomething :: (MonadState m s a) => m s a
doSomething = get >>= compute >>= compute2 >>= put


updateState = (runState . runFoo) doSomething initState

20090325: Feeling Overwhelmed

I like this English word: Overwhelmed, which have the nice property of closely matching form and meaning. For a French native speaking, it is quite uneasy to pronounce with its cluster of v, w and h in the middle. It better conveys than surchargé the feeling one has when faced with an apparently never-ending pile of problems, popping up one after another, or nested one in another like an onion peel. I just feel like living inside the onion, trying to dig my way out without the slightest idea of the number of skins the onion has.  

20090218: JMXTerm

Found an interesting project for command-line connecting to a JMX console: JMXTerm. It works nicely and can even be used from a script shell !

20090213: What a Sysadmin reads ?

Gildas:  je lis un peu ce qui me passe devant les yeux
mais y a des classiques
http://highscalability.com/ par ex pour les infos sur comment font les acteurs "referents"
y a qq publis de temps a autre mais assez peu sur des sujets connexes
y a pas trop de pointeurs precis
je lis aussi des trucs comme le blog de Kris Buytaert
mais en ce moment je trouve que sa qualite a baisse
(et pourtant c'est un pote :))
le blog de Luke Kanies
(le mec qui a dev puppet)
des retours d'experience lorsque j'en trouve via des sites d'informations "generalistes" en informatique
(de /. a osnews en passant par linuxfr)
desole de pas avoir de reponse plus precise
j'en suis le premier mortifie!
me:  non. si tu me posais la question, je serais aussi embêté :-)

20090208: Pourquoi je veux être "Agile"

Question: Pourquoi veux-tu être 'Agile' ?

Réponse: Parce que cela me procure  de la joie

Q: Qu'entends-tu par joie ?

R: Le sentiment d'être plus vivant

Q: En quoi être 'Agile' te rend-il plus vivant ?

R: En accroissant ma connaissance du monde avec lequel j'interagis. Etre Agile m'invite à dépasser croyances et opinions pour rechercher des idées vraies par la raison et l'intuition.

Q: En quoi mieux connaître le monde rend-il plus vivant ?

R: En me permettant d'éviter ce qui me nuit, ces choses qui affaiblissent mon corps et mon esprit, et de rechercher ce qui les renforce.

Q: Ne peut-il y avoir de joie sans connaissance ?

R: C'est possible, mais cela arrive par accident. Et de même pour la tristesse.

Q: Et alors ?

R: Alors je suis le jouet de ces sentiments accidentels. Ils me guident plutôt que je ne les suscite, et je leur remets ma liberté.

Q: Mais ne vaut-il pas mieux parfois vivre dans une ignorance bienheureuse que d'avoir connaissance de choses douloureuses ou inévitables ?  

R: Encore une fois, l'ignorance ne conduit qu'à l'esclavage des passions, les siennes ou celles des autres.

Q: Pourquoi souhaiter que les autres acquièrent aussi cette connaissance ? Ne peux-tu les laisser en paix ?

R: Ma joie est d'autant plus grande qu'y participe un plus grand nombre d'individus. Il ne peut y avoir de vrai joie partagée sur la base d'une fausse connaissance (croyance ou opinion). Donc je souhaite partager la connaissance vraie avec les autres pour augmenter ma joie.

Q: Comment le fait d'être 'Agile' t'apporte cette connaissance ?

R: En valorisant les individus et les interactions, j'accrois mes sources de connaissance. En produisant d'abord des logiciels fonctionnels, je suis plus proche de l'essence des choses. En substituant la confiance au contrat, je fuis les passions tristes. En répondant au changement, je suis en prise avec le réel et non le fantasme.

Q: Mais concrètement, comment cela se passe-t'il ?

R: Par exemple, être 'Agile' m'a appris à utiliser TDD: savoir pourquoi j'écris un bout de code avant de l'écrire. Ainsi, je sais toujours ce qu'est sensé faire le logiciel, et s'il le fait réellement. Partager le code avec l'ensemble de l'équipe m'oblige à être plus rigoureux, à bien choisir mes mots et les termes que j'utilise, et à éviter les constructions alambiquées. Ou bien encore, automatiser les tests me libère de tâches fastidieuses et répétitives qui ne m'apportent aucune connaissance supplémentaire. Mais aussi, avoir un client ou son représentant dans l'équipe me permet de ne pas me fier à mes opinions et mes croyances sur ce que doit faire le logiciel.  

Q: Mais c'est difficile et ce n'est pas pour tout le monde ?

R: Oui c'est difficile. Et si, c'est pour tout le monde. Il y faut de l'envie, du désir pour la rigueur et la connaissance qu'elle peut offrir.  

Q: Ce n'est pas un peu radical ?

R: Si bien sûr. Mais il n'est pas dit que tout changement doive advenir en un jour. Rien n'est imposé, tout est proposé. C'est comme un virus qui se propage, des fois il trouve un terrain favorable, des fois il meurt avant de pouvoir se reproduire et se transmettre. Mais il est toujours là car il est multiple.

20090205: Google Gadgets & Shindig RPC Enlightment

We are having some issues at Leroy-Merlin trying to create Google Gadgets whose content is drawn from a CMS (ie. Nuxeo Web Engine). The problem lies in the way RPC is done and how the rpcToken that authenticates gadgets RPC requets is transmitted from the gadget container to the gadget. In order to solve these issues, I tried to understand how the whole things work which yielded me to delve deeply into RPC code and container/gadgets interaction. A summary of the picture may be useful for others who are developping containers or gadgets.

The problem

A gadget is (usually) loaded by a container within an iframe in the container's page. Loading means here that:

  1. gadget container creates an iframe to contains the gadget
  2. this iframe's src attribute is set to a URL which is constructed to point to the gadget renderer of the server (ie. shindig server)
  3. the URL contains, among other things, a fragment that itself contains pairs of key/values and most notably the <code>rpcTokenXXX</code> value. The XXX= value is generated by the container.

The gadget's content is a <Module> XML file that contains various data useful for gadget rendering. The actual content of the gadget may either be:

  1. inline within the XML description,
  2. or external at a URL pointed at by a url tag.

A URL-type gadget is useful to gadgetize an existing application or HTML page easily.

The gadget, whatever its type, may load features (ie. Gadget libraries) and more generally any kind of javascript and elements. Being loaded in an iframe, gadget's content and code is partially isolated from its parent's through sandboxing and same-domain policy which says that an iframe's code may access parent's code and context iff it comes from the same domain than the parent. A domain is a triple (protocol, host, port).

What happens when one's gadget needs a feature or a functionality from its container ? For example, what happens when one wants its gadget's content to be resized dynamically ? Enters RPC (Remote Procedure Call) which is a mechanism for gadget-to-container and gadget-to-gadget communication.

Gadget initialization and RPC

The following figure sketches how RPC works for a 'standard' container which hosts gadgets within an iframe

Shindig RPC Scenario

The container is the web page that hosts the gadgets. It is also usually a concrete Javascript object (here gadget.container) that creates javascript objects representing gadgets (here gadget.Gadget): These details are of course container-specific.

gadget.rpc is a common shindig feature which lays in rpc.js file.  

When the container loads a gadget, the following happens:

  1. a security token is registered for the gadget within the gadget.rpc scope. Note that as we are in the container scope, there is no parent so the token for .. which is used to identify the gadget within its parent's context is set to 0,
  2. the container creates the Gadget instance,
  3. which in turns creates an iframe instance to hold the gadget's content. This is done by setting the iframe's src property to a URL which points to the gadget rendering servlet hosted by Shindig, here: <code>http://server1/shindig/ifr?specUrlhttp://server2/gadget/gadget.xml#authToken1234</code>. Note the use of a URI fragment to allow cross-domain communication between the iframe and its parent,
  4. the browser requests the iframe's content from the shindig server,  
  5. which in turns request it from the gadget's server. The rendering servlet then produces a HTML page from the gadget's definition, including necessary features like
  6. rpc.js which, once loaded,  
  7. initializes the authentication token associated to this gadget from the URI fragment drawn from its deocument.location.href property.

The gadget is then ready to operate.  

RPC calls

Uasge of gadgets RPC is specified in the OpenSocial API and is quite simple: A provider registers a service upon intialization, which may then be used by a later call. As an example, we take a look at the dynamic-height feature which provides gadget.window.adjustHeight API function.

The code for this feature is the following (comments and technical details removed): previous

var gadgets = gadgets || {};


gadgets.window = gadgets.window || {};


(function() {


  var oldHeight;


  gadgets.window.getViewportDimensions = function() { ..  };


  gadgets.window.adjustHeight = function(opt_height) {
    ..
    // Only make the IFPC call if height has changed
    if (newHeight != oldHeight) {
      oldHeight = newHeight;
      gadgets.rpc.call(null, "resize_iframe", null, newHeight);
    }
  };
}());

This code is loaded in the context of the gadget hence the need for the rpc.call: only the iframe's parent can know how to resize properly the gadget.

In the container context (ie. in our case a file called gadgets.js which is greatly inspired from Shindig's sample container), we find the following code:  previous

gadgets.ContentGadgetService = function() {
  gadgets.GadgetService.call(this);
  gadgets.rpc.register('resize_iframe', this.setHeight);
};


gadgets.ContentGadgetService.prototype.setHeight = function(height) {
  var element = document.getElementById(this.f);
  if (element) {
    element.style.height = height + 'px';
  }
};

The resize_iframe call is bound to a local function which adjusts the gadget's DOM element's height. The gadget's identity is equal to the iframe's identity and stored in this context object for the setHeight function. The object this is constructed in the gadget.rpc.call method: previous

      var rpc = {
        s: serviceName,
        f: from,
        c: callback ? callId : 0,
        a: Array.prototype.slice.call(arguments, 3),
        t: authToken[targetId]
      };
and used in the gadget.rpc.process method: previous
 var result = (services[rpc.s] ||
               services[DEFAULT_NAME]).apply(rpc, rpc.a);

Handling Cross-domains calls

As said before, browsers enforce strict security delimitations between iframes and their parent's context. To allow gadgets to communicate with their context and with each other, RPC uses various hacks depending on the browser context. The details can be found in the rpc.js source code.

The most important cross-domain communication channel used is called IFPC which stands for Inter Frame Procedure Call, because it is the fallback mechanism for all browsers. IFPC uses a realyUrl on the same domain as the container to trampoline rpc.call

20090129: Embedding JUnit tests within JMX

I finally managed to get my JUnitMBean up and running, and to test it within our Nuxeo based project from a remote jconsole. Although the process is rather well documented on Sun's site and in the JDK documentation, this is not entirely trivial. Furthermore, the idea seems rather new: I found references on the web about an old package offering this kind of service, but this was quite cold trail. So I think this warrants a small article :-)

Motivation

I briefly explained the basic idea in another post: I want to be able to execute JUnit compliant tests within a running Java application using a JUnitMBean controlled through a JMX console. There are numerous reasons I want this, among others:

Implementation

The basic features of the bean are the following:

Here is the bean's interface: previous


public interface JUnitMonitor {


    JUnitResult run();


    void addClass(String className) throws ClassNotFoundException;


    String[] getTestClasses();
}

The JUnitResult class returned by the run() method is a Serializable transformation of the basic Result class from JUnit. Note that in the current state of the system, I do not know how to produce meaningful messages from the JMX jconsole. I wrote toString() method but it is not displayed when run() gets executed, so I guess I will have to return a basic String. It is possible to create portable data structures within JMX through MBeanInfo customization and various complex operations I am not sure I want to undertake.

The implementation class is very straightforward and boils down to storing class instances in a list and calling the JUnit framework on this list:

previous

public class JUnitMBean implements JUnitMonitor {


  private List<Class<? extends Object>> testClasses = new ArrayList<Class<? extends Object>>();


  public void addClass(String className) throws ClassNotFoundException {
    testClasses.add(Class.forName(className));
  }


  public JUnitResult run() {
    return map(JUnitCore.runClasses(testClasses.toArray(new Class[testClasses.size()])));
  }


  private JUnitResult map(Result result) { ... }


  [...]
}

Deployment

As this is supposed to work in a Nuxeo/OSGi based environment, all of this is packaged in a bundle that will deployed within a server. Bean's registration occurs when the component is deployed in the container, using the following code:

previous

mbean = new JUnitMBean();
mbs = ManagementFactory.getPlatformMBeanServer();
mbs.registerMBean(new StandardMBean(mbean, JUnitMonitor.class), mbean
        .getName());

Note the wrapping in a StandardMBean that allows JUnitMBean to not inherit from the latter class.

JMX access is configured through various parameters. I use the following ones that defines the port to be non-privileged 54321 port and do not use SSL for communication between jconsole and the running system. 

JMX_OPTS=" -Dcom.sun.management.jmxremote.port=54321 -Dcom.sun.management.jmxremote.ssl=false "

There are jmxremote.passwd and jmxremote.access files that define, respectively, the passwords for each identified user/role and the access rights (readonly or readWrite) for each user/role. Location of these files may be parameterized using command-line switch or a global configuration file. The runtime system requires the password file to be accessible only to the user running it.

Note: I am currently facing some strange issues on one of my machine that refuses to let me access JMX although it seems that everything is setup fine.

To access the console from a remote host, I have to augment the classpath so that the JUnitResult class could be found: 

$> jconsole -J-Djava.class.path=/usr/lib/jvm/java-6-sun/lib/jconsole.jar:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/nono/lm-portal-jmxtest-1.0-SNAPSHOT.jar

Then in the console, on the MBean tab, a new entry appears in the MBeans tree for oqube.jmxtest that offers attributes and operations through the standard console interface.

20090124: Haskell Pitch

From haskell-cafe mailing-list: 


>  Such a database would help me counter by boss's
> argument that "it's impossible to find and hire Haskell programmers."
>
>   
There was a thread last week where someone asked who would be
interested in a hypothetical Haskell job.  He got about 20 positive
responses.  This agrees with the experience of Microsoft Research in
2006 when they advertised for a third person to help with GHC
development.   They also had about 20 applicants.


So next time I hear the "you can't get the programmers" line I'm going
to respond with something like this:


   "If you post an advert for a Haskell developer you will get 20
   applicants.  All of those people will be the kind of developer who
   learns new programming languages to improve their own abilities and
   stretch themselves, because nobody yet learns Haskell just to get a job.


   "If you post an advert for a Java developer you will get 200
   applicants.  Most of them will be the kind of developer who learned
   Java because there are lots of Java jobs out there, and as long as
   they know enough to hold down a job then they see no reason to learn
   anything."


Paul.

The question is: among the 200 java developers you will get answers from, is it not possible that about 10% of them will be of the kind that want to improve their own abilities and stretch themselves ?

20090123: Back from Agile Open France 2009

I am back from Agile Open France 2009 and much like previous year, I am back energized and at the same time exhausted: Full of the energy I gathered through the numerous discussions and interactions I had with the other attendants, and exhausted by lack of sleep. This event gathered around 50% more people than previous year and at the same time I was able to talk, albeit sometimes briefly with nearly all the other participants.

This is the main reason I like this kind of events and the open-space format: As most session proposals take the form of workshop, discussions and experiences reports, one gets plenty of opportunities to not only be a spectator but also an actor of what's going on. The state of unpreparedness most of us are in is really a good thing: As a session proponent, you are not able to claim more than a fraction of the session's available time which means you need the energy of other people to make something interesting happen.

Some salient point that may or may not be developped later on:

Au final j'ai le sentiment, qui n'est ni positif ni négatif mais un simple constat, que cet événement, comme d'autres mais avec des possibilités d'interactions plus fortes, montre que l'agilité réunit sous une même bannière (au moins) deux groupes de personnes caractérisés par des centres d'intérêts et des préoccupations assez différents.

Le premier groupe, que j'appelerais sans aucune connotation péjorative les mystiques serait plus attiré par les caractères humain que l'agilité permet d'exprimer, réunir, valoriser et faire progresser et par l'émergence d'un tout plus grand que ses parties que l'interaction de ces caractères produit, en l'occurence un groupe ou une équipe.

Le second, disons les matérialistes, serait caractérisé par son amour du faire, par le plaisir et la joie que procurent à chaque membre de l'équipe un travail de valeur dans un environnement épanouissant pour chacun. Pour les membres de ce groupe, l'équipe serait le moyen de multiplier des interactions enrichissantes et d'atteindre un but inaccessible pour un individu isolé.

Les deux positions ne sont pas inconciliables et se retrouvent à des degrés divers en chaque participant. Mais on pourrait les caractériser certainement au moyen de la répartition entre les différents styles de sessions classées selon leur sujet et leur proximité à ces deux pôles.  

20090114: Embedding tests

Thinking about embedding tests within Java programs using JMX capabilities. A JUnitMBean would expose a run() method for running a test suite defined at compile time and would expose the results in the console.  

20090109: Iteratees

Some really nice work on high-level constructs for looping over input data: Iteratees by Oleg Kiselyov.

20081225: Layers of testing

It is easy to test software at the unit level: function, object, class, numerous frameworks exist that make this task seamless within the development process. There exists also quite numerous frameworks and systems for testing at the system level, whether through a GUI, command-line tool or using behind-the-gui requirements driven testing tools. While not always easy to setup and stabilize, implementing automated testing of whole system is becoming more commonplace.

There seems however to be a gap that I find difficult to fill: Testing at the component level. Because the component is a unit of deployment, as defined by C. Szyperski in its seminal book, and hence implicitly or explicitly relies on a context provided by its container or the system it is supposed to be part of, testing a whole component requires lot of scaffolding, stubbing and mocking to be effective.

Careful design and architecture can help a lot by decoupling the glue code which allows interaction between the ambient medium (the container) and the business code: The latter is unit-testable and there is only a small layer of proxying/transforming that is not tested directly (but still is at the system level).

Yet the component's components often relies on services provided by the container that are very difficult to isolate completely, or that are simply needed for proper functionning: dependency injection, non-functional services (eg. trace and logs, monitoring), data(base) retrieval and storage... Aspects, annotations, meta-programming are useful techniques that gives the developer the ability to properly isolate the various dimensions of a component's behaviour within units of computation.  

20081224: Sous contrainte

"Les chaînes et le silence, qui auraient dû la ligoter au fond d' elle-même, l'étouffer, l'étrangler, tout au contraire la délivraient d'elle-même." Histoire d'O, Pauline Réage

Sans atteindre ces extrêmes de l'âme humaine où se réfugie le plaisir d'O dans l'extraordinaire roman sus-cité, je crois profondément vraie qu'une grande jouissance, particulièrement une jouissance créative, peut être le fruit de grandes contraintes quand celles ci sont acceptées, incorporées, digérées et retransformées en une source de création.

Au coeur de cette joie se niche le plaisir du jeu

20081223: Martin Fowler on DSLs

In a recent blog post, Martin Fowler discusses the peculiarities of external Domain Specific Languages interpretation with respect to general purpose languages. He notes that tradeoffs are quite different although the techniques are similar: DSLs interpretation is not - and should not be - tied to the efficiency constraints that general purpose languages must support, so one can use heavy weight techniques such as the building of a semantic model of the parsed expressions to work with, making parsing much simpler and maintainable (ie. much like parsing an XML allows construction of a DOM).

An interesting side-effect of semantic models is that they can be a way to mix external and internal DSLs patterns, whether the latter evolves to the former with complexity pressure, or the former provides the latter as an embeddable way of working in the underlying model.

Another important difference is in the handling of abstractions: while GPLs must support a way to build new abstractions, DSLs should ususally not support them. The basic tenet of a Domain Specific language is that it models a specific domain, and only that domain, so all abstractions are builtin and given to the developpers to use, but not to extend.  

20081222: Mock et conception

Les simulacres induisent-ils une conception trop fortement couplée  et sont ils fondamentalement inutiles ?

Cette critique ressurgi fréquemment, et l'on trouvera des  discussions détaillées sur le Wiki C2 ou encore sur le site http://www.mockobjects.com. Une critique particulière que je souhaite traiter ici, non pas réfuter car il entre dans ces matières beaucoup de convictions et de pratiques personnelles, concerne l'inutilité des simulacres

TDD sans simulacres

D'une part, si chaque objet encapsule correctement un comportement et un état, alors il doit être possible de contrôler ce comportement et cet état uniquement en interagissant avec lui au travers de ses méthodes, soit qu'il produise un résultat à partir de données qui lui sont transmises, soit qu'il produise des données partielles à traiter par un autre objet.

Ce schéma correspond à une conception en rateau: un objet chef d'orchestre se charge d'organiser, voire de médiatiser, les interactions entre plusieurs objets et contient donc toute la logique de cette interaction.  

Un tel comportement rend caduque l'utilisation de simulacres puisque l'objet testé ne fait appel à aucun autre objet, hormis les plus triviaux.

Par contre, sur le plan de la conception et de la structure du système, il peut présenter l'inconvénient de produire d'un côté beaucoup de petits objets autonomes mais relativement stupides, de l'autre quelques gros objets "intelligents" mais complexes.  

Même en dehors de ce schéma, si chaque objet possède les propriétés requises d'indépendances et d'encapsulation, alors toute collaboration entre objets peut être adéquatement testée en utilisant les vrais objets et non leurs simulacres.

Du rateau on passe à l'arbre ou au DAG, une hiérarchie d'objets constitués en couches homogènes. Du point de vue du test, on se contentera de bouchonner le comportement des vrais objets, par exemple pour court-circuiter des accès à des ressources externes.  

Là encore, on évitera les simulacres, mais au prix probablement d'une complexité croissante des tests à mesure que l'on s'élevera dans les couches, chaque niveau nécessitant de plus en plus de code de mise en place pour que les tests puissent fonctionner.

TDD avec simulacres

L'utilisation des simulacres, non pas comme de simples bouchons, mais comme des simulateurs de rôles et d'interfaces induit une conception en graphe dans laquelle les différents objets sont liés entre eux par des dépendances contractuelles exprimées sous forme d'interfaces (dans le cas de Java) ou de types avec contraintes (dans le cas de langages avec un système de types plus sophistiqué).

Un tel graphe peut contenir sans problèmes des cycles: les objets liés par un cycle le sont au travers d'interfaces décrivant un protocole d'interaction testé justement par les simulacres.

Cette structure induit comme principal avantage un fort découplage entre les différents objets qui dépendront principalement d'interfaces ou éventuellement d'autres objets fortement cohésifs.  

Dans un modèle fonctionnel de test standard, le testeur est responsable de:

  1. la sensibilisation de l'objet testé (OUT ou Object Under Test) au travers d'appels de méthode
  2. la validation du comportement de l'OUT par l'examen des retours d'appels.  

Modèle de test

Comme on le voit dans le modèle de test ci-dessus, un objet collaborant avec un autre au travers d'une interface possède un deuxième point de contrôle et d'observation: la référence à l'interface de l'objet collaborant. Sur ce deuxième PCO, les rôles sont inversés: le testeur valide les appels réalisés par l'OUT et le sensibilise par les valeurs de retours qu'il lui fournit. Ce rôle est parfaitement symétrique et, d'un certain point de vue, quand on identifie les appels et retours de méthodes à de simples messages échangés entre deux objets, il est identique au premier.

Les outils de création de simulacres tel que mockito, easymock ou jmock, permettent de simplifier la gestion de cette partie du test lorsque c'est nécessaire.  

20081221: To mock or not to mock

Les bibliothèques de mock objects sont elles utiles pour tester et développer du code en TDD ?  

Telle est en substance la question vers laquelle a dérivée une discussion sur la liste de diffusion XP France. Il me semble que deux arguments (au moins) ont été donnés contre l'utilisation des simulacres

  1. dans ce message, Dominic Williams préconise a) que l'on utilise les objets réels pour tester des interactions lorsque c'est nécessaire et/ou b) que les objets soient autonomes, c'est-à-dire que soit ils font un traitement eux-mêmes, soit ils produisent des informations permettant à un autre objet de poursuivre le traitement;
  2. dans cet autre message, Laurent Bossavit fait remarquer que les simulacres ne sont pas nécessaires quand le langage permet de redéfinir le comportement d'un objet.

Un Exemple avec Mockito

Suivant le conseil de Laurent, voici quelques lignes de code Java utilisant la bibliothèque mockito qui, il me semble, justifient l'utilisation d'une bibliothèque de simulacres. On veut tester un Client qui utilise un Server pour accéder à une ressource nommée en lecture/écriture. Le protocole du serveur prévoit que:

Ce scénario est, en très simplifié, ce qui se passe dans de très nombreux systèmes aujourd'hui en production.

Voici le code du serveur:

previous

public interface Server {
  String read();
  void write(String string);
  void open(String string);
  void close();
}

Par convention, toutes les méthodes peuvent lancer une exception ServerException de la famille des RuntimeException

Voici les tests. D'abord la mise en place, on crée le simulacre et le client:

previous

public class ClientTest {
  private Server server;
  private Client client;


  @Before
  public void setup() {
    this.server = mock(Server.class);
    this.client = new Client();
    this.order = inOrder(this.server);
    client.setServer(server);
  }

Un premier test vérifiera que le client appele bien les différentes méthodes dans l'ordre prescrit par le protocole

previous

  @Test
  public void clientProperlyClosesAResourceOnServer() throws Exception {
    when(server.read()).thenReturn("data");
    client.update("toto", "datum");
    order.verify(server).open("toto");
    order.verify(server).read();
    order.verify(server).write("datum");
    order.verify(server).close();
  }
On peut noter que l'on touche là une limitation de la bibliothèque (ou de ma compréhension d'icelle): ce que je voudrais pouvoir dire, c'est qu'entre un open() et un close(), l'ordre des read/write importe peu du point de vue du protocole. La critique d'une surspécification des tests n'est donc pas totalement infondée, puisque l'on m'impose de préciser l'ordre de mon read et de mon write.

Un second test vérifie ce qui se passe en cas d'erreur du serveur signalée par une ServerException. Le client doit impérativement fermer la ressource ouverte avant de relancer l'exception (dans la réalité, on encapsulerait probablement l'exception de bas-niveau produite par le serveur).

previous

  @Test
  public void clientClosesResourcesUponExceptionAndRethrowsIt()
      throws Exception {
    when(server.read()).thenThrow(new ServerException("I/O Exception"));
    try {
      client.update("toto", "datum");
    } catch (ServerException e) { }
    order.verify(server).open("toto");
    order.verify(server).close();

Voici enfin le code du client dérivé à partir de ces deux tests:

previous

public class Client {


  private Server server;


  public void setServer(Server server) {
    this.server = server;
  }


  public String update(String resource, String value) {
    server.open(resource);
    try {
      String old = server.read();
      server.write(value);
      return old;
    } finally {
      server.close();
    }
  }
}

Discussion

Ce petit exemple illustre la philosophie qui sous-tend l'approche par simulacres: la vérification de contrats, d'où l'importance attachée au fait de:

  1. distinguer les simulacres des simples bouchons (stubs);
  2. simuler des rôles (ie. des interfaces) et non des objets.  

Un simulacre n'est pas un bouchon et je ne reprendrai pas l'argumentation détaillée et pertinente de Martin Fowler sur la question.

La notion de contrats, au sens de programmation par contrats, me paraît essentiel et un peu négligée dans toute cette histoire (encore que le BDD n'en est qu'une version moderne). La puissance du simulacre vient de ce qu'il me permet de vérifier la partie cliente d'un contrat, de m'assurer que le Client dans mon exemple respecte bien le protocole du serveur, sans avoir besoin de vrai serveur

Le simulacre n'est donc que la moitié de l'histoire, il faut aussi tester, avec les mêmes exigences la partie fournisseur du contrat. C'est un inconvénient des outils de siumulacres actuels qui impliquent de dupliquer ces tests: un jeu pour le client, un jeu pour le fournisseurs, sans disposer de moyen simple de vérifier la cohérence entre les deux.

C'est pourquoi la remarque de Dominic Williams sur le fait q'un simulacre peut valider un test avec une implantation incorrecte est vraie. Mais elle n'implique pas de devoir jeter le bébé avec l'eau du bain, juste de redoubler de rigueur dans l'utilisation des simulacres.

Sur le plan technique, en Java, les simulacres appliqués à des interfaces simplifient considérablement le code des tests, ce qui répond à la critique de Laurent Bossavit.  

20081219: Testing interactions

While developping various components of our systems, I am trying to spread the use of unit testing within the team, mainly through rotating pair programming with all the teammates. This week, while developing an important feature of the system, we were bitten by some nasty integration bugs. I use the term bug which have been sanctified by usage but of course, the real meaning is error. This lead me to some thought about make interaction testing more systematic.

The Issue

The problem was quite complex, but not uncommon in java enterprise development where there seems to be some untold law about necessary level of complexity. We have three components that are interacting using various protocols:

We developed the three main parts in parallel:

We had an integration session Thursday afternoon where everything was set up and the build process was tuned to produce the right artifacts. We had some issues with cryptographic key handling used in encoding the token, but at the end of the day, integration was done, calls were routed back and forth from the client and we could retrieve some attribute of the Person object.

Things went awry today when we did more testing and tried to polish the interface: incorrect data was transferred between shindig and content server. Tracking the bug, we found that content server and the shindig server did not agree on the JSON structure they exchanged: They lacked common definition of the protocol.

First solution

A first level explanation was that the content server side was not tested enough: There were no unit tests that demonstrated expected JSON structure, something which could have been compared to the tests used in the shindig server. Second level explanation is that there is no consistent definition of the protocol used, ie. no contract definition or demonstration that could be used by both parties.

A common way to deal with this problem is to define some common model structure, eg. class structure in Java, along with predefined mapping functions that could be used to test and transform data on both sides of the wire. This is known, I think, as the Data Transfer Object pattern or Transfer Object, although the use case for DTO is usually to relieve the system from too many network communications.

More generally, it is common in J2EE application to have a model component that defines all the objects and data structures used to exchange data between the components of the system. One major drawback, I think, of this design is that it makes all components dependent ultimately of some concrete implementations of objects. And we all know components should depend on abstractions, not

implementations. EVery modification of the model have potential impacts on numerous components of the system, either at build or run time.

Test based solution

I am not sure this may be relevant to other settings, but in the particular context described, it may be much more rewarding to use a contract checking system than a common business objects layer. A contract checking system would be a test level component that could be used:

A common usage pattern could be the following:

previous



@RunWith(Parameterized.class)
class ConsumerCorrectlyHandlesResponsesTest {


   @Parameters public static Collection<String> jsonPersons() {
      return new PersonTestProvider().encodedData();
   }


   CorrectDataServiceTest(String responseBody) {
      this.server = mock(Server.class);
      this.response  = new HttpResponse(responseBody);
      when(server.request(anyObject())).andReturn(this.response);
      this.service = new Service(server);
   }


   @Test
   public void serviceHandlesData() {
      service.doService();
   };


}


@RunWith(Parameterized.class)
class ProducerCorrectlyGeneratesDataTest {


   @Parameters public static Collection<Object[]> personsAndEncoding(){
      return new PersonTestProvider().decodedData();
   }


   ProducerCorrectlyGeneratesDataTest(Person person, String encoding) {}


   @Test
   public void producerCorrectlyEncodesData() {
      assertEquals(this.encoding,producer.encode(this.person));
   }
}

The idea is that the consumer and producers should not be linked by an implementation but by tied by a contract which is enforced at the testing level. It is up to the test module to provide all the evidence that both sides respect their part of the contract. This would promote lose coupling as no runtime dependency would be needed between the two modules.  

20081216: Automating tests - Part 3

Functional testing

The number of functional tests is slowly increasing, but while they are rather easy to write, thanks to WebDriver's API we find it is  hard to maintain them, as the underlying pages layout is constantly changing and even the flow of control is changing. As an example, we have a WelcomePage on which it is possible to create some item. This reation used to take the user back to the item created in view mode, which meant user had to take some action to edit the item:

previous

    WelcomePage page = new WelcomePage(..);
    NewsEditPage news = page.createNews(title,description).edit();

This fragment is extremely frequent in the code (and should probably be factored out), and when the logic changed so that creating an item lead user to WelcomePage, we had to change all of our tests:

previous

    WelcomePage page = new WelcomePage(..);
    NewsEditPage news = page.createNews(title,description)
                            .lookupNews(title)
                            .view()
                            .edit();

Then everything changed again and creating a news lead user directly to edit mode:

previous

    WelcomePage page = new WelcomePage(..);
    NewsEditPage news = page.createNews(title,description);

I am conscious this is a tribute we are paying for sloppy refactoring and duplication of the code. Such a common pattern should be factored out in a method.

Another issue we are running on is the classical problem of timeouts: It takes time for the browser to display its pages and this may lead to randomly failing tests. WebDriver offers a waitFor() method that waits for the presence of some element in the HTML.

Unit Testing JS

I managed to wrap jqUnit tests within JUnit3 style test cases. The code of the TestCase subclass is very simple:

previous

    private final String testHtml;
    private static final String basedir = System.getProperty("basedir", ".");


    public JQUnitTest(String testHtml) {
        this.testHtml = testHtml + ".html";
    }


    @Override
    public void run(TestResult result) {
        HtmlUnitDriver driver = new HtmlUnitDriver();
        driver.setJavascriptEnabled(true);
        File f = new File(basedir);
        driver.get("file:///" + f.getAbsolutePath() + "/target/test-classes/"
                + this.testHtml);
        List<WebElement> tests = driver.findElements(By.xpath("//ol[@id='tests']/li"));
        for (WebElement element : tests) {
            JQTest jqTest = new JQTest(element);
            jqTest.run(result);
        }
        driver.quit();
    }

The JQtest wrapper class is:

previous

  public class JQTest implements Test {


        private final WebElement element;


        public JQTest(WebElement element) {
            this.element = element;
        }


        public int countTestCases() {
            return 1;
        }


        public void run(TestResult result) {
            result.startTest(this);
            if (element.getAttribute("class")
                    .contains("fail"))
                result.addFailure(this, new AssertionFailedError(
                        element.findElement(By.xpath("//li[@class='fail']"))
                                .getText()));
            result.endTest(this);
        }


        @Override
        public String toString() {
            return element.getText()
                    .toString();
        }
    }

THe basic idea is to reconstruct Test instances and populate a TestResult while scrapping an HTML page which executes jqUnit tests. I use HtmlUnitDriver with javascript enabled to speed up things.  

20081213: Installing happs-tutorial

I am trying to install happs-tutorial as per the instructions given on the site. Installing the tutorial from cabal works fine, except the package is not up-to-date. So I downloaded its sources and try to compile it, using 

runghc Setup.lhs configure

Surprisingly, it refuses to build arguing that its dependencies are not satisfied. The required packages are however installed (with cabal) and available when running 

ghc-pkg list

I tried to reinstall cabal-install from sources, which works fine, but does not solve my problem. Trying a simple 

cabal install

in the happs-tutorial tells me that it requires base >= 3.0.3.0 which, i suspect, is provided by a greater version of ghc than mine (6.8.2 on ubuntu). So I just downloaded the sources and tried to compile them by hand.

Compilation of ghc-6.10.1 worked like a breeze with the usual configure/make/make install.

While compiling happs-tutorial, I ran into an error compiling Crypto-4.1.0. There was an error message about trying an option for GHC, so I went to the cached archive package, unpacked it, added the option to the cabal file and installed it.

Then everything ran fine.

20081212: Automating tests - Part 2

Functional testing

Virtual framebuffer

I added the capacity to drive tests within an X vritual framebuffer server, which allows running the tests on the server side without needing screen or a real server. This is done with the maven-selenium-plugin and some additional configurations within the webdriver based functional tests:

Of course, all this works only for Linux/Unix based systems. I had some issues with the authentication stuff, something that also bited me on an earlier project. I should learn how this works really on X servers.

On the webdriver mailing-list, Nat Pryce pointed at the recordMyDesktop utility that can be used to record a movie of the test session and display it should it fails.

Writing test cases

Writing test cases with webdriver is quite easy. I just have had issues with handling multiple windows, something that should be solved in next releases of webdriver but is now part of the FirefoxDriver interface only.

The Page Object pattern was quite easy to grasp by all the team and produces really easy to read test cases, with all the technical stuff hidden inside the pages logic. At present, we have problems with page design consistency and robustness: identifiers are not used everywhere they should be (ie. where they could be used for user actions or as test handles), and sometimes are not even unique in a page (!). This means the mapping between high-level actions (Page objects interfaces) and webdriver actions is brittle as we have to rely on complex xpath expressions or even xpath plus code navigation.

For the first time this morning, all the 10 test cases passed using a pristine copy of the sources (not on the CI server with Xvfb however). This may sound a bit ridiculous, but that's really a big step for the team and should improve confidence in the stability of features which is presently a major concern.

Unit testing JS

For the JS code, I investigated moving from the current jqUnit based tests to jsUnit (the Jörg Schaible one) as the latter support a maven2 plugin. But I resolved to stay with jqUnit as the jsUnit plugin uses only a Rhino based environment for its execution.

Given my recent experiences with webdriver, I will write a small JUnit wrapper that shall do the following:

Or maybe a single test method for each page delegating to a private helper would be much simpler, with the firefox launching in @Before/@After methods.

Collecting test results should be simple with webdriver: wait for the p.result object to appear and collect inner spans containing informations about the run.

20081204: Automating tests

Started to work on test automation. I did 2 experiments:

  1. unit testing javascript code for some UI element,
  2. functional testing the application.

Unit testing

The system we are developing uses an OpenSocial server and proposes OpenSocial gadgets to its users, who can place them on their own sites and pages. The first gadget to be developed is a turnover report for the identified user. In short, this gadget does the following:

This gadget existed and I proposed the developer to write unit tests for it while refactoring the code. The latter was completely embedded within a <[CDATA[ tag in an XML file that is a specialized format used by the gadget container to produce desired output within a page (ie. using Shindig open-source OpenSocial server).

First step was simply to extract the javascript to a single file, so that we can test it in isolation. Then we created the necessary elements for unit testing this code using jqUnit, a small unit-testing framework based on jQuery:

Our first test was simply reading a single line data file, but it entailed quite a lot of refactoring, something that is not very comfortable to do when you don't have a sound test harness at your disposal. Yet this was necessary just to be able to test things in isolation because things were deeply entangled: loading the data, parsing the file, formatting the data.

With 5 or 6 tests, we were able to remove quite a lot code, creating subsidiary functions in the process and removing a lot of duplication accumalated through various layers of copy-paste.

A few more tries were necessary to reinject the modified javascript into the original context and do some (minor) functional testing. I plan to integrate those tests into CI using WebDriver.  

Functional testing

Thanks to the list feedback from a previous question, I started writing functional tests using WebDriver. I do not want yet to add another layer of language, so I discarded using ruby based frameworks just because our current technology stack is complex enough.

On the technical side, the setup is quite standard. We use maven as our build system and I created two side projects to the main modules comprising the system:

The data module is updated by the developers on a regular basis to provide data according to expected features.

The it module is where everything happens. Using a combination of maven-exec, maven-dependency and surefire plugins, it does the following:

There is one missing piece: starting/stopping a virtual X server so that running the tests on a build server works fine. I am working on this, which is just a matter of configuring the selenium-maven-plugin.  

The latter are plain jUnit tests encapsulating webdriver usage.  I followed the Page Object pattern detailed on the project's wiki to create high-level tests exposing business logic rather than detailed click and type interactions. Each state of the possible dialogue with the user is a distinct class offering methods consistant with the possible actions of the user at that state. For example, the starting page is simply WelcomePage from which a user can login providing a name/password combination. login is a single method that takes 2 parameters, the page taking care of injecting those values into the required fields and triggering form's submission through clicks

An example of this kind of test is: previous

    @Test
    public void canCreateNewsWithComplementaryFields() {
        NewsPage news = accueil.createNews("Some news", "some description");
        news = news.edit()
                .champsComplementaires()
                .optionalFields()
                .chapeau("chapo")
                .encadre("encadre")
                .contact("contact")
                .save();
        accueil.reload();
        news = accueil.navigateTo(news.getHref());
        NewsEditPage editChampsComplementaires = news.edit()
                .champsComplementaires();
        assertEquals("chapo", editChampsComplementaires.chapeau());
        assertEquals("encadre", editChampsComplementaires.encadre());
        assertEquals("contact", editChampsComplementaires.contact());
    }

I like to use Fluent Interface style which is examplified in the sequence of actions above for editting and setting various fields in a form. With some massaging, it could be possible to produce mere-mortals readable scenarios from those tests.  

Tests are run as part of 2 deployments tasks:

  1. the continuous build deployment that produces a new version of the system for each commit,
  2. the customer test deployment which is built on demande by the customer representative, basically when a story is done-done.

The dark side

We have just begun setting this process up, and while the developers are rather enthusiastic with the idea, the tests are still broken most of the time. This is something I have already seen on other projects: there is an initial rampup period for CI to really root itselfy at the heart of the developers work-cycle, so that breaking the build becomes something else than an annoying email from a robot.

Usually, it takes time for things to settle but there is always a point, at least in my experience, where the builds stay green most of the time and everybody cares to fix failing tests, so I am not worried much about that.

20081208: Forum PHP 2008

Ma session

J'avais une session acceptée au Forum PHP cette année, intitulée Mon Javascript est Plus Gros que le Tien et dont on pourra trouver les sources ici. Je ne connais pas du tout PHP, mais l'AFUP a souhaité ouvrir le forum cette année, au moins lors de la première journée, et comme j'ai l'honneur et le plaisir de connaître Perric Penet, membre du bureau  de l'AFUP et créateur de No Parking, celui-ci m'a invité à soumettre cette session dont il avait pu voir une version lors de XP Day Paris 2008.

L'objectif de cette session est d'illustrer quelques techniques de programmation peu ou mal connues, et de mettre en valeur les possibilités du javascript:

La (petite) salle était bondée, nombre de personnes étaient debout dans le fond. J'avais réquisitionné une chaise pour pouvoir inviter un participant à être mon co-pilote, ce qui fût fait et a permis par ailleurs d'illustrer, ne fusse qu'incidemment, la pratique du binômage.

La session s'est bien déroulée, très bien même selon mes impressions (j'ai eu quelques retours très positifs par la suite, j'attends avec impatience le compte-rendu général). J'avais bien répété et repris, sur les conseils avisés de l'équipe de No Parking, mon idée initial de centré la session sur un Kata, avec un minimum de dispositif statique. Nous sommes presque allé jusqu'au bout du kata, nous arrêtant au seuil de la fonction checkWith étendant jQuery. Il aurait fallu une demi-heure supplémentaire pour le finir complètement.  

La conférence

Je n'ai vu que deux autres présentations:

La première était de fort bonne facture: un support basé quasi exclusivement sur des photos légendées, un discours fluide et structuré, un propos intéressant. Z.Greant compare les logiciels libres à la Loi, considérant qu'aujourd'hui les logiciels régissent de plus en plus notre vie (les Literate Machines du titre) au même titre qu'un code de loi. Et une société est d'autant plus libre que tous ses membres connaissent et approuvent les lois qui la régisse. La plus grande part du support est consacrée à l'histoire des progrès de notre connaissance de ces lois, depuis la première stèle d'Hammourabi jusqu'au net, en passant par John Wyclif et la traduction de la Bible en anglais. Il conclut nécessairement en la nécessité de promouvoir les logiciels libres et la mise à disposition générale du code source de tous les logiciels qui deviennent un constituant indispensable de nos libertés civiles.

J'ai particulièrement apprécié une image qui compare deux nuages de mots: celui du code d'Hammourabi et celui de la déclaration universelle des droits de l'homme. On voit très bien la montée en abstraction faite en 4 millénaires, l'un se préoccupant de détails matérielsn, de propriété, de commerce, de distinction de statut, quand l'autre évoque essentiellement des droits universels, des entités abstraites et générales.

La seconde présentation était nettement moins intéressante, et pour tout dire assez soporiphique. A la décharge des orateurs, c'était le début de l'après-midi et j'ai peut-être souffert de la baisse de vigilance induite par la digestion. On en est resté à des généralités sur les processus de déploiements et l'historique de PHP à la BNP, sans identifier réellement des avantages ou inconvénients, mesurer des gains en terme de souplesse et/ou de performance, synthétiser des points de vue des équipes.  

Il semble que la répartition des salles n'ait pas été optimale, la petite salle prévue pour des ateliers plus intimes se retrouvant régulièrement bondée au point de refuser du monde. Probablement y avait-il une palette trop étroite d'offres: deux sessions en parallèles pour étaler le public sur ses divers centres d'intérêt.  J'ai eu d'ailleurs l'impression que, si l'AFUP souhaitait donner à l'événemnt une dimension plus business, la majeure partie du public était des geek.  

L'after

Je suis parti très tôt pour faire le tour des librairies (Le Monde en Tique, Eyrolles et Gibert) et aller au dojo. Sur la lancée, nous avons refait en randori le kata que j'avais présenté le matin, la plupart des participants étant très intéressés par le javascript. Puis Emmanuel et moi nous sommes dirigés vers l'after de la conférence, dans un bar sympathique du 10°, où j'ai eu l'occasion de discuter avec Gilles Vanwormhoudt qui faisait aussi une présentation le lendemain.

Chose intéressante, dans le cadre de son projet Cocoa, il a développé une interface permettant de scripter un (méta)modèle avec du javascript.  

20081202: Saff Squeeze

Thanks to Uncle Bob's Blog, I learnt a new testing technique: the Saff Squeeze. The idea is to isolate a bug through progressive inlining from high-level to low-level method calls that make it manifest, until one finds the smallest method that foots the bill.

Assertions are moved and adjusted during the progressive inlining to prune irrelevant chunks of code. IDE support to inlining helps of course.