From Programs to Mograms

From Programs to Mograms [1]

Changes in Software Engineering

The apparently smooth and constant evolution in software production and maintenance practices, over nearly half a century, may be hiding some forthcoming radical changes, more qualitative than quantitative. There are some indications that we are currently crossing some new frontiers in technology and practices and it may be helpful to make these situations explicit. Several initiatives like SEMAT initiative comes out very timely to provide an open and rich discussion forum where these issues of change management may be identified and their possible consequences on the future of software engineering discussed. Three important past transitions may be remembered and these may help understanding the possible impact of more recent and even future changes.

Past Ruptures

We have already seen several ruptures in the young history of software engineering. The first one was discussed at the Garmisch NATO meeting in October 1968 (the copy above that I preciously keep in my office corresponds to the second meeting, one year later). The emergence of complex systems obliged to realize that the period of the individual (and isolated) programmer was over, and that the target was “large programming systems of more than 30,000 instructions, produced by more than 25 programmers, in more than 6 months’ development time, with more than one level of management”. The move from individual programming to collective software production was then initiated. Recent interest in agile programming shows that the debate cannot yet be considered as completely closed, but one of the indirect consequences of this NATO meeting was probably the recognition of the importance of team issues, process issues and software architecture issues. The work done later on “programming in the large vs. programming in the small” (DeRemer, Kron) may also be linked to this influence and one may notice the ubiquity of language issues in all these discussions (mainly specification, architecture and process languages).

A second important rupture could be observed in the early 80’s with the paradigm change from procedural to object-oriented programming. The analysis of this move to object-oriented technology is complex and has many possible interpretations mentioned by several authors (Dahl, Nygaard, Kay, Meyer, etc.) like:

• The inadequacy of the procedural paradigm to easily describe real world situations,
• The inadequacy of the procedural paradigm to allow large scale software reusability,
• The inadequacy of the procedural paradigm to build stable software architectures.

Here again the importance of language issues (Smalltalk, Eiffel, Objective-C, C++, etc.) appears central in raising the abstraction level.

A third rupture was triggered by the OMG MDA initiative in November 2000. The initial motivation was the need for portability of information systems in regard to the rapid evolution of technological platforms (CORBA, DotNet, XML, etc.). Some ideas suggested by the OMG in this MDA vision were again about raising the abstraction level with the help of a family of languages named “modeling languages” by opposition to “direct programming languages”, and the corresponding impact on:

• The use of these abstract modeling languages to describe business and application systems independently of the underlying technical platforms,
• The use of transformational techniques to map these business systems on various present or future platforms,
• The separation and composition of facets (e.g. platform-dependent and -independent facets) by various domain-specific modeling languages.
• The handling of most interoperability issues at the language level (abstract syntax or metamodel) and not at the middleware level.

Present and Future Ruptures

These three examples have contributed to shape the software engineering landscape that we know today. However new important ruptures may be arriving, for example:

• The emergence of the end-user programmer as a key actor and the redefinition of the role of the professional programmer (an Excel user may be seen as an end-user programmer when producing spreadsheet content for example),
• The emergence of multiple Domain Specific Languages (DSLs) as an alternative to general purpose languages for handling specific tasks in specific contexts,
• The rapid arrival on the market of huge amounts of software applications in so-called application stores (Apple, Google, etc.) not only limited to smartphone users,
• The increasing diversity of technological solutions and the need to make them interoperate at a reasonable cost in a continuum integrating legacy assets[2] and more advanced solutions (e.g., cloud computing),
• The need to constantly adapt and even synchronize the information systems to more and more rapidly evolving business organizations.

The question is now to suggest an ambitious new vision for software engineering in the 21^st century, encompassing theory and practice and able to cope with the aforementioned ruptures and additional ones that will certainly show up in the future.

We know that the number of applications that will have to be produced in the next 30 years for individual, professional or social needs will be in exponential grow. We know that the total number of professional programmers in the world going to implement them is likely to be stable or in a very slow linear increase. We learnt from the past that whatever progresses we may achieve in software tools, the productivity of software professionals to develop applications will not improve by a major factor. The equation seems thus impossible to solve if we stay in the same context. We need to be imaginative to find realistic solutions to this apparently impossible equation. The key to solving this problems probably lies in the collaboration between professional and end-user programmers, but do we need to invent a specific software engineering set of practices for end-users?

So there will be a huge number of rapidly evolving interacting applications, often written by different people in a number of different languages, and usually not developed with classical software development life-cycle. Each of these applications has a specific data model, state model and event model. The result is high fragmentation and the need to make heterogeneous parts interoperable has never been so acute. We need paradigm-agnostic integration methods that will allow coping with language coordination and large-scale application interoperability.

Plus ça change, plus c’est la même chose
(Software Engineering is Language Engineering)

In software engineering also, the more things change, the more they remain the same. The invariant that may be noticed in practices and technology ruptures is that they are very often related to language issues. The only new thing is that the definition of language has been extended. Textual languages stay very important but many kinds of visual languages are appearing and some languages offer an abstract syntax with various concrete syntaxes, sometimes textual, tabular or visual. These languages are defined not only by grammars, but also by metamodels, schemas, ontologies, etc. The application scope of these languages will be considerably extended, mainly to specific tasks or domains. Most of these languages will allow capturing facets that will have to be combined with other facets to be meaningful. Programmers (professional or end-users) will use these languages to write programs, or terminal models, or domain code or “mograms”. Precision of these mograms is of paramount importance even if very often they are not executable. Computer executability is not necessary to achieve precision. Languages will allow describing products as well as processes, code and data, requirements, specifications and algorithms, test, architecture, etc. Applications will be built with a number of different languages, usually more than ten or twenty for each. More important, the evolution and maintenance of these applications will have to be carried on with all these languages.

The practical and theoretical core of software engineering is language engineering. But we need to revisit the new scope of definition, implementation and usage of languages in a more general way than the old general purpose executable grammar-based textual programming languages.

Model Driven Engineering (MDE) is one of the proposals that are currently on the table as a proposed core contribution. MDE tries to imagine a new set of practices to produce and manage precise mograms, conforming to open and evolving metamodels, within a network of precisely related artifacts. Milner’s vision in his “Towers of Models” grand challenge sets up a very exciting and ambitious target for a new software engineering kernel that could be built from MDE. In this perspective, more than inventing new languages, the difficulty will be in establishing precise and operational semantic relations between all these DSLs that will constitute the core of the new software engineering practices for the 21^st century. Language interoperability will anyway be an important part of the corresponding landscape.

---

[1] Mogram is a contraction of “model” and “program” suggested by A. Kleppe. A mogram is not always computer-excutable, but it is nevertheless precise and conforms to a given metamodel.

[2] Making the future interoperate with the past will be an important software issue for this century, considering not only the COBOL, but also the Java legacy that is being accumulated. Many applications will not been built from scratch, but by evolution of a previous legacy.

________________________

This post is based on a contribution I made to the SEMAT initiative in March 2010.

Teaching Model Driven Engineering

Four lectures on MDE

I am currently working on a set of four general lectures  on Principles and Applications of  Model Driven Engineering. The content will be evolving, but the general organization is the following:

• Course #1: History and context of MDE

This first course will present the modern history of MDE (Model Driven Engineering) as a specific branch of software modeling. The differences with simulation will be first outlined. A survey of semi-formal notations will first show how the rich variety of these formalisms has been important in the history of computer science (SADT, Flow-charts, State-charts, Petri nets, Entity-association diagrams, to name only a few of them). Unfortunately the lack of unification and automation has quite often limited the utility of these formalisms but they have allowed gaining a good understanding of visual and textual semi-formal notations.  In the long history of contemplative modeling, there were probably some missed opportunities but one outcome was certainly the progressive convergence of Object Oriented Analysis and Design methods towards the consensual and normative definition and use of the UML software artifacts description language.

This UML proposal triggered in fact a lot of innovations in the area of software modeling. Many of these innovations will probably outlast the UML language itself, and are probably much more important. Since UML was not a method, another language like SPEM was necessary to describe the various software development processes. Having two languages was like having an arbitrary number of them: a need for a metalanguage (i.e. a language to define languages) became necessary. The MOF was then defined and the idea of precise characterization of languages by metamodels came then to life, even if there are still much progresses to achieve in this field. Then the evolution of software sytems, for example  when changing platforms, was described as possible transformations from languages more or les dependent on these platforms. This suggestion was made in the MDA white paper in November 2000 and became quite popular. Today MDE may be seen as a generalization of MDA.

Course #2: Theory and Basic Principles of MDE

The second course will present the basic principles of MDE from a conceptual and theoretical point of view. MDE uses typed graphs for most of the representation tasks and rule-based declarative transformations for most of the operation tasks. It relies essentially on the concepts of metamodels and model transformations.

Sometimes called Modelware, MDE is thus a graph-based and transformation based technical space. The notion of technical space will be defined and used to compare similarities and differences with other technical spaces like Grammarware.

Metamodels provide some sort of typing system on top of the models themselves. The essence of MDE is however most related to the unification of abstract models. Terminal models, metamodels and metametamodels are the most important categories of abstract models. Different operations like storage, retrieval and transformation may be applied on all abstract models.

One of the important characteristics of MDE is to represent systems by models. This relation of representation which is central to all modeling activities will be analyzed and discussed.

• Course #3: Applications of MDE

The subject of applications of MDE is vast and rapidly evolving. Usually one may consider three important areas: Generation of software artefacts,  Discovery of models from structured systems and Interoperability between heterogeneous systems. The course will show how these three areas of increasing complexity are constantly evolving.

Another classification of MDE applications considers various domains like health care, automotive, aeronautics, embedded systems, information systems, etc.

An interesting way to look at MDE applications is also through the various forms of model transormations. These transformation may be performed at system design, system maintenance or evolution and at system operation.

The course will also show how the typology of applications may be related to a classification of metamodels.

• Course #4: Migration to MDE

The last course recognizes that there are more than technical factors to characterize the move to MDE in the given context of a company or an organization: we have also to consider the organizational, social and human factors.

Starting from a list of success and failures, this lecture will analyze the process of migration to MDE from a situation where direct programming (typically in Java) is the common practice. Some indications on MDE maturity levels will be sketched. When starting an MDE project, risk factors will be analyzed. Some of these risks may be related to the maturity of the tools being employed, but others may be relaated to the human reluctance to loose some control on the direct production, maintenance or execution process.

References will be made to the migration process from procedural technology to object-oriented technology in the 80′s.  The increased difficulties in MDE come from the fact that the code is no more the reference.

These four courses may be seen as a reorganization of the content of some of the posts presented in this blog.

The impossible equation

Let us define a professional programmer as one who is making a life of programming, for different subjects or domains, usually in a General Purpose Language (GPL)  like Java or C#.

1. Slow increase of professional programmers

The total number of professional programmers in the world is only slowly increasing. The extended capacity of universities and engineer schools is limited and does not overtake the number of professional programmers that are going to retire. The additional number of professional programmers in emerging countries (India, Brazil, China, Russia, etc.) is compensated by the lack of interest by technical programming careers in many industriellay developed countries. 

These trends are likely to stay similar in the coming 25 years period.

2. Rapid increase of applications to develop

The number of application that will have to be devloped in the next years, for professional, social, ludical, individual or other needs is likely to follow an exponential increase. 

3. Solving the impossible equation

An application is mainly made of software and this is not likely to change in the future. Whatever the language used, a professional programmer is able to produce a limited amount of software par day. Since fifty years, the individual productivity of the progrmmers has increased with better programming languages, but in an order of magnitude that let us pessimistic on the chances of making a breakthrough in the coming years, with another technique like automatic programming, program genaration from formal specification or another supposed silver bullet. 

If we draw these two observations on the same diagram, we see that with the population of professional programmers in 2015, it will not be possible to deal with the production of all needed software-based applications.

Massively educating non-programmers into programming techniques means that we may miss these competencies for other important tasks (health care, building, commerce, law and police, manufacturing, etc.). 

The problem is quite serious and will rapidly hit industrialized countries. Outsourcing will not be an acceptable solution. 

4. Interdisciplinarity consider harmful.

One solution could be to develop systematic interdiscipliary education. This means that every professional (chemist, architect, artist, farmer, lawyer, book-keeper, etc.) should develop a competency to use general purpose programming languages as part of her/his education. Then s/he will be able to develop specialized computer programs for her/his corporation, in addition to the day to day work in the specialty. 

This has been tried in some countries where special curicula have been set up (computers and chemistry, computers and management, computer and biology, etc. ). The net result is that people educated in such a way often perform poorly in both computer and the additional discipline. 

5. An old success story.

If we want to seriouly plan for the future, we should analyze the past. A long time ago this kind of situation was already met once. This is nicely recorded in Dan Bricklin’s Web site :   http://www.bricklin.com/. Dan Bricklin invented a language for accountants called Visicalc in the late seventies. The first advertisement for this language appeared in the May 1979 issue of Byte. 

Visicalc came as a nice solution to the needs of many acounting people that had the repetitive task of establishing several similar lines (number of items, unit price, total price) and then a summarizing gran total line.  Instead of asking a professional programmer to develop such a program with the correct formula each time in Pascal, Fortran, Cobol or C, the great idea was then to provide the accounting people with a system in which they would not need this help.

This was a great idea at the time and when we see the current success of Excel (the successor of VisiCalc), we can imagine the catastrophic situation we would face if each excel user were to ask a Java programmer to write a program for the particular task.

This invention was one of the most important in computer science. Only now we start realizint that it introduced a new kind of actor in the system, namely the end-user programmer. On the contrary of the professional programmer, the end-user programmer does not belong to the computer corporation  and has an objective of doing a work in a particular domain  with the tools suited top this domain. In many cases s/he will work with a DSL (Domain Specific Language). Visicalc and Excel are domain specific languages.

6. The importance of End-User Programming

The importance of end-user programming is now widely recognized. See for example the presentation of Brad Myers in http://www.cs.cmu.edu/~bam/papers/EUPchi2006overviewColor.pdf.

Brad Myers in this presentation provides many definitions, as the following ones:

He also gives some figures as the number of actors in the USA:

USA:

90 Millions computer users;

50 Millions Spreadsheet & DB users;

12 Millions self described programmers;

3 Millions professional programmers

These definitions are quite helpful in undertanding the current situation and coping with the problem aforementioned. The only solution to the impossible equation mentioned earlier is to bring end-user programmers in the scope. But of course we cannot ask them to work with professional programmers tools like Java or C#. And this is where the DSLs arrive.

7. DSLs at the rescue.

Building languages for end-user programmer is a difficult task as reported in  the paper “Why Johnny can’t program” (http://www.bricklin.com/wontprogram.htm).

U-Languages or M-Languages?

I cannot resist quoting Stefan Tilkov’s blog : http://www.innoq.com/blog/st/

——————————
” … Once upon a time, I used to believe the OMG shared the vision of modeling based on custom metamodels — after all, what MS now claims as its innovation is something that is very much aligned with the original MOF concept. Sadly, though, with OMG’s ridiculous monster of a modeling language that aims to be everything to everybody, this seems to be no longer the case. (If anybody needs proof that Tim Bray is right in asserting that standards bodies should never invent anything, and is tired of using another well-known monster to do so, UML 2 would be a great candidate.)

So in the end, it seems to come down to MOF in its sort-of-working semi-compatible non-standardized incarnation (EMF), embedded into Eclipse, vs. Microsoft’s proprietary DSL stuff, embedded into Visual Studio. And the strangest thing is: I’d rather place a bet on something that is backed by eclipse.org than on an OMG standard nobody gives a fuck about. Care for another example? Take this.”
——————————

So seems that the old debate on (Unique/Unified/Universal) languages or U-languages is starting again. After the failure of languages like PL/1 and ADA to name only two of them, it seems that what happened on the programming scene is on the verge of happening again on the modeling scene with UML 2.0. [By the way it is funny to notice how old ADA unconditional supporters are easily tempted to back UML 2.0. ]

The OMG camp has always been divided between the tenants of U-languages (UML) and the tenants of M-languages (Meta-Languages, e.g. MOF based). Until now, at the cost of constant compromises and aligments, these views have been cohabiting in the MDA approach.

But today, with the emergence of the very strong DSL trend in industry and research and the clear commitment of Microsoft “Software Factories” into the M-language camp, the cursor seems to move away from the UML 2.0 users and profilers.

This does not come as a surprise since it is only an aligment on what is already happening in the XML technical space. After all, the co-existence of the various XML schemas and DTDs is not apparently such a catastrophic move, at least from the point of view of the users.

Departing from the apparent comfort of huge, unprecise and difficult to specialize monolithic languages like UML 2.0, lead us to the apparent creative and productive environment of multiple coordinated languages. It is quite easy to see what could be achieved with the help of small, well focused and precise DSLs. However we have still many problems beyond of us to solve. The big challenge is how to cope with the fragmentation problem, i.e. the co-existence of hundreds or even thousands of these DSLs. For the time being we have only two ways of coping with this: the existence of a well founded M3-level metalanguage (i.e. the MOF) and the possible forthcoming QVT standard for model transformation language, in the case that this effort finally produces a satisfactory and realistic proposal in 2005 (still to be proved). However, even with a very precise metametamodel and a very well designed model transformation language, we we still have a long way to go before demonstrating the practicality of the approach.