Three levels for managing software

Three levels for managing software

We may deal with solutions to various problems in software. These solutions, at various levels of achievement, may be considered in different ways. It is interesting to clearly distinguish the the three main branches or levels for managing software: representation, reasoning, and execution. We can have representation only. When we have reasoning or execution, this is based on a given representation.

1. Representation.  Pure representation may be used for rendering of the information on external devices for independent external interpretation.
2. Reasoning. Reasoning is based on a given representation. It is mainly a process of questions and answers. For example one may wish to get the static call matrix for the methods of a program.
3. Execution. Execution may be seen as the strongest form of operation on a given represention. The most known situation is program execution where the program usually corresponds to a  tree representation. The classical public static void main allows to give all information for launching the program to the virtual machine.

Between the levels of Reasoning and Execution, we may find some intermediate specific forms, for example animation, operationalization or enactment. Operationalization is the process of defining a fuzzy concept so as to make the concept measurable  and to understand it in terms of empirical observations.  The term enactment corresponds to the situation when people act they bring structures and events into existence and set them in action.

More technology adoption curves

Akira Tanaka nicely extended my Mgram request by adding “Service Oriented” and “Enterprise architecture”:

@JBezivin How about this? http://goo.gl/374Sk—
Akira Tanaka (@atanaka) December 20, 2010

The lesson I draw from this data is that “model driven engineering” and “service engineering” are approximately following the same increase rate.

But I am also impressed by the important grow of the “enterprise architecture” subject.

Of course this is only one of the possible indicators of technology adoption, just only one.

Thanks Akira

Interpreting technology adoption curves

Interpreting technology adoption curves

Following the current Google Ngramming fashionable trend, I started to look at the evolution of references to some technological buzzwords. I started by drawing the comparative curves for « object oriented » and « model driven ». The result was quite interesting:

There are many interpretations of this curve.  I propose the following ones:

1. Object technology is now mainstream and thus explicit references are decreasing. Object technology started in 1980 and had an initial development until 1985, and then a much more rapid one until 1995, followed by a slower decrease.
2. Model driven engineering is gaining momentum since 1996 and steadily increasing its apparent impact.
3. The development rate of object technology between 1985 and 1995 was much more rapid (steep curve) than the later development of model driven engineering in the 1996 to 2006 decade.
4. So the two phenomenon are not comparable. Probably this is due to the fact that there were some obvious killer applications in object technology that triggered the rapid development just before 1985, while this is  not so obvious with model driven engineering.

I am interested by any other observation.

University courses on programming for Mobile platforms

Following the previous post on new software engineering courses, I am getting some news on US Universities that are offering courses for programming mobile platforms like iOS (Apple) or Android (Google). Jeff Gray has mentioned the following:

• Stanford (Paul Hegarty)
• Virginia Tech (Jules White)
• Vanderbilt V-MAT group
• Columbia University
• MIT (Hal Abelson)
• Columbia University
• University of Maryland (Adam Porter)
• Carnegie Mellon – Silicon Valley( Tony Wasserman) - (Tony gave recently a really neat talk at FSE FOSER )
• Harding University (Frank McCown)
• University of Alabama (Jeff Gray)

I do not know European universities that are engaging in such specific courses, but probably there are some.

There is an interesting reference to Hal Abelson on this subject. I cannot resist quoting him:

“Mobile applications are triggering a fundamental shift in the way people experience computing and use mobile phones. Ten years ago, people “went to the computer” to perform tasks and access the Internet, and they used a cell phone only to make calls. Today, smartphones let us carry computing with us, have become central to servicing our communication and information needs, and have made the web part of all that we do. Ten years ago, people’s use of computing was largely dissociated from real life. With the ubiquity of social networking, online and offline life are becoming fused.”

It would be difficult to express this evolution more accurately.

On a related subject, some comparative information on iPhone and Android development environments is becoming available:

Comparing the iPhone and Android development environments http://j.mp/aaKWqC—
guillaume (@gsempe) November 21, 2010

New software engineering courses

Developing applications for mobile devices

Software engineering is rapidly evolving. A huge amount of the software that is developed today is produced as a set of so-called applications for App Stores.

Only in the Apple App store there is currently more than 300,000 applications available [ref] . When we take into account the Android market (170,000), the Windows phone market, the Nokia market and all the emerging platforms for mobile devices [ref], this corresponds to a huge software production silo. This is happening very rapidly since in July 2008 there were only 500 applications in the Apple App Store. The development curve on the Wikipedia site looks exponential.

I have several questions about this rapid evolution, and among them:

• What are the programming languages used for these development? Java? Objective-C?
• What is the average size (in LOC or in number of classes) of an application?
• What are the methods (if any) used to develop these applications?
• Are formal methods used?
• Are the requirements explicitly represented?
• How is the maintenance of these applications organized?
• What does the total development effort of all these applications represent in the total software development effort in the world?
• What is the status of the application developers (employees of big companies, individual programmers, etc.)?

I do not know the answers to these questions, but I have the impression that this phenomenon is going to deeply impact the way we consider software production. I know several companies (in the bank domain for example) that are already changing the vision of their information system architecture to integrate this.  They are looking at their old legacy application portfolio to see how this could evolve to accommodate new applications written for mobile platforms. The old client-server organization is impacted by the mobile-equipped client. The way they internally produce software or acquire software (applications) from external vendors after adequate testing is also impacted.

We have already seen several interesting projects in the Eclipse modeling area targeting the production of mobile applications. Of course this is mainly Android since Apple forbid such practices (applications have apparently to be developed natively in Objective-C and cannot be generated).

One interesting observation is that the universities that have always been in advance in the domain of software engineering are today recognizing that the application production for mobile devices requires different skills and different methods. As a consequence they are rapidly opening specific courses for mobile application development. If we want to see which universities are most innovative, most open on their environment, most adaptive, we may look at those that are opening such courses. Apparently the first to take such initiative was Stanford university.  Today they are sometimes offering free online courses.  And they seem to have a lot of success as reported by many Twitters:

New blog post: Stanford – iPhone App Dev. Learning Notes (2) http://is.gd/hwP54 #iOS—
Wei CHEN (陈炜) (@weichen815) November 21, 2010

Really enjoying the Stanford U series on iOS development by Paul Hegarty. I'm on lecture #4 so far. http://bit.ly/b8tGNb—
Evan Lenz (@evanlenz) November 21, 2010

Latest iTunesU course on iOS from Stanford is much better. http://tinyurl.com/2ds5kn2 Big improvement over last semesters instructor.—
John Olson (@yapiodesign) November 21, 2010

Attended my first virtual class at Stanford on iPhone/iPod Touch :: iOS development ! very excited to develop my first app…IIMB MBA dev —
  (@parashar) November 20, 2010

Really digging the Stanford lectures on iOS dev (http://bit.ly/bwhtgA) Definitely covers material in depth!—
Ryan Wersal (@ryanwersal) November 20, 2010

Fantastic! RT: The Fall 2010 Stanford course 'Developing Apps for iOS' is on iTunes http://bit.ly/adYiA8 (via @applespotlight)—
  (@danielwu) November 18, 2010

Paul Hegarty's iOS/iPhone/iPad dev course on Stanford is MUCH BETTER than the previous semester courses http://bit.ly/bwhtgA—
Nitesh Dhanjani (@nitesh_dhanjani) November 14, 2010

etc., etc.

I would also be interested by other answers, like:

• Are these courses targeting only iOS, or Android or are some of them platform-agnostic?
• What are the universities that are opening such courses?

From Programs to Mograms

From Programs to Mograms [1]

Changes in Software Engineering

The apparently smooth and constant evolution in software production and maintenance practices, over nearly half a century, may be hiding some forthcoming radical changes, more qualitative than quantitative. There are some indications that we are currently crossing some new frontiers in technology and practices and it may be helpful to make these situations explicit. Several initiatives like SEMAT initiative comes out very timely to provide an open and rich discussion forum where these issues of change management may be identified and their possible consequences on the future of software engineering discussed. Three important past transitions may be remembered and these may help understanding the possible impact of more recent and even future changes.

Past Ruptures

We have already seen several ruptures in the young history of software engineering. The first one was discussed at the Garmisch NATO meeting in October 1968 (the copy above that I preciously keep in my office corresponds to the second meeting, one year later). The emergence of complex systems obliged to realize that the period of the individual (and isolated) programmer was over, and that the target was “large programming systems of more than 30,000 instructions, produced by more than 25 programmers, in more than 6 months’ development time, with more than one level of management”. The move from individual programming to collective software production was then initiated. Recent interest in agile programming shows that the debate cannot yet be considered as completely closed, but one of the indirect consequences of this NATO meeting was probably the recognition of the importance of team issues, process issues and software architecture issues. The work done later on “programming in the large vs. programming in the small” (DeRemer, Kron) may also be linked to this influence and one may notice the ubiquity of language issues in all these discussions (mainly specification, architecture and process languages).

A second important rupture could be observed in the early 80’s with the paradigm change from procedural to object-oriented programming. The analysis of this move to object-oriented technology is complex and has many possible interpretations mentioned by several authors (Dahl, Nygaard, Kay, Meyer, etc.) like:

• The inadequacy of the procedural paradigm to easily describe real world situations,
• The inadequacy of the procedural paradigm to allow large scale software reusability,
• The inadequacy of the procedural paradigm to build stable software architectures.

Here again the importance of language issues (Smalltalk, Eiffel, Objective-C, C++, etc.) appears central in raising the abstraction level.

A third rupture was triggered by the OMG MDA initiative in November 2000. The initial motivation was the need for portability of information systems in regard to the rapid evolution of technological platforms (CORBA, DotNet, XML, etc.). Some ideas suggested by the OMG in this MDA vision were again about raising the abstraction level with the help of a family of languages named “modeling languages” by opposition to “direct programming languages”, and the corresponding impact on:

• The use of these abstract modeling languages to describe business and application systems independently of the underlying technical platforms,
• The use of transformational techniques to map these business systems on various present or future platforms,
• The separation and composition of facets (e.g. platform-dependent and -independent facets) by various domain-specific modeling languages.
• The handling of most interoperability issues at the language level (abstract syntax or metamodel) and not at the middleware level.

Present and Future Ruptures

These three examples have contributed to shape the software engineering landscape that we know today. However new important ruptures may be arriving, for example:

• The emergence of the end-user programmer as a key actor and the redefinition of the role of the professional programmer (an Excel user may be seen as an end-user programmer when producing spreadsheet content for example),
• The emergence of multiple Domain Specific Languages (DSLs) as an alternative to general purpose languages for handling specific tasks in specific contexts,
• The rapid arrival on the market of huge amounts of software applications in so-called application stores (Apple, Google, etc.) not only limited to smartphone users,
• The increasing diversity of technological solutions and the need to make them interoperate at a reasonable cost in a continuum integrating legacy assets[2] and more advanced solutions (e.g., cloud computing),
• The need to constantly adapt and even synchronize the information systems to more and more rapidly evolving business organizations.

The question is now to suggest an ambitious new vision for software engineering in the 21^st century, encompassing theory and practice and able to cope with the aforementioned ruptures and additional ones that will certainly show up in the future.

We know that the number of applications that will have to be produced in the next 30 years for individual, professional or social needs will be in exponential grow. We know that the total number of professional programmers in the world going to implement them is likely to be stable or in a very slow linear increase. We learnt from the past that whatever progresses we may achieve in software tools, the productivity of software professionals to develop applications will not improve by a major factor. The equation seems thus impossible to solve if we stay in the same context. We need to be imaginative to find realistic solutions to this apparently impossible equation. The key to solving this problems probably lies in the collaboration between professional and end-user programmers, but do we need to invent a specific software engineering set of practices for end-users?

So there will be a huge number of rapidly evolving interacting applications, often written by different people in a number of different languages, and usually not developed with classical software development life-cycle. Each of these applications has a specific data model, state model and event model. The result is high fragmentation and the need to make heterogeneous parts interoperable has never been so acute. We need paradigm-agnostic integration methods that will allow coping with language coordination and large-scale application interoperability.

Plus ça change, plus c’est la même chose
(Software Engineering is Language Engineering)

In software engineering also, the more things change, the more they remain the same. The invariant that may be noticed in practices and technology ruptures is that they are very often related to language issues. The only new thing is that the definition of language has been extended. Textual languages stay very important but many kinds of visual languages are appearing and some languages offer an abstract syntax with various concrete syntaxes, sometimes textual, tabular or visual. These languages are defined not only by grammars, but also by metamodels, schemas, ontologies, etc. The application scope of these languages will be considerably extended, mainly to specific tasks or domains. Most of these languages will allow capturing facets that will have to be combined with other facets to be meaningful. Programmers (professional or end-users) will use these languages to write programs, or terminal models, or domain code or “mograms”. Precision of these mograms is of paramount importance even if very often they are not executable. Computer executability is not necessary to achieve precision. Languages will allow describing products as well as processes, code and data, requirements, specifications and algorithms, test, architecture, etc. Applications will be built with a number of different languages, usually more than ten or twenty for each. More important, the evolution and maintenance of these applications will have to be carried on with all these languages.

The practical and theoretical core of software engineering is language engineering. But we need to revisit the new scope of definition, implementation and usage of languages in a more general way than the old general purpose executable grammar-based textual programming languages.

Model Driven Engineering (MDE) is one of the proposals that are currently on the table as a proposed core contribution. MDE tries to imagine a new set of practices to produce and manage precise mograms, conforming to open and evolving metamodels, within a network of precisely related artifacts. Milner’s vision in his “Towers of Models” grand challenge sets up a very exciting and ambitious target for a new software engineering kernel that could be built from MDE. In this perspective, more than inventing new languages, the difficulty will be in establishing precise and operational semantic relations between all these DSLs that will constitute the core of the new software engineering practices for the 21^st century. Language interoperability will anyway be an important part of the corresponding landscape.

---

[1] Mogram is a contraction of “model” and “program” suggested by A. Kleppe. A mogram is not always computer-excutable, but it is nevertheless precise and conforms to a given metamodel.

[2] Making the future interoperate with the past will be an important software issue for this century, considering not only the COBOL, but also the Java legacy that is being accumulated. Many applications will not been built from scratch, but by evolution of a previous legacy.

________________________

This post is based on a contribution I made to the SEMAT initiative in March 2010.

From Unified Method to Unified Language

Products and Processes in Software Engineering

Here is a diagram illustrating the history of UML.

When we look at this picture, one remark comes immediately to mind.

Why this change, between 1995 and 1996 from “Unified Method” to “Unified Language“? The reason is that, at this period people realized that there was no possibility to propose a unique software development method. The goal of the OOAD at OMG was not only too ambitious, but is was an impossible mission.

There is an infinity of possible software processes answering the question of who is doing what, when, how and why in the software development and maintenance world.

The decision of OMG to reduce the ambition of this initiative was most clever. The goal was then reduced only to define a language for describing a subset of software artefacts. The name was chosen to be Unified Modeling Language. And the discussions on process description were ignored and delayed for further studies. Only when UML got stabilized in its first versions, people at OMG came back to the problem of process description and this effort finally produced the SPEM contribution (Software Process Engineering Metamodel).

This move was not a surprize and corresponded to the classical separation between process and product in most advanced engineering disciplines (think of chemistry for example).

But the language to describe software artefacts (UML) and the language to define software processes (SPEM) could not be completely independent. The idea was thus to consider them as two languages conforming to a common language as a matter of fact a metalnguage. And this metalanguage already existed at OMG: the MOF (Meta Object Facility).

This is not sufficient to make explicit the relations between SPEM and UML, but at least having both languages expressed in terms of a common metalanguage should facilitate in the future the precise expression of their relations.

Towards Reproducibility of Scientific Results in Software Engineering

Open Source, Open Data, but no Reproducible Scientific Results?

There is a very interesting paper in the preface of the last SPLASH/OOPSLA conference by Martin Rinard from MIT:

OOPSLA/SPLASH 2010 Program Chair Statement http://bit.ly/d7OcK7 (pdf, p.iv)

Among the thoughtful reflections by OOSPLA/SPLASH program chair on conference reviewing processes, one remark of particular interest is about the serious lack of reproducibility in the software engineering community. Quoting Martin Rinard about the SPLASH/OOPSLA reviewing:

The ability to reproduce the presented results is a fundamental property of scientific publication. Given the complexity of the systems that many computer scientists work with, the fact that many of the components in such systems change rapidly (with older versions quickly becoming effectively unavailable), and the strict page limits that conferences impose, reproducibility has always been a problematic issue for the field [5]. Specific issues include whether or not the paper presents sufficient detail for another researchers to reimplement the system (almost all do not, given the complexity of the systems involved), whether the researchers make there implementation issue moot by making their data and/or artifacts publicly available via dissemination mechanisms such as the Internet (many but by no means all researchers do this), or whether, given the speed with which the field moves, reproducibility is even a relevant goal or not for most papers. The program committee did not attempt to address this issue in any detail other than to simply apply prevailing reviewing standards.

One issue did, however, come up during the program committee meeting. The computer science research community in general, and the programming languages/software engineering community in particular, currently has strong participation from communities (for example, researchers working in industrial research labs) that often work with proprietary software systems. Because such systems are not available to the broad research community, it is in general not possible for others in the community to reproduce the reported results — the data and/or artifacts required to reproduce the results are not available (and will never become available) to others. While the program committee was not entirely comfortable accepting papers that present such results, it was at least as uncomfortable with requiring industry researchers to present only results that others could reproduce.

Given the central importance of reproducibility to scientific inquiry and the current ambiguous status of reproducibility in the field, I think it is important for the field to come to amore explicit understanding of what degree of reproducibility is acceptable. In particular, I believe the field needs to come to a decision on the acceptability of results that others are inherently unable to reproduce because of the proprietary nature of the relevant data and/or artifacts.

[5] T. Mudge. Report on the panel: How can computer architecture researchers avoid becoming a society for irreproducible results? Computer Architecture News, 24(1), 1996.

This is finely observed and nicely presented, but what can we do in practice to help our discipline reach the level of nearly all other scientific disciplines? Until we take the necessary steps to ensure that self-proclaimed results may be checked against strong and openly available experimental data, we can difficultly claim, in academic committees and circles, that computer science and software engineering has reached the level of maturity of other more mature engineering fields.

I remember a discussion in the PC of a software engineering conference on this subject. Someone proposed that any paper claiming novelty on the basis of experiments should provide a stable pointer to the data that was used to produce the results. Then the discussion moved on how to ensure stability and transparency in accessing these experimental data necessary to show reproducibility of results. There was a claim that the conference organizers (was it ACM, IEEE or another organization, I don’t remember well) could provide this neutral access to stable experimental data. Suprizingly the initial answer of the representatives of the organizers was that they were not opposed at all to organizing this, for example as a service to their members. The funny story is that a small number of people started then claiming this a bad idea because it was unfair. And do you know what their arguments were? This way of doing would be unfair because it would shed suspicion on papers that were not providing the reproducible data to reviewers. It would give a too important advantage to the papers submitted to reviewing together with the reproducible data. The discussion was then stopped.

Our discipline claims respectability because we are using peer-revieving in our journals and conferences. But we all know how fragile this argument is. We will reach the maturity level of other disciplines when we have achieved a sound process for reproducible results in software engineering. There is no excuse in 2010, with the broad web access, the low-cost of disk storage and the knowledge and experience accumulated in the open source and open data communities, to escape this necessary sanity move. All conference organizers or PC chairs should state in their call for papers, how the experimental data for non theoretical could be stored and retrieved.

The good news is that some colleagues in the community are already organizing themselves for a move in the good direction.

Pieter Van Gorp and his colleagues for example are already organizing the campaign for reproducible results in software engineering. Read their paper on “Supporting the internet_based evaluation of research software with cloud infrastructure” at: http://www.springerlink.com/content/y1488178640l6412/

Read also Pieter’s answer to the “Where are the zoos” blog at: http://modelseverywhere.wordpress.com/2010/11/06/where-are-the-zoos/trackback/

SEMAT now an OMG activity

The new SEMAT Kernel RFP

On December 8, there will be a presentation of a new RFP at OMG by Ivar Jacobson and Arne J. Berre:

14:00 “A domain-specific language and a kernel of essentials for Software Engineering”, Draft RFP presentation and discussion (Ivar Jacobson and Arne J. Berre)

This is on the preliminary agenda of the Analysis and Design Task Force Meeting,  in Santa Clara, on 8 December 2010.

This means that SEMAT is going one step further to becoming a standard OMG activity by the proposal of a RFP (Request For Proposal). This is good news since it will give more stability and visibility to the SEMAT initiative.

End User Programming

FSE’09  Keynote

The keynote slides of Mary Shaw at the The 7th joint meeting of the European Software Engineering Conference (ESEC)
and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE) August 24-28 2009 Amsterdam, The Netherlands are available at: Keynote

Referencing their seminal work on end-user programming:

C. Scaffidi, M. Shaw, and B. Myers. Estimating the Numbers of End Users and End User Programmers.VL/HCC’05: Proc 2005 IEEE Symposium on Visual Languages and Human-Centric Computing, pp. 207-214, 2005.

Mary shaw gives some figures on the number of end-user programmers: