How Many software professional in the United States?

In my last blog, I published the following figure:

One percent seems a lot.  Let’s check this:

In the Bureau of Labor Statistics of May 2011, we find the following:

15-0000 Computer and Mathematical Occupations (Major Group)

Total employment = 3,406,720 as of May 2011

15-0000 Computer and Mathematical Occupations 3,406,720

This covers the following categories:

Computer and Information Research Scientists ; Computer Systems Analysts ;Computer Programmers ; Software Developers, Applications ; Software Developers, Systems Software ; Database Administrators ;Network and Computer Systems Administrators* ; Computer Support Specialists ; Information Security Analysts, Web Developers, and Computer Network Architects ; Computer Occupations, All Other* ; Actuaries ; Mathematicians ; Operations Research Analysts ;Statisticians ; Mathematical Technicians ; Mathematical Science Occupations, All Other

So it is not fair to include  Actuaries ; Mathematicians ; Operations Research Analysts ;Statisticians ; Mathematical Technicians ; Mathematical Science Occupations, All Other.

Let us count only the computer science categories:

15-1111 Computer and Information Research Scientists 25,160
15-1121 Computer Systems Analysts 487,740
15-1131 Computer Programmers 320,100
15-1132 Software Developers, Applications 539,880
15-1133 Software Developers, Systems Software 387,050
15-1141 Database Administrators 108,500
15-1142 Network and Computer Systems Administrators* 341,800
15-1150 Computer Support Specialists 632,490
15-1179 Information Security Analysts, Web Developers, and Computer Network Architects 272,670
15-1799 Computer Occupations, All Other* 177,630

And add them:

25,160

487,740

320,100

539,880

387,050

108,500

341,800

632,490

272,670

177,630

———-

3,293,020

The total population in USA as of January 1st 2013  is estimated to :

I have not found the population in May 2011, but should not be very different.

So I come to the conclusion:

Number of software professionals = 3,293,020

Total population = 315,091,138

Meaning that there are more than 1% of the population in these computer science categories, if my way of counting is right.

Of course we can discuss if some categories mentioned above should be considered as software professional. This is particularly true of 15-1150 Computer Support Specialists.  

According to the way we count them, we may be a bit below the 1% of the population considered as software professionals. But this level may be kept as a broad indication. It has been used by many authors. For example in his NPUC’09 presentation entitled ‘End-User Design and Development” at IBM Almaden research center (July 9, 2009, the future of design and software development), Brad Myers presented the following data on End-User Developers (EUDs), mentioning again this 3 million of professional programmers in US estimated for 2012:

But, the problem is not to discuss this number. It is rather to understand how the 99% rest of the population are using software based systems.

——

One percent software professionals in advanced countries

As a rule of thumb, we may consider that about one percent of the population in advanced countries like USA are computer scientists.

• Can we/ Should we  go beyond this 1% level?

We know that computer scientists do not directly produce goods like farmers, or build houses like bricklayers, electricians, plumbers, carpenters, etc. How far can an advanced society go? Can we imaagine a society with 30% computer scientists? If everybody is just skilled to produce software, then we’ll starve and we’ll have no house to live.  More computer scientists we have in a society, less skilled people remains for other specialized essential tasks like medical doctors, bus drivers, hair dressers, etc.

• Should we educate the remaining 99% so that they also learn a general purpose programming language like Java

Does this make sense? Should everybody be able to program in Java, Python, Basic, Perl, Ruby, Javascript or in a similar language? Should we teach them to all,  from elementary school to college, to educate everybody in such programming languages?

My personal opinion is that we should answer a clear no to both these questions. No we don’t need more than 1% of the population as software professionals. No we should not try to educate the remaining 99% of the population with Java-like languages.

But of course having said this, we face the problem of having to produce in the coming decades an exponentially growing number of new software applications for all the needs of society: personnal, professional, familial, social, etc.  And we know that the productivity of individual prrogrammers will not increase too much in the future. So, this is the impossible equation:

How, with the same amount of people (code producer and managers, about 1% of the total population), to produce an order of magnitude more of software applications than now?

The basic statement of this impossible equation may be sketched as follows:

We need to find a solution to this problem. For me the keywords that may help here are EndUserProgramming, DSLs (Domain Specific Languages, a branch of language engineeing), and Higher Order Generation. Computer scientists should not anymore write “programs that solve problems“, but “write programs that generate programs (that generate programs) that solve problems”.

This will not be easy, but this is the only way to go if we want to propose a solution to the impossible equation.

So, the army of computer scientists (1% of the population) should focus on the definition and implementation of DSLs and the remainder of the people (99%) should be educated in using these DSLs in helping to perform the tasks they are skilled for. This is of course very simplistic and needs to be discussed and refined, but there are many arguments in favor of this solution.

Why programming is much harder than modeling?

Why programming is much harder than modeling?

Yesterday Rafael Chaves wrote on Google+:

1:48 AM (edited) – Public
“There is something confusing about the idea of good modeling being hard. After all, all one is doing is describing how the system is supposed to work without having to worry about the implementation details. If one can’t do that, then how is one supposed to manually create a correct, working system?”
Well put, +Thomas Mercer-Hursh!

https://groups.google.com/d/msg/umlforum/mZkvOQrfLF0/VGMZ26wjAjkJ

Just wrote a post about that on my blog:

http://abstratt.com/blog/2013/02/09/how-can-modeling-be-harder-than-programming/

It’s interesting to see that some people believe that modeling is harder than pure and classical programming. Not only I agree with Rafael that this is not true, but I have the opposite opinion. IMHO, Programming is much harder than Modeling. When you build a program, you are performing (at least) five tasks:

1. Modeling the surrounding world (or the domain)
2. Modeling thee underlying executable platform.
3. Modeling the applicative user needs (application or user requirements)
4. Merging these three models (synthesis) into one that will be executable.
5. Deploying the solution in a real environment.

On the contrary, when you are modeling, you concentrate only on one aspect. The problem of merging models comes only later.

So this is why programming is a noble and difficult art, much more complex than modeling. Programming (a contraction of “program engineering”) is a very complex synthesis of the three models pictured below, and probably much more:

This idea is not new. I remember a guest talk given by Michael Jackson, a long time ago,  at the second OOPSLA conference mentioning it. I was very lucky to listen again to these ideas at the Cambridge celebration of the work of Tony Hoare in 1999 where Michael gave a talk on “The Real World“:

http://mcs.open.ac.uk/mj665/Hoare99a.pdf

In this talk, Michael Jackson refers again to the inspirational work of Brian Cantwell-Smith by borrowing the now classical representation of the central nature of programming:

Looking at this illustration, we understand that the central issue of programming is to build a complex composite model that will bridge the world and the machine, under some difficult constraints.

All this also explains why we should pursue all efforts on software modeling. The most difficult task is merging all models to produce one program. We know two things:

• in specific cases this is possible and has been achieved by simple code generation techniques
• in the most general cases, this is likely to be beyond reach for a long time

But we know also that having clean, clear and rigorous models of the domain world, of the platform, of the application requirements, of the QoS constraints, etc. will make easier the life of the programmer, specially if these models may by automatically processed.

In this view, the OMG tentative to propose in 2000 the MDA(tm) initiative with fuzzy correspondences between CIMs, PIMs, and PSMs may be considered now as a very naïve simplification, not very helpful. But it had the advantage of emphasizing the hard problem of explicitly dealing with the world (domain or CIM/PIM) and the machine (or platform, PSM).

—

YADOMAP (Yet Another Discussion On Modeling And Programming)

YADOMAP (Yet Another Discussion On Modeling And Programming)

There are several issues that are related but not completely equivalent. For example we may separate the two discussions threads:

1. What is the relation between models and programs?
2. What is the relation between model engineering (or modeling) and program engineering (or programming)?

In the first case we only compare mere artifacts. In the second case, the comparison is much more general, about processes and contexts. Of course models are the main artifact of model engineering and programs are the main artifact of program engineering.

There is another interesting dimension that could be considered in this discussion. Model and program artifacts have the particularity of conforming to languages. So we could ask the question:

3. What is the relation between a Programming Language (PL) and a Modeling Language (ML)?

A good indication of the consensual community agreement on this could be obtained by presenting a list of languages or notations to members of the community, asking them the following question: For each language do you thing of it as a PL (directly contributing to executable code) or as a ML, i.e. a language that may somewhat contribute or help producing code but that does not directly produce executable code for applications (i.e. not a PL). Let us suppose people have no choice of other categories of languages.

	PL	ML
Java	X
UML		X
C	X
Assembler 360	X
Flowchart		X
SADT/IDEF		X
Prolog	X
BPMN		X
Z		X
Lisp	X
B		X
SQL	X
R	X
Javascript	X
SysML		X
HTML5	X
JSON		X
XML	X
Petri nets		X
Excel	X
DFD		X
ADA	X
xUML	X
SPEM		X

Do these answers correspond to what most people consider as Programming Languages (PL) or as languages that are not PL and that we may name Modeling Languages (ML)?

The history of programming languages is well known; there has been plenty of papers and books written on this.

The history of modeling languages is much more obscure. The reason is that most of these languages have not been originally considered as modeling languages because the expression is quite recent. They have been associated to:

• methods (formal or semi-formals)
• requirements
• specifications
• etc.

In the simplified drawing below, one can see that such general “modeling languages” are nearly as old as programming languages. Furthermore the history of modeling languages parallels the history of programming languages.

So what we need is some kind of consolidated history of modeling languages. If we look at all these notations from the modern point of view of DSLs, we may find there a rich matter for reflection, that does not reduce to UML only
—

Modeling and Programming

Some time ago we had a Workshop on Modeling and Programming at SPLASH2011 in Portland.

It was a pleasure to meet many nice  old chaps there.

In particular I remember some pleasant discussions with my modeling colleague and nevertheless good friend Ed Seidewitz. After the workshop, we chatted again about the discussion cliché of modeling vs. programming (in french we don’t say cliché, but tarte à la crème or custard tart). We came to the conclusion that it would be a positive service to the community to write a consensual paper on this subject.  We are loosing so much time endlessly arguing on this that it takes a significant energy that could be spent on more constructive debates.

French philosopher Albert Camus used to say:

“Mal nommer les choses, c’est ajouter au malheur du monde”   which could be translated by “To misname things is to add misery to the world“

And I believe this perfectly applies to the “Programming vs. Modeling” debate.  From time to time I tried to list the pro and cons arguments. Why modeling is programming and why programming is modeling?   Why both are similar and why both are different? If you want to have some fun in software circles, launch the discussion after a couple of beers on this subject and then just observe and enjoy. The nice property of this debate is that nearly everybody has strong opinions on the subject.

In december, at the CHOOSE Forum in Bern we had again this discussion.  Another respected colleague and my good friend Markus Voelter gave a talk with the title: “Modeling and Programming – isn’t all the same?” and with the following summary:

“Modeling and programming are often considered two different things. Programming languages are (perceived to be) different from modeling languages. DSLs are different from GPLs. But are they? Should there be a difference between the two worlds? How closely should the two approaches be integrated? In this talk I will argue that programming and modeling are really essentially the same thing; they are only different in terms of the abstraction level ….”

The more I look at the problem and the more I become convinced that organizing debates on the subject will lead us nowhere. We are in a blocked situation where the notion of modeling … and also the notion of programming have too many different definitions and meanings, usually very loose and sometimes completely contradictory.

So I came to the conclusion that we should propose a moratorium on any discussion on this subject. That obviously does not mean that the subjects of modeling and programming are taboo. That only means that we should give more time to provide precise definitions of  ”program engineering” (or programming) on one side and “model engineering” (or modeling) on the other side.

Proceeding in that way, we may be able to somewhat cool off the debate while allowing all camps to build more solid definitions. Hopefully at some time, these definitions will allow avoiding sterile, futile, and non productive  arguments and leaving the discussion by the top and not by the bottom (i.e. not reaching Godwin’s point !).

We can study the differences and the similarities between these two forms of engineering. After all, if we find only similarities at some point, that may be good news. This seems a very respectful and scientific approach:

Let us let the champions of both approaches producing good and precise definitions. This is not so easy. I have been very careful to use “model engineering” and not “model driven engineering” which is a completely different subject. The relation of program engineering with language engineering (or software language engineering as it is often called now ) has also to be studied very carefully. But the relation of model engineering and language engineering has similarly to be made much more precise.  Markus Voelter recently published an excellent book on “DSL Engineering” which can be read as a  contribution to the subject because it touches the three topics of program engineering, model engineering and language engineering.

Finally we will also perhaps agree that software engineering is some kind of composite engineering encompassing program engineering, model engineering, language engineering, and many other forms of engineering. But this is a more ambitious subject.

Software Engineering Resurrection

In December I gave a talk at the CHOOSE forum in Bern

The title was: “Should we Resurrect Software Engineering?”

The abstract may be found here: Last Post

The slides are here: PDF file

Since this presentation, I got a number of very interesting remarks. Thanks.

I will work on an update of this talk in the coming weeks.

There is currently a trend to say that, perhaps, what we call “computer science” is not a science at all. It may be some kind of generic engineering. I don’t know if this is true, but the debate is growing and the discussion becoming hot.

Should We Resurrect Software Engineering

“Should we Resurrect Software Engineering” : This is the title of a talk I will give soon.

The abstract is as follows:

Software engineering is dead or at least critically ill. Moving from the individual creation of small programs to the collective production of complex and evolutionary software systems was rightly identified as a serious problem more than forty years ago. But the attempts to solve it have recently been quite deceiving. One of the last innovative ideas was the notion of the Smalltalk class browser at the PARC in the early 80′s.  There has been limited improvement in the programming productivity and this is becoming serious at a time when the number of necessary applications for professional, individual, social, and other needs is exponentially growing. Hopes created by formal methods have not scaled up in industrial solutions to increasing reliability. Recent efforts have only addressed the most apparent flaws by a more agile organization of working teams, which is probably necessary, but far from sufficient.

Clearly the present situation is not satisfactory and we do not see any forthcoming silver bullet that could help on the horizon. Most proposals are only small incremental deltas to known programming techniques and will not produce the large-scale desired effects. We need to rapidly find a way out of this situation and this can only be achieved by an important epistemological rupture or at least by a significant paradigm shift.  It is the collective responsibility of our discipline to find an operational solution. In order to do this, we need to have a critical look at the young history of software engineering in order to propose a radical departure from current practices.

The talk will first propose to completely forget about classical software engineering. It will instead advocate concentrating on the important subject of “engineering” that has changed a lot in the last half-century.  We need to realize that computers are now omnipresent and software ubiquitous. Old expressions like “computer-assisted” have lost any discriminant meaning just because most engineering processes are now more or less assisted by computers and software. We need to revisit a lot of beliefs that have been with us in the last decades and start thinking out of the box. The presentation will argue that we should first concentrate on defining some “unifying theory of engineering“. If this is successful, then we may propose to view software engineering as a specialization of this conceptual framework, with several expected benefits.

To make things concrete, we will consider two broad categories of engineering called “support engineering” and “domain engineering“. The first category defines a set of technical spaces like service engineering, system engineering, model engineering, constraint engineering, data engineering, process engineering, language engineering, mathematical engineering, and many more. At the opposite of this solution space, we find the problem space with a lot of domain engineering like electrical, mechanical, civil, health, telecommunication, avionics, biological and many more.  There are many commonalities of domain engineering that would gain to be exposed. Starting with the building of models conforming to some ontology, a process usually defines some model validation or verification followed by a manufacturing or production step intended to augment or transform the real world.

The talk will propose an initial cartography of support engineering, insisting on model, service, process, data, and program engineering and their relations to a possible redefinition of software engineering.  The talk will also mention many possible benefits of exhibiting similarities and synergies between urban architecture, mechanical, biological, energy or other forms of domain engineering and here again to a possible impact on a possible redefinition of software engineering.

Software Engineering is Dead

Software Engineering is Dead

From many places, authors have been suggesting in the past years that SE is dead.

http://blogs.msdn.com/b/elee/archive/2009/09/08/tom-demarco-software-engineering-is-dead.aspx

Tom DeMarco’s article in IEEE Software entitled “Software Engineering: An Idea Whose Time Has Come and Gone?”  In it he recants his early writing on the topic of metrics and control in software engineering projects and says that software projects are fundamentally experimental and uncontrollable.  Consistency and predictability in software development are largely unattainable goals.  Much of what we think of when we think about “software engineering” is a dead end.

This paper has raised several questions, i.e.:

http://www.codinghorror.com/blog/2009/07/software-engineering-dead.html

or:

http://www.itwire.com/opinion-and-analysis/the-linux-distillery/26370-software-engineering-is-dead-long-live-engineering-of-software

or:

http://edweissman.com/software-engineering-dead

or:

http://blogs.msdn.com/b/marcod/archive/2011/02/04/thedeadofsoftwareengineering.aspx

or:

http://www.rspa.com/commentary/SEarticle7.html

or:

http://leonardwoody.com/2011/02/25/software_engineering_isnt_dead_yet_but/

or:

http://mountainbikerak.blogspot.fr/2009/07/is-software-engineering-really-dead.html

http://blogs.msdn.com/b/devschool/archive/2009/07/27/is-software-engineering-dead.aspx

The paper itself may be found at:

http://www2.computer.org/cms/Computer.org/ComputingNow/homepage/2009/0709/rW_SO_Viewpoints.pdf

I was very lucky to attend a keynote given by Michele Lanza in Sao Paolo last year with the title: Requiem for Software Engineering

http://www.each.usp.br/cbsoft2011/ingles/sbes/palestras_sbes_en.html

Unfortunately I do not find anymore the slides of this presentation, but it was really convincing.

Endangered (technical) species

I am amazed by the rapid disappearance of many references to important technical contributions. It is normal life that new proposals appear and that old proposals disappear. But I find annoying that direct Web references vanishes so quickly. It is sad because some of these proposals were not bad or low quality, they just did not meet their public at the time for reasons that were often more political than technical. It is a problem because so many people now believe that something that is not on the Web does not exist and has never existed.

I started doing some research on pre-MOF metamodeling techniques and my first idea was to look at a very interesting Web site that was built a long time ago by my friend Johannes Ernst. This site was simply  ”http://www.metamodel.com“. But unfortunately this site appears to have completely disappeared and the Web pointer now redirects to Adaptive, the company of another good friend Pete Rivett.

Fortunately some people have saved small parts of this initial site, for example I managed to find track of the discussion What is metamodeling, and what is it good for? on another site: infogrid.org.

This made me realize that we lost most memories of an important metamodeling initiative called CDIF. This was a real achievement in metamodeling. Paraphrasing Tony Hoare, we can say that CDIF was an improvement on most of its successors, including the multiple variants of MOF and also ECORE. It is a pity that the memory of CDIF is disappearing so quickly because there are many lessons that we can still draw today from this interesting proposal.

Fortunately we can still find some interesting archives like the paper by Oscar Nierstrasz and his colleagues on the use of CDIF for reengineering Tools. This paper was published in 1998. Is that so old that we should completely forget about this technology?

I am convinced that MOF and UML have not done so well as to justify that we should delete all references to previous modeling proposals. On the contrary I believe that we can still find a lot of inspiration in older technologies. As we say in french, “C‘est dans les vieux pots que l’on fait la meilleure soupe” (experience always wins the day).

Does someone have a list of pointers on CDIF or better has saved some part of this very valuable technology heritage?

My main reference is still the paper by Rony Flatscher:  ”An Overview of the Architecture of EIA’s CASE Data Interchange Format (CDIF)“. I know that Rony also published a book on CDIF, but I understand it was in German and was not translated in English.

A history of Metamodeling

In 2010 I was happy to welcome Sridhar Yyengar to give a seminar in Nantes with the following title: A (Meta)modeling Odyssey : Future of modeling, metamodeling, tools integration and the semantic web. The abstract is below (1).

Sridhar Yyengar leads the technical strategy group for Software Tools & Methods at the IBM TJ Watson Research Centre and led the definition of OMG MOF and XMI standards, among many other contributions as member of the OMG Board of Directors. Sridhar has been very influential in developing the MOF vision at OMG (Meta Object Facility).

But what we need now is something much broader that the strict MOF historical vision to understand the pertinence of these reflective languages to describe metamodels, their deep goal, structure, usage, rationale and evolution. We need to revisit these metamodel description DSLs and to look carefully at their positioning towards other DSLs and GPLs.

In January 2012 I was giving a course at the National Institute of Informatics in Tokyo and I started looking at the history of these metamodel description languages. I will try to write down some of my notes on this blog in the future. In the meantime, I will be happy to exchange and discuss on this blog  or on Twitter or Google+ with anyone interested in the history of MOF-like languages.

______________________________

(1) Abstract: The past 15 years have seen some very interesting advances in the world of modeling, metamodeling and their use in developing tools and middleware across the complete application lifecycle. Industry modeling standards from OMG and web infrastructure standards like XML, WSDL and more recently semantic web standards like RDF and OWL have made their way into tools that cover the entire application lifecycle for both custom developed as well as packaged application implementations. As we start a new decade what can we expect in the next 10 years? This presentation will summarize the historical journey, take a snapshot of where the industry and related standards are as we begin the new decade. We will have a look at how we expect the world of the internet and semantic web will shape modeling, metamodeling and tools integration over the next decade. We will look at how emerging collaborative application lifecycle platforms like Jazz (www.jazz.net) and emerging standards like SMOF (Semantic MOF) and future versions of UML and BPMN are evolving to address the challenges of the new decade.