I received an email yesterday pointing me to a bill, introduced on March 27, that would require all Executive branch agencies in the state of Minnesota to "use open standards in situations where the other requirements of a project do not make it technically impossible to do this." The text of the bill is focused specifically on "open data formats," and would amend the existing statute that establishes the authority of the Office of Enterprise Technology (OET), and the duties of the states Chief Information Officer. While the amendment does not refer to open source software, the definition of "open standards" that it contains would be conducive to open source implementations of open standards. The text of the affected sections of Minnesota Statutes Chapter 16E, showing the amendments proposed, can be found here.
The fact that such a bill has been introduced is significant in a number of respects. First, the debate over open formats will now be ongoing in two U.S. states rather than one. Second, if the bill is successful, the Minnesota CIO will be required to enforce a law requiring the use of open formats, rather than be forced to justify his or her authority to do so. Third, the size of the market share that can be won (or lost) depending upon a vendor's compliance with open standards will increase. And finally, if two states successfully adopt and implement open data format policies, other states will be more inclined to follow.
The amendment, and the statutory framework in which it exists, provides an interesting contrast to the open standards policy adopted by the Massachusetts Information Technology Division (ITD) in many respects. For example, it appears from the existing text of the statute that the Minnesota OET already has clear authority to mandate the use of open data formats, reading in part as follows:
When state agencies have need for the same or similar public data, the chief information officer, in coordination with the affected agencies, shall manage the most efficient and cost-effective method of producing and storing data for or sharing data between those agencies. The development of this information architecture must include the establishment of standards and guidelines to be followed by state agencies. The office shall ensure compliance with the architecture.
Second, the proposed amendment contains an extremely detailed (and rather eclectic) definition of “open standards,” including not only the traditional concepts of availability to all on reasonable and nondiscriminatory (“RAND”) terms, but a good deal more besides. For example, it requires that all permitted standards must permit royalty free implementation, and also includes a number of requirements that are far more detailed than would normally be found in the description of an open standard, but which might be agreed upon as necessary by the members of a working group developing a standard. The following is one example taken from the proposed definition of an “open standard” found in the amendment: “[An open standard] is documented, so that anyone can write software that can read and interpret the complete semantics of any data file stored in the data format.” (I’ve included the full text of the proposed definition at the end of this blog entry.)
In addition to the very restrictive definition of open standards and open data formats, the amendment is also intolerant of making exceptions, providing:
For a particular project involving the access, storage, or transfer of data, a restricted data format may be chosen when satisfaction of essential project requirements precludes the use of an open data format. Neither the current storage format of previously collected data, nor current utilization of specific software products, is a sufficient reason, in absence of other specific overriding functional requirements, to use a restricted format;
Moreover, the amendment would require periodic review of all “existing data stored in a restricted format, to which the state of Minnesota does not own the rights, every four years to determine if the format has become open and, if not, whether an appropriate open standard exists;” The amendment, if enacted, would therefore impose a very tight collar on what types of software could be purchased and used.
On its face, the amendment is vendor neutral. It does, however, include one provision that may have been directed at Microsoft, which has at times been criticized for adding proprietary extensions to otherwise standards-compliant product features. That provision is found in the definition of an “open standard,” and requires that if a standard, “allows extensions, ensures that all extensions of the data format are themselves documented and have the other characteristics of an open data format;”
There are a number of other interesting points that I note in reading the amendment at the Minnesota site, one being that the state legislation portal is set up to provide data on a bill’s sponsors and current status – something that is sadly lacking in Massachusetts. There also appear to be a number of differences between the responsibilities and authority of the CIO in Minnesota, as compared to Massachusetts, that Peter Quinn might have appreciated. I may look more deeply into those differences and return to them in a future entry.
Curiously, I have been able to find out almost nothing on the Web about the amendment, indicating that thus far it has received little public notice. The news item submission (written by the submitter) that pointed me to the bill, however, reads in part as follows:
A consortium of Minnesota businesses and citizens has moved to put forward legislation that promises to assist the State in overcoming the negative effects of time, innovation and the market. Referred to as the Open Data Formats Bill, House File 3971 defines the means by which Minnesota could take control over how it stores information so as to not be bound, technically or legally, to anything other than its own technical objectives.
The Bill is an attempt to counter the fact that searches on the State website display information that is primarily stored in formats owned by vendors… that could go out of business, get acquired, or turn into the next Enron. It is also a response to what some refer to as the ‘eight track tape effect’ where information is stored in a way that is both popular and looks permanent, but then is quickly replaced by newer technologies. In some instances the information has been lost forever.
The Bill is not biased towards any one technology and advances a policy where at all times and in all instances the State has the ability and legal right to review, fix or improve the information it uses to conduct business. The Bill is not expected to increase State spending on technology. It is, nonetheless, expected to receive stiff opposition despite improving competition for State contracts, enhancing the ability of Minnesotans to access State services and data, and improving communication between State systems.
I will keep an eye on this new bill and report further as additional information becomes available. For now, however, it is significant to note that the debate over open data formats has now begun in a second state. It will be interesting to watch how the forces align, and the discussion becomes focused as the process moves forward.
The full text of the two new statute sections appears below.
For further blog entries on ODF, click here
subscribe to the free Consortium Standards Bulletin
(and remember to Buy Your Books at Biff’s)
Language to be added to Minnesota Statutes 2005 Supplement, section 16E.03, subdivision 1:
(f) “Open standards” means specifications for the encoding and transfer of computer data that:
(1) is free for all to implement and use in perpetuity, with no royalty or fee;
(2) has no restrictions on the use of data stored in the format;
(3) has no restrictions on the creation of software that stores, transmits, receives, or accesses data codified in such way;
(4) has a specification available for all to read, in a human-readable format, written in commonly accepted technical language;
(5) is documented, so that anyone can write software that can read and interpret the complete semantics of any data file stored in the data format;
(6) if it allows extensions, ensures that all extensions of the data format are themselves documented and have the other characteristics of an open data format;
(7) allows any file written in that format to be identified as adhering or not adhering to the format;
(8) if it includes any use of encryption, provides that the encryption algorithm is usable on a royalty-free, nondiscriminatory manner in perpetuity, and is documented so that anyone in possession of the appropriate encryption key or keys is able to write 2.20 software to unencrypt the data.
(g) “Restricted format” means any data format that is accessed, stored, or transferred 2.22 and is not open standards compliant.
Language to be added to Minnesota Statutes 2005 Supplement, section 16E.04, subdivision 2:
(g) The office shall assist state agencies to avoid the purchase or creation of data processing devices or systems that do not comply with open standards for the accessing, storing, or transferring of data. The office shall:
(1) ensure any new data standards which the state of Minnesota defines and to which it owns all rights are open standards compliant;
(2) use open standards in situations where the other requirements of a project do not make it technically impossible to do this. For a particular project involving the access, storage, or transfer of data, a restricted data format may be chosen when satisfaction of essential project requirements precludes the use of an open data format. Neither the current storage format of previously collected data, nor current utilization of specific software products, is a sufficient reason, in absence of other specific overriding functional requirements, to use a restricted format;
(3) reexamine existing data stored in a restricted format, to which the state of Minnesota does not own the rights, every four years to determine if the format has become open and, if not, whether an appropriate open standard exists;
(4) make readily accessible, from a central location on the Internet, documentation on open data formats used by the state of Minnesota. When data in open format is made available through the state’s Web site, a link shall be provided to the corresponding data format documentation.
Hi,
definitely a good move. Open playground enforces more competetion and
that’s (after cooperation) the second best way of working. I am affraid
M$ is in the “ignoration” working category ;(.
Take care,
Jan
As I read this, this goes WAY beyond what Massachusetts has done. This affects backups, archives and potentially even file systems.
Is the ZIP format an open standard? Would that qualify for us under these proposed changes?
The deepth or the difficulty of fixing the problem should not be a consideration as whether to do something or not. Rather, how. B/c zip is used a great deal should not exempt it. By the same reasoning asbestos should simply be ignored as it is everywhere.
You missed my point.
The Zip format is not something that has been endorsed by any standards body that I’m aware of. However, it is a known format. Does that count as an “open standard”? As I read the text of the bill it does because the bill doesn’t really require standards, it requires openness. A vendor could meet the letter of the bill (and perhaps the spirit as well) IF it made available the data formats that it uses to store data (and there were no patents on any of those formats).
For example, (and this would never happen) Microsoft could get Word 97 thru this by documenting the format and granting royalty-free rights to any patents it has (assuming it didn’t license any patents as well). Is that the intent of this bill? I kinda doubt it, but I don’t know.
The Zip format is not something that has been endorsed by any standards
body that I’m aware of. However, it is a known format.
the metric the bill sets forth is not ‘known’ when determining the accessibility of a data format. it is much more specific.
Does that
count as an “open standard”? As I read the text of the bill it does
because the bill doesn’t really require standards, it requires
openness.
again, you apply a word -here it is ‘openness’- that doesn’t appear in
the bill as its put forth above. the wording, to me, is very specific
and does not leave the wiggle room for interpretation like the verbiage
you’re saying it does.
A vendor could meet the letter of the bill (and perhaps the
spirit as well) IF it made available the data formats that it uses to
store data (and there were no patents on any of those formats).
patents are a big part of it. but as I”m sure you’re aware there are
many ways for a software company to encumber a standard. with
regard to spirit, hmm, i wish the irs would accept the spirit of
my intent to pay as renumeration.
For
example, (and this would never happen) Microsoft could get Word 97 thru
this by documenting the format and granting royalty-free rights to any
patents it has (assuming it didn’t license any patents as well). Is
that the intent of this bill? I kinda doubt it, but I don’t know.
the bill, to me, isn’t just about payments. it’s about making sure
anyone with the inclination and the time can look at how information is
organized by a software application or whatever and extract
understanding. patents and roytalty payments are components of this,
but the bill is way broader.
The bill is not that specific and there’s PLENTY of wiggle room. I repeat: this bill (as currently worded) is about openness of documentation and implementation, it’s not about standards.
What the bill calls an “open standard” does not have to be a standard in any conventional sense. It doesn’t have to be a de jure standard. It doesn’t have to be a de facto standard. Until the day the software is put up for consideration, the documentation for the data formats the software uses do not have to be public.
Hypothetical: Under this bill, my company could submit some newly developed software for consideration and on the day of submission make available documentation on the data formats the software uses. Until that day, no one outside my company has seen the documentation; no one outside the company has had any input on the data formats used. There could be formats that are used that only one person had any input on. The data formats have no patents so there’s no problem there.
Thus, my company could meet the letter of the law, but the data formats used by the software are not standard in any meaningful sense of the word. Thus, the spirit of “open standards” is certainly not met. And that’s called wiggle room.
The zip file format is not an open standard. But it is well-known.
By my reading, it complies. If it does not comply, I am not aware of any compression and archival program that does – as memory serves, there are a couple open standard archival programs, and there’s at least one open standard compression program, but I don’t believe the union of the two exists. The standard use for those programs is for the archival program to then use the compression program, when one wants to have the archive compressed – which isn’t quite the same thing.
If this law were to pass, and the zip file format were to be found lacking, I think there would be ample time for it to be standardized – after all, I saw nothing which dictated the standards bodies in question, or even what type of standards bodies one could use.
I already feel sorry for whatever representative introduced or sponsored this bill. It is almost guaranteed that he or she will be personally attacked in the media, etc. before this process is over, just as Peter Quinn in MA was.
Don’t publicize this!
Great. Now MS is mobilizing their army of lobbyists to kill this thing before it can pass.
don’t worry about that, MS is fully aware of what happens when it comes to anything that threatens its monopoly positions, especially with windows desktop and office suite.
What’s so entertaining is that MS has no rational position to stand on. Consider the alternative bill: documents must be kept in closed or semi-closed formats that require proprietary software that has been purchased at a high price to display.
No, what’s actually going to happen at some point in the next year or so is that MS is going to release the newest office suite with open document support. I will bet you that they already have that support built in, and all they have to do is turn it on.
Of course government data and information should be kept in open formats, so that they can be accessed for as long we need. Any other claim is completely ludicrous, and will fail as momentum builds. It figures that this started with Mass and Minn though, that’s not a surprise at all.
The idea that a state body should require you to use expensive proprietary software to access the data that you as a tax payer essentially paid for is so completely ridiculous that it has to fail eventually. It doesn’t matter how much money MS throws at this particular issue, they are so clearly wrong, and demonstrably so, even to dimwitted state politicians, that their only choice is bribery, which I’m sure they are busy engaging in as I type this. But even that has its limits, as recent events in washington show.
My reading is that clause G. (2) implies that, if passed, nobody in Minnesota government will be able to produce and store any new documents in .doc (Ms-Word) or .xls (Excel) format. If that is truly the case, then the CIO will be very busy doing a lot of behavior / change management to “ensure compliance”
fix it now or fix it later. There are some great studies out about the
cost assocaited with dealing with old systems and how the older and
more entrenched, vis-a-vis less interoperable, the greater the cost to
move to the newer system. interestingly one of the factors the study
talks about is that what really drives up costs is fear of the moving
from the status quo. not to be funny, but i don’t know many
people scared of delimited text files.
Possibly.
Microsoft could decide to publicly document the .doc and .xls formats and grant rights on any patents it has or has filed. If there are no other patents that Microsoft had to license for the files (and I’m guessing that’s the case) then as the proposed law is currently written no changes would be needed.
If this hadn’t been reported, maybe it could have been snuck through without notice from Microsoft… 🙁
Might be better to hear only of its passing (unless Microsoft was already obstructing it somehow, in which case, the more noise the better!) …
Nah, that’s silly. MS monitors all state legislatures for bills that will affect them in any way. Don’t be naive, they have an army of lawyers protecting their interests in just this way.
the bill is not trying to create a
million ISO-like organizations or processes every time someone
releases a protocol, application, etc.. This is not
its objective explicitly or implied. Given this, you are right:
your company would be conforming to the letter of the law if it acted
as you described. But that fact is what your company has
released IS a standard/specification. That they didn’t go and get
some sort of certification is not as important as ensuring
they don’t encumber the standard with patents, copyrights,
encryption, etc. At first this requirement may seem almost silly.
Certainly not worth of a bill. But if the application becomes popular
-popular in the same way PDF, HTML or Word has- this lack of
encumbrances will be critical. And this is what the bill is getting
at. Not the creation of committees.
“the bill is not trying to create a
million ISO-like organizations or processes every time someone
releases a protocol, application, etc.. This is not
its objective explicitly or implied.”
Agreed. I certainly didn’t say that nor did I mean to imply that.
“But that fact is what your company has
released IS a standard/specification.”
It is a specification; it is NOT a standard.
“That they didn’t go and get
some sort of certification is not as important as ensuring
they don’t encumber the standard with patents, copyrights,
encryption, etc.”
I’m not concerned about “certification,” but standards are a lot more than certification. Some people believe that only buying products that (supposedly) comply with standards (when standards are available) is a good thing. It’s not clear to me if those proposing this change believe that or not.
The phrase “open standards” used in the proposed change has a fuzzy definition, but a standard in one form or another is a part of it. None of the supporting verbiage in the proposed change promotes or supports standards. Thus, the true intent of this change is not clear. Once the true intent is clear we can discuss the merits of that intent. Until then…
Perhaps I’m a bit sensitive on the standards part as I’ve represented several companies in a standards
organization and was the editor of a document that’s going through ISO
review right now. But, by my definition, an open standard must have input from
multiple vendors and there must be group control of the standard. The verbiage in the bill does nothing to promote this.
“the bill is not trying to create a
million ISO-like organizations or processes every time someone
releases a protocol, application, etc.. This is not
its objective explicitly or implied.”
Agreed. I certainly didn’t say that nor did I mean to imply that.
“But that fact is what your company has
released IS a standard/specification.”
It is a specification; it is NOT a standard.
“That they didn’t go and get
some sort of certification is not as important as ensuring
they don’t encumber the standard with patents, copyrights,
encryption, etc.<“
I’m not concerned about “certification,” but standards are a lot more than certification. Some people believe that only buying products that comply with standards (when standards are available) is a good thing. It’s not clear to me if those proposing this change believe that or not.
The phrase “open standards” used in the proposed change has a fuzzy definition, but a standard in one form or another is a part of it. None of the supporting verbiage in the proposed change promotes or supports standards. Thus, the true intent of this change is not clear. Once the true intent is clear we can discuss the merits of that intent. Until then…
Perhaps I’m a bit sensitive on the standards part as I’ve represented several companies in a standards
organization and was the editor of a document that’s going through ISO
review right now. But, by my definition, an open standard must have input from
multiple vendors and there must be group control of the standard. The verbiage in the bill does nothing to promote this.
“The Bill is an attempt to counter the fact that searches on the State
website display information that is primarily stored in formats owned
by vendors… that could go out of business, get acquired, or turn into
the next Enron. It is also a response to what some refer to as the
‘eight track tape effect’ where information is stored in a way that is
both popular and looks permanent, but then is quickly replaced by newer
technologies. In some instances the information has been lost forever.”
my favorite part – finally someone gets it – maybe other states will get smart and follow suit like any competent cio should.
like the part especially about the next enron.
sorry to get the reply thing messed up. and not that it’s an excuse, but typing anything at a waterpark in wisconsin is tough.
nonetheless, i see your point. and frankly the bill doesn’t really
addess the time/how/why/who at which a specification becomes
a standard.
don’t get me wrong: standards bodies and what they try to do is
a really good thing. that said, this bill should not get
dumped becuase it doesn’t go one more step past removing encumbrances
and dictate the the mechanisms involved in elevating specifications to the status of ‘Standard’.
it probably shouldn’t. and why should it? if it did i have no doubt it would die a quick death.
But a better way to respond this critiscm, though, is the
quesiton: what good is a review/maint. committee if the specifications
are patented and at any point can be used to sue anyone who attempts to
implment the standard? this is kinda the nature of the agreement
between sun and microsoft now relative to open office.
there is probably not enough room in the word ‘standard’ for all of us.
i might be ok with ‘open specifications’ but i’m sure this would be a
problem for others.
“what good is a review/maint. committee if the specifications
are patented”
From the perspective of the FOSS/OSS community: none.
That’s why the standards have to be “open”. The FOSS/OSS community believe that an “open standard” means that either there are no patents or that a perpetual, royalty-free license is available for any patents that apply to the standard (see http://www.fsf.org/news/oasis.html). Vendors are typically satisified if they can get RAND terms.
i wonder if this has been tested in court?
I doubt it, this is a relatively new concept in government.
On what basis do you think someone would challenge it in court?