Back to all updates

almost 10 years ago

Judging

The goal of this contest is to encourage creation of systems that help Veterans make appointments to receive outpatient and ambulatory care[1] from the Veterans Health Administration.  VA also seeks to obtain information which will allow it to reduce the risks inherent in procurement and deployment of a replacement medical scheduling product.

Numbers and Amounts of Prizes. 

a. VA will award monetary prizes of as much as $3 Million to as many as three entrants that deliver demonstration software or service which the judges determine delivers the required functionality and is compatible with Open Source VistA, as described in this notice.  

b. Judges will determine the number and amount of monetary prizes.  Judges may determine that no prize will be awarded.

c. VA may consider compatibility with Open Source VistA as demonstrated in this contest when considering proposals in any subsequent procurement of MSP-related product(s).

Basis on which a winner will be selected.  (See section 6 for judging procedures and point system.)  Winners will be selected from entries that demonstrate[2] to the satisfaction of the judges that they:

a. Perform as required by these rules while interfaced with VA-provided VistA instances running on VA-provided virtual machines.  Performance with other versions or instances of VistA will not be considered.

b. Perform each of the scheduling-centric business functions defined by VA on Attachment A to this notice, including particularly:

    1. Automated scheduling of an appointment at any VA site from any location by a Veteran using online or mobile devices or by a VA employee acting as a scheduler,[3] and

    2. Where semi-automated scheduling is performed, use of a calendar-view format presentation of available resource(s), with support for point-and-click scheduling when the calendar view shows needed resources to be available.[4]

c. Perform all or some of the designated Scheduling-Related/VA-Specific business functions set forth on Attachment B to this notice, including functions related to workload capture, data analysis and reporting, operational and capital planning and travel reimbursement.

d. Perform each of the non-functional requirements set forth on Attachment C to this notice.

e. Do so using open APIs, including open source code for the interfaces between the product and VistA, and at the option of the entrant, using open source modules to support some or all of the product’s additional functions.  Note:   Entries that consist of proprietary code are not precluded, so long as interfaces with VistA use open APIs and the software implementing the interfaces is made available as open source code.

[1]Scheduling of inpatient, surgical, and extended nursing care is excluded.   

[2]Winning products will be used for demonstration only and will not be deployed in VA facilities except for testing or demonstration purposes. 

[3] See, e.g., Attachment A, sections 1.4.1, 1.4.5, 2.7, 3

[4] See, e.g., Attachment A, sections 1.2.5, 3, 3.5, 3.7.  

Judging of Entries

a. Method.  VA may use any technical means it determines suitable to evaluate any entry, award points and determine any winner in this contest.  All or any part of the evaluation may be conducted by third parties under VA supervision.  VA may use an entrant’s submitted testing scripts, routines and software to perform all or part of any evaluation and judging.  VA may, at its option, request an entrant to demonstrate its or their entry either in-person or via web-based technologies.  

b. Technical Evaluation.  VA contemplates it will judge entries as follows.

    1.    Step 1 Testing will determine Open Source VistA Compatibility, i.e.,:

           a) Whether it is compatible with the three separate VistA instances on the VA-supplied virtual machines, and

           b) Whether the interfaces (APIs) between the product and VistA are based on software code that either is or can become open source.

           c) VA contemplates that substantial portions of this evaluation will be conducted by OSEHRA, using Contestant-supplied automated scripts.  These scripts will address a set of eight (8) use cases defined in documentation available on the Contest Web Site.  Any entry that does not demonstrate Open Source VistA Compatibility to the satisfaction of the judges will be disqualified and will not be further considered for a prize.

      2.  Step 2A testing will further evaluate each qualified submission to determine whether it can meet the following requirements:

           a) Scheduling-centric functions defined on Attachment A.  An entry that does not demonstrate to the satisfaction of the judges that it can perform the Scheduling-centric functions defined on Attachment A will be disqualified and will not be further considered for a prize.

           b) Non-Functional Requirements on Attachment C.  Any entry which does not demonstrate to the satisfaction of the judges that it can perform the Non-Functional Requirements on Attachment C will be disqualified and will not be further considered for a prize.

           c) Scheduling-related or VA-specific functions defined on Attachment B. Judges will award the designated points for each function defined on Attachment B that they determine is performed by the entry.  A maximum of 120 points can be awarded for performance of functions defined on Attachment B.

      3.  Step  2B - Open Source Content.  An entry that demonstrates Open Source VistA Compatibility and also demonstrates it can perform all of the Scheduling-centric functions on Attachment A will also be evaluated by the judges to determine the extent to which it uses software modules, as designated by the entrant, which are or will become open source.

           a) Interfaces (APIs) between the product and VistA will not be considered in this phase of the judging, as they are a required component of Open Source Compatibility.

           b) Judges will evaluate the significance of the designated open source modules in delivering required functions.  For example, judges may determine that modules that provide common platform services or business logic are more significant than modules that deliver the presentation layer.

           c) Judges may award a maximum of 30 points for significant open source modules incorporated in an entry.

c. Points awarded by the judges will determine the winners of this contest. 

Contestants, in order to be judged, will contribute the open APIs and any open source content in their entries to the Open Source Electronic Health Record Agent (OSEHRA).  Contestants will apply the Apache 2.0 license to the designated open source modules (including the interfaces between the contestant’s product and VistA) and contribute the modules to OSEHRA.[1]

Judges and Judging Procedures.

a. Subject to the requirements of 15 U.S.C. 3719(k), the VA Assistant Secretary for Information Technology, acting on behalf and with the authority of the Secretary of VA, will appoint one or more qualified individuals to act as judges of this contest and may appoint himself as a judge.  Judges may include individuals from outside VA, including from the private sector. Judges will operate in a transparent manner.

b. A judge may not have a personal or financial interest in, or be an employee, officer, director, or agent of any entity that is a registered entrant in this contest, and may not have a familial or financial relationship with an individual who is a registered entrant.

c. Specific tasks related to the judging process may be delegated to VA employees or employees of a collaborating Federal agency.  Third parties may perform judging tasks subject to supervision by VA or by a collaborating Federal agency.

d. Judges shall have the authority to disregard any minor error in any entry that does not create any substantial benefit or detriment to any entrant.

e. Judges shall have the authority to obtain from any entrant additional information, clarification of information, or assistance in resolving any technical issues relating to the installation, use, testing or evaluation of any entry, so long as no substantial benefit or detriment to any entrant occurs thereby.

f. Decisions of the judges are final.

[1]http://www.osehra.org/page/osehra-code-repository.