1988; rev. 1991, rev.1992, editorial corrections 1999.
Prepared by the VLDB Endowment, with copyrights.
This document is also available in Microsoft Word format or in PDF format.
- 1. Introduction
2. Setting up VLDB Program Committees
- 2.1 The Committee Structure
- 2.2 One or two level organization?
- 2.3 Selecting a good committee
3. The Conference Content and Structure
- 3.1 Conference topics
- 3.2 Technical papers
- 3.3 Tutorials
- 3.4 Panel discussions
- 3.5 Roles of participants (authors, panelists)
4. Program Committee Work
- 4.1 Planning
- 4.2 The reviewing process
- 4.3 Program committee meetings
- 4.4 Proceedings management
- 4.5 Publicity, exposure
- 4.6 Miscellaneous
5. Concluding Remarks
During the last 10 years, VLDB Conferences have been highly successful. The number of full length papers submitted has consistently been over 200. In some recently held conferences, the number of submitted papers exceeded 300. Papers submitted to a VLDB conference must be refereed by three to four high quality referees, implying the need to handle a huge number of referee reports. This requires careful preparations and quality control considerations when setting up and planning the work of VLDB Program Committees (PCs).
The purpose of these guidelines is to assist future VLDB Program Committee chairpersons and other VLDB officers to plan their work. The guidelines can be seen as a part of VLDB's "corporate knowledge base", assembled during many years of running VLDB conferences and program committees.
The guidelines are organized as follows. In Chapter 2, we outline the structure of a VLDB Program Committee and the process of establishing it. Chapter 3 describes the VLDB Conference, its structure and contents in terms of different kinds of contributions and events. Chapter 4 describes the work of a VLDB program committee and discusses standards and criteria for reviewing submitted papers.
VLDB Conference Proceedings are internationally marketed by Morgan Kaufman Publishers, Inc., San Mateo, California. They have developed a detailed set of instructions and guidelines for the conference organizers to follow in order to achieve a high quality and a uniform appearance of the proceedings each year. These instructions, called "The VLDB Proceedings Management Guide", are available from the Endowment.
The Program Committee guidelines are a supplement to the "VLDB Annual Conference Principles, Policies, and Guidelines" (issued by the VLDB Endowment, revised version, September 1991). Program committee chairpersons should also study these recommendations carefully.
The program committee is organized in two or three regional subcommittees, each headed by a PC co-chairperson. The PC is then collectively managed by these co-chairpersons. The number of PC members should be adequate so as to avoid an excessive refereeing load. A PC member should not be responsible for reviewing more than 10 papers himself. In a two-level organization this number might be increased to a maximum of 20. The number of members in each regional subcommittee should be proportional to the expected number of papers from that region.
The following are examples of the structure of typical regional subcommittees:
For roles and responsibilities of program co-chairpersons and other officers, please consult the "VLDB Annual Conference Principles, Policies, and Guidelines".
2.2 One- or two-level organization?
A one-level organization implies that each PC member himself reviews all papers allocated to him. No other referees are involved. In a two-level organization, PC members act mainly as dispatchers of papers to referees they have selected. PC members are, however, fully responsible for the timely collection of the reviews (and forwarding them to the PC chair), and for the quality of the work performed.
Recent VLDB conferences have had about 300 submissions and, consequently, the need for about 900 referee reports. The total size of the program committee has been 70 - 90 PC members. This has implied a load of more than 10 papers per PC member. The Program Committees have obviously followed a mixed one and two level approach, depending on the availability of second level reviewers of acceptable experience and competence.
The Endowment does not wish to prescribe any particular level structure to follow. The overall goal of the PC should be high quality reviewing work and a deep consideration of those who have made the effort to submit a paper.
2.3 Selecting a good committee
VLDB conference organizers bear the burden and honour of the organization and quality of the programs presented. The program committees are carrying a substantial part of the burden. The program committee chairpersons should select their program committee members very carefully. They should explain to potential members what is required of them and solicit their pledges.
VLDB conferences have been in existence for a number of years. Over the years, we have developed a "corporate memory" of good program committee members, as well as of program committee members who seem to be over-committed and not very responsive. This memory is vested in the previous program committee chairs. One way to take advantage of the "corporate memory" is to consult previous program committee chairpersons, to identify conscientious former committee members, and to solicit their help. Pre-screening program committee members is one way to have successful committees.
The following recommendations can be suggested regarding the selection of PC members (not in order of priority):
A VLDB conference normally runs for 3.5 to 4 full days. The program committee chairpersons are responsible for designing the program. The conference program consists of:
See also Section 20 of "VLDB Annual Conference Principles, Policies, and Guidelines" (issued by the VLDB Endowment, revised version, September 1991).
The number of accepted technical papers is normally about 50. A typical VLDB conference program has three parallel streams. One stream is for tutorials, if they are run during the conference. The other two are used for presentation of technical papers and for panel discussions. It is, however, up to the program committee chairs, together with the main officers of the conference, to make an adequate structure for a particular VLDB conference.
The Endowment does not wish to formulate any particular recommendation about the detailed conference structure. The Endowment recommends, however, that the conference officers establish close cooperation with the VLDB Journal editors on advertising and paper solicitation. Conference organizers may also elect to cooperate with the Journal editors by considering the best conference papers for publication in the Journal.
The acronym VLDB indicates the main conference theme must focus on the broad topic of databases, their design, implementation, operation, maintenance, software support, use/application, and related disciplines.
It is up to the organizing body and the program chairpersons to set up the solicited topic list of a particular conference and/or to encourage submissions of a particular kind. The only policy rule the Endowment wishes to state is to select a set of topics which is up-to-date with respect to the state of the art and important directions n database research, and which exhibit constructive relationships to research and development in related disciplines. Theoretical, conceptual, as well as systems- and engineering oriented papers of relevance to business and industry should be solicited.
A typical VLDB conference contains 45 - 50 technical papers. This is not a limit. Depending on the quality of the submissions, the number of accepted papers can be adjusted by, for example, introducing more parallel sessions. Experience says, however, that the number of submitted papers where all three or four referees recommend unconditional acceptance, is normally less than 30. It is therefore normally not a problem to accommodate all good papers if the target is about 45 - 50 accepted papers.
A problem may arise in the rare case where there are more than 45 papers with excellent reviews. The policy rule here is to "never reject a good paper". The PC has a free hand to structure the conference program to conform to this rule. The only policy statement the Endowment wishes to make is to allocate a minimum of 30 minutes to each accepted paper (including discussions). Furthermore, the Endowment does not wish to encourage the acceptance of extended abstracts, or to schedule abstract sessions, poster sessions, and the like. There should be only one type of accepted paper at a VLDB conference. This does not bar the excellent idea of introducing a "best paper award" and "best paper session" in order to stimulate submission of high quality research and development work.
Additional recommendations regarding technical papers are given in Section 20 of the "VLDB Annual Conference Principle, Policies, and Guidelines (issued by the VLDB Endowment, revised version, September 1991).
Tutorials can be made an attraction of the Conference. Although they are primarily designed for an audience from business and industry, many seasoned researchers and developers of database systems also enjoy tutorials. Database experts tend to be highly specialized in their own sub-fields and need to be introduced via tutorials into other related areas of database systems outside their specialities. In other words, tutorials are good for all participants. Good tutorials on timely topics, and having a common underlying theme of high industrial relevance, will greatly improve the conference's attendance. Half-day tutorials of a three-hour lecture and half-hour discussion should, therefore, be seriously considered.
Tutorials can be part of the conference, be scheduled before, during, or after the conference. Tutorials can be of two kinds: general tutorials and technical tutorials. General tutorials cover broad areas and are normally intended for practitioners. They are normally scheduled as pre- or post-conference events. Technical tutorials, addressing more focused research areas, are usually run in parallel with the rest of the conference program.
The tutorial program must meet the high quality standard of a VLDB conference. Therefore, the PC is in charge of deciding the structure and the content of the tutorial program, as well as of actually running the tutorial program. The PC is responsible for naming a tutorial chairperson(s), and other tutorial officers needed, and for issuing particular Guidelines for that conference's tutorial program.
Additional recommendations for planning and organizing tutorials are:
Panel discussions may substantially contribute to the conference by creating interesting and enlightened discussions on novel or controversial topics. Too often, however, panel discussions have been boring and contributed little. If the program committees are considering panel discussions, they must be fully aware of the need to prepare the discussions well, and to select good topics as well as good discussants. (!) Preparations for a panel should not be at the last minute, but done well in advance at the latest, at the time of the final program committee meeting. The Endowment's recommendations are:
Additional hints are:
3.5 Roles of participants (authors, panelists)
To increase participation (and quality), we should ask participants to play only one role in the VLDB conference. Unless they are indispensable to a panel, authors with a paper or tutorial to present should not be included as panelists. There are so many talented people in the database and related fields, we surely can spread the roles, including session chairs, among a large number of individuals.
The conference PC should give special consideration to the authors of rejected papers as well. Many rejected papers have interesting ideas. The rejection may be due to inadequate development of material, unsubstantiated claims of results, poor English, or other unfavourable facts. A paper nevertheless indicates the authors' background, beliefs, and results on the subject. The PC should keep the authors of rejected papers in mind as potential contributors to panels, etc.
The work of the program committees is tightly integrated with the work of preparing the VLDB conference. Appendix II of the "VLDB Annual Conference Principles, Policies, and Guidelines" (issued by the VLDB Endowment, revised version, September 1991) gives a sample conference plan and lists a number of checkpoints to be considered. The PC Chairs should schedule submission deadlines and letters of acceptance or rejection so as to provide the authors with adequate time to plan their submissions or participation in other related conferences.
The reviewing process is based on issuing, completing, and communicating a number of working documents. These are:
The detailed design of these documents is the responsibility of the PC chairs. It is important that these documents are conceptually equivalent and consistent in all regional program committees, in order to make integration of results possible. Objectives and guidelines regarding these documents are outlined below.
The letter of invitation to join the program committee should:
The letter requesting reviewing work is dispatched together with a number of copies of submitted papers and a corresponding number of referee report forms. This letter should:
The referee report form should be carefully designed to reflect the refereeing quality criteria adopted by the program committee. The following objectives are recommended:
The referee should be definitely discouraged from making a sloppy job by only ticking a number of boxes.
Acceptance as well as rejection letters must be accompanied with copies of those parts of referee reports that are to be disclosed to the authors. The acceptance letter should not present any problem to write. On the other hand, the letter indicating a rejected paper should be designed with the greatest care. It is the obligation and the responsibility of a VLDB program chairperson to give special consideration to authors of rejected papers in particular papers which show promise but are not yet mature, contain interesting ideas, and show future potential. It is advised that the PC chairpersons use the rejection letter to add personal, constructive, and encouraging comments.
The last in our set of typical working documents is the reminder letter. It is a short letter, issued about two weeks before the referee report deadline to all PC members. It simply states that the reviews are due in two weeks. Experience shows that this action results in almost a 100% return rate of the refereeing work. Of course, e-mail communication is ideal for reminder letters.
4.3 Reviewing standards and criteria
General criteria and recommendations for reviewing scientific and technical papers can be found in the editorial sections of almost all computing periodicals and journals. In this sense, VLDB does not differ from other publications. For instance, the referee's task is very illustratively described by Alan Jay Smith in IEEE Computer, April, 1990. The VLDB program committees' chairs and members are strongly advised to study this generally applicable document.
Regarding VLDB submission reviewing, the following special rules and recommendations are given:
For additional information, the handling of papers submitted elsewhere, and the policy regarding invited papers, see Section 17 of the "VLDB Annual Conference Principles, Policies, and Guidelines" (issued by the VLDB Endowment, revised version, September 1991).
4.4 Program committee meetings
The time available for a program committee meeting is usually limited to one day. It is therefore necessary to plan and to exploit this time as effectively as possible. Remember that most of the attending program committee members have travelled quite a distance to attend the meeting. A minor part of the meeting has to be reserved to discuss the general layout of the program, i.e., the type and number of tutorials, parallel sessions, time slots, topics for panels, invited speakers, etc. The major part of the meeting should be devoted to a discussion of the papers and referee reports. We should ensure that most of the discussion is concentrated on controversial papers, i.e., the papers where the referees do not agree.
A well proven technique is to prepare three lists and to make them available for program committee members attending the meeting. The first list contains the names of the authors, titles of the papers, and the referee reports, in condensed form. The second list ranks the papers in the order of overall judgement. The ranking could be made using the final result of each referee's report, e.g., "reject, weak accept, ... strong accept", or following some other numerical score, possibly a weighted average of grades of different assessment factors. The third list contains all papers that have been judged in the category of accepted by the PC members of the final meeting. This list is important in keeping track of the accepted papers during the meeting. The second list can be used to start the discussions.
Unfortunately, experience shows that program committee members often send their referee reports at a very late date, sometimes just days before the meeting starts. Previous program committee chairpersons have recommended issuing a reminder letter, or e-mail, to all program committee members about two weeks before the deadlines for the referee reports. In most cases, this has resulted in almost a 100% response rate for referee reports.
VLDB Conference Proceedings are internationally marketed by Morgan Kaufman Publishers, Inc., San Mateo, California. They have developed a detailed set of instructions and guidelines for the conference organizers to follow in order to achieve a high quality and a uniform appearance of the proceedings each year. The instructions, called "The VLDB Proceedings Management Guide" are included as Appendix I to these PC guidelines.
It is strongly recommended that each year's VLDB organization appoints one particular person as that conference's Proceedings Manager. The Proceedings Manager will act as a liaison between the current year's VLDB conference organization in particular its PC chairpersons the VLDB Endowment's Publications Liaison, and Morgan Kaufman Publisher's, Inc., and be in charge of the process of production, shipping, and distribution of the proceedings. The VLDB Endowment's Publication Liaison is a permanent officer with the purpose of following all aspects of the Endowments publications affairs, such as contacts with Morgan Kaufman, Proceedings of Conferences and Tutorial Series, and the VLDB Journal. The Publications Liaison advises conference organizers and the Endowment on said affairs.
VLDB is a high-quality conference aimed at professionals from the scientific community, as well as for professionals from business and industry. There is a "core" or professionals who almost always attend a VLDB conference. But this core is often not sufficient to have a conference economically "break even". The participation of additional business and industry people is therefore welcome. These people, however, do not go to a conference only for its name. They want to go and listen to interesting and relevant topics, and well prepared presentations. It is, therefore, strongly recommended that the conference organizers
For additional recommendations and Guidelines related to a VLDB program committee, see also the "VLDB Annual Conference Principles, Policies, and Guidelines" (issued by the VLDB Endowment, revised version, September 1991). Section 8 of these guidelines gives advice regarding the responsibilities of conference chairpersons. Section 20 gives recommendations regarding the conference program; and Section 23 C covers the issue of supporting PC member travel. Furthermore, Appendix I gives miscellaneous hints for running a conference, and Appendix II shows a sample of a VLDB plan and checkpoints.
The quality of future VLDB Conferences depend upon the way the present VLDB program committees conduct their refereeing practices. If committees do not address the issues of scheduling tutorials and papers, volume of submissions, refereeing criteria and standards, and responses to authors (particularly the rejected authors), then the committee will prejudice future VLDB Conferences. Both the quality and number of future submissions will likely drop. It is with the future in mind that we have compiled these guidelines and recommendations for VLDB program committees. We would like all future program committee chairpersons, members, and referees to read this document carefully, and take it to hear.
The Endowment also wishes to extend its thanks to chairpersons and members of past program committees, and other past conference officers as well as supporters, for many valuable comments and suggestions. They have contributed considerably to the development of this document.
The Endowment would like to endorse the building of collective experiences concerning the organization of the VLDB conference and encourage constructive input from all future VLDB conference officers, as well as conference attendees, in order to improve this document. Suggestions for improvement should be sent to the secretary of the Endowment.