PhysioNetWorks Project Guide

If you would like to publish content on PhysioNet, please use our new site: https://alpha.physionet.org

This page documents the structure and purpose of PhysioNetWorks and its project types. It instructs users on how to develop their own projects, and provides a reference for owners of existing projects.

PhysioNetWorks provides access-controlled collaborative workspaces for researchers to work on software solutions and data preparation, so that the content may be made publicly available via PhysioNet when complete. Unlike other areas of PhysioNet, these workspaces are password-protected.

New users should view the first two sections below: Life cycle of a project and Project and Archive Types, to get an overview of how PhysioNetWorks is organized.

When you are ready, you can get started on your own PhysioNetWorks project as described in Creating a new project below.

Life cycle of a project

PhysioNetWorks projects move through three phases, as illustrated schematically below:

[PhysioNetWorks project lifecyle schematic illustration]

Phase 1 - Private Project
Private projects can be created by any PhysioNetWorks member. During phase 1, the project owner can compose a home page for the project and can upload a limited amount of content to its archive. Each member's private projects occupy space within their owner's personal workspace; they cannot be shared with anyone else.

Transition to active project: The owner of a private project requests activation of the project. A PhysioNet team member reviews the project. If the project satisfies the active project requirements, the project is copied (not moved) to a sharable workspace where the copy becomes an active project. The owner may delete the private project, or keep it as a workspace for previewing new material.

Phase 2 - Active Project
The owner of an active project can accept or invite other PhysioNetWorks members to join the project and share access to the content uploaded by the owner and by any members designated by the owner as collaborators. Non-members can view the home pages of active projects but none of the other uploaded content.

Publishing an active project: When the project satisfies the published project requirements, the project owner can request a review of it so that it can be made public. A PhysioNet team member then works with the project owner to make any final changes, such as the removal of PHI. The project becomes published when the owner and the PhysioNet team agree that it is ready for open access.

Phase 3 - Published Project
The project's contents are copied to an appropriate location within the PhysioNet web site, where they become freely available to all visitors. The project owner can either choose to maintain the active PhysioNetWorks project in order to support interaction with users and make further developments, or to declare the project as complete, in which case the project will be archived. In either case, there will be an indicator on the physionetworks project list directing users to the published database or toolkit.

This is what an example database should look like/contain: https://physionet.org/physiobank/database/example/

top

Project and Archive Types

Before seeking to start a certain project type, it is important to understand the purpose of PhysioNetWorks, and the distinctions between the two PhysioNetWorks project types and public PhysioNet databases and toolkits.

  1. Private Projects

    A private project belongs to and is only accessible by a single owner. Given its limited accessibility and file space allocation (100 MB), it is used as a draft staging area for an intended active project. The PhysioNet team performs basic quality control for the activation of private projects in order to ensure that all active projects have a clear purpose before allocating the project much more space and making it visible to the PhysioNetWorks community

  2. Active Projects

    Active projects are the core of PhysioNetWorks. They are shared projects visible to the entire PhysioNetWorks community (anyone with an account). Their purpose is to provide collaborative workspaces for researchers and developers to work and share solutions for software writing and data preparation. They also act as an essential staging area for PhysioNet public data and software contributions which users are encouraged to provide. Access to an active project's files is controlled by the owner who allows collaborators and reviewers.

  3. Public Databases and Toolkits

    PhysioNetWorks is the only account+password controlled area of PhysioNet. The rest of PhysioNet including PhysioBank and PhysioToolkit is publicly accessible by anyone on the internet. The database and software contributions in these areas have been validated for quality and are freely provided for the benefit of the entire scientific community.

The table below summarizes key features of the two PhysioNetWorks project types, and compares them with PhysioBank data collections:

Private Project → Active (Shareable) Project → PhysioBank Data Collection
Creation By any PhysioNetWorks member, instantly By PhysioNet, from an existing private project that has been submitted by its owner for activation By PhysioNet, from an active project that has been submitted by its owner for posting in PhysioBank
Membership Owner only Owner sets access policy (self-enroll, apply to join, or no access);
owner may add or remove members and change their roles
Open access (no membership required)
Project Title and Home Page Visible to and editable by owner only Visible to all PhysioNetWorks members; owner may edit home page but title may not be changed Visible to anyone
Downloads from Archive Owner via browser Owner via browser or rsync;
collaborators and reviewers via browser
Anyone via browser or rsync
Uploads to Archive Owner via browser Owner via browser (individual files, tarballs or zip archives);
collaborators via browser (no overwriting)
None
Size 100 MB by default, up to 1 GB by request (shared with personal workspace and owner's other private projects) 10 GB by default, up to 100 GB by request (more by arrangement) No preset limit
Backup None Daily and monthly; mirroring by owner optional Daily and monthly; may be mirrored by anyone

top

Project Activation Requirements

These requirements must be satisfied in order for us to activate your project

Publishing Requirements

These requirements must be satisfied in order for us to publish your project.

Physiobank Databases

This is what an example database should look like/contain: https://physionet.org/physiobank/database/example/. Please provide the indicated descriptive fields.

Physiotoolkit Software Packages

In order to add or edit descriptive information about a project, go to your PhysioNetWorks home page, click for the specified project, click , and fill in the Project Abstract section.

top

Creating a New Project

All projects begin as private projects. In that state, they are visible only to their creators and cannot be shared with others.

You can create a private project by clicking near the top of your PhysioNetWorks home page. When you do so, your browser shows instructions and a short form to fill in and submit. (In brief, you will need to enter a name for your project in order to create it, and the form allows you to enter various other information about the project at this time if you wish. You may edit any of this information later on.)

If you have created any private projects, they are visible near the bottom of your PhysioNetWorks home page, below your personal workspace. The project names are shown as links; click on one to view your project as it would appear to a visitor. To the right of the name is a set of three buttons:

Think of a private project as a rough draft. When you're ready to share it (or when it outgrows the available room in your personal workspace):

  1. Make a final check of your project's home page and additional information and make any needed corrections. To do so, go to its Manage page and click . Your browser will show a preview of the home page and additional information about the project, above a form that allows you to make changes. Proofread the project abstract and aim to make it comprehensible to readers who don't already know what you are doing. Make sure that you have selected the access policy that you want initially (you can change it later if you wish), and that you have estimated the storage that will be needed and the completion date. Once the project has been activated, its name cannot be changed; choose its name carefully.
  2. Request activation of your project by clicking button on its Manage page. Your request will be reviewed, and you will receive a reply by email, usually within a day or two.

top

Access and roles

The Projects section of your PhysioNetWorks home page lists all active projects, followed by your personal workspace. If you have created any private projects, they are listed below your personal workspace. Click on any project in the list to visit its home page, which displays its owner's profile, a brief description of the project written by its owner, and the project's access policy.

Your role with respect to each project, which is shown to the left of each project on your home page, determines what you can do within the project:

If you are not a member of a project, your role is shown as '-'; you can visit its home page, but not its archive.

Any private projects you have created appear below the entry for your personal workspace. Private projects use storage within your personal workspace, and neither your personal workspace nor any private projects within it are visible to anyone else. You are the owner of your private projects, for which your role is shown as '*'.

top

Access policy

When creating or editing a project's home page, its owner can set or change the project's access policy. Six choices are available:

  1. PhysioNetWorks members may be allowed to enroll themselves as reviewers, without individual approval by the project owner.
  2. As above, but members enrolling themselves agree to terms and conditions determined by the project owner.
  3. PhysioNetWorks members apply for membership.
  4. As above, but members agree to terms and conditions determined by the project's owner when submitting their applications.
  5. Members may add themselves to a list of those interested in joining the project in the future, when its owner opens it.
  6. Membership applications are not accepted.

If policy 3, 4, or 5 is chosen, the project owner finds a list of interested members on the project's Manage page, and can approve or decline each application after viewing the member's profile (see The roster below).

Regardless of the policy chosen, the owner can view and edit the project roster, and has the option of adding selected PhysioNetWorks members to it at any time.

top

Managing membership applications

Policies 3 and 4 above require frequent attention from the project owner. When a member applies to a project with either of these policies, this notice appears in his or her browser window:

[User]'s application to Test has been received. The owner of the '[Project]' project may request additional information (by email to [User]) before making a decision about your application. Please be patient! The project owner may not be able to review your application for several days (expect longer delays on weekends and holidays).

PhysioNet provides password-protected workspaces for researchers who are developing data sets and software resources that will eventually become available freely via PhysioBank and PhysioToolkit. While this project is hosted on PhysioNetWorks, access to its archive is controlled solely by its owner, who may allow pre-publication access to the archive contents to some, all, or no others. Please respect the decisions made by the owner of this project.

You will receive an email when your application has been reviewed. If it has been approved, your PhysioNetWorks home page will include a 'Review' button in the 'Activities' column, next to the name of this project; click this button to enter the project archive.

If you have chosen policy 3 or 4 for your project, you will receive an automatically generated email reminder on any day when there is at least one pending application for membership. Visit your project's Manage page to review and accept or deny these applications (see The roster below for details).

Applicants may have unrealistic expectations about how rapidly their applications will be approved, despite the message that suggests that several days may be needed. If your project is related to an event that involves a deadline (for example, a competition), be sure that potential applicants understand that they need to allow a reasonable time for you to review their applications, unless you are prepared to sit by your computer day and night without interruption. Alternatively, you may wish to consider using access policy 1 or 2, which allows anyone to join your project as a reviewer immediately, with no review required; you will still be able to see who has joined your project in any case.

If you know that you will be unable to review applications for an extended period, it may help to add a short note to your project's abstract (e.g., "Membership applications received during August will be reviewed in early September.").

top

Summary of access privileges

Roles
PrivilegesOwnerCollaboratorsReviewers
Downloading
Browser downloads
Batch downloads
Uploading
Browser uploads
Batch uploads
Project management
Replace, unpack, rename, delete files
Download from backup archives
Invite collaborators and reviewers
Change roles of collaborators and reviewers
Hand off ownership

top

Restrictions on file names

The names of files stored in PhysioNetWorks projects may contain upper- and lower-case unaccented Roman letters (A-Z), numerals (0-9), and the characters .-+_ (full stop, hyphen, plus, underscore). Any other characters in the names of uploaded files will be replaced by hyphens in the names of the stored files. Examples:

Original nameName in PhysioNetWorks project
My data - set 11.xml My-data---set-11.xml
E:\xfer\notes.doc E--xfer-notes.doc

top

File formats

To make it as easy as possible to share data using PhysioNetWorks, we do not require use of specific formats (although naturally those with whom you wish to share data will need to know how to read them). We encourage use of open formats in PhysioNetWorks projects, and we require open formats in PhysioBank data collections. If you need assistance in converting your data into suitable open formats, we will help.

By an open format, we mean one that has a freely available and complete specification, permitting creation of open-source software that reads files written in that format. This is necessary for open-access files, since we must be able to verify that PHI and other content that may not be redistributed have been removed from such files.

We also strongly encourage the use of common open formats when possible. Many researchers looking for data find multiple collections of interest in PhysioBank, and those collections that can be accessed using familiar software are most likely to be used and cited. Your data set is more likely to be useful to others if it does not require learning how to use a unique set of tools for exploring and analyzing its contents.

See Creating PhysioBank (WFDB-compatible) Records for a tutorial on this subject.

top

Reading files from an archive (downloading)

Two methods of downloading are available. Browser downloads of archive files work in the same way as any other downloads, and are convenient if you have only a small number of files to copy at a time. Batch downloads are available to the project owner only; they are more efficient if you have many files, they allow you to mirror the organization of your archive, and they avoid unnecessary downloads of files that have not changed since the last time you downloaded them.

top

Browser downloads

Navigate from your PhysioNetWorks home page to the home page of the project that contains the file you want to download. If your files are organized in directories (folders) in your archive, you may need to navigate further until the file is visible.

The details of downloading a file depend on your browser, not on any special property of the archive or of PhysioNetWorks. General tips on downloading using popular browsers can be found in the PhysioNet FAQ.

top

Batch downloads (mirroring a project)

Only the owner of an project may perform batch downloads. The first time you make a batch download, perform these preparatory steps:

  1. Install rsync if you have not already done so.
  2. Make a backup directory (folder) on your computer.
  3. On your project's Manage page, find the link to its secure batch download key, and download and save this short text file.
  4. On Linux, Mac OS X, Cygwin/MS-Windows, and other platforms with Unix-style access controls, run this command to secure the key file:

         chmod 600 keyfile
    

    replacing keyfile with the name of the text file you saved in the previous step. It should not be possible for other users of your computer to read this file (rsync will refuse to accept it if it does not have sufficiently restricted access).

You won't need to repeat these steps subsequently.

To make a complete local backup of your project, run this command in a terminal window:

    rsync -avz -e "ssh -i keyfile -l pnw" physionet.org::ProjectName BackupName

replacing ProjectName with the abbreviated name of your project (see below), and BackupName with the name of your backup directory.

The abbreviated name of your project is its name, with spaces and punctuation deleted. Thus, for example, the abbreviated name of a project named HRV in Ex-Parrots is HRVinExParrots. If you are unsure, run this command:

    rsync -avz -e "ssh -i keyfile -l pnw" physionet.org::

which will display the abbreviated name followed by the full name in the terminal window.

If you have made a batch download of your project previously, using the same backup directory, rsync will download only those files in the project that are new or have changed since your last download. rsync will preserve the organization of the files as they are arranged in your project, creating subdirectories as needed if they do not exist already.

If a batch download is interrupted for any reason, simply repeat the rsync command, and rsync will complete the download without resending data that have been transferred already.

If you have software such as cron that can perform scheduled tasks, you can add the first rsync command above to your daily or weekly task lists, so that your mirror can be kept up-to-date automatically.

top

Sending files to an archive (uploading)

[WARNING] All transfers are logged. You are responsible for the contents of any files you upload. Be sure that you have permission to copy them. Do not share files containing protected health information (PHI).

Two methods of uploading are available, and are described below. Browser uploads are convenient if you have only a small number of files to upload at a time. Batch uploads are available only to the project owner; they are more efficient if you have many files, and they allow you to organize your files in directories. You can use whichever method you prefer at the time of each upload. The entire transfer is encrypted in either case.

top

Browser uploads

To upload a file using your browser, click or to the right of your project's name on your your PhysioNetWorks home page. (If you are a collaborator of the project, is not present; if you are a reviewer, neither button is present, and you may not upload files to the project.)

If you use , a new page containing , , and buttons appears. (Your browser may display or similar rather than .) Click on and select the file on your own computer from the dialog box, then click on to begin the transfer. You may not overwrite an existing file of the same name in this way; rename it on your own computer first if necessary. Use to return to your home page.

The project owner has additional options using button. Near the top of the Manage page, check Allow overwriting in order to enable these options. In addition to and buttons, more controls allow the owner to rename or remove files that are already in the archive.

Files uploaded using your browser are collected into the top-level directory (folder) in your data archive.

Tip: Viewing progress during long browser uploads

Although all modern browsers support the secure SSL-encrypted (HTTPS) uploads described above, most do not include status indicators. A notable exception is Chrome, which displays a progress indicator at the lower left corner while an upload is in progress, although this indicator may freeze for 20-30 seconds while uploading continues. Firefox users can install the UploadProgress add-on, which displays a progress bar in the lower right corner, and also adds an Uploads window (similar to the Downloads window) to the Tools menu.

top

Batch uploads

To upload a batch of files in a single transfer:

  1. Organize your files into directories (folders) as you would like to have them appear on-line in your project.
  2. Pack them into a tarball or zip file (using software such as tar, PKZIP, or WinZip; see Wikipedia's list of file archivers for links to software that can create compatible archives).
  3. Upload the tarball or zip file to your project as described above.
  4. On your project's Manage page, select the tarball or zip file and unpack it into the project.

Note that step 4 will not overwrite existing files in the project unless you have checked Allow overwriting first (see above).

If you are a collaborator in a project, you can do steps 1-3 and ask the owner to perform step 4.

The original tarball or zip archive remains in the project's archive unless it is deleted by the owner.

top

Project management

From your PhysioNetWorks home page, click on next to your project's name to go to its Manage page. ( is visible only to the project's owner.)

top

Replacing a project file

To protect against accidental data loss, overwriting is disabled by default. To replace a file with another of the same name:

  1. Check (click on) the Allow overwriting checkbox near the top of the Manage page.
  2. Upload the replacement file.

Overwriting remains allowed until you leave the Manage page.

top

Unpacking a project file

The project owner may unpack (extract the files contained within) a previously uploaded archive file if that file is in a supported format:

  1. Scroll down to the heading Operate on an existing file.
  2. Choose the file you wish to unpack from the drop-down list below the heading.
  3. Click .

Currently supported formats include uncompressed .tar archives; gzip (.tar.gz or .tgz), bzip2 (.tar.bz2), and compress (.tar.z) compressed tar archives, and .zip (PKZIP, WinZip, etc.) archives.

Existing files will not be overwritten by unpacked files unless Allow overwriting has been checked.

top

Renaming a project file

To rename an existing file:

  1. Scroll down to the heading Operate on an existing file.
  2. Choose the file you wish to rename from the drop-down list below the heading.
  3. Type the new file name in the text box provided.
  4. Click .

If a file with the chosen new name exists already, renaming will fail unless Allow overwriting has been checked.

top

Deleting a project file

To delete an existing file:

  1. Scroll down to the heading Operate on an existing file.
  2. Choose the file you wish to delete from the drop-down list below the heading.
  3. Click .

top

Snapshots (daily and monthly backups)

A snapshot is a copy of some or all of the files in an active project. Only the project's owner has access to the project's snapshots, and that access is limited to browser downloads. Snapshots for private projects are not available.

For each data project, the PhysioNetWorks server automatically creates a snapshot daily at about 3am US Eastern time. Daily snapshots are kept for one month, and the last snapshot taken each month is kept for at least a year.

At the end of the Archive section of your project's Manage page, find the heading Snapshots (daily and monthly backups) and follow the link there to previous versions of your project.

To restore a previous version of a file to a data archive, simply download it from the backup archive in which it appears, and then upload it to the data archive. To replace an existing file with a previous version, delete the existing file, then restore it.

top

The roster (active projects only)

A project's roster is a list of all those who have access to its archive (the owner and his or her collaborators and reviewers). The roster appears at the end of the project's Manage page, below the heading Current roster. Since a private project cannot be shared, it does not have a roster.

Those who will be listed on the roster of an active project must become members of PhysioNetWorks first (by creating personal PhysioNetWorks accounts) if they have not already done so. Initially, the only entries on the roster are the owner and one reviewer (the PhysioNet team member who reviewed and approved the project). Once the project is active, PhysioNet's reviewer will not examine the contents of the project's archive unless the owner specifically requests help.

The project owner can change the roles of collaborators and reviewers, and may remove them from the roster entirely. Do any of these things by selecting an email address on the roster, and then selecting one of the available actions. Note that you cannot remove yourself from your project's roster (accidentally or otherwise).

If you are the owner of a project, you may give any other PhysioNetWorks member access to the archive as a collaborator or reviewer, with the privileges outlined above.

If you chose an access policy for your project that requires interested members to apply for access, or to indicate their interest in joining the project when access becomes available (policy 3, 4, or 5 above), the email addresses (PhysioNetWorks account names) of any such members will be listed immediately below the heading Add, reassign, or remove a member. Several actions are possible:

You may add members to your project at any time, even if they have not applied, and even if your access policy does not offer a way for other members to join or apply, by entering the member's email address in the text box provided in the Add, reassign, or remove a member section. Use the buttons below this text box to set the member's role in your project. If necessary, a member can be removed from a roster using . Members can appear in only one location in the roster, so you can reassign a reviewer as a collaborator (for example) simply by entering the reviewer's email address and clicking on ; the email address will be removed from the list of reviewers and added to the list of collaborators.

Note, however, that you cannot add an address that has not been registered. If you find that you are unable to add an address of someone known to be a PhysioNetWorks member, you will need to find out which address was registered by contacting the member directly. (For privacy reasons, PhysioNetWorks does not maintain a publicly-accessible directory of members; for security reasons, PhysioNetWorks requires verification of all email addresses and acceptance of its terms of use as a condition of registration.)

Whenever you add members to your project, change their roles, or remove them from your project's roster, these changes are reflected on their PhysioNetWorks home pages immediately, and they receive email notifications of the changes.

top

Transferring archive ownership

As noted above, the owner of an archive has privileges that are not shared with any other member, but an archive has only one owner at any given time. It may be useful to allow more than one member to exercise the owner's privileges. This is possible if the owner hands off ownership to another member, a process that can be repeated as often as necessary.

This feature is available by request.

top

Making a completed project archive public

When your work is complete and you are ready to make it publicly available via PhysioNet, request that PhysioNet review it. Do this by clicking the mailto: link in the Request review of this active project section of your project's Manage page (or send a note to PhysioNet's reviewer for your project, or to editor@physionet.org). The reviewer will conduct a final screen for PHI, and will work with you to identify any changes that may be needed in order for your data to satisfy requirements for inclusion in PhysioBank. You may be asked to supply additional information that will help future users of your archive to understand its contents and how to make use of them. You will also have an opportunity to edit the brief description of your archive that will be used to announce its availability.

When you and the reviewer are satisfied with the state of your work, the reviewer will copy its contents into a publicly accessible area of PhysioBank and announce its availability. Your protected archive will continue to be available as well, so that if you wish you may continue to add to it and to correct any errors discovered after its initial public release.

Often a completed archive is made public at the time a paper that refers to the archive's contents is accepted for publication. Such a paper is an ideal location in which to document the availability and location of the data. The deadlock that may develop in this situation (the data aren't available until the paper is accepted, and the paper won't be accepted until the data are available) can often be resolved with the help of the PhysioNet reviewer, who can provide a publicly accessible, permanent location that can be cited in the manuscript, and who can store a sample of the data in that location pending acceptance of the paper.

top

Sharing works-in-progress publicly

Some researchers may wish to share their projects' archives before they are in final form. Among the reasons for doing so are:

If these or other reasons encourage you to make a work-in-progress public, here are two routes to consider:

Either route can help you to identify potential collaborators with the interest and expertise to help you complete your work-in-progress.

top

rsync

rsync is required for project owners wishing to mirror (maintain complete copies of) their PhysioNetWorks projects. rsync is free and open source software. If you don't have it already, get it here (link opens in another window).

The algorithm used by rsync identifies a minimal set of data that must be copied to achieve the intended result, a set that includes:

For the typical case in which the sets of source and destination files are similar, rsync provides a huge bandwidth usage and speed advantage (often a factor of 10,000 or more) over alternatives such as ftp and similar methods.

Although rsync is capable of uploading as well as downloading files, PhysioNetWorks projects are protected against uploading using any means other than browser uploads.

When used with the option string -avz, as in all of the examples on this page, rsync does several useful things as it transfers your files from your archive. The a option in rsync's option string causes it to preserve the timestamps and other attributes of the files transferred. The v option causes rsync to display the names of the files it transfers, and the z option causes them to be compressed before transfer and decompressed by the receiver. rsync has many other options that might be useful; for brief descriptions of all of them, see the rsync manual (link opens in another window).

top

Questions and Comments

If you would like help understanding, using, or downloading content, please see our Frequently Asked Questions.

If you have any comments, feedback, or particular questions regarding this page, please send them to the webmaster.

Comments and issues can also be raised on PhysioNet's GitHub page.

Updated Friday, 8 March 2019 at 16:08 EST

PhysioNet is supported by the National Institute of General Medical Sciences (NIGMS) and the National Institute of Biomedical Imaging and Bioengineering (NIBIB) under NIH grant number 2R01GM104987-09.