Command Line Tools¶
Example Use Cases¶
Download public files for a given project/user¶
You can use ohpub-download
on the command line to download all public files
associated with a project. Here is an example that will download all public
23andMe files:
mkdir 23andme # create new folder to store the data in
ohpub-download --source direct-sharing-128 --directory 23andme
To do the same for all public files of a given user:
mkdir gedankenstuecke # create new folder to store the data in
ohpub-download --username gedankenstuecke --directory gedankenstuecke
Download private files from members of your project¶
If you are running a project yourself you can use a master_access_token
you can get from the Open Humans website
to download the files shared with you.
Here is an example of how to do this:
mkdir my_downloaded_data # create new folder to store the data in
ohproj-download --master-token my_master_access_token --directory my_downloaded_data
Upload files to the accounts of your project members¶
Uploading data into your members accounts is two-step procedure using both
ohproj-upload-metadata
as well as ohproj-upload
, as you first need to draft
some metadata that will go along with the files.
Both commands expect a directory that contains sub-directories named after
your members project_member_id
. An example directory structure with files could look like this:
- member_data/
- 01234567/
- testdata.json
- testdata.txt
- 12345678/
- testdata.json
- testdata.txt
- 01234567/
To upload this data we need to have a CSV
file that contains the metadata.
We can draft one using the following command:
ohproj-upload-metadata -d member_data --create-csv member_data_metadata.csv
The resulting CSV will look like this:
$ cat member_data_metadata.csv =>
project_member_id,filename,tags,description,md5,creation_date
01234567,testdata.txt,,,fa61a92e21a2597900cbde09d8ddbc1a,2016-08-23T15:23:22.277060+00:00
01234567,testdata.json,json,,577da9879649acaf17226a6461bd19c8,2016-08-23T16:06:16.415039+00:00
12345678,testdata.txt,,,fa61a92e21a2597900cbde09d8ddbc1a,2016-09-20T10:10:59.863201+00:00
12345678,testdata.json,json,,577da9879649acaf17226a6461bd19c8,2016-09-20T10:10:59.859201+00:00
Edit this CSV with a text or spreadsheet editor of your choice to contain the metadata and then save it as a CSV again:
$ cat member_data_metadata.csv =>
project_member_id,filename,tags,description,md5,creation_date
1234567,testdata.txt,"txt, verbose-data",Complete test data in text format.,fa61a92e21a2597900cbde09d8ddbc1a,2016-08-23T15:23:22.277060+00:00
1234567,testdata.json,"json, metadata",Summary metadata in JSON format.,577da9879649acaf17226a6461bd19c8,2016-08-23T16:06:16.415039+00:00
12345678,testdata.txt,"txt, verbose-data",Complete test data in text format.,fa61a92e21a2597900cbde09d8ddbc1a,2016-09-20T10:10:59.863201+00:00
12345678,testdata.json,"json, metadata",Summary test data JSON.,577da9879649acaf17226a6461bd19c8,2016-09-20T10:10:59.859201+00:00
We filled in the tags as a "
-escaped and comma-separated list as well as a text-description.
With this we can now perform the actual upload like this:
ohproj-upload -T YOUR_MASTER_ACCESS_TOKEN --metadata-csv member_data_metadata.csv -d member_data
This will upload the data from the member_data
directory along with the metadata specified in member_data_metadata.csv
.
Using an OAuth2 access_token
¶
The CLI tools also accept the usage of an OAuth2 access_token``s instead of an ``master_access_token
.
This enables you to use the CLI tools in your own OAuth2 applications if you’re not using Python for your development.
Here is an example to download all the data shared by a single member through the command line:
ohproj-download -t personal_access_token -m '12345678' --directory download_directory
List of commands and their parameters¶
ohpub-download¶
Command line tools for downloading public data.
ohpub-download [OPTIONS]
Options
-
-s
,
--source
<source>
¶ the source to download files from
-
-u
,
--username
<username>
¶ the user to download files from
-
-d
,
--directory
<directory>
¶ the directory for downloaded files
-
-m
,
--max-size
<max_size>
¶ the maximum file size to download
-
-q
,
--quiet
¶
Report ERROR level logging to stdout
-
--debug
¶
Report DEBUG level logging to stdout.
ohproj-download¶
Command line function for downloading data from project members to the
target directory. For more information visit
download
.
ohproj-download [OPTIONS]
Options
-
-d
,
--directory
<directory>
¶ Target directory for downloaded files. [required]
-
-T
,
--master-token
<master_token>
¶ Project master access token.
-
-m
,
--member
<member>
¶ Project member ID.
-
-t
,
--access-token
<access_token>
¶ OAuth2 user access token.
-
-s
,
--source
<source>
¶ Only download files from this source.
-
--project-data
¶
Download this project’s own data.
-
--max-size
<max_size>
¶ Maximum file size to download. [default: 128m]
-
-v
,
--verbose
¶
Report INFO level logging to stdout
-
--debug
¶
Report DEBUG level logging to stdout.
-
--memberlist
<memberlist>
¶ Text file with whitelist IDs to retrieve
-
--excludelist
<excludelist>
¶ Text file with blacklist IDs to avoid
-
--id-filename
¶
Prepend filenames with IDs to ensure uniqueness.
ohproj-download-metadata¶
Command line function for downloading metadata.
For more information visit
download_metadata
.
ohproj-download-metadata [OPTIONS]
Options
-
-T
,
--master-token
<master_token>
¶ Project master access token. [required]
-
-v
,
--verbose
¶
Show INFO level logging
-
--debug
¶
Show DEBUG level logging.
-
--output-csv
<output_csv>
¶ Output project metedata CSV [required]
ohproj-upload-metadata¶
Command line function for drafting or reviewing metadata files.
For more information visit
upload_metadata
.
ohproj-upload-metadata [OPTIONS]
Options
-
-d
,
--directory
<directory>
¶ Target directory [required]
-
--create-csv
<create_csv>
¶ Create draft CSV metadata
-
--review
<review>
¶ Review existing metadata file
-
--max-size
<max_size>
¶ Maximum file size to consider. [default: 128m]
-
-v
,
--verbose
¶
Show INFO level logging
-
--debug
¶
Show DEBUG level logging.
ohproj-upload¶
Command line function for uploading files to OH.
For more information visit
upload
.
ohproj-upload [OPTIONS]
Options
-
-d
,
--directory
<directory>
¶ Target directory for downloaded files. [required]
-
--metadata-csv
<metadata_csv>
¶ CSV file containing file metadata. [required]
-
-T
,
--master-token
<master_token>
¶ Project master access token.
-
-m
,
--member
<member>
¶ Project member ID.
-
-t
,
--access-token
<access_token>
¶ OAuth2 user access token.
-
--safe
¶
Do not overwrite files in Open Humans.
-
--sync
¶
Delete files not present in local directories.
-
--max-size
<max_size>
¶ Maximum file size to download. [default: 128m]
-
-v
,
--verbose
¶
Report INFO level logging to stdout
-
--debug
¶
Report DEBUG level logging to stdout.
ohproj-oauth2-token-exchange¶
Command line function for obtaining the refresh token/code.
For more information visit
oauth2_token_exchange
.
ohproj-oauth2-token-exchange [OPTIONS]
Options
-
-cid
,
--client_id
<client_id>
¶ client id of user. [required]
-
-cs
,
--client_secret
<client_secret>
¶ client secret of user. [required]
-
-re_uri
,
--redirect_uri
<redirect_uri>
¶ redirect_uri of user [required]
-
--base_url
<base_url>
¶ base url of Open Humans [default: https://www.openhumans.org/]
-
--code
<code>
¶ code of user
-
-rt
,
--refresh_token
<refresh_token>
¶ refresh token of user
ohproj-oauth2-url¶
Command line function for obtaining the Oauth2 url.
For more information visit
oauth2_auth_url
.
ohproj-oauth2-url [OPTIONS]
Options
-
-r
,
--redirect_uri
<redirect_uri>
¶ Redirect URL for project
-
-c
,
--client_id
<client_id>
¶ Client ID for project [required]
-
--base_url
<base_url>
¶ Base URL
ohproj-message¶
Command line function for sending email to a single user or in bulk.
For more information visit
message
.
ohproj-message [OPTIONS]
Options
-
-s
,
--subject
<subject>
¶ subject [required]
-
-m
,
--message_body
<message_body>
¶ compose message [required]
-
-at
,
--access_token
<access_token>
¶ access token [required]
-
--all_members
<all_members>
¶ all members [default: False]
-
--project_member_ids
<project_member_ids>
¶ list of comma-separated project_member_ids. Example argument: “ID1, ID2”
-
-v
,
--verbose
¶
Show INFO level logging
-
--debug
¶
Show DEBUG level logging.
ohproj-delete¶
Command line function for deleting files.
For more information visit
delete_file
.
ohproj-delete [OPTIONS]
Options
-
-T
,
--access_token
<access_token>
¶ Access token [required]
-
-m
,
--project_member_id
<project_member_id>
¶ Project Member ID [required]
-
-b
,
--file_basename
<file_basename>
¶ File basename
-
-i
,
--file_id
<file_id>
¶ File ID
-
--all_files
<all_files>
¶ Delete all files [default: False]
-
ohapi.command_line.
download
(directory, master_token=None, member=None, access_token=None, source=None, project_data=False, max_size='128m', verbose=False, debug=False, memberlist=None, excludelist=None, id_filename=False)¶ Download data from project members to the target directory.
Unless this is a member-specific download, directories will be created for each project member ID. Also, unless a source is specified, all shared sources are downloaded and data is sorted into subdirectories according to source.
Projects can optionally return data to Open Humans member accounts. If project_data is True (or the “–project-data” flag is used), this data (the project’s own data files, instead of data from other sources) will be downloaded for each member.
Parameters: - directory – This field is the target directory to download data.
- master_token – This field is the master access token for the project. It’s default value is None.
- member – This field is specific member whose project data is downloaded. It’s default value is None.
- access_token – This field is the user specific access token. It’s default value is None.
- source – This field is the data source. It’s default value is None.
- project_data – This field is data related to particular project. It’s default value is False.
- max_size – This field is the maximum file size. It’s default value is 128m.
- verbose – This boolean field is the logging level. It’s default value is False.
- debug – This boolean field is the logging level. It’s default value is False.
- memberlist – This field is list of members whose data will be downloaded. It’s default value is None.
- excludelist – This field is list of members whose data will be skipped. It’s default value is None.
-
ohapi.command_line.
download_metadata
(master_token, output_csv, verbose=False, debug=False)¶ Output CSV with metadata for a project’s downloadable files in Open Humans.
Parameters: - master_token – This field is the master access token for the project.
- output_csv – This field is the target csv file to which metadata is written.
- verbose – This boolean field is the logging level. It’s default value is False.
- debug – This boolean field is the logging level. It’s default value is False.
-
ohapi.command_line.
set_log_level
(debug, verbose)¶ Function for setting the logging level.
Parameters: - debug – This boolean field is the logging level.
- verbose – This boolean field is the logging level.
-
ohapi.command_line.
upload
(directory, metadata_csv, master_token=None, member=None, access_token=None, safe=False, sync=False, max_size='128m', mode='default', verbose=False, debug=False)¶ Upload files for the project to Open Humans member accounts.
If using a master access token and not specifying member ID:
(1) Files should be organized in subdirectories according to project member ID, e.g.:
main_directory/01234567/data.json main_directory/12345678/data.json main_directory/23456789/data.jsonThe metadata CSV should have the following format:
1st column: Project member ID 2nd column: filenames 3rd & additional columns: Metadata fields (see below)
If uploading for a specific member: (1) The local directory should not contain subdirectories. (2) The metadata CSV should have the following format: 1st column: filenames 2nd & additional columns: Metadata fields (see below)
The default behavior is to overwrite files with matching filenames on Open Humans, but not otherwise delete files. (Use –safe or –sync to change this behavior.)
If included, the following metadata columns should be correctly formatted: ‘tags’: should be comma-separated strings ‘md5’: should match the file’s md5 hexdigest ‘creation_date’, ‘start_date’, ‘end_date’: ISO 8601 dates or datetimes
Other metedata fields (e.g. ‘description’) can be arbitrary strings. Either specify sync as True or safe as True but not both.
Parameters: - directory – This field is the target directory from which data will be uploaded.
- metadata_csv – This field is the filepath of the metadata csv file.
- master_token – This field is the master access token for the project. It’s default value is None.
- member – This field is specific member whose project data is downloaded. It’s default value is None.
- access_token – This field is the user specific access token. It’s default value is None.
- safe – This boolean field will overwrite matching filename. It’s default value is False.
- sync – This boolean field will delete files on Open Humans that are not in the local directory. It’s default value is False.
- max_size – This field is the maximum file size. It’s default value is None.
- mode – This field takes three value default, sync, safe. It’s default value is ‘default’.
- verbose – This boolean field is the logging level. It’s default value is False.
- debug – This boolean field is the logging level. It’s default value is False.
-
ohapi.command_line.
upload_metadata
(directory, create_csv='', review='', max_size='128m', verbose=False, debug=False)¶ Draft or review metadata files for uploading files to Open Humans. The target directory should either represent files for a single member (no subdirectories), or contain a subdirectory for each project member ID.
Parameters: - directory – This field is the directory for which metadata has to be created.
- create_csv – This field is the output filepath to which csv file will be written.
- max_size – This field is the maximum file size. It’s default value is None.
- verbose – This boolean field is the logging level. It’s default value is False.
- debug – This boolean field is the logging level. It’s default value is False.