Python

0.02.2

Datastore#

Cloud Datastore in 10 seconds#

Install the library#

The source code for the library (and demo code) lives on GitHub, You can install the library quickly with pip:

$ pip install gcloud

Run the demo#

In order to run the demo, you need to have registred an actual gcloud project and so you’ll need to provide some environment variables to facilitate authentication to your project:

  • GCLOUD_TESTS_PROJECT_ID: Developers Console project ID (e.g. bamboo-shift-455).
  • GCLOUD_TESTS_DATASET_ID: The name of the dataset your tests connect to. This is typically the same as GCLOUD_TESTS_PROJECT_ID.
  • GCLOUD_TESTS_CLIENT_EMAIL: The email for the service account you’re authenticating with
  • GCLOUD_TESTS_KEY_FILE: The path to an encrypted key file. See private key docs for explanation on how to get a private key.

Run the example script included in the package:

$ python -m gcloud.datastore.demo

And that’s it! You just read and wrote a bunch of data to the Cloud Datastore.

Try it yourself#

You can interact with a demo dataset in a Python interactive shell.

Start by importing the demo module and instantiating the demo dataset:

>>> from gcloud.datastore import demo
>>> dataset = demo.get_dataset()

Once you have the dataset, you can create entities and save them:

>>> dataset.query('MyExampleKind').fetch()
[<Entity{...}, ]
>>> entity = dataset.entity('Person')
>>> entity['name'] = 'Your name'
>>> entity['age'] = 25
>>> entity.save()
>>> dataset.query('Person').fetch()
[<Entity{...} {'name': 'Your name', 'age': 25}>]

Note

The get_dataset method is just a shortcut for:

>>> from gcloud import datastore
>>> from gcloud.datastore import demo
>>> dataset = datastore.get_dataset(
>>>     demo.DATASET_ID, demo.CLIENT_EMAIL, demo.PRIVATE_KEY_PATH)

gcloud.datastore#

Shortcut methods for getting set up with Google Cloud Datastore.

You’ll typically use these to get started with the API:

>>> from gcloud import datastore
>>> dataset = datastore.get_dataset('dataset-id-here',
...                                 'long-email@googleapis.com',
...                                 '/path/to/private.key')
>>> # Then do other things...
>>> query = dataset.query().kind('EntityKind')
>>> entity = dataset.entity('EntityKind')

The main concepts with this API are:

gcloud.datastore.__init__.SCOPE = ('https://www.googleapis.com/auth/datastore ', 'https://www.googleapis.com/auth/userinfo.email')#

The scope required for authenticating as a Cloud Datastore consumer.

gcloud.datastore.__init__.get_connection(client_email, private_key_path)[source]#

Shortcut method to establish a connection to the Cloud Datastore.

Use this if you are going to access several datasets with the same set of credentials (unlikely):

>>> from gcloud import datastore
>>> connection = datastore.get_connection(email, key_path)
>>> dataset1 = connection.dataset('dataset1')
>>> dataset2 = connection.dataset('dataset2')
Parameters:
  • client_email (string) – The e-mail attached to the service account.
  • private_key_path (string) – The path to a private key file (this file was given to you when you created the service account).
Return type:

gcloud.datastore.connection.Connection

Returns:

A connection defined with the proper credentials.

gcloud.datastore.__init__.get_dataset(dataset_id, client_email, private_key_path)[source]#

Establish a connection to a particular dataset in the Cloud Datastore.

This is a shortcut method for creating a connection and using it to connect to a dataset.

You’ll generally use this as the first call to working with the API:

>>> from gcloud import datastore
>>> dataset = datastore.get_dataset('dataset-id', email, key_path)
>>> # Now you can do things with the dataset.
>>> dataset.query().kind('TestKind').fetch()
[...]
Parameters:
  • dataset_id (string) – The id of the dataset you want to use. This is akin to a database name and is usually the same as your Cloud Datastore project name.
  • client_email (string) – The e-mail attached to the service account.
  • private_key_path (string) – The path to a private key file (this file was given to you when you created the service account).
Return type:

gcloud.datastore.dataset.Dataset

Returns:

A dataset with a connection using the provided credentials.

Connections#

Connections to gcloud datastore API servers.

class gcloud.datastore.connection.Connection(credentials=None)[source]#

Bases: gcloud.connection.Connection

A connection to the Google Cloud Datastore via the Protobuf API.

This class should understand only the basic types (and protobufs) in method arguments, however should be capable of returning advanced types.

Parameters:credentials (oauth2client.client.OAuth2Credentials) – The OAuth2 Credentials to use for this connection.
API_URL_TEMPLATE = '{api_base}/datastore/{api_version}/datasets/{dataset_id}/{method}'#

A template for the URL of a particular API call.

API_VERSION = 'v1beta2'#

The version of the API, used in building the API call’s URL.

allocate_ids(dataset_id, key_pbs)[source]#

Obtain backend-generated IDs for a set of keys.

Maps the DatastoreService.AllocateIds protobuf RPC.

Parameters:
  • dataset_id (string) – The dataset to which the transaction belongs.
  • key_pbs (list of gcloud.datastore.datastore_v1_pb2.Key) – The keys for which the backend should allocate IDs.
Return type:

list of gcloud.datastore.datastore_v1_pb2.Key

Returns:

An equal number of keys, with IDs filled in by the backend.

begin_transaction(dataset_id, serializable=False)[source]#

Begin a transaction.

Maps the DatastoreService.BeginTransaction protobuf RPC.

Parameters:dataset_id (string) – The dataset over which to execute the transaction.
classmethod build_api_url(dataset_id, method, base_url=None, api_version=None)[source]#

Construct the URL for a particular API call.

This method is used internally to come up with the URL to use when making RPCs to the Cloud Datastore API.

Parameters:
  • dataset_id (string) – The ID of the dataset to connect to. This is usually your project name in the cloud console.
  • method (string) – The API method to call (ie, runQuery, lookup, ...).
  • base_url (string) – The base URL where the API lives. You shouldn’t have to provide this.
  • api_version (string) – The version of the API to connect to. You shouldn’t have to provide this.
commit(dataset_id, mutation_pb)[source]#

Commit dataset mutations in context of current transation (if any).

Maps the DatastoreService.Commit protobuf RPC.

Parameters:
  • dataset_id (string) – The dataset in which to perform the changes.
  • mutation_pb (gcloud.datastore.datastore_v1_pb2.Mutation.) – The protobuf for the mutations being saved.
Return type:

gcloud.datastore.datastore_v1_pb2.MutationResult.

Returns’:

the result protobuf for the mutation.

dataset(*args, **kwargs)[source]#

Factory method for Dataset objects.

Parameters:args – All args and kwargs will be passed along to the gcloud.datastore.dataset.Dataset initializer.
Return type:gcloud.datastore.dataset.Dataset
Returns:A dataset object that will use this connection as its transport.
delete_entities(dataset_id, key_pbs)[source]#

Delete keys from a dataset in the Cloud Datastore.

This method deals only with gcloud.datastore.datastore_v1_pb2.Key protobufs and not with any of the other abstractions. For example, it’s used under the hood in the gcloud.datastore.entity.Entity.delete() method.

Parameters:
  • dataset_id (string) – The dataset from which to delete the keys.
  • key_pbs (list of gcloud.datastore.datastore_v1_pb2.Key) – The keys to delete from the datastore.
Return type:

boolean (if in a transaction) or else gcloud.datastore.datastore_v1_pb2.MutationResult.

Returns:

True

lookup(dataset_id, key_pbs)[source]#

Lookup keys from a dataset in the Cloud Datastore.

Maps the DatastoreService.Lookup protobuf RPC.

This method deals only with protobufs (gcloud.datastore.datastore_v1_pb2.Key and gcloud.datastore.datastore_v1_pb2.Entity) and is used under the hood for methods like gcloud.datastore.dataset.Dataset.get_entity():

>>> from gcloud import datastore
>>> from gcloud.datastore.key import Key
>>> connection = datastore.get_connection(email, key_path)
>>> dataset = connection.dataset('dataset-id')
>>> key = Key(dataset=dataset).kind('MyKind').id(1234)

Using the gcloud.datastore.dataset.Dataset helper:

>>> dataset.get_entity(key)
<Entity object>

Using the connection class directly:

>>> connection.lookup('dataset-id', key.to_protobuf())
<Entity protobuf>
Parameters:
  • dataset_id (string) – The dataset to look up the keys.
  • key_pbs (list of gcloud.datastore.datastore_v1_pb2.Key (or a single Key)) – The key (or keys) to retrieve from the datastore.
Return type:

list of gcloud.datastore.datastore_v1_pb2.Entity (or a single Entity)

Returns:

The entities corresponding to the keys provided. If a single key was provided and no results matched, this will return None. If multiple keys were provided and no results matched, this will return an empty list.

mutation()[source]#

Getter for mutation usable with current connection.

Return type:gcloud.datastore.datastore_v1_pb2.Mutation.
Returns:the mutation instance associated with the current transaction (if one exists) or or a new mutation instance.
rollback(dataset_id)[source]#

Rollback the connection’s existing transaction.

Maps the DatastoreService.Rollback protobuf RPC.

Raises a ValueError if the connection isn’t currently in a transaction.

Parameters:dataset_id (string) – The dataset to which the transaction belongs.
run_query(dataset_id, query_pb, namespace=None)[source]#

Run a query on the Cloud Datastore.

Maps the DatastoreService.RunQuery protobuf RPC.

Given a Query protobuf, sends a runQuery request to the Cloud Datastore API and returns a list of entity protobufs matching the query.

You typically wouldn’t use this method directly, in favor of the gcloud.datastore.query.Query.fetch() method.

Under the hood, the gcloud.datastore.query.Query class uses this method to fetch data:

>>> from gcloud import datastore
>>> connection = datastore.get_connection(email, key_path)
>>> dataset = connection.dataset('dataset-id')
>>> query = dataset.query().kind('MyKind').filter('property =', 'val')

Using the fetch` method...

>>> query.fetch()
[<list of Entity unmarshalled from protobuf>]
>>> query.cursor()
<string containing cursor where fetch stopped>

Under the hood this is doing...

>>> connection.run_query('dataset-id', query.to_protobuf())
[<list of Entity Protobufs>], cursor, more_results, skipped_results
Parameters:
  • dataset_id (string) – The ID of the dataset over which to run the query.
  • query_pb (gcloud.datastore.datastore_v1_pb2.Query) – The Protobuf representing the query to run.
  • namespace (string) – The namespace over which to run the query.
save_entity(dataset_id, key_pb, properties, exclude_from_indexes=())[source]#

Save an entity to the Cloud Datastore with the provided properties.

Note

Any existing properties for the entity identified by ‘key_pb’ will be replaced by those passed in ‘properties’; properties not passed in ‘properties’ no longer be set for the entity.

Parameters:
  • dataset_id (string) – The dataset in which to save the entity.
  • key_pb (gcloud.datastore.datastore_v1_pb2.Key) – The complete or partial key for the entity.
  • properties (dict) – The properties to store on the entity.
  • exclude_from_indexes (sequence of str) – Names of properties not to be indexed.
transaction(transaction=<object object at 0x106affe40>)[source]#

Getter/setter for the connection’s transaction object.

Parameters:transaction (gcloud.datastore.transaction.Transaction, (setting), or omitted (getting).) – The new transaction (if passed).
Return type:gcloud.datastore.transaction.Transaction, (getting) or gcloud.datastore.connection.Connection (setting)
Returns:the current transaction (getting) or self (setting).

Helper functions#

Helper functions for dealing with Cloud Datastore’s Protobuf API.

The non-private functions are part of the API.

gcloud.datastore.helpers.entity_from_protobuf(pb, dataset=None)[source]#

Factory method for creating an entity based on a protobuf.

The protobuf should be one returned from the Cloud Datastore Protobuf API.

Parameters:pb (gcloud.datastore.datastore_v1_pb2.Entity) – The Protobuf representing the entity.
Return type:gcloud.datastore.entity.Entity
Returns:The entity derived from the protobuf.
gcloud.datastore.helpers.key_from_protobuf(pb)[source]#

Factory method for creating a key based on a protobuf.

The protobuf should be one returned from the Cloud Datastore Protobuf API.

Parameters:pb (gcloud.datastore.datastore_v1_pb2.Key) – The Protobuf representing the key.
Return type:gcloud.datastore.key.Key
Returns:a new Key instance