Retrieve items from a datastore

There are a few ways you can retrieve items from a datastore. You can:

• Retrieve items with get and bulkGet
• Find unknown items with query

Retrieve items with get and bulkGet

There are two methods for retrieving items in datastores:

• The apps.datastore.get method is used for single items.
• The apps.datastore.bulkGet method is used for multiple items.

They work quite similarly. Regardless of what you named your primary_key, the query will always use the id key.

// /functions/create_draft/interactivity_handler.ts
...
export const openDraftEditView: BlockActionHandler<
  typeof CreateDraftFunction.definition
> = async ({ body, action, client }) => {
  if (action.selected_option.value == "edit_message_overflow") {
    const id = action.block_id;

    // Get the draft
    const getResp = await client.apps.datastore.get<
      typeof DraftDatastore.definition
    >(
      {
        datastore: DraftDatastore.name,
        id: id,
      },
    );
...

{
  "ok": true,
  "datastore": "drafts",
  "item": {
    "id": "906dba92-44f5-4680-ada9-065149e4e930",
    "created_by": "U045A5X302V",
    "message": "This is a test message",
    "channels": ["C039ARY976C"],
    "channel": "C038M39A2TV",
    "icon": "",
    "username": "Slackbot",
    "status": "draft",
  }
}

{
  "ok": true,
  "datastore": "drafts",
  "item": {}
}

// /functions/create_draft/interactivity_handler.ts
...
export const openDraftEditView: BlockActionHandler<
  typeof CreateDraftFunction.definition
> = async ({ body, action, client }) => {
  if (action.selected_option.value == "edit_message_overflow") {
    const id = action.block_id;

    // Get the draft
    const getResp = await client.apps.datastore.bulkGet<
      typeof DraftDatastore.definition
    >(
      {
        datastore: DraftDatastore.name,
        ids: [id,"41"]
      },
    );
...

{
  "ok": true,
  "datastore": "drafts",
  "items": [
    {
      "id": "906dba92-44f5-4680-ada9-065149e4e930",
      "created_by": "U045A5X302V",
      "message": "This is a test message",
      "channels": ["C039ARY976C"],
      "channel": "C038M39A2TV",
      "icon": "",
      "username": "Slackbot",
      "status": "draft",
    },
    {
      "id": "906dba92-44f5-4680-ada9-065149e4e930",
      "created_by": "U045A5X302V",
      "message": "This is a test message",
      "channels": ["C039ARY976C"],
      "channel": "C038M39A2TV",
      "icon": "",
      "username": "Slackbot",
      "status": "draft",
    }
  ]
}

{
  "ok": true,
  "datastore": "drafts",
  "items": []
}

For both methods, if the call was unsuccessful, ok will be false and you'll see some information on the error.

{
  "ok": false,
  "error": "datastore_error",
  "errors": [
    {
    "code": "datastore_config_not_found",
    "message": "The datastore configuration could not be found",
    "pointer": "/datastores"
    }
  ]
}

It is possible to have records with undefined values, and it's important to be proactive in expecting those situations in your code. Here are some examples of how to code around a potential undefined field while retrieving an item.

This example snippet supports the case where the function returns an optional output:

const getResponse = client.apps.datastore.get<typeof DraftsDatastore.definition>({...});
const announcementId = getResponse.item.id; // this is the primary key
const announcementIcon = getResponse.item.icon; // icon could be undefined

return {
  outputs: {
    id: announcementId, // id is always defined
    icon: announcementIcon,  // icon must be an optional output of the function
  }
}

This example snippet supports the case where the function assigns a default:

const getResponse = client.apps.datastore.get<typeof DraftsDatastore.definition>({...});
const announcementId = getResponse.item.id; // this is the primary key

// icon could be undefined, so use a fallback
const announcementIcon = getResponse.item.icon ?? "n/a";

return {
  outputs: {
    id: announcementId, // id is always defined
    icon: announcementIcon,  // email is always defined
  }
}

And finally, this example snippet supports the case where the function should error:

const getResponse = client.apps.datastore.get<typeof DraftsDatastore.definition>({...});
const announcementId = getResponse.item.id; // this is the primary key

if (getResponse.item.icon) {
  const announcementIcon = getResponse.item.icon;
  return {
    outputs: {
      id: announcementId,
      icon: announcementIcon }
  }
} else {
  return {
    error: "Announcement doesn't have an icon assigned"
  }
}

Find items with query

If you need to find data without already knowing the item's id, you'll want to run a query. Querying a datastore includes knowledge of a few different components. First let's look at the fields of a datastore query, how it might look in code, and then break down the details of each bit.

A Slack datastore query includes the following arguments:

Here's an example of how to query our drafts datastore and retrieve a list of all the announcements with messages containing "timesheet":

const result = await client.apps.datastore.query({
  datastore: "drafts",
  expression: "contains (#message_term, :message)",
  expression_attributes: { "#message_term": "message" },
  expression_values: { ":message": "timesheet" },
});

If that example looks wonky to you; read on while we explain. Under the hood, the apps.datastore.query API method is a DynamoDB scan, and thereby uses DynamoDB filter expression syntax.

Let's break down that previous query example:

The expression is the search criteria. The expression_attributes object is a map of the columns used for the comparison, and the expression_values object is a map of values. The expression_attributes property must always begin with a #, and the expression_values property must always begin with a :.

To break that down further, #message_term seen here is a variable representing the message datastore attribute. So, why not just use message in the expression, such that it would be expression: "message = :message"? We do this to safeguard against anything that might break the search query, like double quotes or spaces in a name, or using DynamoDB's reserved words as attribute names. The second such variable used in the expression is :message. We see that defined in expression_values as the hard-coded value of "timesheet", but it's more likely that you'll use a variable here, perhaps a value obtained from a user interaction.

In summary, this query searches for items in the drafts datastore that have a value of "timesheet" (represented by :message) in their message attribute (represented by #message_term).

Let's take a look at another example, this one exploring searching a datastore by timestamp. Given this set of data in a datastore:

{
  "id": "foo5",
  "message": "bar5",
  "timestamp": 1671752648
}

{
  "id": "foo4",
  "message": "bar4",
  "timestamp": 1670975048
}

{
  "id": "foo3",
  "message": "bar3",
  "timestamp": 1702511048
}

If we run the query:

hermes datastore query '{
  "datastore": "messages",
  "expression": "#timestamp between :time_start AND :time_end",
  "expression_attributes": {"#timestamp":"timestamp"},
  "expression_values": {":time_start":1670975049,":time_end":1702511047}
  }'

We will see this object as a result:

{
  "id": "foo5",
  "message": "bar5",
  "timestamp": 1671752648
}

You can use filter expression operators with any of the date types (Schema.slack.types.date, Schema.slack.types.timestamp, and Schema.slack.types.message_ts), so long as the values passed match the underlying type and format.

Here is another example of a date query, this one using the Schema.slack.types.date field.

Given this set of data in a datastore:

{
  "date": "2022-01-02",
  "message": "First message",
  "id": "1"
}

{
  "date": "2023-04-11",
  "message": "Second message",
  "id": "2"
}

{
  "date": "2024-01-01",
  "message": "Third message",
  "id": "3"
}

Running this query:

hermes datastore query '{
  "datastore": "messages", 
  "expression": "#date < :date_end", 
  "expression_attributes": {"#date": "date"}, 
  "expression_values": {":date_end": "2023-01-01"}
  }'

Will yield this result:

{
  "date": "2022-01-02",
  "message": "First message",
  "id": "1"
}

Filter expressions

Because datastore query is a DynamoDB scan, all query expressions are essentially filter expressions. It's what you put in the value of the expression argument. Filter expressions are applied post-hoc. This is important context to understand because it can yield some confusing results; i.e. return fewer results than requested yet have additional pages of results to be queried and paginated (see the pagination section below). Each query can return a maximum of 1MB of data per page of results, and it returns all results of the datastore before applying any filter conditions. The filter conditions are applied to each page of results individually. This is how you could end up with the first page of zero results, yet still have a cursor for a following page of results.

Here is the full list of comparison operators to use in a filter expression, followed by some examples:

Revisiting our drafts datastore, here we retrieve all the announcements created by user C123ABC456:

const result = await client.apps.datastore.query({
  datastore: "drafts",
  expression: "#announcement_creator = :user",
  expression_attributes: { "#announcement_creator": "created_by"},
  expression_values: {":user": "C123ABC456"},
});

If you wanted to verify the query before putting it in your app code, the CLI query for that same search would be:

slack datastore query '{
  "datastore": "drafts",
  "expression": "#announcement_creator = :user",
  "expression_attributes": { "#announcement_creator": "created_by"},
  "expression_values": {":user": "C123ABC456"}
}'

Here's an example of a function that receives a string message via an input and queries for the announcement record that matches the provided message:

  const result = await client.apps.datastore.query({
  datastore: "drafts",
  expression: "contains (#message_term, :message)",
  expression_attributes: { "#message_term": "message" },
  expression_values: { ":message": input.message },
});

You could also chain expressions together to narrow your results even further:

  const result = await client.apps.datastore.query({
  datastore: "drafts",
  expression: "contains (#message_term, :message) AND #announcement_creator = :creator",
  expression_attributes: { "#message_term": "message", "#announcement_creator": "created_by" },
  expression_values: { ":message": input.message, ":creator": input.creator },
});

Pagination

For cursor-paginated methods, use the cursor parameter to retrieve the next page of your query results.

If your initial query has another page of results, the next_cursor response parameter is the key returned that will unlock your next page of results. Use this key to query the datastore again and set cursor to the value of next_cursor.

That request might look like:

const result = await client.apps.datastore.query({
  datastore: "drafts",
  cursor: "eyJ...n19"
});

Remember that filters are applied post-hoc, so you should always be sure to check subsequent pages for results, even if the initial page has fewer results than expected. Read the filter section above for context.

Count items with count

As mentioned above, querying a datastore uses a DynamoDB scan to return an array of matching items for your query results. We also mentioned that each query, i.e. each DynamoDB scan, can return a maximum of 1MB of data per page of results. For that reason, if you have over 1MB of data in your datastore, multiple scans are necessary to paginate through your entire datastore.

DynamoDB accomplishes this by returning a cursor to start a new scan where you left off with your previous one. Therefore if you wanted to use the query method to count all of the matching items in your datastore, you would need to call the query command several times, then manually add together the sizes of each array of matching items returned.

Instead, you can use the count method to paginate through your datastore and sum up the count of all the items matching your query. If a query is not provided, the count will be equal to the number of items in the entire datastore.

Using the DynamoDB style of syntax, the following example would retrieve the number of records from a datastore called "good_tunes", where "You" is in the song title:

{
	"datastore": "good_tunes",
	"expression": "contains (#song, :keyword)",
	"expression_attributes": { "#song": "song" },
	"expression_values": { ":keyword": "You" }
}

The response to the request might look like the following:

{
    "ok": true,
    "datastore": "good_tunes",
    "count": 2
}