Retrieve items from a datastore

Retrieve items with get and bulkGet

There are two methods for retrieving items in datastores:

• The apps.datastore.get method is used for single items.
• The apps.datastore.bulkGet method is used for multiple items.

They work quite similarly. Regardless of what you named your primary_key, the query will always use the id key.

// /functions/create_draft/interactivity_handler.ts
...
export const openDraftEditView: BlockActionHandler<
    typeof CreateDraftFunction.definition
> = async ({ body, action, client }) => {
    if (action.selected_option.value == "edit_message_overflow") {
        const id = action.block_id;

        // Get the draft
        const getResp = await client.apps.datastore.get <
            typeof DraftDatastore.definition
            > (
                {
                    datastore: DraftDatastore.name,
                    id: id,
                },
    );
...

{
    "ok": true,
    "datastore": "drafts",
    "item": {
        "id": "906dba92-44f5-4680-ada9-065149e4e930",
        "created_by": "U045A5X302V",
        "message": "This is a test message",
        "channels": [
            "C039ARY976C"
        ],
        "channel": "C038M39A2TV",
        "icon": "",
        "username": "Slackbot",
        "status": "draft",
    }
}

{
    "ok": true,
    "datastore": "drafts",
    "item": {}
}

// /functions/create_draft/interactivity_handler.ts
...
export const openDraftEditView: BlockActionHandler<
    typeof CreateDraftFunction.definition
> = async ({ body, action, client }) => {
    if (action.selected_option.value == "edit_message_overflow") {
        const id = action.block_id;

        // Get the draft
        const getResp = await client.apps.datastore.bulkGet <
            typeof DraftDatastore.definition
            > (
                {
                    datastore: DraftDatastore.name,
                    ids: [id, "41"]
                },
    );
...

{
    "ok": true,
    "datastore": "drafts",
    "items": [
        {
            "id": "906dba92-44f5-4680-ada9-065149e4e930",
            "created_by": "U045A5X302V",
            "message": "This is a test message",
            "channels": [
                "C039ARY976C"
            ],
            "channel": "C038M39A2TV",
            "icon": "",
            "username": "Slackbot",
            "status": "draft",
        },
        {
            "id": "906dba92-44f5-4680-ada9-065149e4e930",
            "created_by": "U045A5X302V",
            "message": "This is a test message",
            "channels": [
                "C039ARY976C"
            ],
            "channel": "C038M39A2TV",
            "icon": "",
            "username": "Slackbot",
            "status": "draft",
        }
    ]
}

{
    "ok": true,
    "datastore": "drafts",
    "items": []
}

For both methods, if the call was unsuccessful, ok will be false and you'll see some information on the error.

{
    "ok": false,
    "error": "datastore_error",
    "errors": [
        {
            "code": "datastore_config_not_found",
            "message": "The datastore configuration could not be found",
            "pointer": "/datastores"
        }
    ]
}

It is possible to have records with undefined values, and it's important to be proactive in expecting those situations in your code. Here are some examples of how to code around a potential undefined field while retrieving an item.

This example snippet supports the case where the function returns an optional output:

const getResponse = await client.apps.datastore.get < typeof DraftsDatastore.definition > ({ ...});
const announcementId = getResponse.item.id; // this is the primary key
const announcementIcon = getResponse.item.icon; // icon could be undefined

return {
    outputs: {
        id: announcementId, // id is always defined
        icon: announcementIcon,  // icon must be an optional output of the function
    }
}

This example snippet supports the case where the function assigns a default:

const getResponse = await client.apps.datastore.get < typeof DraftsDatastore.definition > ({ ...});
const announcementId = getResponse.item.id; // this is the primary key

// icon could be undefined, so use a fallback
const announcementIcon = getResponse.item.icon ?? "n/a";

return {
    outputs: {
        id: announcementId, // id is always defined
        icon: announcementIcon,  // email is always defined
    }
}

And finally, this example snippet supports the case where the function should error:

const getResponse = await client.apps.datastore.get < typeof DraftsDatastore.definition > ({ ...});
const announcementId = getResponse.item.id; // this is the primary key

if (getResponse.item.icon) {
    const announcementIcon = getResponse.item.icon;
    return {
        outputs: {
            id: announcementId,
            icon: announcementIcon
        }
    }
} else {
    return {
        error: "Announcement doesn't have an icon assigned"
    }
}

Find items with query

If you need to find data without already knowing the item's id, you'll want to run a query. Querying a datastore requires knowledge of a few different components. It's also helpful to brush up on how to use pagination and filter expressions.

First, let's look at the fields of a datastore query and how they might look in code, then break down the details of each bit.

A Slack datastore query includes the following arguments:

Here's an example of how to query our drafts datastore using the Slack CLI and retrieve a list of all the announcements with messages containing "timesheet":

const result = await client.apps.datastore.query({
    datastore: "drafts",
    expression: "contains (#message_term, :message)",
    expression_attributes: { "#message_term": "message" },
    expression_values: { ":message": "timesheet" },
});

If that example looks wonky to you; read on while we explain. Under the hood, the apps.datastore.query API method is a DynamoDB scan, and thereby uses DynamoDB's filter expression syntax.

Let's break down that previous query example:

The expression is the search criteria. The expression_attributes object is a map of the columns used for the comparison, and the expression_values object is a map of values. The expression_attributes property must always begin with a #, and the expression_values property must always begin with a :.

To break that down further, #message_term seen here is a variable representing the message datastore attribute. So, why not just use message in the expression, such that it would be expression: "message = :message"? We do this to safeguard against anything that might break the search query, like double quotes or spaces in a name, or using DynamoDB's reserved words as attribute names. The second such variable used in the expression is :message. We see that defined in expression_values as the hard-coded value of "timesheet", but it's more likely that you'll use a variable here, perhaps a value obtained from a user interaction.

In summary, this query searches for items in the drafts datastore that have a value of "timesheet" (represented by :message) in their message attribute (represented by #message_term).

Let's take a look at another example, this one exploring searching a datastore by timestamp. Given this set of data in a datastore:

{
    "id": "foo5",
    "message": "bar5",
    "timestamp": 1671752648
}

{
    "id": "foo4",
    "message": "bar4",
    "timestamp": 1670975048
}

{
   "id": "foo3",
    "message": "bar3",
    "timestamp": 1702511048
}

If we run the CLI query:

slack datastore query '{
  "datastore": "messages",
  "expression": "#timestamp between :time_start AND :time_end",
  "expression_attributes": {"#timestamp":"timestamp"},
  "expression_values": {":time_start":1670975049,":time_end":1702511047}
  }'

We will see this object as a result:

{
    "id": "foo5",
    "message": "bar5",
    "timestamp": 1671752648
}

You can use filter expression operators with any of the date types (Schema.slack.types.date, Schema.slack.types.timestamp, and Schema.slack.types.message_ts), so long as the values passed match the underlying type and format.

Here is another example of a date query, this one using the Schema.slack.types.date field.

Given this set of data in a datastore:

{
    "date": "2022-01-02",
    "message": "First message",
    "id": "1"
}

{
    "date": "2023-04-11",
    "message": "Second message",
    "id": "2"
}

{
    "date": "2024-01-01",
    "message": "Third message",
    "id": "3"
}

Running this query:

slack datastore query '{
  "datastore": "messages", 
  "expression": "#date < :date_end", 
  "expression_attributes": {"#date": "date"}, 
  "expression_values": {":date_end": "2023-01-01"}
  }'

Will yield this result:

{
    "date": "2022-01-02",
    "message": "First message",
    "id": "1"
}

Pagination

It is strongly recommended to always handle pagination when implementing a query so that you can easily view all of your query results.

The following code snippet from the Virtual Running Buddies sample app shows how to do this:

export async function queryRunningDatastore(
    client: SlackAPIClient,
    expressions?: object,
): Promise<{
    ok: boolean;
    items: DatastoreItem<typeof RunningDatastore.definition>[];
    error?: string;
}> {
    const items: DatastoreItem<typeof RunningDatastore.definition>[] = [];
    let cursor = undefined;

    do {
        const runs: DatastoreQueryResponse<typeof RunningDatastore.definition> =
            await client.apps.datastore.query < typeof RunningDatastore.definition > ({
                datastore: RUN_DATASTORE,
                cursor,
                ...expressions,
            });

        if (!runs.ok) {
            return { ok: false, items, error: runs.error };
        }

        cursor = runs.response_metadata?.next_cursor;
        items.push(...runs.items);
    } while (cursor);

    return { ok: true, items };
}

Essentially, you'll use the cursor parameter to retrieve the next page of your query results.

Filter expressions

Because datastore query is a DynamoDB scan, all query expressions are essentially filter expressions: it's what you put in the value of the expression argument. Filter expressions are applied post-hoc. This is important to understand because it can yield some confusing results; i.e. return fewer results than requested yet have additional pages of results to be queried and paginated. Each query can return a maximum of 1MB of data per page of results, and returns all results of the datastore before applying any filter conditions. The filter conditions are applied to each page of results individually. This is how you could end up with the first page of zero results, yet still have a cursor for a following page of results.

Here is the full list of comparison operators to use in a filter expression, followed by some examples:

Revisiting our drafts datastore, here we retrieve all the announcements created by user C123ABC456:

const result = await client.apps.datastore.query({
    datastore: "drafts",
    expression: "#announcement_creator = :user",
    expression_attributes: { "#announcement_creator": "created_by" },
    expression_values: { ":user": "C123ABC456" },
});

If you wanted to verify the query before putting it in your app code, the CLI query for that same search would be:

slack datastore query '{
  "datastore": "drafts",
  "expression": "#announcement_creator = :user",
  "expression_attributes": { "#announcement_creator": "created_by"},
  "expression_values": {":user": "C123ABC456"}
}'

Here's an example of a function that receives a string message via an input and queries for the announcement record that matches the provided message:

const result = await client.apps.datastore.query({
    datastore: "drafts",
    expression: "contains (#message_term, :message)",
    expression_attributes: { "#message_term": "message" },
    expression_values: { ":message": input.message },
});

You could also chain expressions together to narrow your results even further:

const result = await client.apps.datastore.query({
    datastore: "drafts",
    expression: "contains (#message_term, :message) AND #announcement_creator = :creator",
    expression_attributes: { "#message_term": "message", "#announcement_creator": "created_by" },
    expression_values: { ":message": input.message, ":creator": input.creator },
});

Count items with count

As mentioned above, querying a datastore uses a DynamoDB scan to return an array of matching items for your query results. We also mentioned that each query, i.e. each DynamoDB scan, can return a maximum of 1MB of data per page of results. For that reason, if you have over 1MB of data in your datastore, multiple scans are necessary to paginate through your entire datastore.

DynamoDB accomplishes this by returning a cursor to start a new scan where you left off with your previous one. Therefore if you wanted to use the query method to count all of the matching items in your datastore, you would need to call the query command several times, then manually add together the sizes of each array of matching items returned.

Instead, you can use the count method to paginate through your datastore and sum up the count of all the items matching your query. If a query is not provided, the count will be equal to the number of items in the entire datastore.

Using the DynamoDB style of syntax, the following example would retrieve the number of records from a datastore called "good_tunes", where "You" is in the song title:

{
    "datastore": "good_tunes",
    "expression": "contains (#song, :keyword)",
    "expression_attributes": { "#song": "song" },
    "expression_values": { ":keyword": "You" }
}

The response to the request might look like the following:

{
    "ok": true,
    "datastore": "good_tunes",
    "count": 2
}