attention

torchhd.memory.attention(query: Tensor, keys: Tensor, values: Tensor, beta: float | None = None) → Tensor[source]

Attention mechanism

Parameters:

query (Tensor) – The query vector to compare the similarity with the keys.
keys (Tensor) – The key vectors to compare with the query.
values (Tensor) – The value vectors containing retrievable values from memory.
beta (float, optional) – Temperature scalar for the attention weights before the softmax. Default: 1/sqrt(d)

Shapes:

Query: \((*, f)\)
Keys: \((n, f)\)
Values: \((n, g)\)
Result: \((*, g)\)

Examples::

>>> items = torchhd.random(6, 512)
>>> read = torchhd.memory.attention(items, items, items).sign()
>>> torchhd.cosine_similarity(read, items)
tensor([[ 1.0000,  0.0625,  0.0117, -0.0625, -0.0078, -0.0430],
        [ 0.0625,  1.0000, -0.0195,  0.0703,  0.0469,  0.0508],
        [ 0.0117, -0.0195,  1.0000,  0.0820,  0.0195,  0.0156],
        [-0.0625,  0.0703,  0.0820,  1.0000, -0.0547, -0.0195],
        [-0.0078,  0.0469,  0.0195, -0.0547,  1.0000, -0.0898],
        [-0.0430,  0.0508,  0.0156, -0.0195, -0.0898,  1.0000]])