Messages unnecessarily fetched via MAM after login
How to reproduce the issue:
- Exchange messages with a contact
- Remove that contact (the contact is removed from the roster and the chat history is removed from the database)
- Log out
- Log in
- Messages that were locally removed but more recent than the last message that remained in the database are fetched again from the server via MAM
Messages that were locally removed from a chat that still exists, are ignored because there is a placeholder entry in the database for each locally removed message. Messages that were locally removed while removing the corresponding roster item, are added again to the database after fetching them. In both cases, messages are fetched unnecessarily from the server. That should be avoided. While the former case is only unnecessary, the latter one leads to locally adding messages that were intentionally locally removed.
The current implementation for fetching messages received by the user's server while the user was offline fetches all messages via MAM starting with the timestamp of the last message in the database. That does not only fetch messages that became irrelevant because they were locally removed but also the last message itself which is already locally stored.
That has the following disadvantages. If the last message in the database is older than messages that were removed locally, all the removed messages are fetched again from the server. Furthermore, that results in adding a formerly removed contact again because of Kaidan's new mechanism to add new contacts on the first received message.
Those issues could be resolved by removing messages on the server-side storage when they are locally removed as well (e.g., via Message Retraction tombstones). But Kaidan does not support that yet and older servers might not support that either. Thus, I suggest to implement that in the mid term but provide a fallback in case the server does not support it.
Here are possible approaches for a solution that would work now and be backwards compatible with servers not implementing the mentioned specification:
- Remove only the content and related data of removed messages (similar to what is done when specific messages are removed from a chat without removing the whole roster item and similar to MAM's tombstone approach) and query offline messages starting with the local timestamp of the last locally stored message plus an offset. That would result in many dead database entries for chats that are no longer present and roster items that do not exist anymore.
- Store an account-wide timestamp of the most recent message, keep it even if the corresponding message is locally removed and query offline messages starting with that timestamp plus an offset (to not fetch the message with that timestamp again).
- Store an account-wide timestamp of the last logout and query offline messages starting with that timestamp. That might result in lost/skipped messages if they were received by the server before the client logged out but after the messages could be relayed to the client. But I am not sure whether that could actually happen.
- Store an account-wide ID of the most recent message, keep it even if the corresponding message is locally removed and query offline messages coming after that message via MAM/RSM (to not fetch the message with that timestamp again). This is what I finally implemented.
If the last message has the timestamp of the time when Kaidan sent it and not the timestamp within the MAM archive (which is, as I understand the corresponding specification, the time when the server received it), each of the solutions based on message timestamps might lead to either fetching irrelevant/removed messages again or lost/skipped messages. The former would be the case if the offset is too small and the latter if the offset is too large.
We need to we ensure that no relevant messages are lost.