README.md

<!--
SPDX-FileCopyrightText: 2021 Melvin Keskin <melvo@olomono.de>

SPDX-License-Identifier: AGPL-3.0-or-later
-->

# [XMPP Providers](https://providers.xmpp.net) - Curated List of Providers for Registration and Autocomplete

[![XMPP Providers: A](https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/raw/badge-count-A.svg?job=badges)](https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/raw/providers-A.json?job=filtered-provider-lists)
[![XMPP Providers: B](https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/raw/badge-count-B.svg?job=badges)](https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/raw/providers-B.json?job=filtered-provider-lists)
[![XMPP Providers: C](https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/raw/badge-count-C.svg?job=badges)](https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/raw/providers-C.json?job=filtered-provider-lists)
[![XMPP Providers: D](https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/raw/badge-count-D.svg?job=badges)](https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/raw/providers-D.json?job=filtered-provider-lists)

[![REUSE status](https://api.reuse.software/badge/invent.kde.org/melvo/xmpp-providers)](https://api.reuse.software/info/invent.kde.org/melvo/xmpp-providers)

This project provides a machine-readable curated list of [XMPP](https://xmpp.org/about/technology-overview/) providers and a script for filtering it.
The JSON list can be used by XMPP clients or other services for provider suggestions.
Visit the [official website](https://providers.xmpp.net) to see it in action!

Each **client** has different requirements on a provider: The script in this repository can be used to create a **statically filtered** provider list.

Each **user** has different requirements on a provider: At runtime, the statically filtered list can be **dynamically filtered** on the user's demands.

## Main Goals

This project has two main goals:
1. Simplifying the onboarding of new XMPP users by a list of XMPP providers that can be integrated into XMPP clients and used for choosing a provider for registration
1. Improving the providers' features, security, support and documentation by defining high quality standards and providing information to achieve them

## Contributing

Help us to make *free communication* possible and contribute to this project!

**Please read the [contribution guidelines](/CONTRIBUTING.md) carefully before creating a merge request.**
That saves you and the reviewers a lot of time in the review process.

## Categories

The script to create a filtered list can be used for specific requirements.
However, to provide a standardized way of categorizing providers, there are three main options for filtering the list.

The following categories are used to distinguish between different quality levels the checked provders can offer.
Category A and B offer the best, C an average and D the worst user experience during onboarding and further usage.
Therefore, **A and B are the recommended categories for registrations**!
The unfiltered provider list corresponds to category D.

Category A is a subset of B, B a subset of C and C a subset of D.
Thus, providers of A can be used for the purposes of B, C and D as well.
Providers of B can be used for the purposes of C and D as well.
Providers of C can be used for the purposes of D as well.

### Category A: Automatically Chosen

Providers in this category can be used for an automatic registration.

### Category B: Manually Selectable

Providers in this category can be used for a manual registration.

### Category C: Completely Customizable

Providers in this category can be used for completely customized filtering.

### Category D: Autocomplete

Providers in this category can be used for autocomplete.

## Usage

If you are an XMPP client developer or would like to create a service based on XMPP Providers, you can create filtered lists on your own or use the daily-updated prefiltered lists.

The prefiltered lists are versioned and available for all categories (replace `<version>` and `<category>` with the desired values):
```
https://data.xmpp.net/providers/v<version>/providers-<category>.json
```

Example for a prefiltered list with providers of category **B**:
```
https://data.xmpp.net/providers/v1/providers-B.json
```

Prefiltered **s**imple lists, which contain only the provider domains, are also available:
```
https://data.xmpp.net/providers/v<version>/providers-<category>s.json
```

Example for a prefiltered simple list with providers of category **D**:
```
https://data.xmpp.net/providers/v1/providers-Ds.json
```

The version is a natural number and determines breaking changes.
A breaking change means that the format of the lists changes in a way an application parsing it could not handle it anymore.
In that case, the version is incremented and lists of older versions are not updated anymore.
That would be the case if properties are deleted or their names changed but not if a new property is added or a property value changed.

### Client Integration

Here are the recommended steps to integrate the provider lists in an XMPP client:
1. Download a prefiltered [provider list for category B](https://data.xmpp.net/providers/v1/providers-B.json).
1. Download a prefiltered [simple provider list for category D](https://data.xmpp.net/providers/v1/providers-Ds.json).
1. Include both lists in your client's source code / executable.
1. If you have a registration process without user interaction (i.e., automatic registration such as [Kaidan's quick onboarding](https://www.kaidan.im/2020/01/08/Easy-Registration/)), retrieve the providers of category A from the list for category B.
1. For a manual registration, use all providers from the list of category B.
1. Use the list for category D to autocomplete chat addresses after registration.

The provider lists are primarily used to choose a provider that suits the user best.
Its data can be displayed or used to filter suitable providers during the manual registration.
But the provider list for category B can also be used during or after registration for speicifc actions (e.g., to open a provider's website or send a support message).

You should also create a script or use another way to update the integrated provider lists on a regular basis.

## Properties

The providers file `providers.json` is the source to create all filtered provider lists.
The properties in this section are in the providers file and in the provider lists.
There is another property `category` which is only in the provider lists.

The properties of a provider are mapped to its server address / domain / JID (e.g., `example.org`).
In the providers file, the JID is the key of the corresponding provider's JSON object.
Whereas, in the provider lists, the JID is a JSON object within the JSON array's value for the corresponding provider.

### Basic Information

The following properties do **not affect** the provider's **category**:

Information (Key in JSON File) | Description / Data Type / Unit of Measurement
---|---
lastCheck | [ISO YYYY-MM-DD date](https://en.wikipedia.org/wiki/ISO_8601#Dates) (e.g., 2021-01-16)
website | mappings from lower-case [two-letter (639-1) language code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) to link of website language version or {} for n/a
[busFactor](https://en.wikipedia.org/wiki/Bus_factor) | minimum number of team members that the service could not survive losing or -1 for n/a
company | true or false
passwordReset | mappings from lower-case [two-letter (639-1) language code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) to link of web page language version used for automatic password reset (e.g., via email) / web page describing how to manually reset password (e.g., by contacting the provider) or {} for n/a

### Criteria

The table shows the following two circumstances for category A and B:

A provider is in a specific category if it meets all of the criteria listed in the table below.
A condition can be `true`, `false` or true if a specific case such as greater or lower than a specific value is applicable.
If the data type is a list or mapping, a condition including numbers corresponds to the count of their elements.
If the data type is a date, a condition including numbers corresponds to the age in days.

Here is an example for the in-band and web registration:
*A provider that has no inBandRegistration is not in category A.*
*But if the provider has a registrationWebPage, it is in category B if it also meets all of the other criteria for B.*

The following properties **affect** the provider's **category**:

Criterion (Key in JSON File) | Description / Data Type / Unit of Measurement | Category A | Category B | Category C
---|---|---|---|---
[inBandRegistration](https://xmpp.org/extensions/xep-0077.html#usecases-register) | true if registration via XMPP client supported, otherwise false | true | true or registrationWebPage >= 1 | true or registrationWebPage >= 1
registrationWebPage | mappings from lower-case [two-letter (639-1) language code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) to link of web page language version or {} for n/a | >= 1 or inBandRegistration | >= 1 or inBandRegistration | >= 1 or inBandRegistration
rating[XmppComplianceTester](https://compliance.conversations.im) | score (number in percentage) | = 100 | = 100 | >= 90
rating[ImObservatory](https://xmpp.net)ClientToServer | score (upper-case letter) | >= A | >= A | >= A
rating[ImObservatory](https://xmpp.net)ServerToServer | score (upper-case letter) | >= A | >= A | >= A
maximum[HttpFileUpload](https://xmpp.org/extensions/xep-0363.html)FileSize | number in megabytes (MB), 0 for no limit or -1 for n/a or less than 1 MB | 0 or >= 20 | 0 or >= 20 | >= 0
maximum[HttpFileUpload](https://xmpp.org/extensions/xep-0363.html)TotalSize | number in megabytes (MB), 0 for no limit or -1 for n/a or less than 1 MB | 0 or >= 100 | 0 or >= 100 | >= 0
maximum[HttpFileUpload](https://xmpp.org/extensions/xep-0363.html)StorageTime | number in days, 0 for no limit or -1 for n/a or less than 1 day | 0 or >= 7 | 0 or >= 7 | >= 0
maximum[MessageArchiveManagement](https://xmpp.org/extensions/xep-0313.html)StorageTime | number in days, 0 for no limit or -1 for n/a or less than 1 day | 0 or >= 7 | 0 or >= 7 | >= 0
professionalHosting | true if hosted with good internet connection speed, uninterruptible power supply, access protection and regular backups, otherwise false | true | true | true or false
freeOfCharge | true if unpaid service, otherwise false | true | true or false | true or false
legalNotice | mappings from lower-case [two-letter (639-1) language code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) to link of legalNotice language version or {} for n/a | >= 1 | >= 1 | >= 0
serverLocations | list of lower-case [two-letter country codes](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) or [] for n/a | >= 1 | >= 1 | >= 0
groupChatSupport | mappings from lower-case [two-letter (639-1) language code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) to list of group chat addresses or {} for n/a | >= 1 or chatSupport >= 1 or emailSupport >= 1 | >= 1 or chatSupport >= 1 or emailSupport >= 1 | >= 1 or chatSupport >= 1 or emailSupport >= 1
chatSupport | mappings from lower-case [two-letter (639-1) language code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) to list of chat addresses or {} for n/a | >= 1 or groupChatSupport >= 1 or emailSupport >= 1 | >= 1 or groupChatSupport >= 1 or emailSupport >= 1 | >= 1 or groupChatSupport >= 1 or emailSupport >= 1
emailSupport | mappings from lower-case [two-letter (639-1) language code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) to list of email addresses or {} for n/a | >= 1 or groupChatSupport >= 1 or chatSupport >= 1 | >= 1 or groupChatSupport >= 1 or chatSupport >= 1 | >= 1 or groupChatSupport >= 1 or chatSupport >= 1
since | [ISO YYYY-MM-DD date](https://en.wikipedia.org/wiki/ISO_8601#Dates) since the provider is available or listed for n/a | > 365 | > 365 | >= 0

## Badges

If you are an XMPP provider, you can embed a badge showing your category in your website.
Visit your details page on the [XMPP Providers website](https://providers.xmpp.net).
It contains the code you have to embed.

If you want to provide a service based on XMPP Providers, you can get a badge showing a providers category (replace `example.org` with the desired domain):
```
https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/raw/badges/example.org.svg?job=badges
```

You can also fetch all badges together:
```
https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/download/?job=badges
```

## Results

If you are an XMPP provider, your details page on the [XMPP Providers website](https://providers.xmpp.net) contains your categorization results.

If you want to provide a service based on XMPP Providers, you can get the categorization results for a provider (replace `example.org` with the desired domain):
```
https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/raw/results/example.org.json?job=filtered-provider-lists
```
The JSON file lists all properties that do not meet the criteria for being in a specific category.

You can also fetch all results together:
```
https://invent.kde.org/melvo/xmpp-providers/-/jobs/artifacts/master/download/?job=filtered-provider-lists
```

## Filter Script

The script `filter.py` can be used to create filtered lists of providers (called **provider lists**).

### Usage

You can see all options for running the script:
```
./filter.py -h
```

Provider lists for all categories are created if you run the script without arguments:
```
./filter.py
```

If you want to create a filtered list which can be used by a client, simply enter the category name.

Example for creating a list of category **A** providers:
```
./filter.py -A
```

You can create a provider list containing all providers for completely customized filtering (e.g., by an own filter script or at runtime):
```
./filter.py -C
```

A **s**imple list containing only the domains of the providers is also possible.

Example for creating a domain list of category **D** providers to use it only for autocomplete:
```
./filter.py -s -C
```

The **c**ategories of the filtered providers can be included in their entries.

Example for creating a list of category **D** providers that includes their best categories.
```
./filter.py -c -D
```

You can create files containing the **r**esults of the filtering (called **categorization results**).
Those files include the providers' properties that do not meet the criteria for being in specific categories.

Example for creating result files for all providers:
```
./filter.py -r
```

If you are interested in specific providers, you can append them to the command.

Example for creating a list of category **B** providers out of *example.org* and *example.com*:
```
./filter.py -B example.org example.com
```

The script can be run in **d**ebug mode to see why providers are not in a specific category.

Example for creating a list of category **A** providers and logging additional information:
```
./filter.py -d -A
```

Furthermore, the arguments can be combined to show which criteria specific providers do not meet for being in a specific category.

Example for creating a list of category **A** providers out of *example.org* and *example.com* and logging additional information:
```
./filter.py -d -A example.org example.com
```

## Badge Script

The script `badge.py` can be used to create badges for the specified categories.
It uses the files created by the filter script.
Thus, the filter script must be run before running the badge script.

Template files are used for generating the badges.
All badges contain the category.
Additionally, it is possible to create badges containing the count of providers in a specific category.

The badge script has two phases:
1. It generates badges for all categories.
1. It creates the directory `badges` and fills it with badges for all providers (called **provider badges**).

### Usage

You can see all options for running the script:
```
./badge.py -h
```

Badges for all categories are created if you run the script without arguments:
```
./badge.py
```

The badges for providers can be created as symbolic **l**inks instead of regular files:
```
./badge.py -l
```

If you want to create a badge which can be used by a provider, simply enter the category name.

Example for creating a badge for category **A**:
```
./badge.py -A
```

Badges containing the **c**ount of providers in specific categories are also possible.

Example for creating a badge containing the count of providers in category **C** in addition to the normal badges:
```
./badge.py -c -C
```

The script can be run in **d**ebug mode to see detailed information about the process.

Example for creating a badge for category **A** and logging additional information:
```
./badge.py -d -A
```

## URL Checker

The script `check_urls.py` can be used to check URLs.
It succeeds if all checked URLs are reachable.

### Usage

You can see all options for running the script:
```
./check_urls.py -h
```

The URLs of all appropriate files are checked if you run the script without arguments:
```
./check_urls.py
```

It is also possible to check only the URLs in specific files.

Example for checking the URLs in `providers.json` and `filter.py`:
```
./check_urls.py providers.json filter.py
```