tubearchivist/docs
1
0
Fork 0
mirror of https://github.com/tubearchivist/docs.git synced 2024-11-22 11:50:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

Technical editing and consistency rewrites.

Browse Source
This commit is contained in:
lamusmaser 2024-04-19 06:33:58 -06:00
parent 8d64b9a797
commit f43cf15276
26 changed files with 521 additions and 430 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

56
mkdocs/docs/advanced.md
View File

@ -4,17 +4,17 @@ description: Collection of advanced concepts and debug info.
# Advanced Notes
!!! note
!!! warning
As a general rule of thumb, make sure your backups are up to date before continuing with anything here.
A loose collection of advanced debug info, may or may not apply to you, only use this when you know what you are doing. Some of that functionality might get implemented in the future in the regular UI.
A loose collection of advanced debug info and actions for experienced users for troubleshooting and resolving issues. The situation or command may or may not apply to you, so only use this when you know what you are doing or as directed. Some of the below functionality might get implemented in the future in the regular UI.
## Reactivate documents
As part of the metadata refresh task, Tube Archivist will mark videos, channels and playlists as deactivated, if they are no longer available on YouTube. For some reasons, that might have deactivated something that shouldn't have, for example if a video got reinstated after a copyright strike on YT. You can reactivate all things in bulk, so the refresh task will check them again and deactivate the ones that are actually not available anymore.
As part of the metadata refresh task, **Tube Archivist** will mark videos, channels and playlists as deactivated if they are no longer available on YouTube. Sometimes, they may be marked as deactivated when they shouldn't have, for example if a video got reinstated after a copyright strike on YT. You can reactivate all items in bulk so the refresh task will check them again and deactivate the ones that are actually not available anymore.
Curl commands to run within the TA container to reactivate documents:
??? Videos
??? bug "Videos"
```bash
curl -XPOST "$ES_URL/ta_video/_update_by_query?pretty" -u elastic:$ELASTIC_PASSWORD -H "Content-Type: application/json" -d '
@ -33,7 +33,7 @@ Curl commands to run within the TA container to reactivate documents:
}'
```
??? Channels
??? bug "Channels"
```bash
curl -XPOST "$ES_URL/ta_channel/_update_by_query?pretty" -u elastic:$ELASTIC_PASSWORD -H "Content-Type: application/json" -d '
@ -52,7 +52,7 @@ Curl commands to run within the TA container to reactivate documents:
}'
```
??? Playlists
??? bug "Playlists"
```bash
curl -XPOST "$ES_URL/ta_video/_update_by_query?pretty" -u elastic:$ELASTIC_PASSWORD -H "Content-Type: application/json" -d '
@ -72,24 +72,27 @@ Curl commands to run within the TA container to reactivate documents:
```
## Corrupted ES index reset
After a hard reset of your server or any other hardware failure you might experience data corruption. ES can be particularly unhappy about that, especially if the reset happens during actively writing to disk. It's very likely that only your `/indices` folder got corrupted, as that is where the regular read/writes happen. Luckily you have your [snapshots](settings/application.md#snapshots) set up.
After a hard reset of your server, or any other hardware failure, you might experience data corruption. ES can be particularly unhappy about that, especially if the reset happens while it is actively writing to disk. It's very likely that only your `/indices` folder got corrupted, as that is where the regular read/writes happen. Luckily you have your [snapshots](settings/application.md#snapshots) set up.
ES will not start, if the data is corrupted. So, stop all containers, delete everything *except* the `/snapshot` folder in the ES volume. After that, start everything back up. Tube Archivist will create a new blank index. All your snapshots should be available for restore on your settings page, you probably want to restore the most recent one. After restore, run a [filesystem rescan](settings/actions.md#rescan-filesystem) for good measures.
!!! warning
Confirm you have an available, recoverable snapshot prior to performing this action.
ES will not start, if the data is corrupted. So, stop all containers, delete everything *except* the `/snapshot` folder in the ES volume. After that, start everything back up. **Tube Archivist** will create a new blank index. All your snapshots should be available for restore on your settings page, you probably want to restore the most recent one. After restore, run a [filesystem rescan](settings/actions.md#rescan-filesystem) for good measure.
## ES mapping migrations troubleshooting
Tube Archivist will apply mapping changes at application startup. That usually is needed when changing how an existing field is indexed. That should be seamless and automatic, but can leave your index in a messed up state if that process gets interrupted for any reason. Common reasons could be that if you artificially limit the memory to the container, disabling the OS to dynamically manage that, or if you don't have enough available storage on the ES volume, or if you interrupt that because of your impatience (don't do that).
**Tube Archivist** will apply mapping changes at application startup for certain versions. That usually is needed when changing how an existing field is indexed. This action should be seamless and automatic, but can leave your index in a messed up state if that process gets interrupted for any reason. Common reasons could be that if you artificially limit the memory to the container, disabling the OS to dynamically manage that, or if you don't have enough available storage on the ES volume, or if you interrupt that because of your impatience (don't do that).
In general the process is:
- Compare existing mapping with predefined expected mapping
- If that is identical, there is nothing to do
- Else create a `_backup` of the existing index
- Delete the original index and create a new empty one with the new mapping in place
- Copy over the previously created `_backup` index to apply the new mappings
- Delete the now leftover `_backup` index.
1. Compare existing mapping with predefined expected mapping
1. If that is identical, there is nothing to do
1. Else create a `_backup` of the existing index
1. Delete the original index and create a new empty one with the new mapping in place
1. Copy over the previously created `_backup` index to apply the new mappings
1. Delete the now leftover `_backup` index.
If you are not sure if anything is happening, you can monitor your index and `docs.count` value for each index, that should change over time during that process and you should get an indicator of progress happening:
If you are not sure if anything is happening, you can monitor your index and `docs.count` value for each index. Those values should change over time during the process and you should get an indicator of progress happening:
From within the ES container:
@ -97,9 +100,9 @@ From within the ES container:
curl -u elastic:$ELASTIC_PASSWORD "localhost:9200/_cat/indices?v&s=index"
```
If that process gets interrupted before deleting the `_backup` index and you try to run this again, you will see an error like `resource_already_exists_exception` for example `index [ta_comment_backup/...] already exists` indicating in this case that your migration previously failed for the `ta_comment` index.
If that process gets interrupted before deleting the `_backup` index and you try to run this again, you will see an error like `resource_already_exists_exception`, for example `index [ta_comment_backup/...] already exists` indicating in this case that your migration previously failed for the `ta_comment` index.
First make sure you have the original index still with the command above, after verifying that, stop the TA container then you can delete the `_backup` index e.g. in the case of `ta_comment_backup`.
First, make sure you still have the original index with the command above. After verifying, stop the TA container, then you can delete the `_backup` index, e.g. in the case of `ta_comment_backup`.
```bash
curl -XDELETE -u elastic:$ELASTIC_PASSWORD "localhost:9200/ta_comment_backup?pretty"
@ -112,12 +115,12 @@ and you should get:
}
```
Then you can start everything again and the migration will run again. If your error persists, the ES and TA logs should give additional debug info.
Then you can restart the container and the migration will run again. If your error persists, the ES and TA logs should give additional debug info.
## Manual yt-dlp update
This project strives for timely updates when yt-dlp makes a new release, but sometimes ideals meet reality. Also sometimes yt-dlp has a fix published, but not yet released.
Doing this is **very likely** going to break things for you. You will want to try this out on a testing instance first. Regularly there have been subtle changes in the yt-dlp API, so only do this if you know how to debug this project by yourself, but obviously share your fixes so any problems can be dealt with before release.
!!! warning
Doing this is **very likely** going to break things for you. You will want to try this out on a testing instance first. There are have regularly been subtle changes in the yt-dlp API, so only do this if you know how to debug this project by yourself, but obviously share your fixes so any problems can be dealt with before release.
This project strives for timely updates when yt-dlp makes a new release, but sometimes ideals meet reality. Also, sometimes yt-dlp has a fix published, but not yet released.
**Build your own image**: Update the version in `requirements.txt` and rebuild the image from `Dockerfile`. This will use your own image, even on container rebuild.
@ -128,21 +131,22 @@ Doing this is **very likely** going to break things for you. You will want to tr
Update to newest regular yt-dlp release:
```
```bash
pip install --upgrade yt-dlp
```
To update to nightly you'll have to specify the correct `--target` folder:
```
```bash
pip install \
--upgrade \
--target=/root/.local/bin \
https://github.com/yt-dlp/yt-dlp/archive/master.tar.gz
```
This is obviously particularly likely to create problems. Also note that the `--version` command will only show the latest regular release, not a nightly mention.
This is obviously particularly likely to create problems. Also note that the `--version` command will only show the latest regular release, not a nightly version mentioned.
## Erase errors from download queue
Sometimes the download queue might have some videos that have errored due to rate limits. Videos that have errors won't be retried in a future download queue re-run unless you individually click "Download now" for each individual video. In order to bulk clear the errors from the download queue one needs to execute the following command from within the Tube Archivist container:
Sometimes the download queue might have some videos that have errored due to rate limits or other issues during the download process. Videos that have errors won't be retried in a future download queue re-run unless you individually click "Download now" for each individual video or until the container is fully restarted. In order to bulk clear the errors from the download queue without restarting the container, you need to execute the following command from within the TA container:
???+ bug "Erase Errors"
```bash
curl -X POST "$ES_URL/ta_download/_update_by_query?pretty" -u elastic:$ELASTIC_PASSWORD -H 'Content-Type: application/json' -d'
{

24
mkdocs/docs/channels.md
View File

@ -7,25 +7,25 @@ description: Subscribe to channels, browse your channels and access additional m
The channels are organized on two different levels, similar to the [playlists](playlists.md):
## Channels Overview
Accessible at `/channel/` of your Tube Archivist, the **Overview Page** shows a list of all channels you have indexed.
Accessible at `/channel/` of your **Tube Archivist** instance, the **Overview Page** shows a list of all channels you have indexed.
- You can filter that list to show or hide subscribed channels with the toggle. Clicking on the channel banner or the channel name will direct you to the *Channel Detail Page*.
- If you are subscribed to a channel an *Unsubscribe* button will show, if you aren't subscribed, a *Subscribe* button will show instead.
- You can filter that list to show or hide subscribed channels with the toggle. Clicking on the channel banner or the channel name will direct you to the [*Channel Detail Page*](channels.md#channel-detail).
- If you are subscribed to a channel, an *Unsubscribe* button will show. If you aren't subscribed, a *Subscribe* button will show, instead.
The **Subscribe to Channels** button <img src="/assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to subscribe to a channel. You have a few options:
- Enter a [channel](urls.md#channel).
- Enter a [video](urls.md#video) and let Tube Archivist extract the channel ID for you.
- Enter a [video](urls.md#video), and let **Tube Archivist** extract the channel ID for you.
- Add one per line.
To search your channels, click on the search icon <img src="/assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> to reach the search page. Start your query with `channel:`, learn more on the [search](search.md) page.
To search your channels, click on the search icon <img src="/assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> to reach the search page. Start your query with `channel:` - learn more on the [search](search.md) page.
## Channel Detail
Each channel gets a set of channel detail pages.
- If you are subscribed to the channel, an *Unsubscribe* button will show, else the *Subscribe* button will show.
- You'll see some statistics of the channel, like how many videos you have, total playback and total size. That aggregation is based your filter, e.g. if you toggle *Hide watched*, the aggregation will be over your unwatched videos only.
- The **Mark as Watched** and **Mark as Unwatched** buttos will mark all videos of this channel as watched/unwatched.
- If you are subscribed to the channel, an *Unsubscribe* button will show, otherwise the *Subscribe* button will show.
- You'll see some statistics of the channel, like how many videos you have, total playback and total size. That aggregation is based on your filter, e.g. if you toggle *Hide watched*, the aggregation will be over your unwatched videos only.
- The **Mark as Watched** and **Mark as Unwatched** buttons will mark all videos of this channel as watched/unwatched.
### Videos
Accessible at `/channel/<channel-id>/`, this page shows all the videos you have downloaded from this channel.
@ -37,22 +37,22 @@ If you have any streams indexed, this page will become accessible at `/channel/<
If you have any shorts videos indexed, this page will become accessible at `/channel/<channel-id>/shorts/`, this page shows all the shorts videos of that channel.
### Playlists
If you have playlists from this channel indexed, this page will become accessible at `/channel/<channel-id>/playlist/`. Activate channel playlist indexing on the about page.
If you have playlists from this channel indexed, this page will become accessible at `/channel/<channel-id>/playlist/`. Activate channel playlist indexing on the [About page](channels.md#about).
### About
On the *Channel About* page, accessible at `/channel/<channel-id>/about/`, you can see additional metadata.
- The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata additionally this will also delete playlists metadata belonging to that channel.
- The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata. Additionally, this will also delete playlists metadata belonging to that channel.
- The button **Reindex** will reindex all channel metadata. This will also categorize existing videos as shorts or streams.
- The button **Reindex Videos** will reindex metadata for all videos in this channel.
If available, you can find the channel description and channel tags there.
If available, you can find the channel description and channel tags here.
The channel customize form gives options to change settings on a per channel basis. Any configurations here will overwrite your configurations from the [settings](settings/application.md) page.
- **Download Format**: Overwrite the download quality for videos from this channel.
- **Auto Delete**: Automatically delete watched videos from this channel after selected days.
- **Index Playlists**: Automatically add all Playlists with at least a video downloaded to your index. Only do this for channels where you care about playlists as this will slow down indexing new videos for having to check which playlist this belongs to.
- **Index Playlists**: Automatically add all Playlists with at least a video downloaded to your index. Only do this for channels where you care about playlists as this will slow down indexing new videos because each video will need to check if it belongs to each indexed playlist.
- **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. Customize per channel: You can *disable* or *enable* SponsorBlock for certain channels only to overwrite the behavior set on the [settings](settings/application.md) page. Selecting *unset* will remove the overwrite and your setting will fall back to the default on the settings page.
### Downloads

12
mkdocs/docs/configuration/cast.md
View File

@ -1,11 +1,11 @@
As Cast doesn't support authentication, enabling this functionality will make your static files like artwork and media files accessible by guessing the links. That's read only access, the application itself is still protected.
As Cast doesn't support authentication, enabling this functionality will make your static files, like artwork and media files, accessible by guessing the links. Those only require read-only access and the application itself is still protected.
Enabling this integration will embed an additional third party JS library from **Google**.
Enabling this integration will embed an additional third-party JS library from **Google**.
**Requirements**:
- HTTPS: To use the cast integration HTTPS needs to be enabled, which can be done using a reverse proxy. This is a requirement by Google as communication to the cast device is required to be encrypted, but the content itself is not.
- Supported Browser: A supported browser is required for this integration such as Google Chrome. Other browsers, especially Chromium-based browsers, may support casting by enabling it in the settings.
- Subtitles: Subtitles are supported however they do not work out of the box and require additional configuration. Due to requirements by Google, to use subtitles you need additional headers which will need to be configured in your reverse proxy. See this [page](https://developers.google.com/cast/docs/web_sender/advanced#cors_requirements) for the specific requirements.
- You need the following headers: `Content-Type`, `Accept-Encoding`, and `Range`. Note that the last two headers, `Accept-Encoding` and `Range`, are additional headers that you may not have needed previously.
- HTTPS: To use the cast integration, HTTPS needs to be enabled. This can be done using a reverse proxy. This is a requirement from Google, as communication to the casting device is required to be encrypted, but the content itself is not.
- Supported Browser: A supported browser is required for this integration, such as Google Chrome. Other browsers, especially Chromium-based browsers, may support casting by enabling it in the settings.
- Subtitles: Subtitles are supported, however they do not work out of the box and require additional configuration. Due to requirements by Google, to use subtitles you need additional headers which will need to be configured in your reverse proxy. See this [page](https://developers.google.com/cast/docs/web_sender/advanced#cors_requirements) for the specific requirements.
- You need the following headers: `Content-Type`, `Accept-Encoding`, and `Range`. Note that the last two headers, `Accept-Encoding` and `Range`, are additional headers that you may not have needed previously.
- Wildcards "*" can not be used for the `Access-Control-Allow-Origin` header. If the page has protected media content, it must use a domain instead of a wildcard.

16
mkdocs/docs/configuration/forward-auth.md
View File

@ -1,16 +1,14 @@
You can enable support for authentication proxies such as Authelia.
This effectively disables credentials-based authentication and instead authenticates users if a specific request header contains a known username.
You must make sure that your proxy (nginx, Traefik, Caddy, ...) forwards this header from your auth proxy to tubearchivist.
You must make sure that your proxy (nginx, Traefik, Caddy, ...) forwards this header from your auth proxy to Tube Archivist.
Check the documentation of your auth proxy and your reverse proxy on how to correctly set this up.
Note that this automatically creates new users in the database if they do not already exist.
- `TA_ENABLE_AUTH_PROXY` (ex: `true`) - Set to anything besides empty string to use forward proxy authentication.
- `TA_AUTH_PROXY_USERNAME_HEADER` - The name of the request header that the auth proxy passes to the proxied application (tubearchivist in this case), so that the application can identify the user.
Check the documentation of your auth proxy to get this information.
Note that the request headers are rewritten in tubearchivist: all HTTP headers are prefixed with `HTTP_`, all letters are in uppercase, and dashes are replaced with underscores.
For example, for Authelia, which passes the `Remote-User` HTTP header, the `TA_AUTH_PROXY_USERNAME_HEADER` needs to be configured as `HTTP_REMOTE_USER`.
- `TA_AUTH_PROXY_LOGOUT_URL` - The URL that tubearchivist should redirect to after a logout.
By default, the logout redirects to the login URL, which means the user will be automatically authenticated again.
Instead, you might want to configure the logout URL of the auth proxy here.
| Environment Variable | Example | Description |
| `TA_ENABLE_AUTH_PROXY` | `true` | Set to anything besides empty string to use forward proxy authentication. |
| `TA_AUTH_PROXY_USERNAME_HEADER` | `HTTP_REMOTE_USER` | The name of the request header that the auth proxy passes to the proxied application (**Tube Archivist** in this case), so that the application can identify the user. Check the documentation of your auth proxy to get this information.
??? note
The request headers are rewritten within **Tube Archivist**: all HTTP headers are prefixed with `HTTP_`, all letters are in uppercase, and dashes are replaced with underscores.For example, for Authelia, which passes the `Remote-User` HTTP header, the `TA_AUTH_PROXY_USERNAME_HEADER` needs to be configured as `HTTP_REMOTE_USER`. |
| `TA_AUTH_PROXY_LOGOUT_URL` | | The URL that **Tube Archivist** should redirect to after a logout. By default, the logout redirects to the login URL, which means the user will be automatically authenticated again. Instead, you might want to configure the logout URL of the auth proxy here. |

28
mkdocs/docs/configuration/ldap.md
View File

@ -1,15 +1,17 @@
You can configure LDAP with the following environment variables:
You can enable and configure LDAP with the following environment variables:
- `TA_LDAP` (ex: `true`) Set to anything besides empty string to use LDAP authentication **instead** of local user authentication.
- `TA_LDAP_SERVER_URI` (ex: `ldap://ldap-server:389`) Set to the uri of your LDAP server.
- `TA_LDAP_DISABLE_CERT_CHECK` (ex: `true`) Set to anything besides empty string to disable certificate checking when connecting over LDAPS.
- `TA_LDAP_BIND_DN` (ex: `uid=search-user,ou=users,dc=your-server`) DN of the user that is able to perform searches on your LDAP account.
- `TA_LDAP_BIND_PASSWORD` (ex: `yoursecretpassword`) Password for the search user.
- `TA_LDAP_USER_ATTR_MAP_USERNAME` (default: `uid`) Bind attribute used to map LDAP user's username
- `TA_LDAP_USER_ATTR_MAP_PERSONALNAME` (default: `givenName`) Bind attribute used to match LDAP user's First Name/Personal Name.
- `TA_LDAP_USER_ATTR_MAP_SURNAME` (default: `sn`) Bind attribute used to match LDAP user's Last Name/Surname.
- `TA_LDAP_USER_ATTR_MAP_EMAIL` (default: `mail`) Bind attribute used to match LDAP user's EMail address
- `TA_LDAP_USER_BASE` (ex: `ou=users,dc=your-server`) Search base for user filter.
- `TA_LDAP_USER_FILTER` (ex: `(objectClass=user)`) Filter for valid users. Login usernames are matched using the attribute specified in `TA_LDAP_USER_ATTR_MAP_USERNAME` and should not be specified in this filter.
| Environment Variable | Default | Example | Description |
| :-------------------- | :-------- | :-------- | :------------ |
| `TA_LDAP` | `false` | `true` | Set to anything besides empty string to use LDAP authentication **instead** of local user authentication. |
| `TA_LDAP_SERVER_URI` | `null` | `ldap://ldap-server:389` | Set to the uri of your LDAP server. |
| `TA_LDAP_DISABLE_CERT_CHECK` | `null` | `true` | Set to anything besides empty string to disable certificate checking when connecting over LDAPS. |
| `TA_LDAP_BIND_DN` | `null` | `uid=search-user,ou=users,dc=your-server` | DN of the user that is able to perform searches on your LDAP account. |
| `TA_LDAP_BIND_PASSWORD` | `null` | `yoursecretpassword` | Password for the search user. |
| `TA_LDAP_USER_ATTR_MAP_USERNAME` | `uid` | `uid` | Bind attribute used to map LDAP user's username |
| `TA_LDAP_USER_ATTR_MAP_PERSONALNAME` | `givenName` |`givenName` | Bind attribute used to match LDAP user's First Name/Personal Name. |
| `TA_LDAP_USER_ATTR_MAP_SURNAME` | `sn` |`sn` | Bind attribute used to match LDAP user's Last Name/Surname. |
| `TA_LDAP_USER_ATTR_MAP_EMAIL` | `mail` |`mail` | Bind attribute used to match LDAP user's EMail address |
| `TA_LDAP_USER_BASE` | `null` | `ou=users,dc=your-server` | Search base for user filter. |
| `TA_LDAP_USER_FILTER` | `null` | `(objectClass=user)` | Filter for valid users. Login usernames are matched using the attribute specified in `TA_LDAP_USER_ATTR_MAP_USERNAME` and should not be specified in this filter. |
When LDAP authentication is enabled, django passwords (e.g. the password defined in TA_PASSWORD), will not allow you to login, only the LDAP server is used.
While LDAP authentication is enabled, the django-managed passwords (e.g. the password defined in TA_PASSWORD), will not allow you to login. Only the LDAP server is used for authentication.

38
mkdocs/docs/configuration/stylesheets.md
View File

@ -1,38 +1,40 @@
You can change the appearance of Tube Archivist by selecting a stylesheet in Settings under User Configurations.
You can change the appearance of **Tube Archivist** by selecting a stylesheet in Settings under User Configurations.
## Adding Stylesheets
Assuming a default configuration, stylesheets are stored in `/app/static/css` in the `tubearchivist` container. This is where additional stylesheets should be added.
The recommended method for adding new stylesheets is to mount them in the `docker-compose.yml` file. New mounts need to be added to the `tubearchivist` volume section. For example, `test.css` is added. `test.css` is located in the same directory as the `docker-compose.yml` file.
The recommended method for adding new stylesheets is to mount them in the `docker-compose.yml` file. New mounts need to be added to the `tubearchivist` volume section. For example, `test.css` is added. If `test.css` is located in the same directory as the `docker-compose.yml` file.
```yaml
volumes:
- media:/youtube
- cache:/cache
- ./test.css:/app/static/css/test.css
```
???+ example
```yaml
volumes:
- media:/youtube
- cache:/cache
- ./test.css:/app/static/css/test.css
```
The container will need to be restarted for changes to take effect, which can be accomplished by running the command `docker compose restart`.
The container will need to be rebuilt for changes to take effect, which can be accomplished by running the command `docker compose up -d`.
## Creating Stylesheets
Tube Archivist applies the `style.css` stylesheet before applying the user's selected stylesheet.
**Tube Archivist** applies the `style.css` stylesheet before applying the user's selected stylesheet.
You can use the default `dark.css` theme as a template to create your own. You can get it from the repo [here](https://github.com/tubearchivist/tubearchivist/blob/master/tubearchivist/static/css/dark.css).
The `:root` pseudo-class contains variables that are frequently used in `style.css` for consistent theming. However, not all changes need to be made in `:root`. Classes, IDs, and HTML tags can have their properties overrridden by simply declaring new properties.
The `:root` pseudo-class contains variables that are frequently used in `style.css` for consistent theming. However, not all changes need to be made in `:root`. Classes, IDs, and HTML tags can have their properties overridden by simply declaring new properties.
For example, the following addition to a custom stylesheet would make all `p` tags have a cursive font.
```css
p {
font-family: cursive;
}
```
???+ example
```css
p {
font-family: cursive;
}
```
To create a stylesheet, any selectable stylesheet should be used as a base. Changes can then be made as needed. Changes can be previewed easily by editing the existing stylesheet in realtime through your browser's developer tools. Below is an example of editing the `dark.css` stylesheet through Firefox's developer tools.
To create a stylesheet, any selectable stylesheet should be used as a base. Changes can then be made as needed. Changes can be previewed easily by editing the existing stylesheet in real-time through your browser's developer tools. Below is an example of editing the `dark.css` stylesheet through Firefox's developer tools.
![TubeArchivist](../assets/stylesheets_example.png)
Note that live changes will be lost when the page is refreshed. Save and copy live changes to prevent data loss.
Note that live changes will be lost when the page is refreshed. Copy and save the changes to prevent data loss.

20
mkdocs/docs/downloads.md
View File

@ -3,26 +3,26 @@ description: Populate your download queue by rescanning your Subscriptions or ma
---
# Downloads Page
Accessible at `/downloads/` of your Tube Archivist, this page handles all the download functionality.
Accessible at `/downloads/` of your **Tube Archivist** instance, this page handles all the download functionality.
## Rescan Subscriptions
The **Rescan Subscriptions** icon <img src="/assets/icon-rescan.png?raw=true" alt="rescan icon" width="20px" style="margin:0 5px;"> will start a background task to look for new videos from the channels and playlists you are subscribed to.
Tube Archivist will get available *videos*, *shorts* and *streams* from each channel, you can define the channel and playlist page size on the [settings page](settings/application.md#subscriptions). With the default page size, expect this process to take around 2-3 seconds for each channel or playlist you are subscribed to. A status message will show the progress.
**Tube Archivist** will get available *videos*, *shorts* and *streams* from each channel. You can define the channel and playlist page size on the [settings page](settings/application.md#subscriptions). With the default page size, expect this process to take around 2-3 seconds for each channel or playlist you are subscribed to. A status message will show the progress.
Then for every video found, **Tube Archivist** will skip the video if it has already been downloaded or if you added it to the *ignored* list before. All the other videos will get added to the download queue. Expect this to take around 2 seconds for each video as **Tube Archivist** needs to grab some additional metadata and artwork. New videos will get added at the bottom of the download queue.
Then for every video found, **Tube Archivist** will skip the video if it has already been downloaded or if you added it to the *ignored* list previously. All other videos will get added to the download queue. Expect this to take around 2 seconds for each video as **Tube Archivist** needs to grab some additional metadata and artwork. New videos will get added to the end of the download queue.
## Download Queue
The **Start Download** icon <img src="/assets/icon-download.png?raw=true" alt="download icon" width="20px" style="margin:0 5px;"> will start the download process. This will prioritize videos added as *auto start* or as *download now*, starting from the top of the queue. Once the process started, a progress message will show with additional details and controls:
The **Start Download** icon <img src="/assets/icon-download.png?raw=true" alt="download icon" width="20px" style="margin:0 5px;"> will start the download process. This will prioritize videos added as *auto start* or as *download now*, starting from the top of the queue. Once the process has started, a progress message will show with additional details and controls:
- The stop icon <img src="/assets/icon-stop.png?raw=true" alt="stop icon" width="20px" style="margin:0 5px;"> will gracefully stop the download process, once the current video has been finished successfully. This will also reset the auto start behavior to avoid confusion.
- [Currenlty broken] The cancel icon <img src="/assets/icon-close-red.png?raw=true" alt="close icon" width="20px" style="margin:0 5px;"> is equivalent to killing the process and will stop the download immediately. Any leftover files will get deleted, the canceled video will still be available in the download queue.
- [Currently broken] The cancel icon <img src="/assets/icon-close-red.png?raw=true" alt="close icon" width="20px" style="margin:0 5px;"> is equivalent to killing the process and will stop the download immediately. Any leftover files will get deleted, the canceled video will still be available in the download queue.
After downloading, Tube Archivist tries to add new videos to already indexed playlists and if activated on the settings page, get comments for the new videos.
After downloading, **Tube Archivist** tries to add new videos to already indexed playlists and, if activated on the settings page, get comments for the new videos.
## Add to Download Queue
The **Add to Download Queue** icon <img src="/assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to manually add videos to the download queue. Add one item per line. The *Add to queue* will add the videos as regular items to the queue, you'll be able to ignore undesired videos before starting the queue. If you add them with *Download Now*, this will start the download automatically with priority.
The **Add to Download Queue** icon <img src="/assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to manually add videos to the download queue. Add one item per line. The *Add to queue* will add the videos as regular items to the queue, where you'll be able to ignore undesired videos before starting the queue. If you add them with *Download Now*, this will start the download automatically with priority.
You have a few options:
@ -36,16 +36,16 @@ Add a [channel](urls.md#channel) to download the complete channel, or a [channel
Add a [playlist](urls.md#playlist) to download all videos in the list. When adding a playlist to the queue, this playlist will automatically get [indexed](playlists.md#playlist-detail).
## The Download Queue
Below the three buttons you find the download queue. New items will get added at the bottom of the queue, the next video to download once you click on **Start Download** will be the first in the list.
Below the three action buttons is the download queue. New items will get added to the end of the queue, the next video to download once you click on **Start Download** will be the first in the list.
You can filter the download queue with the **filter** dropdown box, the filter will show once you have more than one channel in the download queue. Select the channel to filter by name, the number in parentheses indicates how many videos you have pending from this channel. Reset the filter by selecting *all* from the dropdown. This will generate links for the top 30 channels with pending videos.
Every video in the download queue has two buttons:
- **Ignore**: This will remove that video from the download queue and this video will not get added again, even when you **Rescan Subscriptions**.
- **Download now**: This will give priority to this video. If the download process is already running, the prioritized video will get downloaded as soon as the current video is finished. If there is no download process running, this will start downloading this single video and stop after that.
- **Download now**: This will give priority to this video. If the download process is already running, the prioritized video will get downloaded as soon as the current video is finished. If there is no download process running, this will start downloading this single video and stop after that [currently broken - after downloading the selected video, it will currently go through the entire queue].
Failed videos will show an error message of what went wrong and will give you additional options with how to continue. Usually this means the video in the queue is no longer available on YouTube. Tube Archivist will not retry to download a failed video.
Failed videos will show an error message of what went wrong and will give you additional options with how to continue. Usually this means the video in the queue is no longer available on YouTube. **Tube Archivist** will not retry to download a failed video.
You can flip the view by activating **Show Only Ignored Videos**. This will show all videos you have previously *ignored*.
Every video in the ignored list has two buttons:

42
mkdocs/docs/faq.md
View File

@ -5,21 +5,21 @@ description: Frequently asked questions about what this project is, what it trie
# Frequently Asked Questions
## What is the scope of this project?
Tube Archivist is *Your self hosted YouTube media server*, which also defines the primary scope of what this project tries to do:
Tube Archivist is *Your self-hosted YouTube media server*, which also defines the primary scope of what this project tries to do:
- **Self hosted**: This assumes you have full control over the underlying operating system and hardware and can configure things to work properly with Docker, it's volumes and networks as well as whatever disk storage and filesystem you choose to use.
- **Self-hosted**: This assumes you have full control over the underlying operating system and hardware and can configure things to work properly with Docker, it's volumes and networks as well as whatever disk storage and filesystem you choose to use.
- **YouTube**: Downloading, indexing and playing videos from YouTube, there are currently no plans to expand this to any additional platforms.
- **Media server**: This project tries to be a stand alone media server in it's own web interface.
- **Media server**: This project tries to be a stand-alone media server in its own web interface.
Additionally to that, progress is also happening on:
Additionally, progress is also happening to provide the following core capabilities:
- **API**: Endpoints for additional integrations.
- **Browser Extension**: To integrate between youtube.com and Tube Archivist.
Defining the scope is important for the success of any project:
- A scope too broad will result in development effort spreading too thin and will run into danger that his project tries to do too many things and none of them well.
- A too narrow scope will make this project uninteresting and will exclude audiences that could also benefit from this project.
- A scope too broad will result in development effort spreading too thin and will run into danger that this project tries to do too many things and none of them well.
- A scope too narrow will make this project uninteresting and will exclude audiences that could also benefit from this project.
- Not defining a scope will easily lead to misunderstandings and false hopes of where this project tries to go.
Of course this is subject to change: The scope can be expanded as this project continues to grow and more people contribute.
@ -27,28 +27,28 @@ Of course this is subject to change: The scope can be expanded as this project c
## How do I import my videos to Emby-Plex-Jellyfin-Kodi?
Although there are similarities between these excellent projects and Tube Archivist, they have a very different use case. Trying to fit the metadata relations and database structure of a YouTube archival project into these media servers that specialize in Movies and TV shows is always going to be limiting.
Part of the scope is to be its own media server, to be able to overcome these limitations, so that's where the focus and effort of this project is. That being said, the nature of self hosted and open source software gives you all the possible freedom to use your media as you wish.
Part of the scope is to be its own media server, to be able to overcome these limitations, so that's where the focus and effort of this project is. That being said, the nature of self-hosted and open source software gives you all the possible freedom to use your media as you wish.
- **Jellyfin**: There is an API to API integration available to sync metadata from Tube Archivist to Jellyfin: [tubearchivist/tubearchivist-jf](https://github.com/tubearchivist/tubearchivist-jf). Follow the instructions there. Please contribute to improve this integration.
- **Plex**: There is a Plex Scanner and Agent combination that allows integration between Tube Archivist and Plex: [tubearchivist/tubearchivist-plex](https://github.com/tubearchivist/tubearchivist-plex). Follow the instructions there. Please contribute to improve this integration.
## How do I install this natively?
This project is a classical Docker application: There are multiple moving parts that need to be able to interact with each other and need to be compatible with multiple architectures and operating systems. Additionally Docker also drastically reduces development complexity which is highly appreciated.
This project is natively a container-based Docker application: There are multiple moving parts that need to be able to interact with each other and need to be compatible with multiple architectures and operating systems. Additionally, Docker also drastically reduces development complexity which is highly appreciated.
Docker is the only supported installation method. If you don't have any experience with Docker, consider investing the time to learn this very useful technology. Alternatively you can find user provided installation instructions for Podman [here](installation/podman.md).
Docker is the only supported installation method. If you don't have any experience with Docker, consider investing the time to learn this very useful technology. Alternatively, you can find user provided installation instructions for Podman [here](installation/podman.md).
## How do I finetune ElasticSearch?
A recommended configuration of Elasticsearch (ES) is provided in the example docker-compose.yml file. ES is highly configurable and very interesting to learn more about. Refer to the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) if you want to get into it.
## How do I finetune Elasticsearch?
A recommended configuration of Elasticsearch (ES) is provided in the example docker-compose.yml file. ES is highly configurable and very interesting to learn more about. Refer to the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) if you want to get into it.
All testing is done with the recommended configuration, changing that can create hard to debug and randomly occurring problems, use at your own risk.
All testing is done with the recommended configuration, changing that can create hard to debug and randomly occurring problems, so use at your own risk.
## Why Elasticsearch?
That might be an unconventional choice at first glance. Tube Archivist is built to scale to 100k+ videos without slowing down and to 1M+ with only minimal impact on performance, and that all with full text indexing and searching over your subtitles and comments. Elasticsearch is the industry standard <sup>[citation needed]</sup> for a task like that and through its structured query language allows for a very flexible interface to query the index.
That might be an unconventional choice at first glance. Tube Archivist is built to scale to 100k+ videos without slowing down and to 1M+ with only minimal impact on performance, with full text indexing and searching over your subtitles and comments. Elasticsearch is the industry standard <sup>[citation needed]</sup> for a task like that and through its structured query language allows for a very flexible interface to query the index.
That comes at a price: ES can use a lot of memory, particularly on a big index, and will heavily use in memory cached queries to be able to respond within milliseconds, even when searching through multiple GBs of raw text.
That comes at a price: ES can use a lot of memory, particularly on a big index, and will heavily use in-memory cached queries to be able to respond within milliseconds, even when searching through multiple GBs of raw text.
## Why does subscribing to a channel not download the complete channel?
For Tube Archivist, these are two different things: To download a complete channel, add it to the [download queue](downloads.md#add-to-download-queue) with the form or with [Tube Archivist Companion](https://github.com/tubearchivist/browser-extension), the browser extension. This is meant for a complete archival.
For Tube Archivist, these are two different actions: To download a complete channel, add it to the [download queue](downloads.md#add-to-download-queue) with the form or with [Tube Archivist Companion](https://github.com/tubearchivist/browser-extension), the browser extension. This is meant for a complete archival.
Subscribing to a channel is for downloading new videos as they come out. That is designed to be as quick as possible, to allow you to efficiently rescan your favourite channels frequently. This will add videos to your download queue based on your [channel page size](settings/application.md#subscriptions).
@ -68,7 +68,7 @@ This project tries to be compatible with as many filesystem/OS variations out th
That's why this project has landed on `<channel-id>/<video-id>.mp4`. These values are guaranteed to be static, are guaranteed to be compatible with every filesystem out there and make things predictable where all files will go on every instance of Tube Archivist indefinitely.
For browsing these files you have the fancy interface provided by this project, or use a supported integration as stated above. If you really want to you could easily also create your own file naming structure with the API and symlinks, but that is not part of the scope of this project.
For browsing these files you have the fancy interface provided by this project, or use a supported integration as stated above. If you really want to, you could easily also create your own file naming structure with the API and symlinks, but that is not part of the scope of this project.
## Does this project implement feature X?
Generic answer to that question is:
@ -83,16 +83,18 @@ Read the [docs](https://docs.tubearchivist.com/) with a comprehensive overview o
## Am I getting blocked/throttled?
Heavy users of this project can run into danger of getting blocked / throttled or soft banned from YT. Symptoms might not be obvious and the error messages cryptic. Users have reported messages and behavior like:
Heavy users of this project can run into the danger of getting blocked / throttled or soft banned from YouTube. Symptoms might not be obvious and the error messages cryptic. Users have reported messages and behavior like:
- `Playlists that require authentication may not extract correctly without a successful webpage download` but you aren't downloading a private playlist, a sign that YT is blocking your requests behind authentication.
- `Playlists that require authentication may not extract correctly without a successful webpage download` but you aren't downloading a private playlist, a sign that YouTube is blocking your requests behind authentication.
- `Requested format is not available. Use --list-formats for a list of available formats` but your download format is valid, a sign that yt-dlp can't extract available streams because your request gets blocked.
- `Got error: HTTP Error 403: Forbidden` a generic HTTP message that YT is blocking your request.
- `Got error: HTTP Error 403: Forbidden` a generic HTTP message that YouTube is blocking your request.
- Your download speed suddenly drops to a few KB/s.
There are other error messages that show up from time to time and may affect only part of the download request. YouTube is frequently changing their tactics to combat the generally larger influx of automated downloads.
**Solutions**
- Wait and try again, these blocks might get lifted somewhere between a day or a week.
- As described above, use a VPN or proxy to change your public IP, rotate every so often.
- In some cases if you are using [your cookie](settings/application.md#cookie), this can be an account level ban and YT will block all requests from that cookie/account. Sometimes refreshing your cookie will work around that, but most likely only temporarily. Only known solution for these cases is to remove your cookie.
- In some cases if you are using [your cookie](settings/application.md#cookie), this can be an account level ban and YouTube will block all requests from that cookie/account. Sometimes refreshing your cookie will work around that, but most likely only temporarily. Only known solution for these cases is to remove your cookie.
- Conversely in some cases, adding your cookie can be a solution if you didn't use your cookie previously, as user authenticated requests are sometimes allowed to pass.

2
mkdocs/docs/index.md
View File

@ -23,7 +23,7 @@ Welcome to the official Tube Archivist Docs. This is an up-to-date documentation
An empty checkbox icon <img src="assets/icon-unseen.png?raw=true" alt="unseen icon" width="20px" style="margin:0 5px;"> will show for videos you haven't marked as watched. Click on it and the icon will change to a filled checkbox <img src="assets/icon-seen.png?raw=true" alt="seen icon" width="20px" style="margin:0 5px;"> indicating it as watched - click again to revert.
When available the <img src="assets/icon-gridview.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> gridview icon will display the list in a grid. A grid row holds 3 items by default, use the <img src="assets/icon-add.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> icon to add more or the <img src="assets/icon-substract.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> icon to remove items per row, depending on your screen size. The <img src="assets/icon-listview.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> listview icon will arrange the items in a list. The sort icon <img src="assets/icon-sort.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> will open additional sort options.
When available, the <img src="assets/icon-gridview.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> gridview icon will display the list in a grid. A grid row holds 3 items by default, use the <img src="assets/icon-add.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> icon to add more or the <img src="assets/icon-substract.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> icon to remove items per row, depending on your screen size. The <img src="assets/icon-listview.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> listview icon will arrange the items in a list. The sort icon <img src="assets/icon-sort.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> will open additional sort options.
You can control the video player with the following keyboard shortcuts:

114
mkdocs/docs/installation/docker-compose.md
View File

@ -1,42 +1,47 @@
# Setting up TubeArchivist with Docker
# Setting up Tube Archivist with Docker
TubeArchivist requires Docker. Please make sure that it is installed and running on your computer before continuing.
Docker is required because Tube Archivist depends on three main components split up into separate docker containers.
**Tube Archivist** requires Docker. Please make sure that Docker is installed and running on your computer before continuing.
Docker is required because **Tube Archivist** depends on three main components split up into separate containers. Solutions that do not utilize the containers are largely untested and unsupported.
For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus.
For minimal system requirements, the **Tube Archivist** stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimum requirements for CPU are usually expected to be dual core with 4 threads, with better performance coming from quad core and higher, and more available threads.
!!! note
For **arm64**: Tube Archivist is a multi arch container, same for redis. For Elasitc Search use the official image for arm64 support. Other architectures are not supported.
!!! info
For **arm64**: **Tube Archivist** is built as a multi-architecture (multi-arch) container, same for Redis. Elasticsearch should use the official image for arm64 support. Other architectures are not supported.
Save the [docker-compose.yml](https://github.com/tubearchivist/tubearchivist/blob/master/docker-compose.yml) file from this reposity somewhere permanent on your system, keeping it named `docker-compose.yml`. You'll need to refer to it whenever starting this application.
Save the [docker-compose.yml](https://github.com/tubearchivist/tubearchivist/blob/master/docker-compose.yml) file from the **Tube Archivist** repository somewhere permanent on your system, keeping it named `docker-compose.yml`. You'll need to refer to it whenever starting this application.
## Overview
The main Python application that displays and serves your video collection, built with Django.
**Tube Archivist** is a Python application that displays and serves your video collection, built with Django.
- Serves the interface on port `8000`
- Needs a volume for the video archive at `/youtube`
- And another volume to save application data at `/cache`.
- The environment variables `ES_URL` and `REDIS_HOST` are needed to tell Tube Archivist where Elasticsearch and Redis respectively are located.
- The environment variables `HOST_UID` and `HOST_GID` allows Tube Archivist to `chown` the video files to the main host system user instead of the container user. Those two variables are optional, not setting them will disable that functionality. That might be needed if the underlying filesystem doesn't support `chown` like *NFS*.
- Set the environment variable `TA_HOST` to match with the system running Tube Archivist. This can be a domain like *example.com*, a subdomain like *ta.example.com* or an IP address like *192.168.1.20*. If you are running Tube Archivist behind a SSL reverse proxy, specify the protocoll. You can add multiple hostnames separated with a space. Any wrong configurations here will result in a `Bad Request (400)` response.
- Needs a volume for the video archive linked to `/youtube`.
- Needs a volume to save application data linked to `/cache`.
- The environment variables `ES_URL` and `REDIS_HOST` are needed to tell **Tube Archivist** from where Elasticsearch and Redis respectively are accessible.
- The environment variables `HOST_UID` and `HOST_GID` allows **Tube Archivist** to `chown` the video files to the main host system user instead of the container user.
- These variables are optional and not setting them will disable that functionality. That might be needed if the underlying filesystem doesn't support `chown`, like *NFS*.
- Set the environment variable `TA_HOST` to match with the expected ways you will access your **Tube Archivist** instance.
- This can be a domain like *example.com*, a subdomain like *ta.example.com* or an IP address like *192.168.1.20*.
- If you are running **Tube Archivist** behind a SSL reverse proxy, specify the protocol (`https://`).
- You can add multiple hostnames separated with a space.
- Any wrong configurations here will result in a `Bad Request (400)` response.
- Change the environment variables `TA_USERNAME` and `TA_PASSWORD` to create the initial credentials.
- `ELASTIC_PASSWORD` is for the password for Elasticsearch. The environment variable `ELASTIC_USER` is optional, should you want to change the username from the default *elastic*.
- Optionally set `ES_SNAPSHOT_DIR` to change the folder where ES is storing it's snapshots. When changing that, make sure you have persistence. That is an absolute path from inside the ES container.
- Set `ES_DISABLE_VERIFY_SSL`, boolean value, to disable SSL verification for connections to ES, e.g. for self signed certificates.
- For the scheduler to know what time it is, set your timezone with the `TZ` environment variable, defaults to *UTC*.
- Set the environment variable `ENABLE_CAST=True` to send videos to your cast device, [read more](../configuration/cast.md).
- `ELASTIC_PASSWORD` is the password for Elasticsearch. The environment variable `ELASTIC_USER` is optional and only if you want to change the username from the default, *elastic*.
- Optionally, set `ES_SNAPSHOT_DIR` to change the folder where ES is storing its snapshots. When changing the location, make sure you have persistence of the new location. That is an absolute path from inside the ES container.
- Set `ES_DISABLE_VERIFY_SSL`, a boolean value, to disable SSL verification for connections to ES, e.g. for self-signed certificates.
- For the scheduler to know what time it is, set your timezone with the `TZ` environment variable, defaults to *UTC*.
- Set `ENABLE_CAST`, a boolean value, to enable sending videos to your cast-ready devices, [read more](../configuration/cast.md).
## Configuring TubeArchivist
Edit the following values from that file:
## Configuring Tube Archivist
Edit the following values in your local `docker-compose.yml` file:
Under `tubearchivist`->`environment`:
- `HOST_UID`: your UID, if you want TubeArchivist to create files with your UID. Remove if you are OK with files being owned by the the container user.
- `HOST_GID`: as above but GID.
- `TA_HOST`: change it to the address of the machine you're running this on. This can be an IP address or a domain name.
- `HOST_UID`: the UID of a local user, for if you want **Tube Archivist** to create files with a specific UID. Remove if default UID is acceptable or required.
- `HOST_GID`: the GID of a local group, same as the `HOST_UID`.
- `TA_HOST`: change it to match the address of the machine you're running this on. This can be an IP address or a domain name.
- `TA_PASSWORD`: pick a password to use when logging in.
- `ELASTIC_PASSWORD`: pick a password for the elastic service. You won't need to type this yourself.
- `ELASTIC_PASSWORD`: pick a password for the Elasticsearch service. You won't need to type this yourself, but it does need to match with the `archivist-es` reference.
- `TZ`: your time zone. If you don't know yours, you can look it up [here](https://www.timezoneconverter.com/cgi-bin/findzone/findzone).
Under `archivist-es`->`environment`:
@ -44,71 +49,68 @@ Under `archivist-es`->`environment`:
- `"ELASTIC_PASSWORD=verysecret"`: change `verysecret` to match the `ELASTIC_PASSWORD` you picked above.
By default, Docker will store all data, including downloaded data, in its own data-root directory (which you can find by running `docker info` and looking for the "Docker Root Dir"). If you want to use other locations, you can replace the `media:`, `cache:`, `redis:`, and `es:` volume names with absolute paths; if you do, remove them from the `volumes:` list at the bottom of the file.
By default Docker will store all data, including downloaded data, in its own data-root directory (which you can find by running `docker info` and looking for the "Docker Root Dir"). If you want to use other locations, you can replace the `media:`, `cache:`, `redis:`, and `es:` volume names with absolute paths; if you do, remove them from the `volumes:` list at the bottom of the file.
From a terminal, `cd` into the directory you saved the `docker-compose.yml` file in and run `docker compose up --detach`. The first time you do this it will download the appropriate images, which can take a minute.
You can follow the logs with `docker compose logs -f`. Once it's ready it will print something like `celery@1234567890ab ready`. At this point you should be able to go to `http://your-host:8000` and log in with the `TA_USER`/`TA_PASSWORD` credentials.
You can bring the application down by running `docker compose down` in the same directory.
Use the *latest* (the default) or a named semantic version tag for the docker images. The *unstable* tag is for intermediate testing and as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md).
Use the *latest* (the default) or a named semantic version tag for the Docker images. The *unstable* tag is for intermediate testing, and, as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md).
### Port Collisions
If you have a collision on port `8000`, best solution is to use dockers *HOST_PORT* and *CONTAINER_PORT* distinction: To for example change the interface to port 9000 use `9000:8000` in your docker-compose file.
If you have a collision on port `8000`, best solution is to use Docker's *HOST_PORT* and *CONTAINER_PORT* distinction: For example, to change the interface to port 9000, use `9000:8000` in your `docker-compose.yml` file under `tubearchivist`->`port`.
Should that not be an option, the Tube Archivist container takes these two additional environment variables:
Should that not be an option, the TA container takes these two additional environment variables:
- **TA_PORT**: To actually change the port where nginx listens, make sure to also change the ports value in your docker-compose file.
- **TA_UWSGI_PORT**: To change the default uwsgi port 8080 used for container internal networking between uwsgi serving the django application and nginx.
- **TA_PORT**: To actually change the port where nginx listens, make sure to also change the ports value in your `docker-compose.yml` file.
- **TA_UWSGI_PORT**: To change the default uwsgi port from 8080, which is used for container internal networking between uwsgi, serving the django application, and nginx.
Changing any of these two environment variables will change the files *nginx.conf* and *uwsgi.ini* at startup using `sed` in your container.
Changing any of these two environment variables will change the files *nginx.conf* and *uwsgi.ini* at startup, using `sed` in your container.
## ElasticSearch
!!! note
Tube Archivist depends on Elasticsearch 8.
!!! success
**Tube Archivist** depends on Elasticsearch 8.
Use `bbilly1/tubearchivist-es` to automatically get the recommended version, or use the official image with the version tag in the docker-compose file.
Use `bbilly1/tubearchivist-es` to automatically get the recommended version, or use the official image with the version tag in the `docker-compose.yml` file.
Use official Elastic Search for **arm64**.
Use the official Elasticsearch image for **arm64**.
Stores video meta data and makes everything searchable. Also keeps track of the download queue.
Elasticsearch stores the video metadata and enables searchable functionality for all fields. Elasticsearch is also used to keep track of the download queue.
- Needs to be accessible over the default port `9200`
- Needs a volume at `/usr/share/elasticsearch/data` to store data
- Needs a volume to store persistent data linked to `/usr/share/elasticsearch/data`
Follow the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html) for additional installation details.
Follow the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html) for additional installation details and options.
### Elasticsearch on a custom port
Should you need to change the port for Elasticsearch to for example `9500`, follow these steps:
Should you need to change the port for Elasticsearch to, for example `9500`, follow these steps:
- Set the environment variable `http.port=9500` to the ES container
- Set the environment variable `http.port=9500` for the ES container
- Change the `expose` value for the ES container to match your port number
- For the Tube Archivist container, change the `ES_URL` environment variable, e.g. `ES_URL=http://archivist-es:9500`
- For the TA container, change the `ES_URL` environment variable, i.e. `ES_URL=http://archivist-es:9500`
## Redis
!!! note
!!! danger
You most likely can't use the same Redis instance between other services, especially if they also use Celery as a task scheduler.
Functions as a cache and temporary link between the application and the file system. Used to store and display messages and configuration variables.
Redis functions as a cache and temporary link between the application and the file system. Redis is used to store and display messages and configuration variables.
- Needs to be accessible over the default port `6379`
- Needs a volume at `/data` to make your configuration changes permanent.
- Needs a volume to allow configuration persistence linked to `/data`.
### Redis on a custom port
For some architectures it might be required to run Redis JSON on a nonstandard port. To for example change the Redis port to `6380`, set the following values:
For some environments, it might be required to run Redis on a non-standard port. For example, to change the Redis port to `6380`, set the following values:
- Set the environment variable `REDIS_PORT=6380` to the *tubearchivist* service.
- For the TA container, set the `REDIS_PORT` environment variable, i.e. `REDIS_PORT=6380`
- For the *archivist-redis* service, change the ports to `6380:6380`
- Additionally set the following value to the *archivist-redis* service: `command: --port 6380 --loadmodule /usr/lib/redis/modules/rejson.so`
- Additionally, set the following value to the *archivist-redis* service: `command: --port 6380 --loadmodule /usr/lib/redis/modules/rejson.so`
## Start the application
To start, `cd` into the directory where the `docker-compose.yml` file is located and run `docker compose up --detach` in terminal. The first time you do this it will download the appropriate images, which can take a minute.
From a terminal, `cd` into the directory you saved the `docker-compose.yml` file in and run `docker compose up --detach`. The first time you do this, Docker will download the appropriate images, which can take an additional minute or so.
You can follow the logs with `docker compose logs -f`. Once it's ready it will print something like `celery@1234567890ab ready`. At this point you should be able to go to `http://your-host:8000` and log in with the `TA_USER`/`TA_PASSWORD` credentials.
You can follow the logs with `docker compose logs -f`. Once it's ready, it will print something like `celery@1234567890ab ready`. At this point you should be able to go to `http://your-host:8000` and log in with the `TA_USER`/`TA_PASSWORD` credentials.
You can bring the application down by running `docker compose down` in the same directory.
### Support
If you're still having trouble, join us on [discord](https://www.tubearchivist.com/discord) and come to the [#support channel.](https://discord.com/channels/920056098122248193/1006394050217246772)

6
mkdocs/docs/installation/helm-charts.md
View File

@ -1,4 +1,8 @@
!!! note
These are beginner's guides/installation instructions for additional platforms generously provided by users of these platforms. When in doubt, verify the details with the [project README](https://github.com/tubearchivist/tubearchivist#installing). If you see any issues here while using these instructions, please contribute.
There is a Helm Chart available at [https://github.com/insuusvenerati/helm-charts](https://github.com/insuusvenerati/helm-charts). Mostly self-explanatory but feel free to ask questions in the discord / subreddit.
There is a Helm Chart available at [https://github.com/insuusvenerati/helm-charts](https://github.com/insuusvenerati/helm-charts).
### Support
If you're still having trouble, join us on [discord](https://www.tubearchivist.com/discord) and come to the [#support channel.](https://discord.com/channels/920056098122248193/1006394050217246772)

40
mkdocs/docs/installation/podman.md
View File

@ -1,28 +1,46 @@
!!! note
These are beginner's guides/installation instructions for additional platforms generously provided by users of these platforms. When in doubt, verify the details with the [project README](https://github.com/tubearchivist/tubearchivist#installing). If you see any issues here while using these instructions, please contribute.
Podman handles container hostname resolving slightly differently than docker, so you need to make a few changes to the `docker-compose.yml` to get up and running.
Podman handles container hostname resolution slightly differently than Docker, so you need to make a few changes to the `docker-compose.yml` to get up and running.
### Follow the installation instructions from the [README](https://github.com/tubearchivist/tubearchivist#installing), with a few additional changes to the `docker-compose.yml`.
### Installation Changes from Compose Instructions
Edit these additional changes to the `docker-compose.yml`:
Follow the installation instructions for [Docker Compose](docker-compose.md), with a few additional changes to the `docker-compose.yml`.
Edit the `docker-compose.yml` with these additional changes:
#### Tube Archivist
- under `tubearchivist`->`image`:
- prefix the container name with `docker.io/` (or the url of your repo of choice).
prefix the container name with `docker.io/` (or the url of your container registry of choice).
- under `tubearchivist`->`environment`:
- `ES_URL`: change `archivist-es` to the internal IP of the computer that will be running the containers.
- `REDIS_HOST`: change `archivist-redis` to the internal IP of the computer that will be running the containers (should be the same as above).
`ES_URL`: change `archivist-es` to reflect the internal IP of the computer that will be running the containers.
`REDIS_HOST`: change `archivist-redis` to reflect the internal IP of the computer that will be running the containers (should be the same as above).
#### Redis
- under `archivist-redis`->`image`:
- prefix the container name with `docker.io/` again.
prefix the container name with `docker.io/` again.
- under `archivist-redis`->`expose`:
- change the whole entry from `expose: ["<PORT>"]` into `ports: ["<PORT>:<PORT>"].
change the whole entry from `expose: ["<PORT>"]` into `ports: ["<PORT>:<PORT>"]`.
???+ example
`ports: ["6379:6379"]`
#### Elasticsearch
- under `archivist-es`->`image`:
- prefix the container name with `docker.io/` again.
prefix the container name with `docker.io/` again.
- under `archivist-es`->`expose`:
- change the whole entry from `expose: ["<PORT>"]` into `ports: ["<PORT>:<PORT>"].
change the whole entry from `expose: ["<PORT>"]` into `ports: ["<PORT>:<PORT>"]`.
???+ example
`ports: ["9200:9200"]`
### Create service files (optional)
Since podman doesn't run as a service, it can't start containers after reboots, at least not without some help.
Since Podman doesn't run as a service, it can't start containers after reboots without some additional configuration.
If you want to enable this behavior, you can follow [this example](https://techblog.jeppson.org/2020/04/create-podman-services-with-podman-compose/) to have `systemd` start up the containers with `podman-compose` when the computer boots up.
### Support
If you're still having trouble, join us on [discord](https://www.tubearchivist.com/discord) and come to the [#support channel.](https://discord.com/channels/920056098122248193/1006394050217246772)

226
mkdocs/docs/installation/synology.md
View File

@ -2,28 +2,35 @@
These are beginner's guides/installation instructions for additional platforms generously provided by users of these platforms. When in doubt, verify the details with the [project README](https://github.com/tubearchivist/tubearchivist#installing). If you see any issues here while using these instructions, please contribute.
There are several different methods to install TubeArchivist on Synology platforms. This will focus on the available `docker` package implementation.<!-- and `docker-compose` implementations. -->
There are several different methods to install **Tube Archivist** on Synology platforms. This will focus on the available `docker` package manager implementation for Synology 7.1 and prior.<!-| and `docker-compose` implementations. -->
!!! note
Synology Package Manager for 7.2 and later should be able to follow the instructions below or can create a project using the [docker compose installation instructions.](docker-compose.md)
### Prepare Directories/Folders
Before we setup TubeArchivist, we need to setup the directories/folders. You are assumed to be logged into the Synology NAS.
Before we setup **Tube Archivist**, we need to setup the directories/folders. You are assumed to be logged into the Synology NAS with a properly permissioned user.
#### 1. Docker Base Folder
1. Open the `File Station` utility.
2. Click on the **Create🔽** button and choose *Create New Shared Folder*.
3. **Name** the folder "Docker".
4. Add a **Description**.
5. Select the **Volume Location**.
> Note: By default, this will be where all data is stored. Change the folders as best meets your requirements.
> !!! note
By default, this will be where all persistent data is stored. Change the folders as best meets your requirements.
6. Select the appropriate options from the remaining checkbox configurations.
![Synology - Create Docker Folder](../assets/Synology_0.2.0_Docker-Folder-Create.png)
![Synology | Create Docker Folder](../assets/Synology_0.2.0_Docker-Folder-Create.png)
7. Click the **Next** button.
8. If you are going to **Encrypt** your folder, check the appropriate box and provide the Encryption Key and its confirmation.
9. Click the **Next** button.
10. On the **Advanced Settings** page, you can select the *Enable data checksum for advanced data integrity* setting. This may cause a performance impact, but will allow for potential file self-healing. **This cannot be changed later.**
> Note: This is not recommended, as we will be hosting databases within this folder.
> !!! warning
This is not recommended as we will be hosting databases within this folder.
11. If you are enabling a quota for how large the folder can get, you can select the *Enabled shared folder quota* setting and choose the maximum size this folder can grow. This can be changed later.
12. Click the **Next** button.
13. Confirm the settings, then click the **Apply** button. This will create the folder.
#### 2. TubeArchivist Base Folder
#### 2. Tube Archivist Base Folder
1. Open the `File Station` utility.
2. Select the "Docker" folder on the left-hand side.
3. Click on the `Create🔽` button and choose *create Folder*.
@ -40,45 +47,53 @@ Before we setup TubeArchivist, we need to setup the directories/folders. You are
3. Select the "TubeArchivist" folder beneath "Docker".
4. Click on the `Create🔽` button and choose *create Folder*.
5. **Name** the folder "es".
#### 5. TubeArchivist Cache
#### 5. Tube Archivist Cache
1. Open the `File Station` utility.
2. Select the "Docker" folder on the left-hand side.
3. Select the "TubeArchivist" folder beneath "Docker".
4. Click on the `Create🔽` button and choose *create Folder*.
5. **Name** the folder "cache".
#### 6. TubeArchivist Output
#### 6. Tube Archivist Output
1. Open the `File Station` utility.
2. Select the "Docker" folder on the left-hand side.
3. Select the "TubeArchivist" folder beneath "Docker".
4. Click on the `Create🔽` button and choose *create Folder*.
5. **Name** the folder "media".
#### 7. Confirm Folder Structure
Once all of the folders have been created, it should have a folder structure within Docker\TubeArchivist that includes "cache", "es", "media", and "redis" folders.
![Synology - Docker Folder Structure](../assets/Synology_0.2.0_Docker-Folder-Structure.png)
Once all of the folders have been created, it should have a folder structure within `Docker\TubeArchivist` that includes "`cache`", "`es`", "`media`", and "`redis`" folders.
![Synology | Docker Folder Structure](../assets/Synology_0.2.0_Docker-Folder-Structure.png)
#### 8. Change Permissions - CLI Required
!!! note
#### 8. Change Permissions | CLI Required
!!! failure
If you do not have SSH access enabled for CLI, [enable it](https://kb.synology.com/en-sg/DSM/tutorial/How_to_login_to_DSM_with_root_permission_via_SSH_Telnet) before continuing.
1. Open the SSH connection to the Synology. Login as your primary `Admin` user, or the user that was enabled for SSH access.
1. Open an SSH connection to the Synology. Login as your primary `Admin` user, or the user that was enabled for SSH access.
2. Elevate your access to `root`. Steps are provided [here](https://kb.synology.com/en-sg/DSM/tutorial/How_to_login_to_DSM_with_root_permission_via_SSH_Telnet).
3. Change directories to the **Volume** where the "Docker" folder resides.
</br>Example: `cd /volume1`
!!! example
`cd /volume1`
4. Change directories to the "Docker" folder.
</br>Example: `cd Docker`
!!! example
`cd Docker`
5. Change directories to the "TubeArchivist" folder.
</br>Example: `cd TubeArchivist`
!!! example
`cd TubeArchivist`
6. Change the owner of the "redis" folder. *If correct, this does not have an output.*
</br>Example: `chown 999:100 redis`
!!! example
`chown 999:100 redis`
7. Change the owner of the "es" folder. *If correct, this does not have an output.*
</br>Example: `chown 1000:0 es`
!!! example
`chown 1000:0 es`
8. Confirm that the folders have the correct permissions.
</br>Example: `ls -hl`
![Synology - Docker Folder Permissions Command](../assets/Synology_0.3.6_Docker-Folder-Permissions-Commands.png)
!!! example
`ls -hl`
![Synology | Docker Folder Permissions Command](../assets/Synology_0.3.6_Docker-Folder-Permissions-Commands.png)
9. Logout from root.
</br>Example: `logout`
!!! example
`logout`
10. Disconnect from the SSH connection.
</br>Example: `exit`
!!! example
`exit`
### Synology Docker Setup
@ -91,128 +106,137 @@ Once all of the folders have been created, it should have a folder structure wit
3. Search for `Docker`.
4. Click `Install`.
![Synology - Install Docker Utility](../assets/Synology_0.2.0_Docker-Install.png)
![Synology | Install Docker Utility](../assets/Synology_0.2.0_Docker-Install.png)
#### 2. Install Docker images
#### 2. Download the Docker images
2. After `Docker` is installed, open the `Docker` utility.
3. Go to the `Registry` tab.
4. Search for the following `images` and download them. Follow the recommended versions for each of the images.
- `redis/redis-stack-server`
![Synology - Redis Image Search](../assets/Synology_0.3.6_Docker-Redis-Search.png)
- `bbilly1/tubearchivist-es`
![Synology - ElasticSearch Image Search](../assets/Synology_0.2.0_Docker-ES-Search.png)
- `bbilly1/tubearchivist`
![Synology - TubeArchivist Image Search](../assets/Synology_0.2.0_Docker-TA-Search.png)
> !!! note
"Upgrades in Synology require use of the `latest` tag."
| `redis/redis-stack-server`
![Synology | Redis Image Search](../assets/Synology_0.3.6_Docker-Redis-Search.png)
| `bbilly1/tubearchivist-es`
![Synology | ElasticSearch Image Search](../assets/Synology_0.2.0_Docker-ES-Search.png)
| `bbilly1/tubearchivist`
![Synology | Tube Archivist Image Search](../assets/Synology_0.2.0_Docker-TA-Search.png)
> !!! info
Upgrades in Synology require use of the `latest` tag.
5. Go to the `Image` tab. From here, create a container based on each image with the associated configurations below.
#### 3. Configure ElasticSearch
5. Go to the `Image` tab. From here, create an container based on each image with the associated configurations below.
- ElasticSearch
| ElasticSearch
1. Select the associated image.
2. Click the **Launch** button in the top.
3. Edit the **Container Name** to be "tubearchivist-es".
4. Click on the **Advanced Settings** button.
5. In the **Advanced Settings** tab, check the box for `Enable auto-restart`.
6. In the **Volume** tab, click the **Add Folder** button and select the "Docker/TubeArchivist/es" folder, then type in `/usr/share/elasticsearch/data` for the mount path.
6. In the **Volume** tab, click the **Add Folder** button and select the "`Docker/TubeArchivist/es`" folder, then type in `/usr/share/elasticsearch/data` for the mount path.
7. In the **Network** tab, leave the default `bridge` Network (unless you have a specific Network design that you know how to implement).
8. In the **Port Settings** tab, replace the "Auto" entry under **Local Port** with the port that will be used to connect to ElasticSearch (default is 9200).
9. In the **Port Settings** tab, select the entryline for port 9300 and **➖ delete** the line. It is not needed for this container.
10. The **Links** tab does not require configuration for this container.
11. In the **Environment** tab, add in the following ElasticSearch specific environment variables that may apply.
- `discovery.type=single-node`
- `ES_JAVA_OPTS=-Xms512m -Xmx512m`
- `UID=1000`
- `GID=0`
- `xpack.security.enabled=true`
- `ELASTIC_PASSWORD=verysecret`
- `path.repo=/usr/share/elasticsearch/data/snapshot`
> !!! note "BE AWARE"
- Do not use the default password as it is very insecure.
- Activating snapshots for backups should only be done *after* setting the `path.repo` setting.
![Synology - ElasticSearch Environment Configurations](../assets/Synology_0.2.0_Docker-ES-Env-Conf.png)
| Environment Variable | Setting |
| `discovery.type | single-node` |
| `ES_JAVA_OPTS | -Xms512m -Xmx512m` |
| `UID | 1000` |
| `GID | 0` |
| `xpack.security.enabled | true` |
| `ELASTIC_PASSWORD | verysecret` |
| `path.repo | /usr/share/elasticsearch/data/snapshot` |
> !!! danger "BE AWARE"
| Do not use the default password, as it is very insecure.
| Activating snapshots for backups should only be done *after* setting the `path.repo` setting.
![Synology | ElasticSearch Environment Configurations](../assets/Synology_0.2.0_Docker-ES-Env-Conf.png)
12. Click on the **Apply** button.
13. Back on the **Create Container** screen, click the **Next** button.
14. Review the settings to confirm, then click the **Apply** button.
#### 4. Configure Redis
| Redis
1. Select the associated image.
2. Click the **Launch** button in the top.
3. Edit the **Container Name** to be "tubearchivist-redis".
4. Click on the **Advanced Settings** button.
5. In the **Advanced Settings** tab, check the box for `Enable auto-restart`.
6. In the **Volume** tab, click the **Add Folder** button and select the "`Docker/TubeArchivist/redis`" folder, then type in `/data` for the mount path.
7. In the **Network** tab, leave the default `bridge` Network (unless you have a specific Network design that you know how to implement).
8. In the **Port Settings** tab, replace the "Auto" entry under **Local Port** with the port that will be used to connect to Redis (default is 6379).
9. In the **Links** tab, select the `tubearchivist-es` container from the **Container Name** dropdown and provide it the same alias, "tubearchivist-es".
10. In the **Environment** tab, add in any Redis specific environment variables that may apply (none by default).
11. Click on the **Apply** button.
12. Back on the **Create Container** screen, click the **Next** button.
13. Review the settings to confirm, then click the **Apply** button.
1. Select the associated image.
2. Click the **Launch** button in the top.
3. Edit the **Container Name** to be "tubearchivist-redis".
4. Click on the **Advanced Settings** button.
5. In the **Advanced Settings** tab, check the box for `Enable auto-restart`.
6. In the **Volume** tab, click the **Add Folder** button and select the "Docker/TubeArchivist/redis" folder, then type in `/data` for the mount path.
7. In the **Network** tab, leave the default `bridge` Network (unless you have a specific Network design that you know how to implement).
8. In the **Port Settings** tab, replace the "Auto" entry under **Local Port** with the port that will be used to connect to Redis (default is 6379).
9. In the **Links** tab, select the "tubearchivist-es" container from the **Container Name** dropdown and provide it the same alias, "tubearchivist-es".
10. In the **Environment** tab, add in any Redis specific environment variables that may apply (none by default).
11. Click on the **Apply** button.
12. Back on the **Create Container** screen, click the **Next** button.
13. Review the settings to confirm, then click the **Apply** button.
#### 5. Configure Tube Archivist
| Tube Archivist
1. Select the associated image.
2. Click the **Launch** button in the top.
3. Edit the **Container Name** to be "tubearchivist".
4. Click on the **Advanced Settings** button.
5. In the **Advanced Settings** tab, check the box for `Enable auto-restart`.
6. In the **Volume** tab, click the **Add Folder** button and select the "`Docker/TubeArchivist/cache`" folder, then type in `/cache` for the mount path.
7. In the **Volume** tab, click the **Add Folder** button and select the "`Docker/TubeArchivist/media`" folder, then type in `/youtube` for the mount path.
8. In the **Network** tab, leave the default `bridge` Network (unless you have a specific Network design that you know how to implement).
9. In the **Port Settings** tab, replace the "Auto" entry under **Local Port** with the port that will be used to connect to **Tube Archivist** (default is 8000).
10. In the **Links** tab, select the `tubearchivist-es` container from the **Container Name** dropdown and provide it the same alias, "tubearchivist-es".
11. In the **Links** tab, select the `tubearchivist-redis` container from the **Container Name** dropdown and provide it the same alias, "tubearchivist-redis".
12. In the **Environment** tab, add in the following **Tube Archivist** specific environment variables that may apply. **Change the variables as is appropriate to your use case. Follow the [README section](https://github.com/tubearchivist/tubearchivist#installing) for details on what to set each variable.**
| Environment Variable | Setting |
| `TA_HOST | synology.local` |
| `ES_URL | http://tubearchivist-es:9200` |
| `REDIS_HOST | tubearchivist-redis` |
| `HOST_UID | 1000` |
| `HOST_GID | 0` |
| `TA_USERNAME | tubearchivist` |
| `TA_PASSWORD | verysecret` |
| `ELASTIC_PASSWORD | verysecret` |
| `TZ | America/New_York` |
> !!! danger "BE AWARE"
| Do not use the default password as it is very insecure.
| Ensure that ELASTIC_PASSWORD matches the password used on the `tubearchivist-es` container.
![Synology | Tube Archivist Environment Configurations](../assets/Synology_0.2.0_Docker-TA-Env-Conf.png)
#### 5. Configure TubeArchivist
13. Click on the **Apply** button.
14. Back on the **Create Container** screen, click the **Next** button.
15. Review the settings to confirm, then click the **Apply** button.
1. Select the associated image.
2. Click the **Launch** button in the top.
3. Edit the **Container Name** to be "tubearchivist".
4. Click on the **Advanced Settings** button.
5. In the **Advanced Settings** tab, check the box for `Enable auto-restart`.
6. In the **Volume** tab, click the **Add Folder** button and select the "Docker/TubeArchivist/cache" folder, then type in `/cache` for the mount path.
7. In the **Volume** tab, click the **Add Folder** button and select the "Docker/TubeArchivist/media" folder, then type in `/youtube` for the mount path.
8. In the **Network** tab, leave the default `bridge` Network (unless you have a specific Network design that you know how to implement).
9. In the **Port Settings** tab, replace the "Auto" entry under **Local Port** with the port that will be used to connect to TubeArchivist (default is 8000).
10. In the **Links** tab, select the "tubearchivist-es" container from the **Container Name** dropdown and provide it the same alias, "tubearchivist-es".
11. In the **Links** tab, select the "tubearchivist-redis" container from the **Container Name** dropdown and provide it the same alias, "tubearchivist-redis".
12. In the **Environment** tab, add in the following TubeArchivist specific environment variables that may apply. **Change the variables as is appropriate to your use case. Follow the [README section](https://github.com/tubearchivist/tubearchivist#installing) for details on what to set each variable.**
- `TA_HOST=synology.local`
- `ES_URL=http://tubearchivist-es:9200`
- `REDIS_HOST=tubearchivist-redis`
- `HOST_UID=1000`
- `HOST_GID=0`
- `TA_USERNAME=tubearchivist`
- `TA_PASSWORD=verysecret`
- `ELASTIC_PASSWORD=verysecret`
- `TZ=America/New_York`
> !!! note "BE AWARE"
- Do not use the default password as it is very insecure.
- Ensure that ELASTIC_PASSWORD matches the password used on the tubearchivist-es container.
![Synology - TubeArchivist Environment Configurations](../assets/Synology_0.2.0_Docker-TA-Env-Conf.png)
### 6. Post-Install Monitoring
13. Click on the **Apply** button.
14. Back on the **Create Container** screen, click the **Next** button.
15. Review the settings to confirm, then click the **Apply** button.
6. After the containers have been configured and started, you can go to the **Container** tab and monitor the containers.
7. To review the logs to ensure that the system has started successfully, select the "tubearchivist" container and click on the **Details** button. In the new window, go to the **Log** tab. Monitor the logs until either an error occurs or the message `celery@tubearchivist ready.` is in the logs. This may take a few minutes, especially for a first time setup.
7. To review the logs to ensure that the system has started successfully, select the `tubearchivist` container and click on the **Details** button. In the new window, go to the **Log** tab. Monitor the logs until either an error occurs or the message `celery@tubearchivist ready.` is in the logs. This may take a few minutes, especially for a first time setup.
> !!! note
Synology Docker presents the logs in a pagination format. If you are not seeing the logs update, check if there are additional pages.
8. After it has started, go to the location in the `TA_HOST`. This should give you the standard TubeArchivist login screen.
<!--
Synology Docker presents the logs in a paginated format, showing in historical order (oldest first). If you are not seeing the logs update, check if there are additional pages.
8. After it has started, go to the location provided in the `TA_HOST`. This should give you the standard **Tube Archivist** login screen.
<!-|
### Docker-Compose Setup -->
<!-- This section is a Work In Progress -->
<!-| This section is a Work In Progress -->
**From there, you should be able to start up your containers and you're good to go!**
### Synology Docker Upgrade
When a new version of the image is available, you can follow the following steps to more easily upgrade your previous instance.
!!!note "If you did not use the `latest` tag, you may have some variances in your upgrade steps. Those are detailed below these instructions."
When a new version of the image is available, you can use the following steps to more easily upgrade your previous instance.
!!! failure
If you did not use the `latest` tag, you may have some variances in your upgrade steps. Those are detailed below these instructions.
1. Go to the Registry Tab and download the newest instance of the `:latest` tag, as seen in the Installation Instructions earlier.
2. Go to Image Tab and confirm that you have the newer version available.
3. Stop the running `tubearchivist` container.
4. Click on the **Action🔽** button and choose "Reset".
5. This will load the newer image we downloaded earlier. This should not delete any files if all of your volumes were setup correctly.
6. If it doesn't start automatically, start the `tubearchivist` container. Monitor the upgrade in the logs and confirm that the service starts up successfully.
7. Once you are able to login successfully to the web page for TubeArchivist, you have successfully upgraded your container!
7. Once you are able to login successfully to the web page for **Tube Archivist**, you have successfully upgraded your container!
!!! note "If you did not use the `latest` tag for the `tubearchivist` container, then you will instead do the following:"
!!! success "If you did not use the `latest` tag for the `tubearchivist` container, then you will instead do the following:"
1. Shut down the old container.
2. Download the new image.
3. Follow the Installation instructions again *for just the TubeArchivist image*, using the same configurations as the existing container. It'll have to be named slightly differently.
3. Follow the Installation instructions again *for just the Tube Archivist image*, using the same configurations as the existing container. It'll have to be named slightly differently.
4. After the image is now running and the upgrade of the backend files occurs, shut down the new container. Rename or delete the old container. Rename the new container to have the intended name.
!!! note "Links are incredibly important if you upgrade or change the ES or Redis container images. You will either need to remove the links, create the new containers, then re-add the links or rebuild all of the images with the same instructions as Installation, starting at Step 3."
!!! info
Links are incredibly important if you upgrade or change the ES or Redis container images. You will either need to remove the links, create the new containers, then re-add the links or rebuild all of the images with the same instructions as Installation, starting at Step 3.
If you're still having trouble, join us on [discord](https://www.tubearchivist.com/discord) and come to the #support channel.
### Support
If you're still having trouble, join us on [discord](https://www.tubearchivist.com/discord) and come to the [#support channel.](https://discord.com/channels/920056098122248193/1006394050217246772)

8
mkdocs/docs/installation/truenas-scale.md
View File

@ -1,10 +1,14 @@
!!! note
These are beginner's guides/installation instructions for additional platforms generously provided by users of these platforms. When in doubt, verify the details with the [project README](https://github.com/tubearchivist/tubearchivist#installing). If you see any issues here while using these instructions, please contribute.
Truenas Scale can be a bit confusing, with its k3s kubernetes implementation.
Truenas Scale can be a bit confusing with its k3s Kubernetes implementation.
However, there is a step by step guide available for it's users here:
There is a step-by-step guide available for its users here:
[heavysetup.info](https://heavysetup.info/applications/tube-archivist/dataset/)
- Ensure you are navigating the columns under `Tube Archivist` on the left hand side of the screen
### Support
If you're still having trouble, join us on [discord](https://www.tubearchivist.com/discord) and come to the [#support channel.](https://discord.com/channels/920056098122248193/1006394050217246772)

55
mkdocs/docs/installation/unraid.md
View File

@ -1,30 +1,34 @@
!!! note
!!! abstract
These are beginner's guides/installation instructions for additional platforms generously provided by users of these platforms. When in doubt, verify the details with the [project README](https://github.com/tubearchivist/tubearchivist#installing). If you see any issues here while using these instructions, please contribute.
Tube Archivist, and all if it's dependencies are located in the [community applications](https://unraid.net/community/apps?q=tubearchivist) store. The three containers you will need are as follows:
**Tube Archivist**, and all if it's dependencies, are located in the [community applications](https://unraid.net/community/apps?q=tubearchivist) store. The three containers you will need are as follows:
- **TubeArchivist-RedisJSON**: This container acts as a cache and temporary link between the application and the file system. Used to store and display messages and configuration variables.
- **TubeArchivist-ES**: ElasticSearch stores video metadata and makes everything searchable. Also keeps track of the download queue.
- **TubeArchivist**: Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface.
- **TubeArchivist-RedisJSON**: Redis functions as a cache and temporary link between the application and the file system. Redis is used to store and display messages and configuration variables.
- **TubeArchivist-ES**: Elasticsearch stores the video metadata and enables searchable functionality for all fields. Elasticsearch is also used to keep track of the download queue.
- **TubeArchivist**: Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where **Tube Archivist** comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle, and offline, through a convenient web interface.
## Installing Tube Archivist Applications
### Install `TubeArchivist-RedisJSON`
![TubeArchivist-RedisJSON](../assets/unraid_redis_install.png)
This is the easiest container to setup of the thee, just make sure that you do not have any port conflicts, and that your `/data` is mounted to the correct path. The other containers will map to the same root directory (/mnt/user/appdata/TubeArchivist).
This is the easiest container to setup of the three, just make sure that you do not have any port conflicts and `/data` is mounted to the correct path. The other containers will map to the same root directory (/mnt/user/appdata/TubeArchivist).
If you need to install `TubeArchivist-RedisJSON`on a different port, you'll have to follow [these steps](docker-compose.md#redis-on-a-custom-port) later on when installing the `TubeArchivist` container.
If you're running into port collision, recreate the steps outline in the [docker compose installation instructions.](https://docs.tubearchivist.com/installation/docker-compose/#redis-on-a-custom-port)
If you need to install `TubeArchivist-RedisJSON` on a different port, you'll have to follow [these steps](docker-compose.md#redis-on-a-custom-port) later on when installing the `TubeArchivist` container.
If you're running into port collision errors, recreate the steps outline in the [docker compose installation instructions.](docker-compose.md/#redis-on-a-custom-port)
Make sure and start Redis and the ElasticSearch containers approximately one minute before starting `TubeArchivist`
!!! tip
Make sure and start the Redis and ElasticSearch containers approximately one minute before starting `TubeArchivist`
### Install `TubeArchivist-ES`
![TubeArchivist-ES](../assets/unraid_es_install.png)
ElasticSeach is also pretty easy to setup. Again, make sure you have no port conflicts, make sure that you mapped the ElasticSearch Data to the same root directory as `RedisJSON` (/mnt/user/appdata/TubeArchivist), and make sure to change the default password to something more secure.
ElasticSeach is also pretty easy to setup. Again, make sure you have no port conflicts, that you mapped the ElasticSearch Data directory to the same root directory as `RedisJSON` (/mnt/user/appdata/TubeArchivist), and make sure to change the default password to something more secure.
There is four additional settings in the "show more settings" area, but don't require any changes.
There are four additional settings in the "show more settings" area, but they don't usually require any changes.
Make sure and start Redis and the ElasticSearch containers approximately one minute before starting `TubeArchivist`
!!! tip
Make sure and start Redis and the ElasticSearch containers approximately one minute before starting `TubeArchivist`
### Install `TubeArchivist`
@ -32,10 +36,10 @@ Make sure and start Redis and the ElasticSearch containers approximately one min
It's finally time to set up TubeArchivist!
- `HOST:`This is a list of IP addresses that you will host TA from. Example, 192.168.0.14 is the IP address of my Unraid server. If I was going to access TA from a VPN, or domain name, I'd put those next to my host IP with just a space separating the different addresses. More information [here.](https://github.com/tubearchivist/tubearchivist#installing-and-updating)
- `HOST:` This is a list of IP addresses that you will host TA from. Example, 192.168.0.14 is the IP address of my Unraid server. If I was going to access TA from a VPN, or domain name, I'd put those next to my host IP with a space separating the different addresses. More information [here.](https://github.com/tubearchivist/tubearchivist#installing-and-updating)
- `Port:`Again, make sure that you have no port conflicts on 8000.
- `Port:` Again, make sure that you have no port conflicts on 8000.
- `Youtube Media Path:` is where you'll download all of your videos to.
Make sure that this is an empty directory to not cause confusion when
@ -43,27 +47,32 @@ It's finally time to set up TubeArchivist!
to import into Tube Archivist, please checkout the [settings
wiki.](https://github.com/tubearchivist/tubearchivist/wiki/Settings#manual-media-files-import)
- `Appdata:` This should be the same base path as the other two containers (/mnt/user/appdata/TubeArchivist).
- `TA Username:`This will be your username for TubeArchivist.
- `TA Username:` This will be your username for TubeArchivist.
- `TA Password:`This will be your password for TubeArchivist.
- `TA Password:` This will be your password for TubeArchivist.
- `Redis` This will be JUST the ip address of your redis container.
- If you want to host Redis on a differnt port than the default 6379, simply add a new a new variable as show below.
![TubeArchivist](../assets/unraid_redis_port.png)
- `Redis:` This will **ONLY** be the ip address of your Redis container.
- If you want to host Redis on a differnt port than the default 6379, simply add a new variable as show below.
![TubeArchivist](../assets/unraid_redis_port.png)
- `ElasticSearch Password:`This is the password you defined in the `TubeArchivist-ES` container.
- `ElasticSearch:` This seems to cause some confusion, but it's a pretty simple step, just replace the IP and Port to match you `TubeArchivist-ES` container.
- `ElasticSearch Password:` This is the password you defined in the `TubeArchivist-ES` container.
- `ElasticSearch:` This seems to cause some confusion, but it's a pretty simple step, just replace the IP and Port to match your `TubeArchivist-ES` container.
(example: if your IP is 192.168.0.14, the value should be http://192.168.0.14:9200)
- `Time Zone:` This is an important step for your scheduler, to find your timezone, use a site like [TimeZoneConverter](http://www.timezoneconverter.com/cgi-bin/findzone.tzc)
- `Time Zone:` This is an important step for your scheduler. To find your timezone, use a site like [TimeZoneConverter](http://www.timezoneconverter.com/cgi-bin/findzone.tzc)
**From there, you should be able to start up your containers and you're good to go!**
## Troubleshooting
### Permissions Errors
If you run into permission errors, try ```'newperms /mnt/user/appdata/TubeArchivist/'``` to reset the permissions to the root of your TubeArchivist appdata folder.
### Support
If you're still having trouble, join us on [discord](https://www.tubearchivist.com/discord) and come to the [#support channel.](https://discord.com/channels/920056098122248193/1006394050217246772)

24
mkdocs/docs/playlists.md
View File

@ -3,30 +3,30 @@ description: Subscribe to playlists, browse your playlists and access additional
---
# Playlist Pages
The playlists are organized in two different levels, similar as the [channels](channels.md):
Playlists are organized in two different levels, similar to [Channels](channels.md):
## Playlist Overview
Accessible at `/playlist/` of your Tube Archivist, this **Overview Page** shows a list of all playlists you have indexed over all your channels.
Accessible at `/playlist/` of your **Tube Archivist** instance, this **Overview Page** shows a list of all playlists you have indexed over all your channels.
- You can filter that list to show only subscribed to playlists with the toggle.
- You can filter that list to show only subscribed playlists with the toggle.
You can index playlists of a channel from the channel detail page as described [here](channels.md#channel-detail).
You can index playlists of a channel from the [channel detail page](channels.md#about).
The **Subscribe to Playlist** button <img src="/assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to subscribe to playlists. You have a few options:
- Enter a playlist [playlist](urls.md#playlist)
- Enter a [playlist](urls.md#playlist)
- Add one per line.
!!! note
It doesn't make sense to subscribe to a playlist if you are already subscribed the corresponding channel as this will slow down the **Rescan Subscriptions** [task](downloads.md#rescan-subscriptions).
It doesn't make sense to subscribe to a playlist if you are already subscribed to the corresponding channel as this will slow down the **Rescan Subscriptions** [task](downloads.md#rescan-subscriptions).
You can search your indexed playlists by clicking on the search icon <img src="/assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;">. This will open a dedicated page.
## Playlist Detail
Each playlist will get a dedicated playlist detail page accessible at `/playlist/<playlist-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this playlist.
Each playlist will get a dedicated playlist detail page accessible at `/playlist/<playlist-id>/` of your **Tube Archivist** instance. This page shows all the videos you have downloaded from this playlist.
- If you are subscribed to the playlist, an Unsubscribe button will show, else the Subscribe button will show.
- The **Mark as Watched** and **Mark as Unwatched** buttos will mark all videos of this playlist as watched/unwatched.
- The button **Reindex** will reindex the playlist metadata.
- The button **Reindex Videos** will reindex all videos from this playlist.
- The **Delete Playlist** button will give you the option to delete just the *metadata* which won't delete any media files or *delete all* which will delete metadata plus all videos belonging to this playlist.
- If you are subscribed to the playlist, an Unsubscribe button will show, otherwise the Subscribe button will show.
- The **Mark as Watched** and **Mark as Unwatched** buttons will mark all videos of this playlist as watched/unwatched.
- The **Reindex** button will reindex the playlist metadata.
- The **Reindex Videos** button will reindex all videos from this playlist.
- The **Delete Playlist** button will give you the option to delete just the *metadata*, which won't delete any media files, or *delete all*, which will delete the metadata plus all videos belonging to this playlist.

24
mkdocs/docs/search.md
View File

@ -1,18 +1,18 @@
---
description: Unified search pagage to query your index.
description: Unified search page to query your index.
---
# Search Page
Accessible at `/search/` of your **Tube Archivist**, search your archive for Videos, Channels and Playlists - or even full text search throughout your indexed subtitles.
Accessible at `/search/` of your **Tube Archivist** instance, search your archive for Videos, Channels and Playlists - or even full text search throughout your indexed subtitles.
Just start typing to start a **simple** search *or* **start your query with a primary keyword** to search for a specific type and narrow down the result with secondary keywords. Secondary keywords can be in any order. Use *yes* or *no* for boolean values.
- This will return 30 results per query, pagination is not implemented yet.
- All your queries are case insensitive and are normalized to lowercase.
- All your queries are analyzed for the english language, this means *singular*, *plural* and word variations like *-ing*, *-ed*, *-able* etc are treated as synonyms.
- All your queries are analyzed for the english language, this means *singular*, *plural* and word variations like *-ing*, *-ed*, *-able*, etc., are treated as synonyms.
- Keyword value parsing begins with the `keyword:` name all the way until the end of query or the next keyword, e.g. in `video:learn python channel:corey`, the keyword `video` has value `learn python`.
Fuzzy search is activated for all your searches by default. This can catch typos in your queries or in the matching documents with one to two letters difference, depending on the query length. You can configure fuzziness with the secondary keyword `fuzzy:`, e.g:
Fuzzy search is activated for all your searches by default. This can catch typos in your queries or in the matching documents with a one to two letters difference, depending on the query length. You can configure fuzziness with the secondary keyword `fuzzy:`, e.g:
- `fuzzy:0` or `fuzzy:no`: Deactivate fuzzy matching.
- `fuzzy:1`: Set fuzziness to one letter difference.
@ -20,10 +20,10 @@ Fuzzy search is activated for all your searches by default. This can catch typos
- All text searches are ranked, meaning the better a match the higher ranked the result. Unless otherwise stated, queries with multiple words are processed with the `and` operator, meaning all words need to match so each word will narrow down the result.
## Simple
Start your query without a keyword to make a simple query (primary keyword `simple:` is implied). This will search in *video titles*, *channel names* and *playlist titles* and will return matching videos, channels and playlists. Keyword searches will return more results in a particular category due to the fact that more fields are searched for matches. Simple queries do not have any secondary keywords.
Start your query without any keywords to make a simple query (primary keyword `simple:` is implied). This will search in *video titles*, *channel names* and *playlist titles* and will return matching videos, channels and playlists. Keyword searches will return more results in a particular category due to the fact that more fields are searched for matches. Simple queries do not have any secondary keywords.
## Video
Start your query with the **primary keyword** `video:` to search for videos only. This will search through the *video titles*, *tags* and *category* fields. Narrow your search down with secondary keywords:
Start your query with the **primary keyword** `video:` to search through only videos. This will search through the *video titles*, *tags* and *category* fields. Narrow down your search with secondary keywords:
- `channel:` search for videos matching the channel name.
- `active:` is a boolean value, to search for videos that are still active on youtube or that are not active any more.
@ -36,7 +36,7 @@ Start your query with the **primary keyword** `video:` to search for videos only
- Note the omitted term after the primary key, this will show all videos from the channel *Tom Scott* that are no longer active on YouTube.
## Channel
Start with the `channel:` **primary keyword** to search for channels matching your query. This will search through the *channel name*, *channel description* and *channel tags* fields. Narrow your search down with secondary keywords:
Start your query with the **primary keyword** `channel:` to search through only channels. This will search through the *channel name*, *channel description* and *channel tags* fields. Narrow down your search with secondary keywords:
- `subscribed:` is a boolean value, search for channels that you are subscribed to or not.
- `active:` is a boolean value, to search for channels that are still active on YouTube or that are no longer active.
@ -47,7 +47,7 @@ Start with the `channel:` **primary keyword** to search for channels matching yo
- `channel: active:no`: Note the omitted term after the primary key, this will return all channels that are no longer active on YouTube.
## Playlist
Start your query with the **primary keyword** `playlist:` to search for playlists only. This will search through the *playlist title* and *playlist description* fields. Narrow down your search with these secondary keywords:
Start your query with the **primary keyword** `playlist:` to search through only playlists. This will search through the *playlist title* and *playlist description* fields. Narrow down your search with secondary keywords:
- `subscribed`: is a boolean value, search for playlists that you are subscribed to or not.
- `active:` is a boolean value, to search for playlists that are still active on YouTube or that are no longer active.
@ -59,9 +59,9 @@ Start your query with the **primary keyword** `playlist:` to search for playlist
- `playlist:html css active:yes`: Search for playlists containing *HTML CSS* that are still active on YouTube.
## Full
Start a full text search by beginning your query with the **primary keyword** `full:`. This will search through your indexed Subtitles showing segments with possible matches. This will only show any results if you have activated *subtitle download and index* on the settings page. The operator for full text searches is `or` meaning when searching for multiple words not all words need to match, but additional words will change the ranking of the result, the more words match and the better they match, the higher ranked the result. The matching words will get highlighted in the text preview and you will see a score indicating how well your term is matching.
Start your query with the **primary keyword** `full:` to perform full text search. This will search through your indexed subtitles showing segments with possible matches. This will only show any results if you have activated *subtitle download and index* on the settings page. The operator for full text searches is `or`, meaning searching for multiple words does not require all words to match, but additional matching words will change the ranking of the result. The more words matching and the better they match, the higher ranked the result. The matching words will get highlighted in the text preview and you will see a score indicating how well your query terms are matched.
Clicking the play button on the thumbnail will open the inplace player at the timestamp from where the segment starts. Same when clicking the video title, this will open the video page and put the player at the segment timestamp. This will overwrite any previous playback position.
Clicking the play button on the thumbnail will open the in-place player at the timestamp from where the segment starts. Same when clicking the video title, this will open the video page and put the player at the segment timestamp. This will overwrite any previous playback position.
Narrow down your search with these secondary keywords:
@ -70,5 +70,5 @@ Narrow down your search with these secondary keywords:
**Example**:
- `full:contribute to open source lang:en` search for subtitle segments matching with the words *Contribute to Open Source* in the language *en*.
- `full:flight simulator cockpit source:user` to search for the words *Flight Simulator Cockpit* from *user* uploaded subtitle segments.
- `full:contribute to open source lang:en` search for subtitle segments matching with the words *Contribute*, *to*, *Open* or *Source* in the language *en*.
- `full:flight simulator cockpit source:user` to search for the words *Flight*, *Simulator* or *Cockpit* from *user* uploaded subtitle segments.

52
mkdocs/docs/settings/actions.md
View File

@ -3,21 +3,21 @@ description: Administration tasks for the application.
---
# Actions Page
Accessible at `/settings/actions/` of your **Tube Archivist**, this page allows admins to perform actions related to the database and other functions.
Accessible at `/settings/actions/` of your **Tube Archivist** instance, this page allows admins to perform actions related to the database and other functions.
## Delete download queue
The button **Delete all queued** will delete all pending videos from the download queue. The button **Delete all ignored** will delete all videos you have previously ignored.
## Manual Media Files Import
!!! note
!!! warning
This is inherently error prone, as there are many variables, some outside of the control of this project. Read this carefully and use at your own risk.
!!! note
The importer will *try* to fetch metadata from YouTube after identifying the video, as it is assumed this is newer, even if you add the info json file as described below.
Add the files you'd like to import to the */cache/import* folder. Only add files, don't add subdirectories. All files you are adding, need to have the same *base name* as the media file. Then start the process from the settings page *Manual Media Files Import*.
Add the files you'd like to import to the `/cache/import` folder. Only add files, don't add subdirectories. All files you are adding need to have the same *base name* as the media file. Then, start the process from the settings page with the *Manual Media Files Import* button.
Valid media extensions are *.mp4*, *.mkv* or *.webm*. If you have other file extensions or incompatible codecs, convert them first to mp4. **Tube Archivist** can identify the videos with one of the following methods.
Valid media extensions are *.mp4*, *.mkv* or *.webm*. If you have other file extensions or incompatible codecs, convert them first to mp4. **Tube Archivist** can identify the videos with one of the following methods:
### Method 1:
Add a matching *.info.json* file with the media file. Both files need to have the same base name, for example:
@ -28,9 +28,11 @@ Add a matching *.info.json* file with the media file. Both files need to have th
The import process then looks for the 'id' key within the JSON file to identify the video.
Sometimes you may need to create this file manually, The following are the absolute minimum required tags for manual importing.
Sometimes you may need to create this file manually. The following are the absolute minimum required tags as keys for manual importing.
!!! note
`thumbnail` can be left blank or null, however it is required to be present. If blank the thumbnail will be extracted from the video file on import.
`thumbnail` can be left blank or null, however it is required to be present. If blank the thumbnail will be extracted from the video file on import.
```json
{
"id": "",
@ -40,6 +42,7 @@ Sometimes you may need to create this file manually, The following are the absol
"thumbnail": null
}
```
However, you may fill out additional tags if they are known for a more complete result.
```json
{
@ -54,11 +57,14 @@ However, you may fill out additional tags if they are known for a more complete
"view_count": null
}
```
### Method 2:
Detect the YouTube ID from filename, this accepts the default yt-dlp naming convention for file names like:
Detect the YouTube ID from filename. This accepts the default yt-dlp naming convention for file names like:
- `<base-name>[<youtube-id>].mp4`
- The YouTube ID in square brackets at the end of the filename is the crucial part.
!!! success "Required Naming Convention"
The YouTube ID in square brackets at the end of the filename is the crucial part.
### Offline import:
If the video you are trying to import is not available on YouTube any more, **Tube Archivist** can import the required metadata:
@ -70,39 +76,41 @@ If the video you are trying to import is not available on YouTube any more, **Tu
### Some notes:
- This will **consume** the files you put into the import folder: Files will get converted to mp4 if needed (this might take a long time...) and moved to the archive, *.json* files will get deleted upon completion to avoid having duplicates on the next run.
- This will **consume** the files you put into the import folder: Files will get converted to *.mp4* if needed (this might take a long time...) and moved to the archive, *.json* files will get deleted upon completion to avoid having duplicates on the next run.
- For best file transcoding quality, convert your media files with desired settings first before importing.
- Maybe start with a subset of your files to import to make sure everything goes well...
- A notification box will show with progress, follow the docker logs to monitor for errors.
!!! tip
Starting with a small subset of the files to import to test and confirm that your settings, configurations, and files will work is recommended.
## Embed thumbnails into media file
This will write or overwrite all thumbnails in the media file using the downloaded thumbnail. This is only necessary if you didn't download the files with the option *Embed Thumbnail* enabled or you want to make sure all media files get the newest thumbnail.
## ZIP file index backup
This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files. For data consistency, make sure there aren't any other tasks running that will change the index during the backup process. This is very slow, particularly for large archives.
This will backup your metadata into a zip file. The file will get stored at `/cache/backup` and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files. For data consistency, make sure there aren't any other tasks running that will change the index during the backup process. This is very slow, particularly for large archives.
!!! note "BE AWARE"
This will **not** backup any media files, just the metadata from the Elasticsearch.
!!! danger "BE AWARE"
This will **not** backup any media files. This is only for the metadata from the Elasticsearch database.
## Restore From Backup
The restore functionality will expect the same zip file in *cache/backup* as created from the **Backup database** function. This will recreate the index from the zip archive file. There will be a list of all available backup to choose from. The *source* tag can have these different values:
The restore functionality will expect the same zip file in `/cache/backup` as created from the **Backup database** function. This will recreate the index from the zip archive file. There will be a list of all available backup to choose from. The *source* tag can have these different values:
- **manual**: For backups manually created from here on the settings page.
- **auto**: For backups automatically created via a sceduled task.
- **update**: For backups created after a Tube Archivist update due to changes in the index.
- **update**: For backups created after a **Tube Archivist** update due to changes in the index.
- **False**: Undefined.
!!! note "BE AWARE"
!!! danger "BE AWARE"
This will **replace** your current index with the one from the backup file. This won't restore any media files.
## Rescan Filesystem
This function will go through all your media files and looks at the whole index to try to find any issues:
This action will go through all your media files and looks at the whole index to try to find any issues:
- Should the filename not match with the indexed media url, this will rename the video files correctly and update the index with the new link.
- When you delete media files from the filesystem outside of the Tube Archivist interface, this will delete leftover metadata from the index.
- When you have media files that are not indexed yet, this will grab the metadata from YouTube like it was a newly downloaded video. This can be useful when restoring from an older backup file with missing metadata but already downloaded mediafiles. NOTE: This only works if the media files are named in the same convention as Tube Archivist does, particularly the YouTube ID needs to be at the same index in the filename, alternatively see above for *Manual Media Files Import*.
- The task will stop, when adding a video fails, for example if the video is no longer available on YouTube.
- When you delete media files from the filesystem outside of the **Tube Archivist** interface, this will delete leftover metadata from the index.
- When you have media files that are not indexed yet, this will grab the metadata from YouTube as if it was a newly downloaded video. This can be useful when restoring from an older backup file with missing metadata but already downloaded mediafiles. NOTE: This only works if the media files are named in the same convention as **Tube Archivist** expects, alternatively see above for *Manual Media Files Import*.
- The task will stop when adding a video fails, for example if the video is no longer available on YouTube.
- This will also check all of your thumbnails and download any that are missing.
!!! note "BE AWARE"
There is no undo.
!!! danger "BE AWARE"
There is no undo. Deleted references and metadata are removed and cannot be brought back without a restore operation.

58
mkdocs/docs/settings/application.md
View File

@ -3,17 +3,18 @@ description: Configure this application.
---
# Application Settings Page
Accessible at `/settings/application/` of your **Tube Archivist**, this page holds all of the general application configuration (minus configuration of the [scheduler](scheduling.md)).
Accessible at `/settings/application/` of your **Tube Archivist** instance, this page holds all of the general application configuration settings (minus configuration of the [scheduler](scheduling.md)).
Click on **Update Application Configurations** at the bottom of the page to apply your configurations.
!!! tip "Saving Configurations"
Click on **Update Application Configurations** at the bottom of the page to apply your configurations.
## Subscriptions
Settings related to the channel management. Disable shorts or streams by setting their page size to 0 (zero).
- **Channel Page Size**: Defines how many pages will get analyzed by **Tube Archivist** each time you click on *Rescan Subscriptions*. The default page size used by yt-dlp is **50**, that's also the recommended value to set here. Any value higher will slow down the rescan process, for example if you set the value to 51, that means yt-dlp will have to go through 2 pages of results instead of 1 and by that doubling the time that process takes.
- **Live Page Size**: Same as above, but for channel live streams.
- **Shorts page Size**: Same as above, but for shorts videos.
- **Auto Start**: This will prioritize and automatically start downloading videos from your subscriptions over regular video added to the download queue.
- **Channel Page Size**: Defines how many pages will get analyzed by **Tube Archivist** each time you click on *Rescan Subscriptions*. The default page size used by yt-dlp is **50**, which is also the recommended value to set here. Any value higher will slow down the rescan process, for example if you set the value to 51, that means yt-dlp will have to go through 2 pages of results instead of 1 and by that doubling the time that process takes. *Cannot be set to 0.*
- **Live Page Size**: Same as above, but for a channel's live streams. *Disable by setting to 0.*
- **Shorts page Size**: Same as above, but for a channel's shorts videos. *Disable by setting to 0.*
- **Auto Start**: This will prioritize and automatically start downloading videos from your subscriptions over regular videos added to the download queue.
## Downloads
Settings related to the download process.
@ -26,65 +27,68 @@ Settings related to the download process.
## Download Format
Additional settings passed to yt-dlp.
- **Format**: This controls which streams get downloaded and is equivalent to passing `--format` to yt-dlp. Use one of the recommended one or look at the documentation of [yt-dlp](https://github.com/yt-dlp/yt-dlp#format-selection). Please note: The option `--merge-output-format mp4` is automatically passed to yt-dlp to guarantee browser compatibility. Similar to that, `--check-formats` is passed as well to check that the selected formats are actually downloadable.
- **Format Sort**: This allows you to change how yt-dlp sorts formats by passing `--format-sort` to yt-dlp. Refere to the [documentation](https://github.com/yt-dlp/yt-dlp#sorting-formats), what you can pass here. Be aware, that some codecs might not be compatible with your browser of choice.
- **Extractor Language**: Some channels provide tranlated video titles and descriptions. Add the two letter ISO language code, to set your prefered default language. This will only have an effect, if the uploader adds translations. Not all language codes are supported, see the [documentation](https://github.com/yt-dlp/yt-dlp#youtube) (the `lang` section) for more details.
- **Format**: This controls which streams get downloaded and is equivalent to passing `--format` to yt-dlp. Use one of the recommended configurations or review the documentation for [yt-dlp](https://github.com/yt-dlp/yt-dlp#format-selection). Please note: The option `--merge-output-format mp4` is automatically passed to yt-dlp to guarantee browser compatibility. Similar to that, `--check-formats` is passed as well to check that the selected formats are actually downloadable.
- **Format Sort**: This allows you to change how yt-dlp sorts formats by passing `--format-sort` to yt-dlp. Refer to the [documentation](https://github.com/yt-dlp/yt-dlp#sorting-formats) to see what you can pass here. Be aware that some codecs might not be compatible with your browser of choice.
- **Extractor Language**: Some channels provide translated video titles and descriptions. Add the two letter ISO language code to set your prefered default language. This will only have an effect if the uploader adds translations. Not all language codes are supported, see the [documentation](https://github.com/yt-dlp/yt-dlp#youtube) (the `lang` section) for more details.
- **Embed Metadata**: This saves the available tags directly into the media file by passing `--embed-metadata` to yt-dlp.
- **Embed Thumbnail**: This will save the thumbnail into the media file by passing `--embed-thumbnail` to yt-dlp.
- **Embed Thumbnail**: This saves the thumbnail into the media file by passing `--embed-thumbnail` to yt-dlp.
## Subtitles
- **Download Setting**: Select the subtitle language you like to download. Add a comma separated list for multiple languages. For Chinese you must specify `zh-Hans` or `zh-Hant`, specifying "zh" is invalid, otherwise the subtitle won't download successfully.
- **Source Settings**: User created subtitles are provided from the uploader and are usually the video script. Auto generated is from YouTube, quality varies, particularly for auto translated tracks.
- **Source Settings**: User created subtitles are provided from the uploader and are usually the video script. Auto generated is from YouTube. The quality varies, particularly for auto translated tracks.
- **Index Settings**: Enabling subtitle indexing will add the lines to Elasticsearch and will make subtitles searchable. This will increase the index size and is not recommended on low-end hardware.
## Comments
- **Download and index comments**: Set your configuration for downloading and indexing comments. This takes the same values as documented in the `max_comments` section for the youtube extractor of [yt-dlp](https://github.com/yt-dlp/yt-dlp#youtube). Add without space between the four different fields: *max-comments,max-parents,max-replies,max-replies-per-thread*. Example:
- **Download and index comments**: Set your configuration for downloading and indexing comments. This takes the same values as documented in the `max_comments` section for the youtube extractor of [yt-dlp](https://github.com/yt-dlp/yt-dlp#youtube). Add, without spaces, between the four different fields: *max-comments,max-parents,max-replies,max-replies-per-thread*. Example:
- `all,100,all,30`: Get 100 max-parents and 30 max-replies-per-thread.
- `1000,all,all,50`: Get a total of 1000 comments over all, 50 replies per thread.
- **Comment sort method**: Change sort method between *top* or *new*. The default is *top*, as decided by YouTube.
- The [Refresh Metadata](scheduling.md#refresh-metadata) background task will get comments from your already archived videos, spreading the requests out over time.
Archiving comments is slow as only very few comments get returned per request with yt-dlp. Choose your configuration above wisely. Tube Archivist will download comments after the download queue finishes, your videos will be already available while the comments are getting downloaded.
Archiving comments is slow as only a few comments get returned per request with yt-dlp. Choose your configuration above wisely. Tube Archivist will download comments after the download queue finishes. Your videos will already be available while the comments are getting downloaded.
## Cookie
!!! note "Cookie Expire"
!!! warning "Cookie Expiry"
Using cookies can have unintended consequences. Multiple users have reported that their account got flagged and cookies will expire within a few hours. It appears that YT has some detection mechanism that will invalidate your cookie if it's being used outside of a browser. That is happening server side on YT. If you are affected, you might be better off to not use this functionality.
Importing your YouTube Cookie into Tube Archivist allows yt-dlp to bypass age restrictions, gives access to private videos and your *watch later* or *liked videos*.
Importing your YouTube Cookie into **Tube Archivist** allows yt-dlp to bypass age restrictions, gives access to private videos and your *Watch Later* or *Liked Videos* playlists.
### Security concerns
Cookies are used to store your session and contain your access token to your google account, this information can be used to take over your account. Treat that data with utmost care as you would any other password or credential. *Tube Archivist* stores your cookie in Redis and will automatically append it to yt-dlp for every request.
Cookies are used to store your session and contain your access token to your Google account. **This information can be used to take over your account.** Treat that data with utmost care, as you would any other password or credential. **Tube Archivist** stores your cookie in Redis and will automatically append it to yt-dlp for every request.
### Auto import
Easiest way to import your cookie is to use the **Tube Archivist Companion** [browser extension](https://github.com/tubearchivist/browser-extension) for Firefox and Chrome.
### Manual import
Alternatively you can also manually import your cookie into Tube Archivist. Export your cookie as a *Netscape* formatted text file, name it *cookies.google.txt* and put it into the *cache/import* folder. After that you can enable the option on the settings page and your cookie file will get imported.
Alternatively, you can also manually import your cookie into Tube Archivist. Export your cookie as a *Netscape* formatted text file, name it *cookies.google.txt* and put it into the *cache/import* folder. After that you can enable the option on the settings page and your cookie file will get imported.
- There are various tools out there that allow you to export cookies from your browser. This project doesn't make any specific recommendations.
- Once imported, a **Validate Cookie File** button will show, where you can confirm if your cookie is working or not.
### Use your cookie
Once imported, additionally to the advantages above, your [Watch Later](https://www.youtube.com/playlist?list=WL) and [Liked Videos](https://www.youtube.com/playlist?list=LL) become a regular playlist you can download and subscribe to as any other [playlist](../playlists.md).
Once imported, in addition to the advantages above, your [Watch Later](https://www.youtube.com/playlist?list=WL) and [Liked Videos](https://www.youtube.com/playlist?list=LL) playlists become a regularly accessible playlist that you can download and subscribe to like any other [playlist](../playlists.md).
### Limitation
There is only one cookie per Tube Archivist instance, this will be shared between all users.
There is only one cookie per Tube Archivist instance. This will be shared between all users.
## Integrations
All third party integrations of TubeArchivist will **always** be *opt in*.
All third party integrations of **Tube Archivist** will **always** be *opt in*.
- **API**: Your access token for the Tube Archivist API.
- **returnyoutubedislike.com**: This will get return dislikes and average ratings for each video by integrating with the API from [returnyoutubedislike.com](https://www.returnyoutubedislike.com/).
- **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. If a video doesn't have timestamps, or has unlocked timestamps, use the browser addon to contribute to this excellent project. Can also be activated and deactivated as a per [channel overwrite](../channels.md#about).
- **API**: Your access token for the **Tube Archivist** API.
- **returnyoutubedislike.com**: This will return dislikes and average ratings for each video by integrating with the API from [returnyoutubedislike.com](https://www.returnyoutubedislike.com/).
- **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to retrieve timestamps for, and skip, sponsored content. If a video doesn't have timestamps, or has unlocked timestamps, use the browser addon to contribute to this excellent project. Can also be activated and deactivated on a per [channel overwrite](../channels.md#about).
## Snapshots
!!! note
This will make a snapshot of your metadata index only, no media files or additional configuration variables you have set on the settings page will be backed up.
!!! info
This will make a snapshot of your metadata index only. No media files or additional configuration variables you have set on the settings page will be backed up.
System snapshots will automatically make daily snapshots of the Elasticsearch index. The task will start at 12pm your local time. Snapshots are deduplicated, meaning that each snapshot will only have to backup changes since the last snapshot. Old snpshots will automatically get deleted after 30 days.
- **Create snapshot now**: Will start the snapshot process now, outside of the regular daily schedule.
- **Restore**: Restore your index to that point in time.
- **Create snapshot now**: Will start the snapshot process now and outside of the regular daily schedule.
- **Restore**: Restore your index to that point in time. Select one of the available snapshots to restore from.
!!! tip "Saving Configurations"
Click on **Update Application Configurations** at the bottom of the page to apply your configurations.

2
mkdocs/docs/settings/dashboard.md
View File

@ -3,4 +3,4 @@ description: Overview and statistics about the application.
---
# Dashboard
Accessible at `/settings/` of your **Tube Archivist**, this page shows the status and various statistics related to your library.
Accessible at `/settings/` of your **Tube Archivist** instance, this page shows the status and various statistics related to your library.

45
mkdocs/docs/settings/scheduling.md
View File

@ -3,43 +3,43 @@ description: Configure the scheduler.
---
# Scheduling Settings Page
Accessible at `/settings/scheduling/` of your **Tube Archivist**, this page holds all the configuration for scheduled tasks.
Accessible at `/settings/scheduling/` of your **Tube Archivist** instance, this page holds all the configuration settings for scheduled tasks.
Click on **Update Scheduler Settings** at the bottom of the page to apply your configurations.
!!! tip "Saving Configurations"
Click on **Update Scheduler Settings** at the bottom of the page to apply your configurations.
## Configuring Schedules
Schedule settings expect a cron like format, where the first value is minute, second is hour and third is day of the week. Day 0 is Sunday, day 1 is Monday etc.
The scheduler settings expect a cron-like format, where the first value is the minute, second is the hour and third is day of the week as a number. Day 0 is Sunday, day 1 is Monday etc.
Examples:
- `0 15 *`: Run task every day at 15:00 in the afternoon.
- `0 15 *`: Run task every day at 3 in the afternoon.
- `30 8 */2`: Run task every second day of the week (Sun, Tue, Thu, Sat) at 08:30 in the morning.
- `0 */3,8-17 *`: Execute every hour divisible by 3, and every hour during office hours (8 in the morning - 5 in the afternoon).
- `0 8,16 *`: Execute every day at 8 in the morning and at 4 in the afternoon.
- `auto`: Sensible default.
- `0 */3,8-17 *`: Run task every hour divisible by 3, and every hour during office hours (8 in the morning - 5 in the afternoon).
- `0 8,16 *`: Run task every day at 8 in the morning and at 4 in the afternoon.
- `auto`: Sensible default. Each configuration has a default that is defined by the application's Schedule Builder.
- `0`: (zero), deactivate that task.
!!! note "BE AWARE"
!!! warning "BE AWARE"
- Changes in the scheduler settings require a container restart to take effect.
- Cron format as *number*/*number* are none standard cron and are not supported by the scheduler, for example `0 0/12 *` is invalid, use `0 */12 *` instead.
- Avoid an unnecessary frequent schedule to not get blocked by YouTube. For that reason, the scheduler doesn't support schedules that trigger more than once per hour.
- Cron format as *number*/*number* are non-standard cron formatting and are not supported by the scheduler. For example `0 0/12 *` is invalid, use `0 */12 *` instead.
- Avoid an unnecessary or frequent schedule to reduce likelihood of being blocked or throttled by YouTube. Because of this, the scheduler doesn't support schedules that trigger more than once per hour.
## Notifications
Some of the tasks support sending notifications at task completion with a short summary message. Tasks can get started through the scheduler or manually from the interface. This uses the amazing [Apprise](https://github.com/caronc/apprise) framework, refer to the wiki about the [basics](https://github.com/caronc/apprise/wiki/URLBasics) how to build links and a list of [supported services](https://github.com/caronc/apprise/wiki#notification-services) for the details.
Some of the tasks support sending notifications at task completion with a short summary message. Tasks can get started through the scheduler or manually from the interface. This uses the amazing [Apprise](https://github.com/caronc/apprise) framework. Refer to the wiki about the [basics](https://github.com/caronc/apprise/wiki/URLBasics) of how to build links and a list of [supported services](https://github.com/caronc/apprise/wiki#notification-services) for the details.
Send yourself a test notification to verify your link works, e.g.:
```bash
docker exec -it tubearchivist apprise -b "Hello from TA" <link>
```
Notes:
- This will only send notifications when a task returns anything, e.g. if a [Rescan Subscriptions](#rescan-subscriptions) task doesn't find any new videos to add, no notification will get sent.
- Due to the fact that apprise is running inside a docker container, [desktop notifications](https://github.com/caronc/apprise/wiki#desktop-notification-services) will not work.
- Add one per line.
!!! note "Notes:"
- This will only send notifications when a task returns anything, e.g. if a [Rescan Subscriptions](#rescan-subscriptions) task doesn't find any new videos to add, a notification will *not* be sent.
- Due to the fact that Apprise is running inside a container, [desktop notifications](https://github.com/caronc/apprise/wiki#desktop-notification-services) will not work.
- Add one link per line.
## Rescan Subscriptions
That's the equivalent task as run from the downloads page looking through your channel and playlist and add missing videos to the download queue.
This initiates the same task that can be initiated from the [Downloads Page](../downloads.md#rescan-subscriptions). This will go through each of your subscribed channels and playlists and will add missing videos to the download queue.
Become a sponsor and join [members.tubearchivist.com](https://members.tubearchivist.com/) to get access to *real time* notifications for new videos uploaded by your favorite channels.
@ -47,14 +47,17 @@ Become a sponsor and join [members.tubearchivist.com](https://members.tubearchiv
Start downloading all videos currently in the download queue.
## Refresh Metadata
Rescan videos, channels and playlists on youtube and update metadata periodically. This will also refresh your subtitles and comments based on your current settings. If an item is no longer available on YouTube, this will deactivate it and exclude it from future refreshes. This task is meant to be run once per day, set your schedule accordingly.
Rescan videos, channels and playlists on YouTube and update metadata periodically. This will also refresh your subtitles and comments based on your current settings. If an item is no longer available on YouTube, this will deactivate it and exclude it from future refreshes. This task is meant to be run once per day, set your schedule accordingly.
The field **Refresh older than x days** takes a number where TubeArchivist will consider an item as *outdated*. This value is used to calculate how many items need to be refreshed today based on the total indexed. This will spread out the requests to YouTube. Sensible value here is **90** days.
The field **Refresh older than x days** takes a number where **Tube Archivist** will consider an item as *outdated*. This value is used to calculate how many items need to be refreshed today based on the total indexed. This will spread out the requests to YouTube. The default value here is **90** days.
Additionally to the outdated documents, this will also refresh very recently published videos. This is to keep metadata and statistics uptodate during the first few days when the video goes live.
In addition to any outdated documents, this will also refresh very recently published videos. This is to keep metadata and statistics up-to-date during the first few days when the video goes live.
## Thumbnail check
This will check if all expected thumbnails are there and will delete any artwork without matching video.
This will check if all expected thumbnails are present and will delete any artwork without a matching video.
## ZIP file index backup
Create a zip file of the metadata and select **Max auto backups to keep** to automatically delete old backups created from this task. For data consistency, make sure there aren't any other tasks running that will change the index during the backup process. This is very slow, particularly for large archives. Use [snapshots](application.md#snapshots) instead.
!!! tip "Saving Configurations"
Click on **Update Scheduler Settings** at the bottom of the page to apply your configurations.

6
mkdocs/docs/settings/user.md
View File

@ -3,12 +3,12 @@ description: Configure the user settings of the application.
---
# User Settings Page
Accessible at `/settings/user/` of your **Tube Archivist**, this page holds all the settings that control the look and feel of the application.
Accessible at `/settings/user/` of your **Tube Archivist** instance, this page holds all the settings that control the look and feel of the application. This is configurable for each individual user of your instance.
Click on **Update User Configurations** at the bottom of the page to apply your configurations.
## Color scheme
Switch between the easy on the eyes dark theme and the burning bright theme.
Switch between the "easy on the eyes" dark theme and the "burning" bright theme.
## Archive View
- **Page Size**: Defines how many results get displayed on a given page. Same value goes for all archive views.
- **Page Size**: Defines how many results get displayed on each page. This value is used for all archive views.

10
mkdocs/docs/urls.md
View File

@ -3,10 +3,10 @@ description: How URLs from YouTube get parsed
---
# URLs
This document describes how Tube Archivist identifies and treats links from YouTube.
This document describes how **Tube Archivist** identifies and treats links from YouTube.
!!! note
Application logic of Tube Archivist is tied only to the IDs, not the names.
Application logic of **Tube Archivist** is tied only to the IDs, not the names.
## Video
A video ID is **11** characters long, e.g. `2tdiKTSdE9Y`.
@ -26,8 +26,8 @@ Channel URLs can have these forms, all will get translated to the ID:
- Channel Handle: Starting with a `@` this handle is personal and unique, e.g. `@TomScottGo`
- Alias URL: Based off the channel handle, e.g. `https://www.youtube.com/@TomScottGo`
### Channel sub pages
Tube archivist can separate between different subpages:
### Channel subpages
**Tube Archivist** can differentiate between the primary subpages:
- Videos only: `https://www.youtube.com/@IBRACORP/videos`
- Shorts only: `https://www.youtube.com/@IBRACORP/shorts`
@ -40,4 +40,4 @@ A playlist ID can be `34`, `26` or `18` characters long, e.g. `PL96C35uN7xGLLeET
- Playlist URLs start with a *playlist* path and has a *list* parameter, e.g. `https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha`
### Playlist vs Video URLs
While browsing YouTube videos in Playlists, you might encounter URLs looking like that: `https://www.youtube.com/watch?v=QPZ0pIK_wsc&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha`. As established above, based on the */watch* path and the *v* parameter, Tube Archivist will treat this as a video with the ID `QPZ0pIK_wsc` and **not** as a playlist. If you mean the playlist, you can easily grab the correct ID from the URL, e.g. `PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha`.
While browsing YouTube videos in Playlists, you might encounter URLs looking like that: `https://www.youtube.com/watch?v=QPZ0pIK_wsc&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha`. As established above, based on the */watch* path and the *v* parameter, **Tube Archivist** will treat this as a video with the ID `QPZ0pIK_wsc` and **not** as a playlist. If you mean the playlist, you can easily grab the correct ID from the URL, e.g. `PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha`.

22
mkdocs/docs/users.md
View File

@ -4,21 +4,27 @@ description: Create users, reset passwords, access the admin interface.
# User Management
For now, **Tube Archivist** is *mostly* a single user application. You can create multiple users with different names and passwords, they will share the same videos and and channels. You can configure some permissions and some configurations are on a per user basis. *More is on the roadmap*.
For now, **Tube Archivist** is *mostly* a single user application. You can create multiple users with different names and passwords, however they will share the same videos and channels. You can configure some permissions, and specific configurations are on a per-user basis. *More is on the roadmap*.
## Superuser
The first user gets created with the environment variables **TA_USERNAME** and **TA_PASSWORD** from your docker-compose file. That first user will automatically have *superuser* privileges.
The first user gets created with the environment variables **TA_USERNAME** and **TA_PASSWORD** from your docker-compose file. This first user will automatically have *superuser* privileges.
## Admin Interface
When logged in from your *superuser* account, you are able to access the admin interface from the settings page or at `/admin/`. This interface holds all functionality for user management.
When logged in from your *superuser* account, you are able to access the admin interface from the settings page or at `/admin/`. This interface holds all functionality for user management.
!!! tip "Superuser status required"
This interface is only accessible by users with the **Superuser status** permission.
## Create additional users
From the admin interface when you click on *Accounts* you will get a list of all users. From there you can create additional users by clicking on *Add Account*, provide a name and confirm password and click on *Save* to create the user.
From the admin interface when you click on *Accounts* you will get a list of all users. From there you can create additional users by clicking on *Add Account*, provide a name and confirm a password. and then click on *Save* to create the user.
## Video Management Permissions
To create a user that has permissions to modify the download queue, delete items (channels, videos, playlists, etc.) and change general configuration settings, the user will require the **Is Staff** permission for the user.
## Read Only User
To create a user with limited permissions, remove the **Is Staff** and **Superuser status** permissons. This will make this user a *read only* user, meaning among others, this user will not be able to add anything to the download queue, delete anything, etc.
To create a user with limited permissions, remove the **Is Staff** and **Superuser status** permissons. This will make this user a *read only* user, meaning this user will not be able to add to the download queue, delete anything, etc.
!!! note "Minimally Tested"
!!! warning "Minimally Tested"
This is mostly meant as a *kids mode* or similar, this will most likely not hold against a sophisticated attacker.
## Changing users
@ -27,5 +33,5 @@ You can delete or change permissions and password of a user by clicking on the u
## Reset
Delete all user configurations by deleting the file `cache/db.sqlite3` and restart the container. This will create the superuser again from the environment variables.
!!! note "BE AWARE"
Future improvements here will most likely require such a reset.
!!! danger "BE AWARE"
Future feature improvements here may require resetting the user administration database.

20
mkdocs/docs/video.md
View File

@ -3,29 +3,29 @@ description: Complete Video metadata with playlist navigation and comments.
---
# Video Page
Every video downloaded gets a dedicated page accessible at `/video/<video-id>/` of your Tube Archivist. Throughout the interface, click on a video title to access the video page.
Every video downloaded gets a dedicated page accessible at `/video/<video-id>/` of your **Tube Archivist** instance. Throughout the interface, clicking on a video title will access the video page.
Clicking on the channel name or the channel icon will bring you to the dedicated channel detail [page](channels.md#channel-detail).
Clicking on the channel name or the channel icon will bring you to the dedicated [channel detail page](channels.md#channel-detail).
- The button **Reindex** will reindex the metadata of this video.
- The button **Download File** will download the media file in the browser.
- The button **Delete Video** will delete that video including the media file.
- The button **Delete Video** will delete that video entry and the media file.
If available, a tag cloud will show, representing the tags set by the uploader.
There you can also find stream metadata like file size, video codecs, video bitrate and resolution, audio codecs and bitrate.
You can also find stream metadata like file size, video codecs, video bitrate and resolution, audio codecs and bitrate.
The video description is truncated to the first few lines, click on *show more* to expand the whole description.
The video description is truncated to the first few lines, clicking on *show more* will expand to show the whole description.
## Playlist
When available, a playlist navigation will show at the bottom. Clicking on the playlist name will bring you to the dedicated [Playlist Detail](playlists.md#playlist-detail) page showing all videos downloaded from that playlist. The number in square brackets indicates the position of the current video in that playlist.
When available, a playlist navigation will show at the bottom. Clicking on the playlist name will bring you to the dedicated [Playlist Detail page](playlists.md#playlist-detail) showing all videos downloaded from that playlist. The number in square brackets indicates the position of the current video in that playlist.
Clicking on the next or previous video name or thumbnail will bring you to that dedicated video page.
Clicking on the next, or previous, video name or thumbnail will bring you to that dedicated video page.
## Similar Videos
Tube Archivist will show up to six similar videos in a grid. Similarity is detected from the **video title** and the **video tags**. This naturally will show some videos from the same channel, but can also return videos about the same topic from other channels.
**Tube Archivist** will show up to six similar videos in a grid. Similarity is detected from the **video title** and the **video tags**. This naturally will show some videos from the same channel, but can also return videos about the same topic from other channels.
When playing a video from the similar section with the inline player, the current video will get replaced, refresh the page to reset that or click on the video title to avoid that behavior.
When playing a video from the similar section with the inline player, the current video will get replaced. Refreshing the page to reset that or clicking on the video title will avoid that behavior.
## Comments
If activated on the settings page, this will show the indexed comments. Reveal the threads by clicking the *+ Replies* button. Comments with a heart symbol are favorited by the uploader, comments by the uploader are highlighted in a different color.
If activated on the [Settings Page](settings/application.md#comments), this will show the indexed comments. Reveal the threads by clicking the *+ Replies* button. Comments with a heart symbol are favorited by the uploader, and comments by the uploader are highlighted in a different color.

1
mkdocs/mkdocs.yml
View File

@ -54,6 +54,7 @@ markdown_extensions:
- admonition
- pymdownx.details
- pymdownx.superfences
- tables
theme:
name: material
logo: assets/logo-dark.jpg