tubearchivist/tubearchivist
1
0
Fork 0
mirror of https://github.com/tubearchivist/tubearchivist.git synced 2024-11-12 23:00:14 +00:00
Code Issues Projects Releases Wiki Activity
Your self hosted YouTube media server
1,398 Commits 3 Branches 41 Tags 19 MiB
Python 66.4%
HTML 16.6%
JavaScript 12%
CSS 3.7%
Shell 1%
Other 0.3%
5e841bf7f8
Go to file
simon 5e841bf7f8 Fixing permissions, squashing bugs #build 
Changed:
- Add recent videos to reindes queue
- Fix channel tv art fallback
- Fix notification callback notification off by one
- Fix missing reindex notification in frontend
- Change new channel folder to honoring UID/GID
- Change new subtitles to honour UID/GID
- Change manual import to honour UID/GID
2023-04-08 17:46:41 +07:00
.github add PR template 2023-04-01 12:24:24 +07:00
assets add ibracorp review video 2022-03-27 19:43:40 +07:00
docker_assets commit migrations 2023-02-02 14:19:06 +07:00
docs remove old wiki pages with redirects 2023-04-01 00:01:26 +07:00
tubearchivist fix reindex notification filter 2023-04-08 17:18:42 +07:00
.dockerignore ignore wiki from container 2021-09-30 17:58:56 +07:00
.eslintrc.js Format/slightly modernize the JS (#345) 2022-10-26 09:43:58 +07:00
.gitignore Format/slightly modernize the JS (#345) 2022-10-26 09:43:58 +07:00
ARCHITECTURE.md add basic architecture overview 2022-12-12 17:10:47 +07:00
CONTRIBUTING.md add toc links 2023-04-01 12:18:23 +07:00
deploy.sh ignore mypy_cache from codespell 2023-03-31 12:00:24 +07:00
docker-compose.yml bump es version 2023-03-04 10:57:13 +07:00
Dockerfile bump python base file 2023-01-07 18:08:35 +07:00
LICENSE minimal viable product 2021-09-06 00:10:14 +07:00
package-lock.json cleanup, fix lint 2022-12-23 20:43:36 +07:00
package.json Format/slightly modernize the JS (#345) 2022-10-26 09:43:58 +07:00
README.md add version check note after update 2023-04-04 15:34:58 +07:00
SHOWCASE.MD add more context to showcase screenshots 2023-04-01 12:32:18 +07:00

README.md

Tube Archivist

Your self hosted YouTube media server

tubearchivist-docker tubearchivist-github-star tubearchivist-github-forks tubearchivist-discord

home screenshot

Table of contents:

Core functionality

Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface. THis includes:

  • Subscribe to your favorite YouTube channels
  • Download Videos using yt-dlp
  • Index and make videos searchable
  • Play videos
  • Keep track of viewed and unviewed videos

Resources

Installing

For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus. This project requires docker. Ensure it is installed and running on your system.

The documentation has additional user provided instructions for Unraid, Synology, Podman and True NAS.

The instructions here should get you up and running quickly, for Docker beginners and full explanation about each environment variable, see the docs.

Take a look at the example docker-compose.yml and configure the required environment variables.

TubeArchivist:

Environment Var Value
TA_HOST Server IP or hostname, without port, without protocol Required
TA_USERNAME Initial username when logging into TA Required
TA_PASSWORD Initial password when logging into TA Required
ELASTIC_PASSWORD Password for ElasticSearch Required
REDIS_HOST Hostname for Redis Required
TZ Set your timezone for the scheduler Required
TA_PORT Overwrite Nginx port Optional
TA_UWSGI_PORT Overwrite container internal uwsgi port Optional
ES_URL URL That ElasticSearch runs on Optional
HOST_GID Allow TA to own the video files instead of container user Optional
HOST_UID Allow TA to own the video files instead of container user Optional
ELASTIC_USER Change the default ElasticSearch user Optional
REDIS_PORT Port that Redis runs on Optional
TA_LDAP Configure TA to use LDAP Authentication Read more
ENABLE_CAST Enable casting support Read more
DJANGO_DEBUG Return additional error messages, for debug only

ElasticSearch

Environment Var Value State
ELASTIC_PASSWORD Matching password ELASTIC_PASSWORD from TubeArchivist Required
http.port Change the port ElasticSearch runs on Optional

Update

Always use the latest (the default) or a named semantic version tag for the docker images. The unstable tags are only for your testing environment, there might not be an update path for these testing builds.

You will see the current version number of Tube Archivist in the footer of the interface. There is a daily version check task querying tubearchivist.com, notifying you of any new releases in the footer. To take advantage of the latest fixes and improvements, make sure you are running the latest and greatest. After updating, check the footer to verify you are running the expected version.

  • This project is tested for updates between one or two releases maximum. Further updates back may or may not be supported and you might have to reset your index and configurations to update. Ideally apply new updates at least once per month.
  • There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. Always check the release notes: Any breaking changes will be marked there.
  • All testing and development is done with the Elasticsearch version number as mentioned in the provided docker-compose.yml file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. Use bbilly1/tubearchivist-es to automatically get the recommended version.

Getting Started

  1. Go through the settings page and look at the available options. Particularly set Download Format to your desired video quality before downloading. Tube Archivist downloads the best available quality by default. To support iOS or MacOS and some other browsers a compatible format must be specified. For example:
bestvideo[vcodec*=avc1]+bestaudio[acodec*=mp4a]/mp4
  1. Subscribe to some of your favorite YouTube channels on the channels page.
  2. On the downloads page, click on Rescan subscriptions to add videos from the subscribed channels to your Download queue or click on Add to download queue to manually add Video IDs, links, channels or playlists.
  3. Click on Start download and let Tube Archivist to it's thing.
  4. Enjoy your archived collection!

Port Collisions

If you have a collision on port 8000, best solution is to use dockers HOST_PORT and CONTAINER_PORT distinction: To for example change the interface to port 9000 use 9000:8000 in your docker-compose file.

For more information on port collisions, check the docs.

Common Errors

Here is a list of common errors and their solutions.

vm.max_map_count

Elastic Search in Docker requires the kernel setting of the host machine vm.max_map_count to be set to at least 262144.

To temporary set the value run:

sudo sysctl -w vm.max_map_count=262144

To apply the change permanently depends on your host operating system:

  • For example on Ubuntu Server add vm.max_map_count = 262144 to the file /etc/sysctl.conf.
  • On Arch based systems create a file /etc/sysctl.d/max_map_count.conf with the content vm.max_map_count = 262144.
  • On any other platform look up in the documentation on how to pass kernel parameters.

Permissions for elasticsearch

If you see a message similar to Unable to access 'path.repo' (/usr/share/elasticsearch/data/snapshot) or failed to obtain node locks, tried [/usr/share/elasticsearch/data] and maybe these locations are not writable when initially starting elasticsearch, that probably means the container is not allowed to write files to the volume.
To fix that issue, shutdown the container and on your host machine run:

chown 1000:0 -R /path/to/mount/point

This will match the permissions with the UID and GID of elasticsearch process within the container and should fix the issue.

Disk usage

The Elasticsearch index will turn to read only if the disk usage of the container goes above 95% until the usage drops below 90% again, you will see error messages like disk usage exceeded flood-stage watermark.

Similar to that, TubeArchivist will become all sorts of messed up when running out of disk space. There are some error messages in the logs when that happens, but it's best to make sure to have enough disk space before starting to download.

Known limitations

  • Video files created by Tube Archivist need to be playable in your browser of choice. Not every codec is compatible with every browser and might require some testing with format selection.
  • Every limitation of yt-dlp will also be present in Tube Archivist. If yt-dlp can't download or extract a video for any reason, Tube Archivist won't be able to either.
  • There is currently no flexibility in naming of the media files.

Roadmap

We have come far, nonetheless we are not short of ideas on how to improve and extend this project. Issues waiting for you to be tackled in no particular order:

  • User roles
  • Podcast mode to serve channel as mp3
  • Implement PyFilesystem for flexible video storage
  • Implement Apprise for notifications (#97)
  • User created playlists, random and repeat controls (#108, #220)
  • Auto play or play next link (#226)
  • Multi language support
  • Show total video downloaded vs total videos available in channel
  • Add statistics of index
  • Download speed schedule (#198)
  • Download or Ignore videos by keyword (#163)
  • Custom searchable notes to videos, channels, playlists (#144)

Implemented:

  • Download video comments [2022-11-30]
  • Show similar videos on video page [2022-11-30]
  • Implement complete offline media file import from json file [2022-08-20]
  • Filter and query in search form, search by url query [2022-07-23]
  • Make items in grid row configurable to use more of the screen [2022-06-04]
  • Add passing browser cookies to yt-dlp [2022-05-08]
  • Add SponsorBlock integration [2022-04-16]
  • Implement per channel settings [2022-03-26]
  • Subtitle download & indexing [2022-02-13]
  • Fancy advanced unified search interface [2022-01-08]
  • Auto rescan and auto download on a schedule [2021-12-17]
  • Optional automatic deletion of watched items after a specified time [2021-12-17]
  • Create playlists [2021-11-27]
  • Access control [2021-11-01]
  • Delete videos and channel [2021-10-16]
  • Add thumbnail embed option [2021-10-16]
  • Create a github wiki for user documentation [2021-10-03]
  • Grid and list view for both channel and video list pages [2021-10-03]
  • Un-ignore videos [2021-10-03]
  • Dynamic download queue [2021-09-26]
  • Backup and restore [2021-09-22]
  • Scan your file system to index already downloaded videos [2021-09-14]

Donate

The best donation to Tube Archivist is your time, take a look at the contribution page to get started.
Second best way to support the development is to provide for caffeinated beverages:

Notable mentions

This is a selection of places where this project has been featured on reddit, in the news, blogs or any other online media, newest on top.

  • noted.lol: Dev Debrief, An Interview With the Developer of Tube Archivist, [2023-03-30] [link]
  • console.substack.com: Interview With Simon of Tube Archivist, [2023-01-29] [link]
  • reddit.com: Tube Archivist v0.3.0 - Now Archiving Comments, [2022-12-02] [link]
  • reddit.com: Tube Archivist v0.2 - Now with Full Text Search, [2022-07-24] [link]
  • noted.lol: How I Control What Media My Kids Watch Using Tube Archivist, [2022-03-27] [link]
  • thehomelab.wiki: Tube Archivist - A Youtube-DL Alternative on Steroids, [2022-01-27] [link]
  • reddit.com: Celebrating TubeArchivist v0.1, [2022-01-09] [link]
  • linuxunplugged.com: Pick: tubearchivist — Your self-hosted YouTube media server, [2021-09-11] [link] and [2021-10-05] [link]
  • reddit.com: Introducing Tube Archivist, your self hosted Youtube media server, [2021-09-12] [link]

Sponsor

Big thank you to Digitalocean for generously donating credit for the tubearchivist.com VPS and buildserver.