diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 06034d5..f2538fb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,7 +5,7 @@ If you haven't already, the best place to start is the README. This will give yo ## Report a bug -If you notice something is not working as expected, check to see if it has been previously reported in the [open issues](https://github.com/bbilly1/tubearchivist/issues). +If you notice something is not working as expected, check to see if it has been previously reported in the [open issues](https://github.com/tubearchivist/tubearchivist/issues). If it has not yet been disclosed, go ahead and create an issue. If the issue doesn't move forward due to a lack of response, I assume it's solved and will close it after some time to keep the list fresh. diff --git a/README.md b/README.md index 1b80457..d3ee806 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,10 @@

Your self hosted YouTube media server

+Tube Archivist has a new home: https://github.com/tubearchivist/tubearchivist + ## Table of contents: -* [Wiki](https://github.com/bbilly1/tubearchivist/wiki) for a detailed documentation, with [FAQ](https://github.com/bbilly1/tubearchivist/wiki/FAQ) +* [Wiki](https://github.com/tubearchivist/tubearchivist/wiki) for a detailed documentation, with [FAQ](https://github.com/tubearchivist/tubearchivist/wiki/FAQ) * [Core functionality](#core-functionality) * [Screenshots](#screenshots) * [Problem Tube Archivist tries to solve](#problem-tube-archivist-tries-to-solve) @@ -47,8 +49,8 @@ Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface. ## Connect -- [Discord](https://discord.gg/AFwz8nE7BK): Connect with us on our brand new Discord server. -- [r/TubeArchivist](https://www.reddit.com/r/TubeArchivist/): Join our brand new Subreddit. +- [Discord](https://discord.gg/AFwz8nE7BK): Connect with us on our Discord server. +- [r/TubeArchivist](https://www.reddit.com/r/TubeArchivist/): Join our Subreddit. ## Installing and updating Take a look at the example `docker-compose.yml` file provided. Use the *latest* or the named semantic version tag. The *unstable* tag is for intermediate testing and as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](CONTRIBUTING.md). @@ -76,7 +78,9 @@ Should that not be an option, the Tube Archivist container takes these two addit Changing any of these two environment variables will change the files *nginx.conf* and *uwsgi.ini* at startup using `sed` in your container. ### Elasticsearch -**Note**: Newest Tube Archivist depends on Elasticsearch version 7.17 to provide an automatic updatepath. +**Note**: Newest Tube Archivist depends on Elasticsearch version 7.17 to provide an automatic updatepath in the future. + +Use `bbilly1/tubearchivist-es` to automatically get the recommended version, or use the official image with the version tag in the docker-compose file. Stores video meta data and makes everything searchable. Also keeps track of the download queue. - Needs to be accessible over the default port `9200` @@ -98,11 +102,11 @@ For some architectures it might be required to run Redis JSON on a nonstandard p ### Updating Tube Archivist You will see the current version number of **Tube Archivist** in the footer of the interface so you can compare it with the latest release to make sure you are running the *latest and greatest*. * There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. *Always* check the **release notes**: Any breaking changes will be marked there. -* All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. +* All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. Use `bbilly1/tubearchivist-es` to automatically get the recommended version. ### Alternative installation instructions: - **arm64**: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, you can use `bbilly1/rejson`, an unofficial rebuild for arm64. -- **Synology**: There is a [discussion thread](https://github.com/bbilly1/tubearchivist/discussions/48) with Synology installation instructions. +- **Synology**: There is a [discussion thread](https://github.com/tubearchivist/tubearchivist/discussions/48) with Synology installation instructions. - **Unraid**: The three containers needed are all in the Community Applications. First install `TubeArchivist RedisJSON` followed by `TubeArchivist ES`, and finally you can install `TubeArchivist`. If you have unraid specific issues, report those to the [support thread](https://forums.unraid.net/topic/114073-support-crocs-tube-archivist/ "support thread"). - **Helm Chart**: There is a Helm Chart available at https://github.com/insuusvenerati/helm-charts. Mostly self-explanatory but feel free to ask questions in the discord / subreddit. @@ -148,22 +152,23 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex - [ ] User roles - [ ] Podcast mode to serve channel as mp3 - [ ] Implement [PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2) for flexible video storage -- [ ] Implement [Apprise](https://github.com/caronc/apprise) for notifications ([#97](https://github.com/bbilly1/tubearchivist/issues/97)) -- [ ] Add [SponsorBlock](https://sponsor.ajay.app/) integration -- [ ] Add passing browser cookies to yt-dlp ([#199](https://github.com/bbilly1/tubearchivist/issues/199)) -- [ ] User created playlists, random and repeat controls ([#108](https://github.com/bbilly1/tubearchivist/issues/108), [#220](https://github.com/bbilly1/tubearchivist/issues/220)) -- [ ] Auto play or play next link +- [ ] Implement [Apprise](https://github.com/caronc/apprise) for notifications ([#97](https://github.com/tubearchivist/tubearchivist/issues/97)) +- [ ] Add passing browser cookies to yt-dlp ([#199](https://github.com/tubearchivist/tubearchivist/issues/199)) +- [ ] User created playlists, random and repeat controls ([#108](https://github.com/tubearchivist/tubearchivist/issues/108), [#220](https://github.com/tubearchivist/tubearchivist/issues/220)) +- [ ] Auto play or play next link ([#226](https://github.com/tubearchivist/tubearchivist/issues/226)) - [ ] Show similar videos on video page - [ ] Multi language support - [ ] Show total video downloaded vs total videos available in channel - [ ] Make items in grid row configurable to use more of the screen - [ ] Add statistics of index -- [ ] Implement complete offline media file import from json file ([#138](https://github.com/bbilly1/tubearchivist/issues/138)) -- [ ] Filter and query in search form, search by url query ([#134](https://github.com/bbilly1/tubearchivist/issues/134), [#139](https://github.com/bbilly1/tubearchivist/issues/139)) -- [ ] Auto ignore videos by keyword ([#163](https://github.com/bbilly1/tubearchivist/issues/163)) -- [ ] Custom searchable notes to videos, channels, playlists ([#144](https://github.com/bbilly1/tubearchivist/issues/144)) +- [ ] Implement complete offline media file import from json file ([#138](https://github.com/tubearchivist/tubearchivist/issues/138)) +- [ ] Filter and query in search form, search by url query ([#134](https://github.com/tubearchivist/tubearchivist/issues/134), [#139](https://github.com/tubearchivist/tubearchivist/issues/139)) +- [ ] Auto ignore videos by keyword ([#163](https://github.com/tubearchivist/tubearchivist/issues/163)) +- [ ] Custom searchable notes to videos, channels, playlists ([#144](https://github.com/tubearchivist/tubearchivist/issues/144)) +- [ ] Download video comments Implemented: +- [X] Add [SponsorBlock](https://sponsor.ajay.app/) integration [2022-04-16] - [X] Implement per channel settings [2022-03-26] - [X] Subtitle download & indexing [2022-02-13] - [X] Fancy advanced unified search interface [2022-01-08] diff --git a/deploy.sh b/deploy.sh index 0818e7a..92ca8de 100755 --- a/deploy.sh +++ b/deploy.sh @@ -12,7 +12,7 @@ # docker buildx inspect --bootstrap # more details: -# https://github.com/bbilly1/tubearchivist/issues/6 +# https://github.com/tubearchivist/tubearchivist/issues/6 set -e @@ -103,8 +103,9 @@ function validate { # update latest tag compatible es for set and forget function sync_latest_es { - printf "\nsync new es version:\n" - read -r VERSION + VERSION=$(grep "bbilly1/tubearchivist-es" docker-compose.yml | awk '{print $NF}') + printf "\nsync new ES version %s\nContinue?\n" "$VERSION" + read -rn 1 if [[ $(systemctl is-active docker) != 'active' ]]; then echo "starting docker" diff --git a/docker-compose.yml b/docker-compose.yml index a3ac505..89257f1 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -11,19 +11,19 @@ services: - media:/youtube - cache:/cache environment: - - ES_URL=http://archivist-es:9200 - - REDIS_HOST=archivist-redis + - ES_URL=http://archivist-es:9200 # needs protocol e.g. http and port + - REDIS_HOST=archivist-redis # don't add protocol - HOST_UID=1000 - HOST_GID=1000 - - TA_USERNAME=tubearchivist - - TA_PASSWORD=verysecret - - ELASTIC_PASSWORD=verysecret - - TZ=America/New_York + - TA_USERNAME=tubearchivist # your initial TA credentials + - TA_PASSWORD=verysecret # your initial TA credentials + - ELASTIC_PASSWORD=verysecret # set password for Elasticsearch + - TZ=America/New_York # set your time zone depends_on: - archivist-es - archivist-redis archivist-redis: - image: redislabs/rejson:latest # For arm64 just update this line with bbilly1/rejson:latest + image: redislabs/rejson:latest # for arm64 use bbilly1/rejson container_name: archivist-redis restart: always expose: @@ -33,12 +33,12 @@ services: depends_on: - archivist-es archivist-es: - image: docker.elastic.co/elasticsearch/elasticsearch:7.17.1 + image: bbilly1/tubearchivist-es # only for amd64, or use official es 7.17.2 container_name: archivist-es restart: always environment: - "xpack.security.enabled=true" - - "ELASTIC_PASSWORD=verysecret" + - "ELASTIC_PASSWORD=verysecret" # matching Elasticsearch password - "discovery.type=single-node" - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: @@ -46,7 +46,7 @@ services: soft: -1 hard: -1 volumes: - - es:/usr/share/elasticsearch/data + - es:/usr/share/elasticsearch/data # check for permission error when using bind mount, see readme expose: - "9200" diff --git a/docs/Channels.md b/docs/Channels.md index b82c660..e6f7654 100644 --- a/docs/Channels.md +++ b/docs/Channels.md @@ -28,4 +28,5 @@ Each channel will get a dedicated channel detail page accessible at `/channel/= 10000: + video_daily = 9999 + total_channels = self._get_total_hits("ta_channel") channel_daily = ceil(total_channels / self.interval * self.MULTIPLY) total_playlists = self._get_total_hits("ta_playlist") diff --git a/tubearchivist/home/src/index/video.py b/tubearchivist/home/src/index/video.py index a58a0cb..969d716 100644 --- a/tubearchivist/home/src/index/video.py +++ b/tubearchivist/home/src/index/video.py @@ -10,7 +10,6 @@ from datetime import datetime import requests from django.conf import settings -from home.src.download.thumbnails import ThumbManager from home.src.es.connect import ElasticWrap from home.src.index import channel as ta_channel from home.src.index.generic import YouTubeItem @@ -417,16 +416,15 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle): def _check_get_sb(self): """check if need to run sponsor block""" - integrate = False - if self.config["downloads"]["integrate_sponsorblock"]: - integrate = True + integrate = self.config["downloads"]["integrate_sponsorblock"] if self.video_overwrites: single_overwrite = self.video_overwrites.get(self.youtube_id) if not single_overwrite: return integrate - integrate = single_overwrite.get("integrate_sponsorblock", False) + if "integrate_sponsorblock" in single_overwrite: + return single_overwrite.get("integrate_sponsorblock") return integrate @@ -438,7 +436,8 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle): upload_date_time = datetime.strptime(upload_date, "%Y%m%d") published = upload_date_time.strftime("%Y-%m-%d") last_refresh = int(datetime.now().strftime("%s")) - base64_blur = ThumbManager().get_base64_blur(self.youtube_id) + # base64_blur = ThumbManager().get_base64_blur(self.youtube_id) + base64_blur = False # build json_data basics self.json_data = { "title": self.youtube_meta["title"], diff --git a/tubearchivist/home/templates/home/about.html b/tubearchivist/home/templates/home/about.html index 26d2416..23e01a8 100644 --- a/tubearchivist/home/templates/home/about.html +++ b/tubearchivist/home/templates/home/about.html @@ -7,13 +7,13 @@

Useful Links

-

This project is in active and constant development, take a look at the roadmap for a overview.

-

For any questions on what a button or a function does, You can find the up-to-date user documentation on Github.

-

All contributions are welcome: Open an issue for any bugs and errors, start a discussion for anything that will require a more indepth look. The contributing page is a good place to get started.

+

This project is in active and constant development, take a look at the roadmap for a overview.

+

For any questions on what a button or a function does, You can find the up-to-date user documentation on Github.

+

All contributions are welcome: Open an issue for any bugs and errors, join us on Discord to discuss details. The contributing page is a good place to get started.

Donate

-

Here are some links, if you want to buy the developer a coffee. Thank you for your support!

+

Here are some links, if you want to buy the developer a coffee. Thank you for your support!

{% endblock content %} diff --git a/tubearchivist/home/templates/home/base.html b/tubearchivist/home/templates/home/base.html index ceb35d5..83b41f9 100644 --- a/tubearchivist/home/templates/home/base.html +++ b/tubearchivist/home/templates/home/base.html @@ -132,7 +132,7 @@
- © 2021 - TubeArchivist {{ version }} About | GitHub | Docker Hub | Discord | Reddit + © 2021 - TubeArchivist {{ version }} About | GitHub | Docker Hub | Discord | Reddit
diff --git a/tubearchivist/home/templates/home/channel_id.html b/tubearchivist/home/templates/home/channel_id.html index a30b4f5..8240132 100644 --- a/tubearchivist/home/templates/home/channel_id.html +++ b/tubearchivist/home/templates/home/channel_id.html @@ -93,6 +93,8 @@

Enable SponsorBlock: {% if channel_info.channel_overwrites.integrate_sponsorblock %} {{ channel_info.channel_overwrites.integrate_sponsorblock }} + {% elif channel_info.channel_overwrites.integrate_sponsorblock == False %} + Disabled {% else %} False {% endif %}

diff --git a/tubearchivist/home/templates/home/login.html b/tubearchivist/home/templates/home/login.html index b19f52a..ecefb42 100644 --- a/tubearchivist/home/templates/home/login.html +++ b/tubearchivist/home/templates/home/login.html @@ -45,7 +45,7 @@ -

Github Donate

+

Github Donate

diff --git a/tubearchivist/home/templates/home/settings.html b/tubearchivist/home/templates/home/settings.html index 58e69cb..0ff1516 100644 --- a/tubearchivist/home/templates/home/settings.html +++ b/tubearchivist/home/templates/home/settings.html @@ -258,7 +258,7 @@

Manual media files import.

-

Add files to the cache/import folder. Make sure to follow the instructions in the Github Wiki.

+

Add files to the cache/import folder. Make sure to follow the instructions in the Github Wiki.

@@ -303,7 +303,7 @@

Rescan filesystem

Danger Zone: This will delete the metadata of deleted videos from the filesystem.

-

Rescan your media folder looking for missing videos and clean up index. More infos on the Github Wiki.

+

Rescan your media folder looking for missing videos and clean up index. More infos on the Github Wiki.

diff --git a/tubearchivist/static/script.js b/tubearchivist/static/script.js index 4031ec0..4287100 100644 --- a/tubearchivist/static/script.js +++ b/tubearchivist/static/script.js @@ -333,7 +333,7 @@ function createPlayer(button) { var videoData = getVideoData(videoId); var sponsorBlockElements = ''; - if (videoData.data.sponsorblock.is_enabled) { + if (videoData.data.sponsorblock && videoData.data.sponsorblock.is_enabled) { sponsorBlock = videoData.data.sponsorblock; if (sponsorBlock.segments.length == 0) { sponsorBlockElements = ` @@ -505,19 +505,21 @@ function getVideoPlayerWatchStatus() { return watched; } -// Runs on video playback, marks video as watched if video gets to 90% or higher, sends position to api +// Runs on video playback, marks video as watched if video gets to 90% or higher, sends position to api, SB skipping function onVideoProgress() { var videoId = getVideoPlayerVideoId(); var currentTime = getVideoPlayerCurrentTime(); var duration = getVideoPlayerDuration(); var videoElement = getVideoPlayer(); - // var sponsorBlockElement = document.getElementById("sponsorblock"); var notificationsElement = document.getElementById("notifications"); - if (sponsorBlock.segments.length > 0) { + if (sponsorBlock && sponsorBlock.segments) { for(let i in sponsorBlock.segments) { - if(sponsorBlock.segments[i].segment[0] <= currentTime + 0.3 && sponsorBlock.segments[i].segment[0] >= currentTime) { + if(currentTime >= sponsorBlock.segments[i].segment[0] && currentTime <= sponsorBlock.segments[i].segment[0] + 0.3) { videoElement.currentTime = sponsorBlock.segments[i].segment[1]; - notificationsElement.innerHTML += `

Skipped sponsor segment from ${formatTime(sponsorBlock.segments[i].segment[0])} to ${formatTime(sponsorBlock.segments[i].segment[1])}.

`; + var notificationElement = document.getElementById("notification-" + sponsorBlock.segments[i].UUID); + if (!notificationElement) { + notificationsElement.innerHTML += `

Skipped sponsor segment from ${formatTime(sponsorBlock.segments[i].segment[0])} to ${formatTime(sponsorBlock.segments[i].segment[1])}.

`; + } } if(currentTime > sponsorBlock.segments[i].segment[1] + 10) { var notificationsElementUUID = document.getElementById("notification-" + sponsorBlock.segments[i].UUID);