From 3ca86ba91dfc735d47f726ba8ebc05100f60e127 Mon Sep 17 00:00:00 2001 From: simon Date: Fri, 6 May 2022 10:17:23 +0700 Subject: [PATCH 01/10] fix typo, improve wording, add ansible docker playbook --- CONTRIBUTING.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f2538fb..2a67c68 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -19,7 +19,7 @@ I have learned the hard way, that working on a dockerized application outside of This is my setup I have landed on, YMMV: - Clone the repo, work on it with your favorite code editor in your local filesystem. *testing* branch is the where all the changes are happening, might be unstable and is WIP. -- Then I have a VM on KVM hypervisor running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. But you could also just run docker on your host system. +- Then I have a VM on KVM hypervisor running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. You can use this [Ansible Docker Ubuntu](https://github.com/bbilly1/ansible-playbooks) playbook to get started quickly. But you could also just run docker on your host system. - The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will just take a few seconds. - Take a look at the `deploy.sh` file. I have my local DNS resolve `tubearchivist.local` to the IP of the VM for convenience. To deploy the latest changes and rebuild the application to the testing VM run: ```bash @@ -32,11 +32,11 @@ This is my setup I have landed on, YMMV: ## Working with Elasticsearch Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries. -If you want to run queries in on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it. +If you want to run queries on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it. ## Implementing a new feature -Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Open up an issue and we try to find a solution together. +Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Reach out on Discord and we'll look into it together. ## Making changes @@ -53,10 +53,10 @@ If you want to see what's in your container, checkout the matching release tag. ## Code formatting and linting -To keep things clean and consistent for everybody, there is a github action setup to lint and check the changes. You can test your code locally first if you want. For example if you made changes in the **download** module, run +To keep things clean and consistent for everybody, there is a github action setup to lint and check the changes. You can test your code locally first if you want. For example if you made changes in the **video** module, run ```shell -./deploy.sh validate tubearchivist/home/src/download.py +./deploy.sh validate tubearchivist/home/src/index/video.py ``` -to validate your changes. If you omit the path, all the project files will get checked. This is subject to change as the codebase improves. \ No newline at end of file +to validate your changes. If you omit the path, all the project files will get checked. This is subject to change as the codebase improves. From 087043811f3418a96240e30a4bea9f4d00071725 Mon Sep 17 00:00:00 2001 From: simon Date: Sat, 7 May 2022 09:34:33 +0700 Subject: [PATCH 02/10] avoid unneeded cookie open --- tubearchivist/home/src/download/yt_cookie.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tubearchivist/home/src/download/yt_cookie.py b/tubearchivist/home/src/download/yt_cookie.py index 3b16ca5..3a7244e 100644 --- a/tubearchivist/home/src/download/yt_cookie.py +++ b/tubearchivist/home/src/download/yt_cookie.py @@ -40,6 +40,9 @@ class CookieHandler: print("no cookie imported") raise FileNotFoundError + if os.path.exists(self.COOKIE_PATH): + return self.COOKIE_PATH + with open(self.COOKIE_PATH, "w", encoding="utf-8") as cookie_file: cookie_file.write(cookie) From 3722f11a65b2f26af36be461983981a7342dae34 Mon Sep 17 00:00:00 2001 From: simon Date: Sat, 7 May 2022 09:34:51 +0700 Subject: [PATCH 03/10] cleanup cookie file after task --- tubearchivist/home/src/download/queue.py | 12 ++++++++++++ tubearchivist/home/src/download/yt_dlp_handler.py | 7 +------ 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/tubearchivist/home/src/download/queue.py b/tubearchivist/home/src/download/queue.py index 8b4322e..27217c4 100644 --- a/tubearchivist/home/src/download/queue.py +++ b/tubearchivist/home/src/download/queue.py @@ -145,6 +145,16 @@ class PendingList(PendingIndex): cookie_path = CookieHandler().use() self.yt_obs.update({"cookiefile": cookie_path}) + def close_config(self): + """remove config after task finished""" + config = AppConfig().config + if config["downloads"]["cookie_import"]: + CookieHandler().hide() + try: + del self.yt_obs["cookiefile"] + except KeyError: + pass + def parse_url_list(self): """extract youtube ids from list""" self.missing_videos = [] @@ -225,6 +235,8 @@ class PendingList(PendingIndex): query_str = "\n".join(bulk_list) _, _ = ElasticWrap("_bulk").post(query_str, ndjson=True) + self.close_config() + def _notify_add(self, idx): """send notification for adding videos to download queue""" progress = f"{idx + 1}/{len(self.missing_videos)}" diff --git a/tubearchivist/home/src/download/yt_dlp_handler.py b/tubearchivist/home/src/download/yt_dlp_handler.py index fd91432..2c24cb0 100644 --- a/tubearchivist/home/src/download/yt_dlp_handler.py +++ b/tubearchivist/home/src/download/yt_dlp_handler.py @@ -41,7 +41,7 @@ class DownloadPostProcess: self.auto_delete_all() self.auto_delete_overwrites() self.validate_playlists() - self.clear_cookie() + self.pending.close_config() def auto_delete_all(self): """handle auto delete""" @@ -141,11 +141,6 @@ class DownloadPostProcess: else: RedisArchivist().set_message("message:download", mess_dict) - def clear_cookie(self): - """hide cookie file""" - if self.download.config["downloads"]["cookie_import"]: - CookieHandler().hide() - class VideoDownloader: """ From 5f63dc93ae28e0612638f212da4dc7351873508b Mon Sep 17 00:00:00 2001 From: simon Date: Sat, 7 May 2022 10:09:53 +0700 Subject: [PATCH 04/10] add Extended Universe section to readme --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 16a6112..0d851fb 100644 --- a/README.md +++ b/README.md @@ -5,11 +5,12 @@ Tube Archivist has a new home: https://github.com/tubearchivist/tubearchivist ## Table of contents: -* [Wiki](https://github.com/tubearchivist/tubearchivist/wiki) for a detailed documentation, with [FAQ](https://github.com/tubearchivist/tubearchivist/wiki/FAQ) +* [Wiki](https://github.com/tubearchivist/tubearchivist/wiki) with [FAQ](https://github.com/tubearchivist/tubearchivist/wiki/FAQ) * [Core functionality](#core-functionality) * [Screenshots](#screenshots) * [Problem Tube Archivist tries to solve](#problem-tube-archivist-tries-to-solve) * [Connect](#connect) +* [Extended Universe](#extended-universe) * [Installing and updating](#installing-and-updating) * [Getting Started](#getting-started) * [Potential pitfalls](#potential-pitfalls) @@ -52,6 +53,9 @@ Once your YouTube video collection grows, it becomes hard to search and find a s - [Discord](https://discord.gg/AFwz8nE7BK): Connect with us on our Discord server. - [r/TubeArchivist](https://www.reddit.com/r/TubeArchivist/): Join our Subreddit. +## Extended Universe +- [Browser Extension](https://github.com/tubearchivist/browser-extension) Tube Archivist Companion, for [Firefox](https://addons.mozilla.org/addon/tubearchivist-companion/) and [Chrome](https://chrome.google.com/webstore/detail/tubearchivist-companion/jjnkmicfnfojkkgobdfeieblocadmcie) + ## Installing and updating Take a look at the example `docker-compose.yml` file provided. Use the *latest* or the named semantic version tag. The *unstable* tag is for intermediate testing and as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](CONTRIBUTING.md). From 8cc6e77169c65f3ea8a128da1c649d905cccb5a7 Mon Sep 17 00:00:00 2001 From: simon Date: Sat, 7 May 2022 18:33:16 +0700 Subject: [PATCH 05/10] switch to compose v2 --- deploy.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/deploy.sh b/deploy.sh index 92ca8de..08874eb 100755 --- a/deploy.sh +++ b/deploy.sh @@ -33,7 +33,7 @@ function sync_blackhole { . -e ssh "$host":tubearchivist echo "$PASS" | ssh "$host" 'sudo -S docker buildx build --platform linux/amd64 -t bbilly1/tubearchivist:latest tubearchivist --load 2>/dev/null' - echo "$PASS" | ssh "$host" 'sudo -S docker-compose up -d 2>/dev/null' + echo "$PASS" | ssh "$host" 'sudo -S docker compose up -d 2>/dev/null' } @@ -69,7 +69,7 @@ function sync_test { fi ssh "$host" "docker buildx build --build-arg INSTALL_DEBUG=1 --platform $platform -t bbilly1/tubearchivist:latest tubearchivist --load" - ssh "$host" 'docker-compose -f docker/docker-compose.yml up -d' + ssh "$host" 'docker compose -f docker/docker-compose.yml up -d' } From f94bbec672e4ccf903b80aed8e44f5808d00d974 Mon Sep 17 00:00:00 2001 From: simon Date: Sat, 7 May 2022 19:00:40 +0700 Subject: [PATCH 06/10] add cookie wiki link --- tubearchivist/home/templates/home/settings.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tubearchivist/home/templates/home/settings.html b/tubearchivist/home/templates/home/settings.html index c529443..d4cd7f5 100644 --- a/tubearchivist/home/templates/home/settings.html +++ b/tubearchivist/home/templates/home/settings.html @@ -118,7 +118,7 @@

Cookie

Import YouTube cookie: {{ config.downloads.cookie_import }}

- Place your cookie file named cookies.google.txt in cache/import before enabling.
+ Place your cookie file named cookies.google.txt in cache/import before enabling. Instructions in the Wiki.
{{ app_form.downloads_cookie_import }}
{% if config.downloads.cookie_import %}
From 0ab809447ac34f7431a6fd6053f3414045cd4963 Mon Sep 17 00:00:00 2001 From: simon Date: Sat, 7 May 2022 19:06:43 +0700 Subject: [PATCH 07/10] update version --- tubearchivist/config/settings.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tubearchivist/config/settings.py b/tubearchivist/config/settings.py index 9a44b06..8787adb 100644 --- a/tubearchivist/config/settings.py +++ b/tubearchivist/config/settings.py @@ -163,4 +163,4 @@ CORS_ALLOW_HEADERS = list(default_headers) + [ # TA application settings TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist" -TA_VERSION = "v0.1.4" +TA_VERSION = "v0.1.5" From ca13ddec267fef033782890eaaef635ee7a5df96 Mon Sep 17 00:00:00 2001 From: simon Date: Sun, 8 May 2022 08:06:37 +0700 Subject: [PATCH 08/10] update es version --- docker-compose.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker-compose.yml b/docker-compose.yml index 89257f1..e5966d9 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -33,7 +33,7 @@ services: depends_on: - archivist-es archivist-es: - image: bbilly1/tubearchivist-es # only for amd64, or use official es 7.17.2 + image: bbilly1/tubearchivist-es # only for amd64, or use official es 7.17.3 container_name: archivist-es restart: always environment: From 619370c67043a9ae9ab4b0937dfc89b75554b1d6 Mon Sep 17 00:00:00 2001 From: simon Date: Sun, 8 May 2022 08:06:59 +0700 Subject: [PATCH 09/10] add metrics to extended universe --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 0d851fb..013f352 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,7 @@ Once your YouTube video collection grows, it becomes hard to search and find a s ## Extended Universe - [Browser Extension](https://github.com/tubearchivist/browser-extension) Tube Archivist Companion, for [Firefox](https://addons.mozilla.org/addon/tubearchivist-companion/) and [Chrome](https://chrome.google.com/webstore/detail/tubearchivist-companion/jjnkmicfnfojkkgobdfeieblocadmcie) +- [Tube Archivist Metrics](https://github.com/tubearchivist/tubearchivist-metrics) to create statistics in Prometheus/OpenMetrics format. ## Installing and updating Take a look at the example `docker-compose.yml` file provided. Use the *latest* or the named semantic version tag. The *unstable* tag is for intermediate testing and as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](CONTRIBUTING.md). From 14387eb8d6e6c9de6d3604ed4b68de8ea930aec3 Mon Sep 17 00:00:00 2001 From: simon Date: Sun, 8 May 2022 08:18:17 +0700 Subject: [PATCH 10/10] update roadmap --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 013f352..fcb1a86 100644 --- a/README.md +++ b/README.md @@ -160,7 +160,6 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex - [ ] Podcast mode to serve channel as mp3 - [ ] Implement [PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2) for flexible video storage - [ ] Implement [Apprise](https://github.com/caronc/apprise) for notifications ([#97](https://github.com/tubearchivist/tubearchivist/issues/97)) -- [ ] Add passing browser cookies to yt-dlp ([#199](https://github.com/tubearchivist/tubearchivist/issues/199)) - [ ] User created playlists, random and repeat controls ([#108](https://github.com/tubearchivist/tubearchivist/issues/108), [#220](https://github.com/tubearchivist/tubearchivist/issues/220)) - [ ] Auto play or play next link ([#226](https://github.com/tubearchivist/tubearchivist/issues/226)) - [ ] Show similar videos on video page @@ -175,6 +174,7 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex - [ ] Download video comments Implemented: +- [X] Add passing browser cookies to yt-dlp [2022-05-08] - [X] Add [SponsorBlock](https://sponsor.ajay.app/) integration [2022-04-16] - [X] Implement per channel settings [2022-03-26] - [X] Subtitle download & indexing [2022-02-13]