mirror of
https://github.com/tubearchivist/tubearchivist.git
synced 2025-03-18 11:50:12 +00:00
readme updates for v0.5.0
This commit is contained in:
parent
680279f6a2
commit
b3376318ac
@ -1,48 +0,0 @@
|
||||
# The Inner Workings of Tube Archivist
|
||||
This is a high level overview of the architecture of Tube Archivist, intended for interested contributors to find your way around quickly.
|
||||
|
||||
```
|
||||
Tube Archivist
|
||||
______________________|_____________________
|
||||
| | |
|
||||
------------------- --------------- -------------------
|
||||
| | | | | |
|
||||
| DjangoProject | | RedisJson | | ElasticSearch |
|
||||
| | | | | |
|
||||
------------------- --------------- -------------------
|
||||
```
|
||||
|
||||
## DjangoProject
|
||||
This is the main Python application. Django serves its data container internally with **Uwsgi** on port 8080, the interface is served with **Nginx** on the public port 8000.
|
||||
|
||||
Users created static files like media files and artwork as well as application artwork like logos and fonts are served directly from Nginx, while the rest of the application uses uwsgi_pass to proxy the requests to uwsgi.
|
||||
|
||||
Config files are located in the `docker_assets` folder. The script `run.sh` is the container `CMD` command and entry point, validating env vars, connection to ElasticSearch (ES) and will start the application.
|
||||
|
||||
Compared to other Django projects, this application doesn't make use of the database models, due to a lack of integration with ES. This project has its own abstractions and integrations, treating ES as a REST API.
|
||||
|
||||
Long running application tasks are handed off to **Celery** - using **Redis** as a broker - to run asynchronously from the main threads.
|
||||
- All tasks are defined in the `home.tasks.py` module.
|
||||
|
||||
There are three Django apps:
|
||||
- **config**: The root app, routing the main endpoints and the main `settings.py` file
|
||||
- **api**: The API app with its views and functionality
|
||||
- **home**: Most of the application logic, templates and views, will probably get split up further in the future.
|
||||
|
||||
The *home* app is split up into packages in the `src` directory:
|
||||
- **download**: All download related classes, interact with yt-dlp, download artwork, handle the download queue and post processing tasks.
|
||||
- **es**: All index setup and validation classes, handles mapping validations and makes mapping changes, wrapper functions to simplify interactions with Elasticsearch, backup and restore.
|
||||
- **frontend**: All direct interactions with the frontend, like Django forms, searching, watched state changes, and legacy api_calls in the process of moving to the api app.
|
||||
- **index**: Contains all functionality for scraping and indexing videos, channels, playlists, comments, subtitles, etc...
|
||||
- **ta**: Loose collection of functions and classes, handle application config and contains redis wrapper classes.
|
||||
|
||||
## RedisJson
|
||||
Holds the main application config json object that gets dynamically edited from the frontend, serves as a message broker for **Celery**. Redis serves as a temporary and thread safe link between Django and the frontend, storing progress messages and temporary queues for processing. Used to store locking keys for threads and execution details for tasks.
|
||||
|
||||
- Wrapper classes to interact with Redis are located in the `home.src.ta.ta_redis.py` module.
|
||||
|
||||
## ElasticSearch (ES)
|
||||
Is used to store and index all metadata, functions as an application database and makes it all searchable. The mapping defines which fields are indexed as searchable text fields and which fields are used for match filtering.
|
||||
|
||||
- The index setup and validation is handled in the `home.src.es.index_setup.py` module.
|
||||
- Wrapper classes for making requests to ES are located in the `home.src.es.connect.py` module.
|
||||
@ -3,7 +3,6 @@
|
||||
Welcome, and thanks for showing interest in improving Tube Archivist!
|
||||
|
||||
## Table of Content
|
||||
- [Next Steps](#next-steps)
|
||||
- [Beta Testing](#beta-testing)
|
||||
- [How to open an issue](#how-to-open-an-issue)
|
||||
- [Bug Report](#bug-report)
|
||||
@ -16,16 +15,6 @@ Welcome, and thanks for showing interest in improving Tube Archivist!
|
||||
- [Development Environment](#development-environment)
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
Going forward, this project will focus on developing a new modern frontend.
|
||||
|
||||
- For the time being, don't open any new PRs that are not towards the new frontend.
|
||||
- New features requests likely won't get accepted during this process.
|
||||
- Depending on the severity, bug reports may or may not get fixed during this time.
|
||||
- When in doubt, reach out.
|
||||
|
||||
Join us on [Discord](https://tubearchivist.com/discord) if you want to help with that process.
|
||||
|
||||
## Beta Testing
|
||||
Be the first to help test new features and improvements and provide feedback! There are regular `:unstable` builds for easy access. That's for the tinkerers and the breave. Ideally use a testing environment first, before a release be the first to install it on your main system.
|
||||
|
||||
@ -87,19 +76,16 @@ This is a quick checklist to help streamline the process:
|
||||
- Show off your progress, even if not yet complete, by creating a [draft](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests#draft-pull-requests) PR first and switch it as *ready* when you are ready.
|
||||
- Make sure all your code is linted and formatted correctly, see below. The automatic GH action unfortunately needs to be triggered manually by a maintainer for first time contributors, but will trigger automatically for existing contributors.
|
||||
|
||||
### Making changes to the JavaScript
|
||||
|
||||
The JavaScript does not require any build step; you just edit the files directly. However, there is config for eslint and prettier (a linter and formatter respectively); their use is recommended but not required. To use them, install `node`, run `npm i` from the root directory of this repository to install dependencies, then run `npm run lint` and `npm run format` to run eslint and prettier respectively.
|
||||
|
||||
### Code formatting and linting
|
||||
|
||||
To keep things clean and consistent for everybody, there is a github action setup to lint and check the changes. You can test your code locally first if you want. For example if you made changes in the **video** module, run
|
||||
This project uses the excellent [pre-commit](https://github.com/pre-commit/pre-commit) library. The [pre-commit-config.yml](https://github.com/tubearchivist/tubearchivist/blob/master/.pre-commit-config.yaml) file is part of this repo.
|
||||
|
||||
```shell
|
||||
./deploy.sh validate tubearchivist/home/src/index/video.py
|
||||
```
|
||||
**Quick Start**
|
||||
- Run `pre-commit install` from the root of the repo.
|
||||
- Next time you commit to your local git repo, the defined hooks will run.
|
||||
- On first run, this will download and install the needed environments to your local machine, that can take some time. But that will be reused on sunsequent commits.
|
||||
|
||||
to validate your changes. If you omit the path, all the project files will get checked. This is subject to change as the codebase improves.
|
||||
That is also running as a Git Hub action.
|
||||
|
||||
---
|
||||
|
||||
@ -132,7 +118,7 @@ Some of you might have created useful scripts or API integrations around this pr
|
||||
|
||||
## Improve to the Documentation
|
||||
|
||||
The documentation available at [docs.tubearchivist.com](https://docs.tubearchivist.com/) and is build from a separate repo [tubearchivist/docs](https://github.com/tubearchivist/docs). The Readme has additional instructions on how to make changes.
|
||||
The documentation available at [docs.tubearchivist.com](https://docs.tubearchivist.com/) and is build from a separate repo [tubearchivist/docs](https://github.com/tubearchivist/docs). The Readme there has additional instructions on how to make changes.
|
||||
|
||||
---
|
||||
|
||||
@ -165,7 +151,7 @@ TZ=America/New_York
|
||||
DJANGO_DEBUG=True
|
||||
```
|
||||
|
||||
Than from look at the container startup script `run.sh`, make sure all needed migrations and startup checks ran, then to start the dev backend server from the same folder as `manage.py` run:
|
||||
Then look at the container startup script `run.sh`, make sure all needed migrations and startup checks ran. To start the dev backend server from the same folder as `manage.py` run:
|
||||
|
||||
```bash
|
||||
python manage.py runserver
|
||||
@ -205,11 +191,11 @@ Make your changes locally and re-run `docker compose up --build`. The `Dockerfil
|
||||
|
||||
### Develop environment inside a VM
|
||||
|
||||
You may find it nice to run everything inside of a VM, though this is not necessary. There's a `deploy.sh` script which has some helpers for this use case. YMMV, this is what one of the developers does:
|
||||
You may find it nice to run everything inside of a VM for complete environment snapshots and encapsulation, though this is not strictly necessary. There's a `deploy.sh` script which has some helpers for this use case:
|
||||
|
||||
- Clone the repo, work on it with your favorite code editor in your local filesystem. *testing* branch is where all the changes are happening, might be unstable and is WIP.
|
||||
- Then I have a VM running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate low end environments by limiting CPU cores and memory. You can use this [Ansible Docker Ubuntu](https://github.com/bbilly1/ansible-playbooks) playbook to get started quickly. But you could also just run docker on your host system.
|
||||
- I have my local DNS resolve `tubearchivist.local` to the IP of the VM for convenience. To deploy the latest changes and rebuild the application to the testing VM run:
|
||||
- This assumes a standard Ubuntu Server VM with docker and docker compose already installed.
|
||||
- Configure your local DNS to resolve `tubearchivist.local` to the IP of the VM.
|
||||
- To deploy the latest changes and rebuild the application to the testing VM run:
|
||||
```bash
|
||||
./deploy.sh test
|
||||
```
|
||||
|
||||
12
README.md
12
README.md
@ -2,9 +2,9 @@
|
||||
[*more screenshots and video*](SHOWCASE.MD)
|
||||
|
||||
<div align="center">
|
||||
<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-docker.png" alt="tubearchivist-docker" title="Tube Archivist Docker Pulls" height="50" width="190"/></a>
|
||||
<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-star.png" alt="tubearchivist-github-star" title="Tube Archivist GitHub Stars" height="50" width="190"/></a>
|
||||
<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-forks.png" alt="tubearchivist-github-forks" title="Tube Archivist GitHub Forks" height="50" width="190"/></a>
|
||||
<a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-docker.png" alt="tubearchivist-docker" title="Tube Archivist Docker Pulls" height="50" width="190"/></a>
|
||||
<a href="https://github.com/tubearchivist/tubearchivist/stargazers" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-star.png" alt="tubearchivist-github-star" title="Tube Archivist GitHub Stars" height="50" width="190"/></a>
|
||||
<a href="https://github.com/tubearchivist/tubearchivist/forks" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-forks.png" alt="tubearchivist-github-forks" title="Tube Archivist GitHub Forks" height="50" width="190"/></a>
|
||||
<a href="https://www.tubearchivist.com/discord" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-discord.png" alt="tubearchivist-discord" title="TA Discord Server Members" height="50" width="190"/></a>
|
||||
</div>
|
||||
|
||||
@ -47,6 +47,8 @@ The instructions here should get you up and running quickly, for Docker beginner
|
||||
|
||||
Take a look at the example [docker-compose.yml](https://github.com/tubearchivist/tubearchivist/blob/master/docker-compose.yml) and configure the required environment variables.
|
||||
|
||||
All environment variables are explained in detail in the docs [here](https://docs.tubearchivist.com/installation/env-vars/).
|
||||
|
||||
**TubeArchivist**:
|
||||
| Environment Var | Value | |
|
||||
| ----------- | ----------- | ----------- |
|
||||
@ -68,8 +70,8 @@ Take a look at the example [docker-compose.yml](https://github.com/tubearchivist
|
||||
| HOST_UID | Allow TA to own the video files instead of container user | Optional |
|
||||
| ELASTIC_USER | Change the default ElasticSearch user | Optional |
|
||||
| TA_LDAP | Configure TA to use LDAP Authentication | [Read more](https://docs.tubearchivist.com/configuration/ldap/) |
|
||||
| ENABLE_CAST | Enable casting support | [Read more](https://docs.tubearchivist.com/configuration/cast/) |
|
||||
| DJANGO_DEBUG | Return additional error messages, for debug only | |
|
||||
| DISABLE_STATIC_AUTH | Remove authentication from media files, (Google Cast...) | [Read more](https://docs.tubearchivist.com/installation/env-vars/#disable_static_auth) |
|
||||
| DJANGO_DEBUG | Return additional error messages, for debug only | Optional |
|
||||
|
||||
**ElasticSearch**
|
||||
| Environment Var | Value | State |
|
||||
|
||||
Loading…
Reference in New Issue
Block a user