mirror of
https://github.com/tubearchivist/tubearchivist.git
synced 2024-11-13 15:20:13 +00:00
New search API endpoint, #build
Changed: - Switched search form to API - Additional validations for RYD - Additional log output for curl ES startup check
This commit is contained in:
commit
1b22e61384
@ -146,12 +146,12 @@ To apply the change permanently depends on your host operating system:
|
||||
- On any other platform look up in the documentation on how to pass kernel parameters.
|
||||
|
||||
### Permissions for elasticsearch
|
||||
If you see a message similar to `AccessDeniedException[/usr/share/elasticsearch/data/nodes]` when initially starting elasticsearch, that means the container is not allowed to write files to the volume.
|
||||
That's most likely the case when you run `docker-compose` as an unprivileged user. To fix that issue, shutdown the container and on your host machine run:
|
||||
If you see a message similar to `failed to obtain node locks, tried [/usr/share/elasticsearch/data]` and `maybe these locations are not writable` when initially starting elasticsearch, that probably means the container is not allowed to write files to the volume.
|
||||
To fix that issue, shutdown the container and on your host machine run:
|
||||
```
|
||||
chown 1000:0 /path/to/mount/point
|
||||
chown 1000:0 -R /path/to/mount/point
|
||||
```
|
||||
This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue.
|
||||
This will match the permissions with the **UID** and **GID** of elasticsearch process within the container and should fix the issue.
|
||||
|
||||
### Disk usage
|
||||
The Elasticsearch index will turn to *read only* if the disk usage of the container goes above 95% until the usage drops below 90% again, you will see error messages like `disk usage exceeded flood-stage watermark`, [link](https://github.com/tubearchivist/tubearchivist#disk-usage).
|
||||
|
@ -33,6 +33,7 @@ until curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL" -fs; do
|
||||
if [[ $counter -eq 12 ]]; then
|
||||
# fail after 2 min
|
||||
echo "failed to connect to elastic search, exiting..."
|
||||
curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL"?pretty
|
||||
exit 1
|
||||
fi
|
||||
sleep 10
|
||||
|
@ -5,6 +5,36 @@ Note:
|
||||
- This is very early stages and will change!
|
||||
- Check the commit history to see if a documented feature is already in your release
|
||||
|
||||
## Table of contents
|
||||
- [Authentication](#authentication)
|
||||
- [Pagination](#pagination)
|
||||
|
||||
**Video**
|
||||
- [Video List](#video-list-view)
|
||||
- [Video Single](#video-item-view)
|
||||
- [Video Single Progress](#video-progress-view)
|
||||
- [Video Single Sponsorblock](#sponsor-block-view) WIP
|
||||
|
||||
**Channel**
|
||||
- [Channel List](#channel-list-view)
|
||||
- [Channel Single](#channel-item-view)
|
||||
- [Channel Video List](#channel-videos-view)
|
||||
|
||||
**Playlist**
|
||||
- [Playlist List](#playlist-list-view)
|
||||
- [Playlist Single](#playlist-item-view)
|
||||
- [Playlist Videos List](#playlist-videos-view)
|
||||
|
||||
**Download queue**
|
||||
- [Download Queue List](#download-queue-list-view)
|
||||
- [Download Queue Single](#download-queue-item-view)
|
||||
|
||||
**Additional**
|
||||
- [Login](#login-view)
|
||||
- [Task](#task-view) WIP
|
||||
- [Cookie](#cookie-view)
|
||||
- [Ping](#ping-view)
|
||||
|
||||
## Authentication
|
||||
API token will get automatically created, accessible on the settings page. Token needs to be passed as an authorization header with every request. Additionally session based authentication is enabled too: When you are logged into your TubeArchivist instance, you'll have access to the api in the browser for testing.
|
||||
|
||||
@ -25,35 +55,17 @@ response = requests.get(url, headers=headers)
|
||||
|
||||
## Pagination
|
||||
The list views return a paginate object with the following keys:
|
||||
- page_size: int current page size set in config
|
||||
- page_from: int first result idx
|
||||
- prev_pages: array of ints of previous pages, if available
|
||||
- current_page: int current page from query
|
||||
- max_hits: reached: bool if max of 10k results is reached
|
||||
- last_page: int of last page link
|
||||
- next_pages: array of ints of next pages
|
||||
- total_hits: int total results
|
||||
- page_size: *int* current page size set in config
|
||||
- page_from: *int* first result idx
|
||||
- prev_pages: *array of ints* of previous pages, if available
|
||||
- current_page: *int* current page from query
|
||||
- max_hits: *bool* if max of 10k results is reached
|
||||
- last_page: *int* of last page link
|
||||
- next_pages: *array of ints* of next pages
|
||||
- total_hits: *int* total results
|
||||
|
||||
Pass page number as a query parameter: `page=2`. Defaults to *0*, `page=1` is redundant and falls back to *0*. If a page query doesn't return any results, you'll get `HTTP 404 Not Found`.
|
||||
|
||||
## Login View
|
||||
Return token and user ID for username and password:
|
||||
POST /api/login
|
||||
```json
|
||||
{
|
||||
"username": "tubearchivist",
|
||||
"password": "verysecret"
|
||||
}
|
||||
```
|
||||
|
||||
after successful login returns
|
||||
```json
|
||||
{
|
||||
"token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
|
||||
"user_id": 1
|
||||
}
|
||||
```
|
||||
|
||||
## Video List View
|
||||
/api/video/
|
||||
|
||||
@ -145,7 +157,7 @@ POST /api/channel/
|
||||
## Playlist List View
|
||||
/api/playlist/
|
||||
|
||||
## Playlists Item View
|
||||
## Playlist Item View
|
||||
/api/playlist/\<playlist_id>/
|
||||
|
||||
## Playlist Videos View
|
||||
@ -192,15 +204,21 @@ Add to queue previously ignored video:
|
||||
DELETE /api/download/\<video_id>/
|
||||
Forget or delete from download queue
|
||||
|
||||
## Ping View
|
||||
Validate your connection with the API
|
||||
GET /api/ping
|
||||
|
||||
When valid returns message with user id:
|
||||
## Login View
|
||||
Return token and user ID for username and password:
|
||||
POST /api/login
|
||||
```json
|
||||
{
|
||||
"response": "pong",
|
||||
"user": 1
|
||||
"username": "tubearchivist",
|
||||
"password": "verysecret"
|
||||
}
|
||||
```
|
||||
|
||||
after successful login returns
|
||||
```json
|
||||
{
|
||||
"token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
|
||||
"user_id": 1
|
||||
}
|
||||
```
|
||||
|
||||
@ -273,3 +291,15 @@ Or returns status code 400 on failure:
|
||||
"cookie_validated": false
|
||||
}
|
||||
```
|
||||
|
||||
## Ping View
|
||||
Validate your connection with the API
|
||||
GET /api/ping
|
||||
|
||||
When valid returns message with user id:
|
||||
```json
|
||||
{
|
||||
"response": "pong",
|
||||
"user": 1
|
||||
}
|
||||
```
|
||||
|
@ -22,7 +22,7 @@ class SearchProcess:
|
||||
self.processed = False
|
||||
|
||||
def process(self):
|
||||
"""dedect type and process"""
|
||||
"""detect type and process"""
|
||||
if "_source" in self.response.keys():
|
||||
# single
|
||||
self.processed = self._process_result(self.response)
|
||||
@ -37,7 +37,7 @@ class SearchProcess:
|
||||
return self.processed
|
||||
|
||||
def _process_result(self, result):
|
||||
"""dedect which type of data to process"""
|
||||
"""detect which type of data to process"""
|
||||
index = result["_index"]
|
||||
processed = False
|
||||
if index == "ta_video":
|
||||
|
@ -363,7 +363,7 @@ class ImportFolderScanner:
|
||||
return new_path
|
||||
|
||||
def get_mp4_thumb_type(self, media_path):
|
||||
"""dedect filetype of embedded thumbnail"""
|
||||
"""detect filetype of embedded thumbnail"""
|
||||
streams = self._get_streams(media_path)
|
||||
|
||||
for stream in streams["streams"]:
|
||||
|
@ -320,15 +320,16 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
|
||||
|
||||
def _get_ryd_stats(self):
|
||||
"""get optional stats from returnyoutubedislikeapi.com"""
|
||||
# pylint: disable=broad-except
|
||||
try:
|
||||
print(f"{self.youtube_id}: get ryd stats")
|
||||
result = ryd_client.get(self.youtube_id)
|
||||
except requests.exceptions.ConnectionError:
|
||||
print(f"{self.youtube_id}: failed to query ryd api, skipping")
|
||||
return False
|
||||
except Exception as err:
|
||||
print(f"{self.youtube_id}: failed to query ryd api {err}")
|
||||
return
|
||||
|
||||
if result["status"] == 404:
|
||||
return False
|
||||
return
|
||||
|
||||
dislikes = {
|
||||
"dislike_count": result.get("dislikes", 0),
|
||||
@ -336,8 +337,6 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
|
||||
}
|
||||
self.json_data["stats"].update(dislikes)
|
||||
|
||||
return True
|
||||
|
||||
def _get_sponsorblock(self):
|
||||
"""get optional sponsorblock timestamps from sponsor.ajay.app"""
|
||||
sponsorblock = SponsorBlock().get_timestamps(self.youtube_id)
|
||||
|
@ -173,13 +173,13 @@ class UrlListParser:
|
||||
_ = self.find_valid_id(youtube_id)
|
||||
return youtube_id, "channel"
|
||||
|
||||
# dedect channel with yt_dlp
|
||||
# detect channel with yt_dlp
|
||||
youtube_id = self.extract_channel_name(parsed.geturl())
|
||||
return youtube_id, "channel"
|
||||
|
||||
@staticmethod
|
||||
def find_valid_id(id_str):
|
||||
"""dedect valid id from length of string"""
|
||||
"""detect valid id from length of string"""
|
||||
str_len = len(id_str)
|
||||
if str_len == 11:
|
||||
id_type = "video"
|
||||
|
@ -10,4 +10,4 @@ requests==2.28.1
|
||||
ryd-client==0.0.6
|
||||
uWSGI==2.0.20
|
||||
whitenoise==6.2.0
|
||||
yt_dlp==2022.8.14
|
||||
yt_dlp==2022.8.19
|
||||
|
@ -1,6 +1,6 @@
|
||||
function initializeCastApi() {
|
||||
cast.framework.CastContext.getInstance().setOptions({
|
||||
receiverApplicationId: chrome.cast.media.DEFAULT_MEDIA_RECEIVER_APP_ID, // Use built in reciver app on cast device, see https://developers.google.com/cast/docs/styled_receiver if you want to be able to add a theme, splash screen or watermark. Has a $5 one time fee.
|
||||
receiverApplicationId: chrome.cast.media.DEFAULT_MEDIA_RECEIVER_APP_ID, // Use built in receiver app on cast device, see https://developers.google.com/cast/docs/styled_receiver if you want to be able to add a theme, splash screen or watermark. Has a $5 one time fee.
|
||||
autoJoinPolicy: chrome.cast.AutoJoinPolicy.ORIGIN_SCOPED
|
||||
});
|
||||
|
||||
|
Loading…
Reference in New Issue
Block a user