tubearchivist
/
tubearchivist
mirror of https://github.com/tubearchivist/tubearchivist.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
846 Commits 3 Branches 37 Tags 16 MiB
Commit Graph

316 Commits

Author SHA1 Message Date
simon 160676acc1
add mapping for channel_tvart_url 2022-01-27 21:56:48 +07:00
simon 87b72a571d
simplify reading json files 2022-01-27 15:32:58 +07:00
simon b6f9fb58ad refactor VideoDownloader class, better obs builder 2022-01-26 19:37:36 +07:00
simon 2fc0cbacf5
update doc strings to represent new module structure 2022-01-23 19:32:08 +07:00
simon 0fc0cc8e87
major refactor, split up modules 2022-01-22 22:13:37 +07:00
simon b64a2f8195
squash index.py refactor commits 2022-01-22 17:48:54 +07:00
simon 2f4d4e715b
refactor index class in to base and video classes 2022-01-19 13:04:15 +07:00
simon 68f19b1719
make SearchHandler use new ElasticWrap class 2022-01-18 13:37:22 +07:00
simon 3eb0353fa9
new base class to make all es calls from 2022-01-18 13:36:41 +07:00
simon af2783c18a
remove unneeded auth 2022-01-18 13:35:48 +07:00
simon c703105c5e
fix ryd 404 error for reindex and deactivate 2022-01-11 18:54:40 +07:00
simon 8ba9ced833
move ryd to avoid celery missing redis var 2022-01-08 21:10:38 +07:00
simon 1a95797997
improve scheduler validation with regex, #126 2022-01-08 20:04:51 +07:00
simon 1cbe0bbfc0
delete playlists of channel when deleting channel, #118 2022-01-05 15:39:41 +07:00
simon 49cd272557
size limit for ryd missing rating query 2022-01-05 15:10:33 +07:00
simon 1c27934d77
remove redundant lock 2022-01-05 14:51:02 +07:00
simon c85be45846
extend ryd to refresh when missing average_rating 2022-01-05 14:43:24 +07:00
simon 3999d679e3
fix linting 2022-01-05 14:09:35 +07:00
simon dc2fb65312
add optin returnyoutubedislike.com integration 2022-01-05 14:06:07 +07:00
simon 6aef3dfb96
remove leftovers 2022-01-05 11:10:11 +07:00
simon fd334bcafe
fix legacy import alias for yt-dlp 2022-01-02 15:38:45 +07:00
simon 4915aa0c11
remove now redundant search forms 2021-12-31 13:42:07 +07:00
simon e8eb7077ed
add initial search endpoint and improve results$ 2021-12-30 20:42:42 +07:00
simon 5fb15b79a4
extend SearchForm class with multisearch method for frontend 2021-12-30 15:58:15 +07:00
simon 2a774c3d3e
add english analyzer for main text fields for serching 2021-12-30 15:54:20 +07:00
simon 88ebb6d648
fix autodelete error on empty download queue 2021-12-18 17:05:49 +07:00
simon ad8d75c3e8
add autodelete videos to ignore list, #112 2021-12-17 23:28:53 +07:00
simon a500f9ec63
add notifications and error messages to settings page 2021-12-17 15:09:21 +07:00
simon e858bba944
fix linting, update doc string 2021-12-16 12:48:12 +07:00
simon 9a2dead76c
implement autodelete watched videos after x days, #56 2021-12-16 12:44:37 +07:00
simon 7924f28ccb
implement backup rotate from config value 2021-12-14 19:40:46 +07:00
simon f7c73f7eba
change backup and restore to tag files 2021-12-14 19:05:58 +07:00
simon 6ad1c8e73b
add reason to backup as file name tag 2021-12-13 21:20:48 +07:00
simon 21f70493bb
add scheduler config to regular redis config 2021-12-13 20:28:41 +07:00
simon 5cb6076a1c use config check_reindex_days for refresh_interval 2021-12-13 19:19:49 +07:00
simon 8c5bb0bc1f
add missing mapping for playlist_last_refresh to fix empty index scan, #101 2021-12-06 16:56:56 +07:00
simon 82a71ed75d
clean up playlist subscribe notification 2021-12-05 22:35:46 +07:00
simon 8874168291
clean up progress hook title and speed up playlist validation 2021-12-05 21:57:33 +07:00
simon 19871d55e2
faster release message timer with reduced redis exipre time on last message 2021-12-05 21:27:43 +07:00
simon 67bde4c7ec
fix notification channel for video thumb download 2021-12-05 17:24:57 +07:00
simon 1d262fc936
fix old status variables for message 2021-12-05 16:41:06 +07:00
simon b35440253d
transform add to queue, rescan, and download progress to channel messages 2021-12-03 19:47:43 +07:00
simon 1691bdadf5
transform progress message to list, change subscribe messages 2021-12-03 19:00:26 +07:00
simon 5a2d7c07bf
variable expire value and simpler REDIS_PORT declaration 2021-12-03 13:51:34 +07:00
simon 0b0634273a
validate form entries, set auto sensible default 2021-12-03 11:27:26 +07:00
simon b2861e0369
ScheduleBuilder config class for celery beat 2021-12-02 18:37:54 +07:00
simon 7e97284374
auto index playlists when adding to download 2021-11-29 15:49:45 +07:00
simon b14a820a4c
add 18 char len to valid playlists 2021-11-29 14:21:19 +07:00
simon eacfd4bc84
refactor and move frontend PostData class to dedicated module 2021-11-28 18:42:04 +07:00
simon d95bf4272c
speedup by only testing selected formats 2021-11-27 14:13:48 +07:00
simon 295ea0cde0
handling deactivating playlist 2021-11-27 11:47:12 +07:00
simon 1f82f0c40d
index playlist channel if doesn't exist and beter user feedback 2021-11-26 17:34:59 +07:00
simon 31515961e1
fix prev and nex playlist order 2021-11-26 16:47:39 +07:00
simon 5d7a609285
skip backing up not yet existing index 2021-11-26 16:19:18 +07:00
simon 32d2659658
handling genering index write errors, #91 2021-11-25 22:10:12 +07:00
simon 3f31e50b69
add delete download queue pending or ignored, #85 2021-11-25 20:02:25 +07:00
simon 2ebd0a3e2f
extend unsubscribe for channels and playlists 2021-11-25 11:52:14 +07:00
simon e3f9bd98de
use --check-formats for downloader, #90 2021-11-25 11:17:25 +07:00
simon b1783d6be3
handle multifeed videos with noplaylist 2021-11-22 22:58:25 +07:00
simon 9fceb98f13
add playlists to reindex class 2021-11-22 20:40:11 +07:00
simon b292a5e309
add delete playlist backend functionality 2021-11-20 14:32:36 +07:00
simon 8d708fcdc8
add mark_playlist_watched and refactor mark_channel_watched to update_by_query 2021-11-20 10:58:25 +07:00
simon fef26a06b2
force refresh for new videos to avoid racing condition 2021-11-19 16:26:05 +07:00
simon 3b6e70b4dc
progress message on validate_playlists 2021-11-19 15:53:06 +07:00
simon 3881d305f9
add subscribe to playlist form and better task management 2021-11-19 11:52:27 +07:00
simon fcddf85245
validate_playlists after run_queue from VideoDownloader 2021-11-18 22:35:08 +07:00
simon d0e0903d48
use remote data for playlist find_missing 2021-11-18 21:50:35 +07:00
simon 3efdc1962d
fix playlist nav builder using correct index 2021-11-18 21:16:18 +07:00
simon a1167bac4e
use same page size for playlist rescan 2021-11-18 16:58:39 +07:00
simon bb889f7f67
use find_missing to also add missing videos from playlist to download queue 2021-11-18 16:37:05 +07:00
simon 1af0196208
add subscribe to playlist backend functionality 2021-11-18 16:00:55 +07:00
simon 8b69be645c
use new IndexPaginate class for get_channels and get_playlists 2021-11-18 14:48:31 +07:00
simon f371a03cc7
use new IndexPaginate class for get_all_indexed videos 2021-11-18 12:16:21 +07:00
simon 7b36cc53e7
refactor get_all_pending to use new IndexPaginate class 2021-11-17 18:30:24 +07:00
simon ed9d5aef0a
fix playlist get_entries when not passing all youtube_ids 2021-11-17 18:16:22 +07:00
simon c4b0f900f8
dedicated search_after class to scroll through index 2021-11-17 17:54:47 +07:00
simon 2fe8936e85
add playlist position index 2021-11-15 13:00:17 +07:00
simon e450b38702
add video playlist navigation, previous and next 2021-11-14 19:34:57 +07:00
simon 8932a6bc02
validate existence of playlist entry in index 2021-11-14 15:46:24 +07:00
simon ac14cd5ebb
playlist relation take 2 2021-11-13 17:34:58 +07:00
simon 7930da5242
fix youtu.be extractor, #40 2021-11-12 19:26:47 +07:00
simon b1c82759bb
extend playlist_id with metadata and links 2021-11-12 15:01:39 +07:00
simon 3485f9b1c8
define playlist field types in index 2021-11-11 20:57:28 +07:00
simon c499a130da
add sync playlist meta data and position to video index 2021-11-11 17:56:29 +07:00
simon 5ef9c5b059
download playlist artwork 2021-11-10 17:55:34 +07:00
simon eaf107d0d2
add new ta_playlist index and task to find playlists of channel 2021-11-08 14:54:17 +07:00
simon bd37b653c2
extend playlists to get all playlists of a channel and upload single playlist to index 2021-11-06 22:30:39 +07:00
simon 15845da3a8
use check_formats to validate thumb url, #83 2021-11-06 20:23:59 +07:00
simon 72bc6a4d02
basic methods of YoutubePlaylist class 2021-11-05 16:24:05 +07:00
simon b2a4f88980 add re_sync_thumbs to frontend 2021-11-01 16:42:07 +07:00
simon 1f0e305527 add re_sync_thumbs backend functionality 2021-11-01 16:20:21 +07:00
simon 283a72308e handle deleting channel without videos 2021-10-31 17:30:29 +07:00
simon 164ee5c523 more valueerrors in url extraction 2021-10-31 17:05:13 +07:00
simon 1ba3090db1 rewrite url_str extractor to convert channel names into channel ids, #40 2021-10-31 16:04:28 +07:00
simon caff9ce189 standardize parse_url_list to only return ids 2021-10-30 17:51:50 +07:00
simon cc90e437a9 show progress message for thumbnail download 2021-10-30 17:06:54 +07:00
simon 29fc5fbda2 improved progress message for frontend feedback 2021-10-30 17:06:22 +07:00
simon 20aab04fbf fix keyerror 2021-10-30 16:35:10 +07:00
simon a8ded25b35 userspace color 2021-10-29 23:43:19 +07:00
simon a4397b5204 validate settings form and userspace archive page_size 2021-10-29 22:37:31 +07:00
simon 254c518505 basic auth for elasticsearch 2021-10-28 15:49:58 +07:00
simon ebd8368856 namespace redis keys to ta: 2021-10-27 18:07:35 +07:00
simon 5f3fc460c1 implementing throttled-rate option 2021-10-26 17:14:26 +07:00
simon 58efe64c5d make chown command optional by omitting HOST_UID and HOST_GID, #58 2021-10-20 18:41:39 +07:00
simon de2371c43c handle running empty dl queue, #55 2021-10-17 11:28:05 +07:00
simon 1a88c85436 skip video if is_live now 2021-10-17 10:49:02 +07:00
simon 63be6afcfe allow duration_sec extraction to fail cleanly for add to queue 2021-10-17 09:40:46 +07:00
simon 08338f3270 add thumbnail download for manual import 2021-10-16 17:03:39 +07:00
simon 0c3c1b650a disable notifiation on rescan, fix keyerror issue in missing_thumbs build 2021-10-15 13:16:35 +07:00
simon fd2fa628e3 make sure to get both banner and icon 2021-10-14 22:48:23 +07:00
simon ba12c2ad25 simple progress message for thumbnail bulk download 2021-10-14 21:28:27 +07:00
simon 0d061d8601 get art work for new subscribed channels 2021-10-14 17:47:07 +07:00
simon 0844f7b91a add delayed retry for non 404 thumb download error 2021-10-14 14:44:50 +07:00
simon 044b12d5a5 remove now unused thumbnail methods from searching module 2021-10-14 14:42:35 +07:00
simon ce5f8b6a38 cleanup method for unused thumbs 2021-10-14 11:59:30 +07:00
simon 9ad42dfa38 add artwork refresh on rescan 2021-10-14 11:18:32 +07:00
simon 72af95acd8 fallback channel and video artwork for 404 errors 2021-10-13 19:34:16 +07:00
simon 29f17680bf consolidate delete thumbnails into ThumbManager class 2021-10-11 19:29:27 +07:00
simon 9f12966ef7 bumping yt_dlp version, requires additional noprogress option 2021-10-11 16:34:19 +07:00
simon 35a5eda36e downloading channel artwork after scraping 2021-10-11 16:03:25 +07:00
simon e202265a5f extended validate_thumbnails for channel art 2021-10-11 14:27:48 +07:00
simon bc84696792 new thumbnails module and new cache layout 2021-10-11 13:33:28 +07:00
simon b53a1763c8 better channel folder path builder with clean_string 2021-10-09 20:32:42 +07:00
simon 11780a0ff0 add delete channel backend 2021-10-09 19:54:36 +07:00
simon 7405e960e5 delete video backend functionality 2021-10-08 16:18:01 +07:00
simon 8c97e6786a chown to fix permissions on new channel folders 2021-10-08 15:16:02 +07:00
simon 64ccd3830e implementing filesystem rescan to clean index 2021-10-08 14:56:07 +07:00
simon 0b88fd8b1f implement watched-unwatched toggle, #39 2021-10-07 23:38:17 +07:00
simon 744780f4bd add optional EmbedThumbnail postprocessor 2021-10-07 22:28:32 +07:00
simon 4d348955a3 allowing custom redis port 2021-09-30 18:03:23 +07:00
simon f5a56fca86 refactored redis functions into dedicated RedisArchivist class 2021-09-28 10:33:00 +07:00
simon 3d79f08311 handle return value of del_message 2021-09-27 17:44:42 +07:00
simon 11067094b2 implement os.listdir sanitizer for hidden files, #30 2021-09-25 18:59:54 +07:00
simon 47020e0cfa implement kill function for dl queue 2021-09-24 23:37:26 +07:00
simon f53391c1bb add limit_queue setting back, make buttons only show up while downlaoding 2021-09-24 21:27:53 +07:00
simon 91a57cc780 fix duplication issue with download_now 2021-09-24 16:58:48 +07:00
simon 78d8bc9d24 changed dl queue to redis, changed sort order to represent actual queue order 2021-09-23 18:09:46 +07:00
simon 7f57cabbc6 initial dynamic dl queue in redis 2021-09-23 16:58:47 +07:00
simon 214a248821 better error handeling in add to download form 2021-09-23 10:50:44 +07:00
simon ecae40d502 refactor PostData class 
* Split up into SearchForm class as part of searching module
* Split up into WatchState class as part of index module
 2021-09-22 11:43:38 +07:00
simon 2433e0e7d8 linting everything in black 2021-09-21 16:25:22 +07:00
Daniel Kempkens 69e6e490f4
Optionally embed metadata into video files (#21) 
This adds basic `postprocessors` support and adds a setting to
optionally enable `FFmpegMetadata`.

It's the same as passing `--add-metadata` and `--add-chapters` via the
`yt-dlp` CLI.
 2021-09-21 10:12:33 +07:00
simon c17e2e6dd1 skip post_bulk_restore if empty 2021-09-20 19:10:39 +07:00
simon 4b9acd4c51 load new defaults from config.json #21 2021-09-20 17:45:01 +07:00
simon ef64100d8f flake8 and isort linting 2021-09-18 20:02:54 +07:00
simon cbedfe2acb running codespell #15 2021-09-18 17:28:16 +07:00
simon eafda731c9 restoring indexes from zip archive 2021-09-18 17:10:16 +07:00
simon f9c09d215e sorting on _id is depreciated, using youtube_id instead 2021-09-17 12:52:00 +07:00
simon e53cfc5a32 backup es db to zip file 
backup es db to zip file
 2021-09-16 17:47:01 +07:00
simon f105a9d9d8 fix last_reindex diff calculation 2021-09-15 22:21:12 +07:00
simon fe33e9178c allowing to subscribe to multiple channels at once 2021-09-15 21:09:46 +07:00
simon 2a256bbd06 raise ValueError if url extraction failes 2021-09-15 17:25:18 +07:00
simon 89c99883c8 make sure youtube_id gets written to cache file 2021-09-15 17:14:17 +07:00
simon 82a91308f5 raise ValueError if vid metatdata extraction failes for new videos 2021-09-15 17:13:10 +07:00
simon f8b5b7c5bb fixing cookie consent issue for EU countries 2021-09-14 19:49:05 +07:00
simon 227183189c fixed weird issue by forcing RGB vid thumbnail 2021-09-14 18:27:38 +07:00
simon 7f6732422b media files import with lock to force single run 2021-09-14 18:25:28 +07:00
simon f60dbde3dc manual video file import celery task 2021-09-13 22:17:36 +07:00
simon 3c003750a8 fixed the last_reindex interval calculation 2021-09-11 19:36:19 +07:00
simon 3376412c8f appending reindex task to rescan subscriptions 2021-09-11 17:55:44 +07:00
simon adafa98568 immediate feedback when adding long lists to queue 2021-09-10 15:21:46 +07:00
simon e3afe3947c allowing to add playlist to queue 2021-09-10 15:07:38 +07:00
simon 641698c350 moove limit_count functionality into VideoDownload class to avoid having to reload the celery worker on config change 2021-09-09 12:28:25 +07:00
simon 0369f4d574 changed add_player function of in preparation for scan_filesystem 2021-09-08 12:32:53 +07:00
simon 6510b5af15 autobackup before making any changes, fixing number_of_replicas datatype 2021-09-08 12:31:33 +07:00
simon dc9fd72cde minimal viable product 2021-09-06 00:10:14 +07:00