simon
6a7a84792e
better feedback when skipping not yet existing index backup
2022-11-20 12:21:25 +07:00
simon
8f5311ff04
delete comments when deleting channel
2022-11-18 11:33:06 +07:00
simon
e590d456a1
delete comments when deleting video
2022-11-18 10:47:48 +07:00
simon
a5be5a719b
reindex comments
2022-11-18 10:31:09 +07:00
simon
31c967cc11
fix linter
2022-11-18 09:46:33 +07:00
simon
2818bd5d52
add comments to video template if available
2022-11-18 09:44:20 +07:00
simon
939cd7fb62
index comments as a DownloadPostProcess
2022-11-18 09:22:58 +07:00
simon
172ced7129
configure comments extraction
2022-11-18 08:59:04 +07:00
simon
79d134cea8
clean format comments date string
2022-11-14 11:59:59 +07:00
simon
f6b6185fb2
rename ta_comment index, implement get comments from es
2022-11-12 12:40:14 +07:00
simon
5b7e3e877b
implement basic comment archive
2022-11-12 11:48:48 +07:00
simon
0f191d8a3e
configure fuzziness in search
2022-11-02 11:59:31 +07:00
simon
dcf317e471
form validate channel pagesize greater than 0, #334
2022-11-02 10:30:21 +07:00
simon
749261c146
take snapshot at 12 localtime
2022-11-02 10:17:27 +07:00
simon
74aa134971
fix next_exec epoch time builder
2022-10-29 17:43:14 +07:00
simon
24b02150a7
handle delete, restore snapshot return values
2022-10-29 17:27:04 +07:00
simon
f717c71c66
implement _parse_single_snapshot, slug compatible naming
2022-10-29 16:54:16 +07:00
simon
1fbd603374
frontend implementation for snapshot integration
2022-10-29 15:49:47 +07:00
simon
749f61aba6
fix empty policy building
2022-10-29 15:32:08 +07:00
simon
59f347c135
implement snapshot startup check
2022-10-28 16:47:50 +07:00
simon
904f449d37
combine snapshot_stats for frontend
2022-10-27 18:40:27 +07:00
simon
e7f960bf46
implement basics snapshot management
2022-10-26 23:17:55 +07:00
simon
725f17bcd2
fix playlist missing channel metadata
2022-10-25 11:13:14 +07:00
simon
f0874b2d02
add timeout for sponsorblock api requests, handle 503
2022-10-23 12:46:10 +07:00
simon
baacd3ee39
better message for running and indexing queue
2022-10-23 12:21:27 +07:00
simon
9cd23c3666
error handeling for playlist_thumbnail extraction
2022-10-23 11:50:26 +07:00
simon
0bba36cbc3
add watched_date for _update_by_query mark as watched, #309
2022-10-22 22:00:35 +07:00
simon
dea2688b49
handle UnidentifiedImageError in ThumbManager, #325
2022-10-17 19:26:01 +07:00
simon
bd7cdb3942
append query parameters to pagination
2022-10-17 18:40:20 +07:00
simon
6eee762d3a
add status for sub refresh
2022-10-14 11:27:27 +07:00
simon
a8c5773f81
fix is_live status check before adding to queue
2022-10-05 16:12:58 +02:00
simon
f28c47323f
cleanup channel info.json file after import
2022-09-19 15:10:41 +07:00
simon
b25818dbc3
extend timeout, add retry to YtWrap
2022-09-17 10:31:51 +07:00
simon
f2a8b070f2
add log output for add to queue
2022-09-17 10:30:27 +07:00
simon
dfacc3039a
fix reindex task, skip inactive in _get_unrated_vids
2022-09-14 16:43:48 +07:00
simon
1c3febc49d
force create thumb folder for manual import
2022-09-06 23:13:29 +07:00
simon
a6dd927860
fix channel extraction to catch all alerts, #312
2022-09-06 23:12:49 +07:00
simon
3c92de8e24
improved logging for manual import
2022-09-06 22:42:55 +07:00
simon
e51a662da9
fix manual import splitext matching invalid extensions, #311
2022-09-06 22:27:18 +07:00
simon
e467beb1c7
fix manual import cleanup metadata, #331
2022-08-26 17:07:55 +07:00
simon
1b22e61384
New search API endpoint, #build
...
Changed:
- Switched search form to API
- Additional validations for RYD
- Additional log output for curl ES startup check
2022-08-19 18:00:35 +07:00
57a9fff82b
Add API endpoint for searching ( #303 )
...
* Add API endpoint for searching + use it in frontend
* Fix linting warnings
* Remove multisearch API call
* Avoid 301 and fix up multiline comment
2022-08-19 17:52:37 +07:00
simon
1896e982b9
codespell lint fixes
2022-08-18 18:51:11 +07:00
simon
0e7e0a0560
fail open on ryd error
2022-08-18 18:45:46 +07:00
simon
0210a97b48
archive vtt files for offline import
2022-08-16 15:42:54 +07:00
simon
0b41d288db
fix mkv cover stream extractor
2022-08-16 12:19:27 +07:00
simon
21f3ee88b9
extract vtt from mkv and mp4 for import
2022-08-15 15:18:38 +07:00
simon
7727f533ee
move subtitle to separate module
2022-08-15 13:55:24 +07:00
simon
bd4710ebdc
improved wording for download form, subscription size, #300
2022-08-13 15:43:23 +07:00
simon
6afe25a4af
fix empty subtitle download and index, #288
2022-08-12 15:53:20 +07:00
simon
cc40f46325
improved ryd error handeling, #283
2022-08-12 14:39:22 +07:00
simon
7c3bfa94f3
handle thumbnail download base exception, #281
2022-08-12 12:58:39 +07:00
simon
474ab22792
handle channel name redirect in UrlListParser, #276
2022-08-12 11:53:31 +07:00
simon
0891c49620
implement offline channel import
2022-08-11 19:57:00 +07:00
simon
c26ef012c1
raise ValueError when not matching media files
2022-08-11 11:10:58 +07:00
simon
31061c0eb0
handle offline import without info.json fail
2022-08-10 22:38:17 +07:00
simon
994f7c2443
use user provided thumb for offline import
2022-08-10 22:31:15 +07:00
simon
7029441f89
use fallback thumb when no url passed
2022-08-10 22:24:02 +07:00
simon
35f1084cc2
download thumbnail for manual import
2022-08-10 21:47:26 +07:00
simon
44473a364f
fix double download of playlist art
2022-08-10 21:20:58 +07:00
simon
fb4d6b7be3
major refactor ThumbManager
2022-08-10 21:03:54 +07:00
simon
b76fa69396
WIP: rewrite ManualImport, identify and archive
2022-08-08 19:18:27 +07:00
simon
8f711d359b
implement youtube_meta and media_path overwritein YoutubeVideo
2022-08-08 18:28:32 +07:00
simon
9fd6f9c58b
use absolute paths for manual import
2022-08-08 15:57:42 +07:00
simon
b70cbfa0a8
convert thumbnail to jpg for manual import
2022-08-08 14:52:33 +07:00
simon
195acd3246
WPI: dump thumbnails from mp4 and mkv
2022-07-31 00:02:09 +07:00
simon
61da0c8af3
WIP: extract thumb from mkv
2022-07-30 21:07:31 +07:00
simon
26cc7846c6
WIP: new import folder parser for offline import
2022-07-26 19:51:47 +07:00
simon
4b1ebf44c9
handle filenotfounderror for manual cookie import
2022-07-21 23:01:32 +07:00
simon
eb229440d4
load config into redis on fresh install at startup
2022-07-21 23:01:01 +07:00
simon
0ec6558282
jump to timestamp from full search
2022-07-21 17:15:36 +07:00
simon
55a73995c8
implement source and lang keywords for full text search
2022-07-21 15:39:27 +07:00
simon
e4dd47aed0
implement fulltext search keyword
2022-07-20 21:49:32 +07:00
simon
eb5a7338ab
search with and operator, increase result size
2022-07-20 18:25:39 +07:00
simon
b520a9bfb0
boost title result scores in multi_match
2022-07-20 16:20:23 +07:00
simon
242ca19586
return query_type from SearchParser
2022-07-20 16:10:57 +07:00
simon
6a6665bdf9
implement simple, aka none keyword query
2022-07-20 15:04:10 +07:00
simon
8e8c00af40
implement playlist keyword search, fix channel subscribed keyword
2022-07-20 14:58:04 +07:00
simon
ae3cf7eb80
add channel keyword search
2022-07-20 14:48:41 +07:00
simon
0a2b6ee90a
add keyword search framework, implement video search
2022-07-19 19:23:59 +07:00
simon
cc2fff8bef
use info_dict for message, simplify cache template, #271
2022-07-18 11:37:43 +07:00
simon
b406250ae2
use shutil.copyfile for shutil.move calls
2022-07-17 07:47:27 +07:00
simon
393e00866b
Desc preview, nginx fix, #build
...
Changed:
- Requires new env var: TA_HOST, hostname of your application
- Showing preview for description text, by @p0358
- Fix nginx user permission, by @p0358
- Refactor IndexPaginate to take loop callback
- Fix backup for large index using loop callback
- Fix deactivating missing channels on reindex
2022-07-16 22:11:29 +07:00
79996f6838
Fix permission/mount issues from #268 ( #270 )
...
* make nginx run as root within the container (#268 )
* move file to videos path with fallback to regular copy with no metadata copying (#268 )
* use `copy_function=shutil.copy` everywhere with `shutil.move`
2022-07-16 21:54:39 +07:00
simon
6bb7f80ea2
Refactor IndexPaginate
...
- add callback function
- implement callback for ElasticBackup
2022-07-16 17:55:18 +07:00
simon
91bccfd057
deactivate non existing channels on reindex
2022-07-12 11:41:03 +07:00
simon
8cca09e0ac
remove ta_json backup file, keep only es_ndjson
2022-07-07 12:34:57 +07:00
simon
8c07932071
delete cookie:valid key when revoking
2022-06-23 19:21:13 +07:00
simon
74d74d95a1
store validation result, return in API
2022-06-23 06:48:55 +07:00
simon
499c47c7a1
move set_cookie to separate method for reusabliity
2022-06-21 08:01:33 +07:00
simon
2cf30e1127
refactor: default set_message in RedisArchivist to True
2022-06-16 10:37:46 +07:00
simon
30abbe9be7
rewrite cookie into redis from io stream, auto validate
2022-06-15 17:54:05 +07:00
simon
2ceb1b701a
consolidate config of cookie revoke
2022-06-15 15:39:24 +07:00
simon
2ad093a9a8
handle cookiejar.LoadError dont import invalid cookie
2022-06-15 10:39:48 +07:00
simon
09a94d0df5
set relative values directly with path arg
2022-06-15 10:27:29 +07:00
bbb16bb3c2
Update TaskAPIView with GET - #257 PR ( #258 )
...
* Initial commit for GET response for TaskAPIView.
* Update for missing space - linting issue.
* Additional linting fixes.
* add is_locked method to check if lock is set
* Update to use `.is_locked` method and add decorator.
* Fix linting issue.
* Fix doubled "rescan" call for locked file.
Removed call to ".owned()" under "is_locked".
* Commenting out GET call in TaskAPI view to revert change for testing.
* Commenting is_locked function to see if Server Response 500 is stopped.
* Reassert is_locked function within RedisArchivist class.
* Create test GET responder.
* Reverting simple GET response change.
* Reapplying simple GET response for `/api/task/`.
* Reapplying change with modification to lock key.
* Documentation update for new GET calls.
* README: fix returned value as `bool` instead of `str`.
* Updating the `is_locked` key reference to "rescan" and "downloading" to fix which key is being retrieved.
* Use dictionary keys, rather than line-by-line statements.
* Fix typo in README for Task View.
Co-authored-by: simon <simobilleter@gmail.com>
2022-06-15 09:15:59 +07:00
simon
c41f0a7cc4
try to LOAD_TRUNCATED_IMAGES, #256
2022-06-01 09:04:55 +07:00
simon
5d262d7d48
skip cookie if no config added to YtWrap
2022-05-30 18:22:17 +07:00
simon
231ac1e012
make channel matching optional for reusability
2022-05-30 17:59:52 +07:00
simon
bbb0c26498
fix subtitle delete overwrite, change filename to .lang.vtt, #195
2022-05-29 11:43:25 +07:00
simon
9f92bf95b4
mark video as downloaded false when deleting video
2022-05-28 21:23:56 +07:00
simon
e638f8b487
skip subtitle segment without segs key, #249
2022-05-28 17:47:21 +07:00
simon
caaed252f9
specify videos page for get_last_youtube_videos of channel
2022-05-28 14:34:01 +07:00
simon
5982932457
implement variable grid row size
2022-05-28 13:57:29 +07:00
simon
9a31243b1c
log message when skipping video
2022-05-28 11:11:09 +07:00
simon
88e526f5e5
refactor new YtWrap class for yt-dlp calls, #build
...
Changed:
- new yt-dlp base class
- new cookie class using io_stream
- MetadataFromField postprocessors
2022-05-25 16:15:38 +07:00
730ac15349
Fix #241 Use title/uploader for embedded metadata not Content ID ( #252 )
2022-05-25 16:13:25 +07:00
simon
0ac9243389
remove unused monitor_cache_dir method
2022-05-24 15:54:41 +07:00
simon
c39f8a9cb8
refactor use cookie io_stream
2022-05-24 15:51:58 +07:00
simon
9dd1e9a29f
extract playlists with new YtBase class
2022-05-24 10:44:18 +07:00
simon
3f5104d6d8
integrate new YtWrap class in download queue and subscriptions classes
2022-05-24 10:36:39 +07:00
simon
4b00f454b7
integrate new YtWrap class in yt_dlp handler
2022-05-24 10:19:02 +07:00
simon
33f27f018f
handle download error
2022-05-24 10:18:33 +07:00
simon
a7d9b4584b
implement new YtBase class for index classes
2022-05-24 10:01:30 +07:00
simon
4e699621a0
obs base and handle extract error
2022-05-24 10:00:40 +07:00
simon
37e6f8656a
yt-dlp base class
2022-05-24 08:53:05 +07:00
simon
3722f11a65
cleanup cookie file after task
2022-05-07 09:34:51 +07:00
simon
087043811f
avoid unneeded cookie open
2022-05-07 09:34:33 +07:00
simon
94fa191fb8
skip premium videos, clean exit for empty bulk_list, #237
2022-05-05 23:38:10 +07:00
simon
9a2b29c2ae
implement channle name change in reindex, #211
2022-05-05 22:59:56 +07:00
simon
7a51368955
fix skip deleting none existing subtitles, add update_media_url
2022-05-05 22:56:22 +07:00
simon
7fa0d0f331
untangle delete subtitles from delete media file, #195
2022-05-05 09:11:37 +07:00
simon
1613b71942
hide cookie after rescan and download process
2022-05-03 17:32:32 +07:00
simon
83beb53860
simplify _add_single to call update_status in PendingInteract directly
2022-05-02 21:38:03 +07:00
simon
8a4c50779a
standardize prints for PostData mapper
2022-05-02 18:39:54 +07:00
simon
ca5b00a373
refresh for PendingInteract delete to avoid race condition, #217
2022-05-02 18:20:56 +07:00
simon
34a1fe9e8e
fix thumb re-embed task, #231
2022-05-02 17:38:20 +07:00
simon
34c18e75f5
add LL and WL as valid playlists with cookie
2022-05-02 11:30:38 +07:00
simon
6fbac12314
fix channel_validate in PlaylistSubscription, #223
2022-05-02 11:27:50 +07:00
simon
ca3ea20283
add socket_timeout when passing check_formats to yt-dlp
2022-05-02 09:48:52 +07:00
simon
bc7d90f1f4
raise FileNotFoundError when trying to use unavailable cookie
2022-04-30 19:09:19 +07:00
simon
3a21690309
implement post_process_updated for settings, import cookie
2022-04-30 18:35:04 +07:00
simon
487c81d50b
fix FileNotFoundError in cookie hide
2022-04-30 18:34:28 +07:00
simon
088e87bccf
add cookiefile to PendingList and VideoDownloader
2022-04-30 17:27:57 +07:00
simon
6cc5fd8890
add cookiefile to yt_obs in YouTubeItem baseclass
2022-04-30 16:40:36 +07:00
simon
4e3d71e0e3
simplify, un-userspaced
2022-04-30 16:38:23 +07:00
simon
9b107b9940
implement cookie validator
2022-04-30 13:05:01 +07:00
simon
43eec5aac0
move CookieHandler to separate module
2022-04-30 12:05:41 +07:00
simon
430fdb6f1c
add cookie import to frontend
2022-04-29 18:38:43 +07:00
simon
1fa26cdc44
added basic cookie import functionality
2022-04-23 21:59:59 +07:00
simon
04fc6ed26a
API: add pagination
2022-04-20 22:43:07 +07:00
simon
3147df20da
skip subtitle segments without duration, take 2
2022-04-18 11:52:13 +07:00
simon
50f4e0e5b7
skip base64 generator, hotfix
2022-04-16 13:08:32 +07:00
simon
59f1c111aa
extend timeout for sleep interval
2022-04-16 08:58:35 +07:00
simon
fe610fdaca
add log output for backup progress
2022-04-15 20:41:38 +07:00
simon
c84fbdfb9c
fix _check_get_sb for per channel deactivate
2022-04-15 20:04:01 +07:00
simon
7231b5d245
implement unset of sb per channel overwrite
2022-04-15 20:03:15 +07:00
dc29c6718c
Fix : #210 - Download UI Box being removed when file is still processing ( #219 )
...
* Fix: https://github.com/bbilly1/tubearchivist/issues/210 - Download UI element is removed when the file has finished downloading, but is still being moved on disk to the storage destination. This means the user is presented with nothing in the UI yet the file is still being processed.
* FIX: Resolving linting issue
* FIX: Set moving message to never expire and then a second message after the move with a 4 second expiry timer.
Co-authored-by: Rob Ainsworth <roba@immjsystems.com>
2022-04-15 15:21:24 +07:00
simon
d4b1d97f5c
add mappings for sponsorblock segments
2022-04-15 15:16:31 +07:00
simon
f71644c8bc
guarantee stats in mapping for sort
2022-04-15 15:03:43 +07:00
simon
4853a8b12b
limit video_daily to below 10k in daily refresh
2022-04-14 16:18:08 +07:00
simon
fd00369859
fix last page pagination link building error, #221
2022-04-14 16:04:21 +07:00
simon
a8a7edb93e
skip subtilte events without duration, #196
2022-04-14 15:53:10 +07:00
simon
3007e02fe5
fix per channel sb integration
2022-04-13 15:53:00 +07:00
simon
8edde732b6
improved sponsorblock key with additional metadata
2022-04-12 17:47:13 +07:00
simon
3df1df8b5a
set explicit mappings for sponsorblock key
2022-04-12 17:45:06 +07:00
simon
31378ac756
better generic schedule validator for impossible input, #209
2022-04-10 15:58:11 +07:00
simon
9d73dbc45a
implement video overwrites index for sponsorblock
2022-04-08 17:19:25 +07:00
simon
365ebf53a5
implement per channel sponsorblock
2022-04-08 17:17:39 +07:00
simon
fe3e3cfacc
fix filenotfounderror for missing subtitles when deleting video
2022-04-08 15:56:34 +07:00
simon
a07d789e66
add base64 blur video thumb
2022-04-08 00:29:09 +07:00
simon
6a1cb15114
validate hours to not be greater than 23, #209
2022-04-07 23:02:07 +07:00
simon
b1a7a6a148
use cleaned_data for config form parser
2022-04-07 22:30:20 +07:00
simon
03dd25cff4
implement vote on sponsorblock segments api
2022-04-05 22:25:40 +07:00
simon
bace7d41af
add random string generator helper function
2022-04-05 21:51:10 +07:00
simon
738b083a7f
create dedicated SponsorBlock class to handle integration
2022-04-05 21:50:40 +07:00
simon
a8b62e5cdc
better error message on sponsorblock fail
2022-04-04 23:28:32 +07:00
simon
730d12212f
implement basic sponsblock indexing, #119
2022-04-04 23:28:32 +07:00
simon
cdc617c382
add datestr generator
2022-03-29 16:47:21 +07:00
simon
f15bb50cfe
fix playlist reindex after PendingList refactor
2022-03-26 18:31:00 +07:00
simon
fcadb5ead8
add random headers for requests outside of yt-dlp
2022-03-26 11:49:53 +07:00
simon
a0ed88580f
fix index validate_mappings for new nested object
2022-03-26 10:52:57 +07:00
simon
8d1d09e698
extend IndexPaginate to optionally return _source key
2022-03-23 17:17:42 +07:00
simon
fb4c11bd38
refactor ElasticIndex and ElasticBackup to use ElasticWrap and IndexPaginate
2022-03-23 17:17:17 +07:00
simon
bfcc538ed1
fix concurrency issue with subtitle indexing
2022-03-23 16:20:33 +07:00
simon
912c19f6cf
use ElasticWrap in FilesystemScanner
2022-03-23 15:56:53 +07:00
simon
fda520ad44
refactor and consolidate Reindex class
2022-03-23 15:48:38 +07:00
simon
1f7d6871cf
use ElasticWrap to delete item from pending
2022-03-23 11:09:21 +07:00
simon
5b1c9c64de
cleanup, use ElasticWrap
2022-03-23 09:44:31 +07:00
simon
9f5c9b17a5
refactor redis connection, fix sync_redis_state task setup issue
2022-03-22 17:50:54 +07:00
simon
75cd9d382d
fix duration builder on reindex by ignoring vtt files
2022-03-22 12:59:39 +07:00
simon
1f1dfcb54f
better playlist scan message, private methods in YoutubeChannel class
2022-03-22 10:07:33 +07:00
simon
3d451e47e5
full playlist refresh and index for channel overwrites
2022-03-22 09:42:41 +07:00
simon
91a9477bba
refactor index_channel_playlists task, move to YoutubeChannel class
2022-03-21 17:59:21 +07:00
simon
fc311a7b7e
refactor validate_playlists and move into DownloadPostProcess class
2022-03-21 11:14:44 +07:00
simon
f803c5298b
fix missing update_status method call in ignore video
2022-03-21 11:09:16 +07:00
simon
7c6041d686
refactor VideoDownloader and implement per channel delete older than
2022-03-19 17:37:28 +07:00
simon
1411262793
remember channel overwrites on rescan, remove unused rescape_all_channels method
2022-03-19 15:12:29 +07:00
simon
9fd7e2b11a
rename channel_overwrites attribute
2022-03-19 15:03:38 +07:00
simon
3ba6e0478d
fix filesysem rescan function
2022-03-19 14:36:46 +07:00
simon
b68a00073f
fix thumbnail validation with new PendingList class
2022-03-19 14:27:37 +07:00
simon
89428297c9
add download format channel overwrite per video
2022-03-18 22:22:41 +07:00
simon
ac9df4e082
map channel overwrite to video id for later efficient mapping
2022-03-18 21:39:33 +07:00
simon
a6937db5fd
use the refactored PendingList class
2022-03-18 18:27:25 +07:00
simon
05b8dbc02f
refacor PendingList class into subclasses
2022-03-18 17:19:21 +07:00
simon
1498fadf27
implement channel playlist index overwrite
2022-03-16 12:32:42 +07:00
simon
f0e82caebb
add bool true to channel overwrite form parser
2022-03-16 12:32:02 +07:00
simon
2ef8823c2d
better channel overwrite update form
2022-03-16 12:09:50 +07:00
simon
544d842a4a
rename channel format overwrite field
2022-03-15 17:45:15 +07:00
simon
774220ab1f
add channel overwrite form to frontend, store in es
2022-03-15 12:00:48 +07:00
simon
PrivateGER
p0358
lamusmaser
Clark
Ainsey11