|
|
bb84940a7a
|
slowly updating documentation with rdoc stuff so we can generate yard server
|
2020-01-02 01:19:10 -06:00 |
|
|
|
606efdf3b2
|
more rdoc stuff
|
2019-11-13 02:06:08 -06:00 |
|
|
|
93935f1a5a
|
working on rdoc support with yard
|
2019-11-13 00:53:41 -06:00 |
|
|
|
b16649bdb5
|
removed uneeded config options
|
2019-10-29 21:33:06 -05:00 |
|
|
|
30e014a2f5
|
converting to standalone gem for basic interaction with pastebin api and cmdline client
|
2019-10-29 21:01:03 -05:00 |
|
|
|
67bc00bb1d
|
trying to add in support for configure block. launching console giving me bug with pb object
|
2019-03-29 01:00:11 -05:00 |
|
booboy
|
fe6cce2fc4
|
added a create_mapping method that does not work and a create_mappings.json. i think its the json but ill keep messing with it.
|
2019-03-20 00:13:37 -05:00 |
|
booboy
|
238d770d5e
|
this is now namedspaced correctly. at least much better than it was before this
|
2019-02-07 01:10:20 -06:00 |
|
booboy
|
f8d790e6df
|
making a sidekiq helper and started on a method to read data from raw json file to ship to es
|
2019-02-06 01:25:35 -06:00 |
|
booboy
|
9b0e11bcd9
|
moved examples and added an example of sinatra running sidekiq to view the paste jobs easier in a web ui without needing all of rails
|
2019-02-06 00:21:56 -06:00 |
|
booboy
|
697b0e7679
|
moved around files for better organization
|
2019-02-05 00:46:13 -06:00 |
|
booboy
|
6c0b401d23
|
added the type of config block i want to have. not ready but setup a skeleton to use with it
|
2019-02-05 00:30:50 -06:00 |
|
booboy
|
c3fe3a2bb7
|
added sidekiq and redis to gemspec
|
2019-02-04 23:13:50 -06:00 |
|
booboy
|
0415c64960
|
made a paste_max limit option so you can query how many pastes you want
|
2019-02-04 22:45:26 -06:00 |
|
booboy
|
8d781e2499
|
made a job to scrape pastes and send to es, i got a rate limit error from pastebin so i need to slow down the calls i think, its 501 calls to the, raw metadata file, raw data file, and scrape index
|
2019-02-04 03:49:06 -06:00 |
|
booboy
|
7d0f304d43
|
made it into a difference method instead.
|
2019-02-04 01:48:50 -06:00 |
|
booboy
|
9916bb5c9d
|
made method to grab differences in keys arrays
|
2019-02-04 01:43:15 -06:00 |
|
booboy
|
07e5c4ac03
|
added 250 as a value to the scrape public paste so we can get more pastes
|
2019-02-03 03:18:03 -06:00 |
|
booboy
|
32fd073bf7
|
removed old comments
|
2019-02-03 03:08:04 -06:00 |
|
booboy
|
dc5ca66549
|
yaml safe load way to go
|
2019-02-03 03:03:33 -06:00 |
|
booboy
|
e104b0b015
|
good additions, got nested object working by loading paste_metadata as an array of hash instead of a string
|
2019-02-03 02:09:51 -06:00 |
|
booboy
|
1b6759952e
|
quick typo
|
2019-02-02 04:35:15 -06:00 |
|
booboy
|
79a139a0fc
|
added examples of how you can automate downloading pastes with the command line binary and some systemd service/timer file pairs
|
2019-02-02 04:28:04 -06:00 |
|
booboy
|
fafd8268c0
|
added the ability to download either raw pastes or the full metadata + pastetext in one json document
|
2019-02-02 04:17:17 -06:00 |
|
booboy
|
e580c572f7
|
removed old mapping from pastebinner and also adjusted download to include metadata and fulltext of paste in the json
|
2019-02-02 03:52:56 -06:00 |
|
booboy
|
bf64a2f60e
|
added method to es helper to delete index and get mappings
|
2019-02-02 03:41:39 -06:00 |
|
booboy
|
6f715c3f86
|
this mapping has to be right now please
|
2019-02-02 02:44:59 -06:00 |
|
booboy
|
7db91e6053
|
pretty sure this mapping is going to work
|
2019-02-02 02:25:50 -06:00 |
|
booboy
|
4a2f27d5f6
|
was getting errors for previous mapping when trying to auto increment index with post requests, hopefully this fixes that.
|
2019-02-02 00:37:05 -06:00 |
|
booboy
|
ee93b17576
|
made a mapping that worked and some more stuff to the elastic search helper
|
2019-02-01 23:49:35 -06:00 |
|
booboy
|
2a39616379
|
added methods to update mappings
|
2019-02-01 22:44:41 -06:00 |
|
booboy
|
50b052a9bd
|
made elastic search helper better and made pastebinner pack json
|
2019-02-01 22:19:36 -06:00 |
|
booboy
|
ba20e8dc41
|
fixed some things with hashing the paste method and sending data to es
|
2019-02-01 20:38:14 -06:00 |
|
booboy
|
107c1ff559
|
added some methods to better prepare data for elasticsearch
|
2019-01-29 03:51:36 -06:00 |
|
booboy
|
c72f8c9a5f
|
put files in pastebinner dir
|
2019-01-28 23:58:30 -06:00 |
|
booboy
|
cac1352895
|
building skeletons of exceptions
|
2019-01-06 20:50:05 -06:00 |
|
booboy
|
0acc26f90e
|
small update
|
2018-11-18 21:10:45 -06:00 |
|
booboy
|
319822fdad
|
almost there with the mappings for metadata
|
2018-11-18 06:44:45 -06:00 |
|
booboy
|
18c1bde0a3
|
made a bunch of hashes that merge to prepare pastes and metadata for ES. pastes seem to work, but metadata is still messed up. i think it is because of the mappings that i did in the elastic_search_helper.rb script
|
2018-11-18 06:04:51 -06:00 |
|
booboy
|
345f826600
|
made the json encoding store metadata of a paste and the fulltext
|
2018-11-17 22:15:36 -06:00 |
|
booboy
|
d713b509d2
|
fixed one character typo in ENDPOINTS constants for scraping raw paste metadata
|
2018-11-17 22:02:22 -06:00 |
|
booboy
|
fa0bd05b06
|
added an encode_json method to prep for db
|
2018-11-17 18:37:26 -06:00 |
|
booboy
|
bd6aec9542
|
an update to readme, update to i dont know anymore i am tired and have to go to sleep look at the diff
|
2018-11-10 04:47:41 -06:00 |
|
booboy
|
fa41c0cb7f
|
encode_json method and a 1 character change in the filename check
|
2018-11-10 04:31:32 -06:00 |
|
booboy
|
ef312d12cf
|
added a method that we will later use to serialize the raw paste into json so we can ship it to a DB
|
2018-11-10 04:27:25 -06:00 |
|
booboy
|
879a897770
|
just added some psuedocode for an exception and a config file method
|
2018-11-10 04:14:32 -06:00 |
|
booboy
|
e49e94e0e0
|
moved examples to its own directory
|
2018-11-09 23:04:27 -06:00 |
|
booboy
|
c4fde291e4
|
made an actual readme to explain how to download pastes
|
2018-11-09 22:53:09 -06:00 |
|
booboy
|
4c43f54e96
|
added an option_parser class to use with the command line binary
|
2018-11-09 01:51:21 -06:00 |
|
booboy
|
ebde6ca0f2
|
messed with the cmd line options, some work now
|
2018-11-04 02:59:25 -06:00 |
|
