booboy
238d770d5e
this is now namedspaced correctly. at least much better than it was before this
2019-02-07 01:10:20 -06:00
booboy
f8d790e6df
making a sidekiq helper and started on a method to read data from raw json file to ship to es
2019-02-06 01:25:35 -06:00
booboy
c3fe3a2bb7
added sidekiq and redis to gemspec
2019-02-04 23:13:50 -06:00
booboy
0415c64960
made a paste_max limit option so you can query how many pastes you want
2019-02-04 22:45:26 -06:00
booboy
7d0f304d43
made it into a difference method instead.
2019-02-04 01:48:50 -06:00
booboy
9916bb5c9d
made method to grab differences in keys arrays
2019-02-04 01:43:15 -06:00
booboy
07e5c4ac03
added 250 as a value to the scrape public paste so we can get more pastes
2019-02-03 03:18:03 -06:00
booboy
32fd073bf7
removed old comments
2019-02-03 03:08:04 -06:00
booboy
dc5ca66549
yaml safe load way to go
2019-02-03 03:03:33 -06:00
booboy
e104b0b015
good additions, got nested object working by loading paste_metadata as an array of hash instead of a string
2019-02-03 02:09:51 -06:00
booboy
e580c572f7
removed old mapping from pastebinner and also adjusted download to include metadata and fulltext of paste in the json
2019-02-02 03:52:56 -06:00
booboy
50b052a9bd
made elastic search helper better and made pastebinner pack json
2019-02-01 22:19:36 -06:00
booboy
ba20e8dc41
fixed some things with hashing the paste method and sending data to es
2019-02-01 20:38:14 -06:00
booboy
107c1ff559
added some methods to better prepare data for elasticsearch
2019-01-29 03:51:36 -06:00
booboy
c72f8c9a5f
put files in pastebinner dir
2019-01-28 23:58:30 -06:00
booboy
0acc26f90e
small update
2018-11-18 21:10:45 -06:00
booboy
18c1bde0a3
made a bunch of hashes that merge to prepare pastes and metadata for ES. pastes seem to work, but metadata is still messed up. i think it is because of the mappings that i did in the elastic_search_helper.rb script
2018-11-18 06:04:51 -06:00
booboy
345f826600
made the json encoding store metadata of a paste and the fulltext
2018-11-17 22:15:36 -06:00
booboy
d713b509d2
fixed one character typo in ENDPOINTS constants for scraping raw paste metadata
2018-11-17 22:02:22 -06:00
booboy
fa0bd05b06
added an encode_json method to prep for db
2018-11-17 18:37:26 -06:00
booboy
fa41c0cb7f
encode_json method and a 1 character change in the filename check
2018-11-10 04:31:32 -06:00
booboy
ebde6ca0f2
messed with the cmd line options, some work now
2018-11-04 02:59:25 -06:00
booboy
9d34d53372
fixed trends. one character fix
2018-11-04 02:39:52 -06:00
booboy
eae20ccd0c
started to work on the command line application
...
added
-- raw
-- scrape_public_pastes
-- list_trending (still broken though. need to fix the api call)
-- added a method to extract just the keys from public pastes
-- moved the previous tests to an examples.rb file
need to mess around with all the different ways to parse arguments for the
command line program.
2018-11-04 02:28:57 -06:00
booboy
092bbb252b
added all of the methods.
...
will need to go back through and fix the scraping api endpoints to use the execute_query private method
2018-11-03 01:57:23 -05:00
kenna-bmcdevitt
2be2d4419c
fixed naming conventions and set this up to begin working on command line calls
2018-08-22 15:03:42 -05:00
kenna-bmcdevitt
eaf6207f22
added a couple more api calls
2018-08-22 14:00:13 -05:00
kenna-bmcdevitt
9da7b16a6e
made execute_query a work on the methods inside of the PasteBinner class
2018-08-22 13:32:53 -05:00
booboy
94568018e3
made execute_query a private method
2018-08-19 20:22:48 -05:00
booboy
447dc1f003
added an api_post method, and a get_api_user_key method for retrieving a session key which is required for listing user pastes.
2018-08-16 20:31:34 -05:00
booboy
da6c789a5c
initial commit
2018-08-16 14:54:08 -05:00
