booboy
ee93b17576
made a mapping that worked and some more stuff to the elastic search helper
2019-02-01 23:49:35 -06:00
booboy
2a39616379
added methods to update mappings
2019-02-01 22:44:41 -06:00
booboy
50b052a9bd
made elastic search helper better and made pastebinner pack json
2019-02-01 22:19:36 -06:00
booboy
ba20e8dc41
fixed some things with hashing the paste method and sending data to es
2019-02-01 20:38:14 -06:00
booboy
107c1ff559
added some methods to better prepare data for elasticsearch
2019-01-29 03:51:36 -06:00
booboy
c72f8c9a5f
put files in pastebinner dir
2019-01-28 23:58:30 -06:00
booboy
cac1352895
building skeletons of exceptions
2019-01-06 20:50:05 -06:00
booboy
9a4f1251dd
fixed an issue that was occurring with downloading the raw pastes
2019-01-06 20:21:35 -06:00
booboy
fc82bd78d1
Merge branch 'master' of https://git.mcdevitt.tech/bpmcdevitt/pastebinner
2019-01-06 19:26:49 -06:00
booboy
46526a21e9
made pry console startup with just a pb object for easy experimentation with the methods
2019-01-06 19:26:16 -06:00
booboy
0acc26f90e
small update
2018-11-18 21:10:45 -06:00
booboy
319822fdad
almost there with the mappings for metadata
2018-11-18 06:44:45 -06:00
booboy
18c1bde0a3
made a bunch of hashes that merge to prepare pastes and metadata for ES. pastes seem to work, but metadata is still messed up. i think it is because of the mappings that i did in the elastic_search_helper.rb script
2018-11-18 06:04:51 -06:00
booboy
345f826600
made the json encoding store metadata of a paste and the fulltext
2018-11-17 22:15:36 -06:00
booboy
d713b509d2
fixed one character typo in ENDPOINTS constants for scraping raw paste metadata
2018-11-17 22:02:22 -06:00
booboy
ce9626655a
made an error if only -k --key is passed
2018-11-17 19:04:12 -06:00
booboy
fa0bd05b06
added an encode_json method to prep for db
2018-11-17 18:37:26 -06:00
booboy
4d0dbb46c4
typo
2018-11-10 04:48:51 -06:00
booboy
bd6aec9542
an update to readme, update to i dont know anymore i am tired and have to go to sleep look at the diff
2018-11-10 04:47:41 -06:00
booboy
fa41c0cb7f
encode_json method and a 1 character change in the filename check
2018-11-10 04:31:32 -06:00
booboy
ef312d12cf
added a method that we will later use to serialize the raw paste into json so we can ship it to a DB
2018-11-10 04:27:25 -06:00
booboy
879a897770
just added some psuedocode for an exception and a config file method
2018-11-10 04:14:32 -06:00
booboy
e49e94e0e0
moved examples to its own directory
2018-11-09 23:04:27 -06:00
booboy
eaf5e306c0
added To add to the bottom of the readme
2018-11-09 22:58:47 -06:00
booboy
c4fde291e4
made an actual readme to explain how to download pastes
2018-11-09 22:53:09 -06:00
booboy
4c43f54e96
added an option_parser class to use with the command line binary
2018-11-09 01:51:21 -06:00
booboy
5aa45e28c3
fixed a part of the commandline options
2018-11-05 03:39:00 -06:00
booboy
4853f92ecd
wrote a method to download all the raw pastes to a file inside of a directory.
...
added a --download and -d shorthand switch to download the raw paste files from a public scrape.
i need to check to make sure that it is the correct way to do it, and also check to see if a filename exists already.
this will prevent duplicate files from being downloaded.
2018-11-05 03:15:29 -06:00
booboy
ebde6ca0f2
messed with the cmd line options, some work now
2018-11-04 02:59:25 -06:00
booboy
9d34d53372
fixed trends. one character fix
2018-11-04 02:39:52 -06:00
booboy
eae20ccd0c
started to work on the command line application
...
added
-- raw
-- scrape_public_pastes
-- list_trending (still broken though. need to fix the api call)
-- added a method to extract just the keys from public pastes
-- moved the previous tests to an examples.rb file
need to mess around with all the different ways to parse arguments for the
command line program.
2018-11-04 02:28:57 -06:00
booboy
092bbb252b
added all of the methods.
...
will need to go back through and fix the scraping api endpoints to use the execute_query private method
2018-11-03 01:57:23 -05:00
kenna-bmcdevitt
2be2d4419c
fixed naming conventions and set this up to begin working on command line calls
2018-08-22 15:03:42 -05:00
kenna-bmcdevitt
eaf6207f22
added a couple more api calls
2018-08-22 14:00:13 -05:00
kenna-bmcdevitt
9da7b16a6e
made execute_query a work on the methods inside of the PasteBinner class
2018-08-22 13:32:53 -05:00
booboy
94568018e3
made execute_query a private method
2018-08-19 20:22:48 -05:00
booboy
447dc1f003
added an api_post method, and a get_api_user_key method for retrieving a session key which is required for listing user pastes.
2018-08-16 20:31:34 -05:00
booboy
da6c789a5c
initial commit
2018-08-16 14:54:08 -05:00
Brendan McDevitt
d667a0b2ce
Initial commit
2018-08-16 14:52:20 -05:00
