booboy
|
238d770d5e
|
this is now namedspaced correctly. at least much better than it was before this
|
2019-02-07 01:10:20 -06:00 |
|
booboy
|
f8d790e6df
|
making a sidekiq helper and started on a method to read data from raw json file to ship to es
|
2019-02-06 01:25:35 -06:00 |
|
booboy
|
9b0e11bcd9
|
moved examples and added an example of sinatra running sidekiq to view the paste jobs easier in a web ui without needing all of rails
|
2019-02-06 00:21:56 -06:00 |
|
booboy
|
c80bd01875
|
forgot to change file path
|
2019-02-05 01:03:08 -06:00 |
|
booboy
|
697b0e7679
|
moved around files for better organization
|
2019-02-05 00:46:13 -06:00 |
|
booboy
|
6c0b401d23
|
added the type of config block i want to have. not ready but setup a skeleton to use with it
|
2019-02-05 00:30:50 -06:00 |
|
booboy
|
c3fe3a2bb7
|
added sidekiq and redis to gemspec
|
2019-02-04 23:13:50 -06:00 |
|
booboy
|
0415c64960
|
made a paste_max limit option so you can query how many pastes you want
|
2019-02-04 22:45:26 -06:00 |
|
booboy
|
8d781e2499
|
made a job to scrape pastes and send to es, i got a rate limit error from pastebin so i need to slow down the calls i think, its 501 calls to the, raw metadata file, raw data file, and scrape index
|
2019-02-04 03:49:06 -06:00 |
|
booboy
|
7d0f304d43
|
made it into a difference method instead.
|
2019-02-04 01:48:50 -06:00 |
|
booboy
|
9916bb5c9d
|
made method to grab differences in keys arrays
|
2019-02-04 01:43:15 -06:00 |
|
booboy
|
07e5c4ac03
|
added 250 as a value to the scrape public paste so we can get more pastes
|
2019-02-03 03:18:03 -06:00 |
|
booboy
|
32fd073bf7
|
removed old comments
|
2019-02-03 03:08:04 -06:00 |
|
booboy
|
dc5ca66549
|
yaml safe load way to go
|
2019-02-03 03:03:33 -06:00 |
|
booboy
|
e104b0b015
|
good additions, got nested object working by loading paste_metadata as an array of hash instead of a string
|
2019-02-03 02:09:51 -06:00 |
|
booboy
|
e43e63b98a
|
im bad with markdown, fixed formatting
|
2019-02-02 04:57:28 -06:00 |
|
booboy
|
2ba1bde3b4
|
typo
|
2019-02-02 04:55:46 -06:00 |
|
booboy
|
dacfc616c4
|
updated readme
|
2019-02-02 04:54:26 -06:00 |
|
booboy
|
1b6759952e
|
quick typo
|
2019-02-02 04:35:15 -06:00 |
|
booboy
|
79a139a0fc
|
added examples of how you can automate downloading pastes with the command line binary and some systemd service/timer file pairs
|
2019-02-02 04:28:04 -06:00 |
|
booboy
|
fafd8268c0
|
added the ability to download either raw pastes or the full metadata + pastetext in one json document
|
2019-02-02 04:17:17 -06:00 |
|
booboy
|
e580c572f7
|
removed old mapping from pastebinner and also adjusted download to include metadata and fulltext of paste in the json
|
2019-02-02 03:52:56 -06:00 |
|
booboy
|
bf64a2f60e
|
added method to es helper to delete index and get mappings
|
2019-02-02 03:41:39 -06:00 |
|
booboy
|
6f715c3f86
|
this mapping has to be right now please
|
2019-02-02 02:44:59 -06:00 |
|
booboy
|
7db91e6053
|
pretty sure this mapping is going to work
|
2019-02-02 02:25:50 -06:00 |
|
booboy
|
4a2f27d5f6
|
was getting errors for previous mapping when trying to auto increment index with post requests, hopefully this fixes that.
|
2019-02-02 00:37:05 -06:00 |
|
booboy
|
ee93b17576
|
made a mapping that worked and some more stuff to the elastic search helper
|
2019-02-01 23:49:35 -06:00 |
|
booboy
|
2a39616379
|
added methods to update mappings
|
2019-02-01 22:44:41 -06:00 |
|
booboy
|
50b052a9bd
|
made elastic search helper better and made pastebinner pack json
|
2019-02-01 22:19:36 -06:00 |
|
booboy
|
ba20e8dc41
|
fixed some things with hashing the paste method and sending data to es
|
2019-02-01 20:38:14 -06:00 |
|
booboy
|
107c1ff559
|
added some methods to better prepare data for elasticsearch
|
2019-01-29 03:51:36 -06:00 |
|
booboy
|
c72f8c9a5f
|
put files in pastebinner dir
|
2019-01-28 23:58:30 -06:00 |
|
booboy
|
cac1352895
|
building skeletons of exceptions
|
2019-01-06 20:50:05 -06:00 |
|
booboy
|
9a4f1251dd
|
fixed an issue that was occurring with downloading the raw pastes
|
2019-01-06 20:21:35 -06:00 |
|
booboy
|
fc82bd78d1
|
Merge branch 'master' of https://git.mcdevitt.tech/bpmcdevitt/pastebinner
|
2019-01-06 19:26:49 -06:00 |
|
booboy
|
46526a21e9
|
made pry console startup with just a pb object for easy experimentation with the methods
|
2019-01-06 19:26:16 -06:00 |
|
booboy
|
0acc26f90e
|
small update
|
2018-11-18 21:10:45 -06:00 |
|
booboy
|
319822fdad
|
almost there with the mappings for metadata
|
2018-11-18 06:44:45 -06:00 |
|
booboy
|
18c1bde0a3
|
made a bunch of hashes that merge to prepare pastes and metadata for ES. pastes seem to work, but metadata is still messed up. i think it is because of the mappings that i did in the elastic_search_helper.rb script
|
2018-11-18 06:04:51 -06:00 |
|
booboy
|
345f826600
|
made the json encoding store metadata of a paste and the fulltext
|
2018-11-17 22:15:36 -06:00 |
|
booboy
|
d713b509d2
|
fixed one character typo in ENDPOINTS constants for scraping raw paste metadata
|
2018-11-17 22:02:22 -06:00 |
|
booboy
|
ce9626655a
|
made an error if only -k --key is passed
|
2018-11-17 19:04:12 -06:00 |
|
booboy
|
fa0bd05b06
|
added an encode_json method to prep for db
|
2018-11-17 18:37:26 -06:00 |
|
booboy
|
4d0dbb46c4
|
typo
|
2018-11-10 04:48:51 -06:00 |
|
booboy
|
bd6aec9542
|
an update to readme, update to i dont know anymore i am tired and have to go to sleep look at the diff
|
2018-11-10 04:47:41 -06:00 |
|
booboy
|
fa41c0cb7f
|
encode_json method and a 1 character change in the filename check
|
2018-11-10 04:31:32 -06:00 |
|
booboy
|
ef312d12cf
|
added a method that we will later use to serialize the raw paste into json so we can ship it to a DB
|
2018-11-10 04:27:25 -06:00 |
|
booboy
|
879a897770
|
just added some psuedocode for an exception and a config file method
|
2018-11-10 04:14:32 -06:00 |
|
booboy
|
e49e94e0e0
|
moved examples to its own directory
|
2018-11-09 23:04:27 -06:00 |
|
booboy
|
eaf5e306c0
|
added To add to the bottom of the readme
|
2018-11-09 22:58:47 -06:00 |
|