pastebinner/lib/paste_to_es.rb

class PasteToEs
  include Sidekiq::Worker
  def perform(es_object, pb_object, paste_max)
    Logger.new(STDOUT).info("PasteToEs started")
    pastes = pb_object.scrape_public_pastes(paste_max)
    keys = pb_object.get_unique_paste_keys(pastes)
    json_data = pb_object.json_paste(keys)
    es_object.json_to_es_bulk(json_data)
  end
end
made a job to scrape pastes and send to es, i got a rate limit error from pastebin so i need to slow down the calls i think, its 501 calls to the, raw metadata file, raw data file, and scrape index 2019-02-04 03:49:06 -06:00			`class PasteToEs`
			`include Sidekiq::Worker`
made a paste_max limit option so you can query how many pastes you want 2019-02-04 22:45:26 -06:00			`def perform(es_object, pb_object, paste_max)`
made a job to scrape pastes and send to es, i got a rate limit error from pastebin so i need to slow down the calls i think, its 501 calls to the, raw metadata file, raw data file, and scrape index 2019-02-04 03:49:06 -06:00			`Logger.new(STDOUT).info("PasteToEs started")`
made a paste_max limit option so you can query how many pastes you want 2019-02-04 22:45:26 -06:00			`pastes = pb_object.scrape_public_pastes(paste_max)`
made a job to scrape pastes and send to es, i got a rate limit error from pastebin so i need to slow down the calls i think, its 501 calls to the, raw metadata file, raw data file, and scrape index 2019-02-04 03:49:06 -06:00			`keys = pb_object.get_unique_paste_keys(pastes)`
			`json_data = pb_object.json_paste(keys)`
			`es_object.json_to_es_bulk(json_data)`
			`end`
			`end`