Vaughn Dickson

Why does Apache Nutch sometimes get stuck using a single thread and crawling slowly?

/ DevOps

Nutch generates a list of urls to fetch from the crawldb. In ./bin/crawl it defaults the size of the fetch list to sizeFetchList=50000. If you use the default setting generate.max.count=-1 which is unrestricted, you can potentially end up with 50000 urls from the same domain in your fetch list. Then the setting fetcher.queue.mode=byHost only creates […]

Why does Apache Nutch sometimes get stuck using a single thread and crawling slowly? Read More »

Centralising Clojure/Java logging with Logback, LogStash, ElasticSearch, and Kibana

/ Clojure, DevOps, Elasticsearch, Java, Linux

Checking logs when you have more than one servers is painful. Use Logback/Logstash-forwarder to send json-formatted logs to a central server running Logstash/ElasticSearch/Kibana, where you can then slice and dice logs to your heart’s content with the power of ElasticSearch and Kibana. Confs and docs available here: https://github.com/vaughnd/centralised-logging

Centralising Clojure/Java logging with Logback, LogStash, ElasticSearch, and Kibana Read More »

Keybinding for emacs helm to recursively grep certain file extensions in your src directories

/ Emacs

Helm for Emacs is a fantastic Quicksilver-like extension, but it gets quite wordy sometimes. Instead of C-u M-x helm-do-grep *nav to dir* *enter extensions* *enter query* to recursively grep, I defined the following in my init.el. Now hitting F1 will grep actual source across all my projects. (defun project-search () (interactive) (helm-do-grep-1 ‘(“/home/vaughn/src”) ‘(4) nil

Keybinding for emacs helm to recursively grep certain file extensions in your src directories Read More »