BuildCausalityNetworkTask --> Too many open files

[1fish2]

BuildCausalityNetworkTask writes 29100 json files into a sim generation's seriesOut/ dir, leading to a FileNotFoundException (Too many open files) in Sisyphus.

That's immediately followed by a java.lang.ClassCastException: class java.lang.Character cannot be cast to class java.util.Map$Entry which might be an error handling bug.

Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]: java.io.FileNotFoundException: /tmp/sisyphus/outputs/data/jerry/20190710.140901/wildtype_000000/000000/generation_000000/000000/seriesOut/8699366258568945.json (Too many open files)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at java.base/java.io.FileInputStream.open0(Native Method)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.java.io$fn__11466.invokeStatic(io.clj:229)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.java.io$fn__11466.invoke(io.clj:229)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.java.io$fn__11379$G__11372__11386.invoke(io.clj:69)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.java.io$input_stream.invokeStatic(io.clj:136)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.java.io$input_stream.doInvoke(io.clj:121)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.lang.RestFn.invoke(RestFn.java:410)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.archive$pack_BANG_.invokeStatic(archive.clj:71)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.archive$pack_BANG_.invoke(archive.clj:58)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.task$push_output_BANG_.invokeStatic(task.clj:64)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.task$push_output_BANG_.invoke(task.clj:60)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.task$perform_task_BANG_.invokeStatic(task.clj:195)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.task$perform_task_BANG_.invoke(task.clj:139)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.core$sisyphus_handle_rabbit.invokeStatic(core.clj:109)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.core$sisyphus_handle_rabbit.invoke(core.clj:100)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.core$partial$fn__5824.invoke(core.clj:2626)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at langohr.consumers$create_default$fn__18445.invoke(consumers.clj:83)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at langohr.consumers.proxy$com.rabbitmq.client.DefaultConsumer$ff19274a.handleDelivery(Unknown Source)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:104)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at java.base/java.lang.Thread.run(Thread.java:834)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]: SEVERE sisyphus: rabbit-task
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]: java.lang.ClassCastException: class java.lang.Character cannot be cast to class java.util.Map$Entry (java.lang.Character and java.util.Map$Entry are in module java.base of loader 'bootstrap')
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.lang.APersistentMap.cons(APersistentMap.java:42)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.lang.RT.conj(RT.java:673)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.core$conj__5375.invokeStatic(core.clj:85)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.core$merge$fn__5943.invoke(core.clj:3049)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.core$reduce1.invokeStatic(core.clj:944)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.core$reduce1.invokeStatic(core.clj:934)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.core$merge.invokeStatic(core.clj:3048)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.core$merge.doInvoke(core.clj:3041)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.lang.RestFn.invoke(RestFn.java:421)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.task$send_BANG_.invokeStatic(task.clj:113)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.task$send_BANG_.invoke(task.clj:106)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.task$exception_BANG_.invokeStatic(task.clj:137)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.task$exception_BANG_.invoke(task.clj:134)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.task$perform_task_BANG_.invokeStatic(task.clj:209)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.task$perform_task_BANG_.invoke(task.clj:139)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.core$sisyphus_handle_rabbit.invokeStatic(core.clj:109)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at sisyphus.core$sisyphus_handle_rabbit.invoke(core.clj:100)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at clojure.core$partial$fn__5824.invoke(core.clj:2626)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at langohr.consumers$create_default$fn__18445.invoke(consumers.clj:83)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at langohr.consumers.proxy$com.rabbitmq.client.DefaultConsumer$ff19274a.handleDelivery(Unknown Source)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:104)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Jul 10 21:55:14 sisyphus-jerry-0 bash[1251]:         at java.base/java.lang.Thread.run(Thread.java:834)

[prismofeverything]

Interesting.... this happens during archiving, which I have a feature branch that no longer archives, but instead uploads each file so this would probably not happen in that case.

That said, 21k files per sim is a lot! Maybe too many. Is it possible to refactor the causality network task to emit a single archive of the 21k json files, then the CN opens that? Object store works best with fewer larger files than multitudes of small files like this.

To solve this in the immediate term we can always raise the open file limit (here is a dumb tutorial for this): https://easyengine.io/tutorials/linux/increase-open-files-limit/

[1fish2]

Storing individual files will be great.

If we want to keep the Sisyphus archive feature, is this just a matter of closing each file after adding it to the archive rather than accumulating open files?

Maybe the Causality builder should put all the json files into an archive, but they might have to get unpacked for the web viewer web page. I suspect the json files are organized for the Causality viewer to open incrementally. That's probably related to the viewer's limitations that keep it from being able to fit more than ≈2 generations in memory.

[1fish2]

See CovertLab/wcEcoli#605