DIANA Meeting - Swordfish and Arrow

40/R-B10 (CERN)



Show room on map
Jim Pivarski (Princeton University), Vincent Alexander Croft (New York University (US))


With swordfish you can quickly and accurately forecast experimental sensitivities without all the fuss with time-intensive Monte Carlos, mock data generation and likelihood maximization.

With swordfish you can

  • Calculate the expected upper limit or discovery reach of an instrument.
  • Derive expected confidence contours for parameter reconstruction.
  • Visualize confidence contours as well as the underlying information metric field.
  • Calculate the information flux, an effective signal-to-noise ratio that accounts for background systematics and component degeneracies.

A large range of experiments in particle physics and astronomy are statistically described by a Poisson point process. The swordfish module implements at its core a rather general version of a Poisson point process, and provides easy access to its information geometrical properties. Based on this information, a number of common and less common tasks can be performed.


Apache Arrow

Arrow is a high-performance cross-system data layer for columnar in-memory analytics.

Like a file format, it allows data to be transferred between data analysis platforms, but through zero-copy shared memory, rather than files on disk.

Arrow is backed by key developers of 13 major open source projects, including Calcite, Cassandra, Drill, Hadoop, HBase, Ibis, Impala, Kudu, Pandas, Parquet, Phoenix, Spark, and Storm making it the de-facto standard for columnar in-memory analytics. It also supports a wide variety of industry-standard programming languages, such as Java, C, C++, Python, Ruby, and JavaScript.

As a columnar format, Arrow is suited for native vectorized optimization of analytical data processing.


Recorded Meeting Video: https://www.youtube.com/watch?v=tfaG503cN3M

    • 5:30 PM 6:00 PM
      Swordfish: Information yield of counting experiments 30m
      Speakers: Christoph Weniger (University of Amsterdam), Thomas Edwards (University of Amsterdam)
    • 6:00 PM 6:30 PM
      Apache Arrow 30m
      Speaker: Wes McKinney (Two Sigma)