The system and method described features mechanisms from a big data analytics platform that provides the performance and energy benefits of integrated acceleration circuits such as field programmable gate arrays (FPGA), application specific integrated circuits (ASIC) or custom circuits without sacrificing the ease of developing applications on distributed cluster-computing frameworks like Apache Spark.