Dell’s parallelization of its PowerScale clustered file system storage is proceeding apace as a Dell spokesperson told us: “We’ve received positive feedback on Project Lightning in early access, stay tuned for more on general availability.”
PowerScale is Dell’s extended and rebranded Isilon scale-out filer technology with its OneFS operating system and 4 to 256 nodes. It competes with Qumulo’s Core system, developed by some of the Isilon founders and a similar clustered, scale-out design. A separate high-performance filer system technology uses parallel file systems, such as IBM’s Storage Scale (rebranded and developed GPFS), and Lustre. These provide higher bandwidth file system access for high-performance computing (HPC) systems. In recent years, the rise of big data analytics and then AI, with multiple-GPU servers, have brought the need for high-performance file access to enterprise computing or AI training and also inference.
This led to the development of VAST Data, with its parallel data access; Hammerspace, with its parallel NFS technology; and WEKA, with its parallel file system technology. In response, Dell set up Project Lightning in 2024 to add parallel access to its PowerScale and also ObjectScale storage systems. At the time Dell SVP for its ISG unit’s marketing, Varun Chhabra, told Dell Technologies World attendees: “Project Lightning … will deliver a parallel file system for unstructured data in PowerScale. Project Lightning will bring extreme performance and unparalleled efficiency with near line rate efficiency – 97 percent network utilization and the ability to saturate thousands of data-hungry GPUs.”
Dell tells us that, with Project Lightning: “PowerScale delivers enterprise-ready parallel file system performance with the simplicity of NAS, ideal for enterprises, from small to massive deployments with built-in security. It delivers massive I/O across many nodes from multiple clients for consistent high-speed access, high concurrency and low latency for AI, HPC and analytics workloads, with distributed metadata with no central bottlenecks.”
As with parallel file systems generally, “PowerScale also takes advantage of an on-client software layer for distributing reads to all nodes in the cluster, giving truly parallel read performance (including benefits like RDMA to avoid TCP overhead). While we don’t want to spoil the surprise, we have some good updates on the way that will make PowerScale more parallel and multiply performance.”
RDMA is remote direct memory access and parallelized PowerScale will use this at the drive level. “Project Lightning delivers extremely low latency and extremely high performance by going directly from client to device without traversing the file system. Because it goes directly to the device, its performance is fabric-bound. Project Lightning will be the world’s fastest parallel file system, delivering up to 2x greater performance than competing systems.”
The main competing parallel file systems are IBM Storage Scale, DDN Lustre, VDURA’s PanFS, Quobyte and BeeGFS. These all have HPC credentials and most are now supporting enterprise AI use cases. They all require accessing systems to have specific client software to translate standard file system access protocols to the internal multi-node processes inside their parallel file system software VAST Data and Hammerspace do not, relying on standard NFS and SMB. WEKA does require its WEKA client software to be installed on systems accessing the WekaFS file system for direct file access.
NetApp has its ONTAP for AI Project with separate compute controllers, running ONTAP instances, “embellished with additional metadata and data services,” and storage nodes filled with NVMe SSDs, forming a single storage pool, accessed across a high-speed, low-latency, Ethernet-based RDMA fabric. We expect news about its progress to arrive in the next few weeks. HPE has a deal with VAST Data for its file system software.
These moves provide an opportunity for Qumulo to either strengthen its middle-ground appeal or to add parallel access to its Core operating system alongside its performance-enhancing NeuralCache predictive caching feature.
We expect to hear more about Project Lightning’s impact on PowerScale in coming weeks and it will probably feature strongly at next year’s Dell Technologies World in May at Las Vegas.
